Behavioural Assessment in EBP: Mostly Accurate Claims Examined

It is concluded that future advances in ecological validity are likely to come from: (a) a greater reliance on assessments of the target behaviors in natural settings and (b) combining several of the more promising tasks and analogue methods ...

Introduction: Ecological assessment of executive functions (EF) with tasks simulating everyday-life difficulties in children remains poorly developed. The Behavioural Assessment of the Dysexecutive Syndrome for Children (BADS-C) is one of the rare tools proposed in this perspective, for which developmental and convergent validity are, however, rather limited.

Two additional features should be highlighted here. Specifically, reliability of measurement within individuals (e.g., consistency of response) as well as ecological validity (e.g., similarity to experience outside the office) is emphasized in behavioral assessment.

These tests may be selected for their purported potential to highlight specific cognitive domains that may be affected by the particular intervention, and the availability, in many cases, of good normative data. However, often they are chosen simply because they were used in the past, with little consideration of the fact that they were developed primarily to fulfill a role in an era that emphasized diagnosis.

A primary objective of these methods is to sample the ongoing stream of behavior or thought in its natural sequence and occurrence, and thereby, provide an accurate and valid representation of individuals’ function that generalizes to real-world situations. By design, through repeated assessment of individuals’ behavior and experience in their natural environments, these approaches prioritize ecological validity.

Furthermore, we believe that it is timely to evaluate to which extent recent advances in stimulus generation, data collection methods, and study design may have improved the EV of experimental findings, and the EV of data obtained from large-scale surveys that strive for “contextual” embeddedness. We would also like to mention that the common treatment of EV in current developmental science, in general, and in the context of ecological momentary assessments, in particular, in our view lacks a certain degree of critical reflection.

To develop an ecologically valid measure of executive functioning (i.e. Planning and Organization, Executive Memory, Initiation, Cognitive Shifting, Impulsivity, Sustained and Directed Attention, Error Detection, Error Correction and Time Management) during a functional chocolate brownie cooking task. In Study 1, the inter-rater reliability of a novel behavioural observation assessment method was assessed with 10 people with traumatic brain injury (TBI).

(Reference Alderman, Burgess, Knight and Henman2003) claimed that their test was inherently ecologically valid due to its high face validity, despite having provided a definition that explicitly stated that EV is defined by the test’s ability to predict functional outcome. In other words, it appears that the term EV is implicitly linked not to empirical evidence of validity, but rather to subjective impressions about test appearance.

Furthermore, assessing the level of control and the ‘naturalness’ of their own work encourages researchers to think about them at the design stage of their future studies, and make them generally more aware of the extent to which the experimental setup they have designed offers high levels of ecological validity for the behavior they are interested in studying.