1 Note that this calculator is based on examples of IOA (Improving and Assessing the Quality of Behavi measurement) data, presented in Chapter 5; 102-124) was tested by Cooper, Heron and Heward (2007). For all algorithms, there was a 100% match between the values derived from the IOA with the computer described in this article and those that were reported in the IOA scores interval. The IOA algorithm with a little interval (also called “non-deposit” agreement in the research literature) is also stricter than simple interval-by-interval approaches, taking into account only intervals in which at least one observer records the lack of response. The justification for pointless IOA is similar to that of the IOA with the scored interval, except that this metric responds best for high rates (Cooper et al., 2007). In the figure 2 examples, the 5th and 6th intervals are ignored for calculation purposes, as both observers have received a response at these intervals. Thus, the IOA statistics are calculated from the remaining five intervals. Since agreement has only been reached on three of the five intervals (the second, third and fourth intervals), the approval rate is 60%. Permanent IOA algorithms evaluate the agreement between the temporal data of two observers. These measures consist of (a) the total duration and (b) the average duration of the incident. Table 3 summarizes the strengths of the two algorithms. Consider as a permanent example of the permanent IOA the hypothetical data flow represented in Figure 3, in which two independent observers recorded the duration of a target response over four deposits. The idea that practicing behavioural analysts should collect and report reliability or interobserver agreement (IOA) in behavioural assessments is demonstrated by the Behavior Analyst Certification Board`s (BACB) assertion that behavioural analysts are responsible for the use of “different methods of evaluating the results of measurement methods such as inter-observer agreement, accuracy and reliability” (BACB, 2005).
In addition, Vollmer, Sloman and St. Peter Pipkin (2008) argue that the exclusion of these data significantly limits any interpretation of the effectiveness of a behavioural change procedure. Validity requirements in a behavioural assessment study should therefore be conditional on the inclusion of insurance data (Friman, 2009). In light of these considerations, it is not surprising that a recent review of articles in the Journal of Applied Behavior Analysis (JABA) from 1995 to 2005 (Mudford, Taylor, Martin, 2009) revealed that 100% of articles reporting continuously recorded dependent variables contained IOA calculations. These data, as well as previously published reports on reliability procedures in JABA (Kelly, 1977), suggest that the inclusion of IOA is in fact a trademark – if not a standard – of behavioural evaluation. Trial-by-trial: compares the agreement between the different studies instead of the total number of IOA points. An approach to improve the accuracy of the agreement between two observers for interval recording is simply to limit agreement analyses to cases where at least one observer has recorded a target response in the meantime. Intervals in which no observer has reported a target response are excluded from the calculation in order to provide stricter agree statistics. Cooper et al. (2007) suggest that the IOA point interval (also known as “deposit agreement” in the research literature) is most advantageous when targeted responses are at low rates. In the figure 2 examples, the second, third and fourth intervals are ignored for calculation purposes, as none of the observed intervals have been answered at these intervals. As a result, IOA statistics are only calculated from the first, fifth, sixth and seventh intervals.