Interobserver Agreement (IOA) refers to the degree to which two or more independent observers report the same observed values after measuring the same events.

4 Benefits of IOA

1. Determine the competence of new observers (when IOA is low)
2. Detect observer drift over the course of a study (when IOA is low)
3. Increases confidence that the target behavior was clearly defined (when IOA is high)
4. Confirms that change in data is due to change in behavior and not in data collection (when IOA is high)

4 Methods for collecting IOA

1. Total count IOA – this is the simplest and least exact method.  IOA = smaller count / larger count * 100.  Caution must be used because there is no guarantee that the observers are recording the same instances of the behavior.
2. Mean count-per-interval IOA – The chance to have a more accurate representation of IOA is by
   1. dividing up the total observation period into a series of smaller counting times and
   2. having the observers record the number of occurrences of behavior within each interval
   3. calculating the agreement between the observer counts within each interval
   4. using the agreements per interval as the basis for calculating the IOA for the total observation period
   5. IOA = int 1 IOA + int 2 IOA …+ int N IOA / n intervals * 100
3. Exact Count-per-interval IOA – is the most exact way to count IOA.  This is the percent of intervals in which observers record the same count.  IOA = # of intervals at 100% IOA / n intervals * 100
4. Trial-by-trial IOA – # of trials items agreement / # of trials * 100

Interval IOA

1. In Scored interval IOA, you determine the number of intervals that have a “yes.”  Then you divide the number of those that agree by the total number of those intervals.
2. In Unscored Interval IOA, you do the same for the Scored Interval IOA, except you take the number of intervals that have a “no.”

Reliable vs. Accurate Data

Reliable data is data that gives the same results each time you measure it.

Accurate data is data that is correct.

Reliable data is not always accurate, but accurate data is always reliable.