Questing for the Gold Standard of IOA Agreement

Question: What is the desired percentage of agreement for interobserver agreement?

Answer: 80% or more.

IOA refers to the degree to which two or more independent observers report the same thing after measuring the same behavior at the same time. IOA helps increase the believability of our data. It also helps determine the competence of new observers or ensure that the operational definition that we’ve written is clear enough to measure the behavior. More importantly, it helps us to see whether or not any behavior change is the product of the intervention, or any lack thereof, is the product of the intervention.

There are a few rules that you should know about IOA. First and foremost, all the observers have to use the same measurement system. Also, those same observers have to measure the same event. They also have to be independent of each other; they can’t be interacting while they’re that collecting data. It’s a little counterintuitive to the whole idea. IOA should be measured for a minimum of 20% of all behavioral sessions, and for the believability to be super believable, the closer to 100% agreement, the better. A minimum of 80%, but obviously, believability increases as you get closer to 100%.

Related Content

Dana Do's: Why IOA Math Should Not Intimidate You

PTB co-founder takes some of the fear out of the interobserver agreement math with simple tips that can help narrow your discrimination.

Dana Do's: How to Define the Markers of Trustworthy Measurement

PTB co-founder goes into the three factors that define trustworthy measurement-- "validity", "reliability" and "accuracy". But what's the difference between each? Dana explains.

Dana Do's: How to Decipher the Many Forms of Validity

It's particularly meaningful to be able to tell validity, social validity, internal and external validity apart. PTB co-founder Dana Meller has some tips to do just that.

ABA Terminology Break Down ➠ Indicators of Trustworthy Measurement

What are the three indicators of trustworthy measurement in ABA? PTB co-founder Dana Meller explains and why they ensure data accuracy and reliability. Refer to BCBA® Task List (5th ed.) Section C-8: Evaluate the validity and reliability of measurement procedures.

Break Down a Mock BCBA® Exam Question: Interobserver Agreement (IOA) Facts & Rules to Live By

Test your Measurement, Data Display, and Interpretation skills with PTB co-founder Dana Meller as she reviews a BCBA® mock exam question about several #interobserveragreement (IOA) facts, with rules to live by. Here, Dana highlights a variety of important IOA benefits, as well as IOA percentage of agreement criteria details and the different reporting options. Refer to BCBA® Task List (5th ed.) Section C-8: Evaluate the validity and reliability of measurement procedures.

Reactivity

Refers to the changes in an individual’s behavior due to the presence of an observer.

Measurement bias

When an observer’s measurement is influenced by an expectation or belief they hold rather than what actually occurred.

Observer reactivity

When an observer’s data collection or performance is influenced by the awareness that they are being monitored and evaluated.

Interobserver agreement (IOA)

A measurement of the degree to which two or more observers report the same values when observing and collecting data for the same behaviors/events.

Validity

Measurement is trustworthy when the measurement that produces data is applicable and specific to the target behavior of interest and the relevant dimension of that behavior; in other words, you …

Questing for the Gold Standard of IOA Agreement Read More

Accuracy

Measurement is trustworthy when the degree to which what was quantitatively measured (observed value) is representative of what actually occurred (true value); in other words, the data collected accurately represents …

Questing for the Gold Standard of IOA Agreement Read More

Reliability

Measurement is trustworthy when the degree to which repeating a measurement procedure under the same conditions produces the same result.

Measurement artifact

Data that represents an unwarranted and misleading picture of behavior because of how it was measured, typically a product of time-sampling, limiting measurement scales, or poorly scheduled measurement periods.

Observer drift

When observers collecting data have a shift/drift in how they interpret the operational definition of the target behavior that affects the validity of their data.

IOA Mini Manual

Math can be daunting, but practice can help. In response to growing calls for more IOA practice opportunities, download the free PTB Interobserver Agreement Mini Manual. Designed to promote retention and reduce anxiety, the PTB IOA Mini Manual features 30 practice questions that cover the 9 IOA formulas you need to know to prep for the exam.

Test your ABA Terminology ➠ Indicators of Trustworthy Measurement

Test your knowledge with PTB co-founder Dana Meller as she explains the specific indicator of trustworthy measurement that corresponds with repeated measurement yielding the same result. Included is a description of all three indicators of trustworthy measurement validity, accuracy, and reliability. Refer to BCBA® Task List (5th ed.) Section C-8: Evaluate the validity and reliability of measurement procedures.

Test your ABA Terminology ➠ Confounding Threats to Internal Validity

Test your knowledge of measurement confounds with PTB co-founder Dana Meller as she discusses one of the four types of threats to internal validity, with corresponding examples. Refer to BCBA® Task List (5th ed.) Section C-8: Evaluate the validity and reliability of measurement procedures.

Breaking Down a BCBA® Mock Exam Question: It's the IOA Battle Royal. Which Formula Gets Knocked Out?

Sharpen your measurement skills with PTB co-founder Dana Meller as she reviews a BCBA® mock exam question about IOA and helps explain the different formulas used for time sampling measurements. Dana covers four IOA formulas: Interval-by-Interval, Scored Interval, Trial-by-Tial, and Unscored Interval; and highlights key considerations and pitfalls associated with each. BCBA® Task List (5th ed.) Section C-8: Evaluate the validity and reliability of measurement procedures.

Breaking Down a BCBA® Mock Exam Question: The #1 Threat to Accuracy and Reliability

Test your measurement knowledge with PTB co-founder Dana Meller as she reviews a BCBA® mock exam question about the key indicators of trustworthy measurement, revealing the most significant threat facing accuracy and reliability. Dana provides insight into the distinctions between reliability, validity, and accuracy and how they relate to measurement. BCBA® Task List (5th ed.) Section C-8: Evaluate the validity and reliability of measurement procedures.

Threats to Validity Exposed: When Misleading Data Plays Tricks on Your Perception

Test your knowledge of measurement procedures with PTB co-founder Dana Meller as she explains, in detail, one of the threats to validity with several corresponding examples. Dana also shares a clever memorization study tip that will help you on the Big exam. Refer to BCBA® Task List (5th ed.) Section C-8: Evaluate the validity and reliability of measurement procedures.

Questing for the Gold Standard of IOA Agreement

PTB co-founder Dana Meller discusses Interobserver Agreement (IOA) and its significance in behavior analysis. Discover the desired percentage of agreement for IOA and its role in increasing data credibility and evaluating intervention effectiveness. Refer to BCBA® Task List (5th ed.) Section C-8: Evaluate the validity and reliability of measurement procedures.

Breaking Down a BCBA® Mock Exam Question: Winning Measurement System to Record Client Tolerance to Losing

Let's review and break down a BCBA® mock exam question about measurement techniques in ABA, specifically percentage of opportunity. PTB co-founder Dana Meller explains how to calculate the percentage of opportunities where the behavior is exhibited and why this method is superior to rate, count, and interval recording in certain situations. Refer to BCBA® Task List (5th ed.) Section C-3: Measure occurrence (e.g., count, frequency, rate, percentage).

Let's Break Down a BCBA® Mock Exam Question: Broadening the Scope of Trials-to-Criterion Measurements

PTB co-founder Dana Meller breaks down a BCBA® mock exam question about measurement used to derive trials to criterion data. Dana explains how this measurement can be a powerful tool in understanding and improving behavior performance across different dimensions. Refer to BCBA® Task List (5th ed.) Section C-6: Measure trials to criterion.

Test your ABA Terminology ➠ Cumulative Record Calculations

PTB co-founder Dana Meller reviews a question about using cumulative records and their significance in behavior analysis. Learn how to utilize cumulative graphs to track behavior over specific time periods and measure and calculate response rates for insightful data analysis. Refer to BCBA® Task List (5th ed.) Section C-11: Interpret graphed data.

Test your ABA Terminology ➠ Identify the Measurement System in This Scenario

Test your measurement, data display, and interpretation knowledge with PTB co-founder Dana Meller as she discusses one specific measurement system that measures the effect of the behavior after its occurrence. Here, Dana details the two ways someone could implement this type of measurement system, provides examples of scenarios and a variety of corresponding measurement methods, and highlights some of the pros of utilizing this type of measurement system. Refer to BCBA® Task List (5th ed.) Section C-2: Distinguish among direct, indirect, and product measures of behavior.

ABA Terminology ➠ You Down With OCC? 3 Essentials For Crafting an Epic Operational Definition

PTB co-founder Dana Meller breaks down the three defining features needed to craft a solid operational definition - Remember "OCC" for short. Refer to BCBA® Task List (5th ed.) Section C-1: Establish operational definitions of behavior.

Test your ABA Terminology ➠ Derivative Measures Explained

Put your knowledge of occurence measurement to the test with PTB co-founder Dana Meller as she discusses the two types of derivative measures, and each of their defining features and applications. Dana focuses on the derivative measure used for comparing two or more procedures, teaching clients new skills, evaluating mastery over a class of concepts, and the various measurement methods. Refer to BCBA® Task List (5th ed.) Section C-6: Measure trials to criterion.

Breaking Down a Mock BCBA® Exam Question: Understanding Equal-Interval Graphs & Logarithmic Scales

Test your knowledge of ABA graphs with PTB co-founder Dana Meller as she reviews a BCBA® mock exam question about equal-interval graphs, exploring their unique properties. Included, Dana explains the concept of proportional values and highlights the graph(s) which operate on a logarithmic scale, offering a different perspective on changes in behavior. Refer to BCBA® Task List (5th ed.) Sections C-10: Graph data to communicate relevant quantitative relations (e.g., equal-interval graphs, bar graphs, cumulative records); C-11: Interpret graphed data.

Test your ABA Terminology ➠ Fundamental Properties of Behavior Change

Test your ability to interpret graphed data with PTB co-founder Dana Meller. Here she explains the three fundamental properties of behavior change, focusing on one specific property akin to an average or median. Included are descriptions of level, trend, variability, and the y-axis. Refer to BCBA® Task List (5th ed.) Section C-11: Interpret graphed data.