Over-the-counter data

src: www.evidentlycochrane.net

Over-the-counter data (OTCD) is a design approach used in data systems, particularly educational technology data systems, in order to increase the accuracy of users' data analyses by better reporting data. The approach involves adhering to standards that are organized by five components: Label, Supplemental Documentation, Help System, Package/Display, and Content.

OTCD was inspired by the varied ways over-the-counter medication supports those using its contents. Just as it would be negligent for over-the-counter medication to contain no labeling, documentation, or other supports helping people to use its contents safely, it is deemed negligent for data systems to display data for educators without providing them with the necessary supports to best ensure it is used correctly when educators use the data to treat students' needs.

Video Over-the-counter data

Background

Inspired by the varied ways over-the-counter medication supports those using its contents, OTCD was created in 2010 and applied to the improvement of education data systems. Consider the way in which the Food and Drug Administration (FDA) requires over-the-counter medication to be accompanied by textual guidance proven to improve its use, deeming it negligent to do otherwise. With such guidance, patients may take over-the-counter medication with the goal of improving wellbeing while a doctor is not present to explain how to use the medication. No or poor medication labels have resulted in many errors and tragedy, as people are left with no way to know how to use the contents wisely.

Labeling conventions can translate to improved understanding on non-medication products, as well. Thus, in the way over-the-counter medicine's proper use is communicated with a thorough label and added documentation, a data system used to analyze student performance can include components to help users better comprehend the data it contains. Using an OTCD approach (i.e., following OTCD Standards) when communicating data involves following research-based recommendations likely to improve educators' understanding, analysis, and use of the data being displayed.

Nonetheless, labeling and tools within data systems to assist analyses are uncommon, even though most educators analyze data alone. Essentially, data systems and reports do not commonly present data in an "over-the-counter" format for educators, whose primary option for using data to treat students is thus compared to ingesting medicine from an unmarked or marginally marked container. Just as it would be negligent for over-the-counter medicine to contain no labeling, documentation, or other supports helping people to use its contents safely, it is negligent for data systems and reports to display data for educators without providing necessary supports to best ensure the data is used appropriately and thus has a desirable impact on students.

The recommendations summarized by OTCD Standards (below) are based on research in education and edtech, as well as research in a variety of other fields (e.g., behavioral economics, design, business analytics, technology, and more). An OTCD approach is not meant to replace educators' professional development or other interventions that improve data use, but it is an added solution that doesn't cost educators more time, money, or stress.

Maps Over-the-counter data

Significance

Educators have widely accepted the importance of using data to inform their treatment of students' needs. This is a good thing, as research touts the benefits of effective data use. Unfortunately, educators' widespread data use is not always a good thing. A significant portion - and some research claims most - of educators analyzing and using data are doing so incorrectly. For example in two U.S. Department of Education studies conducted in districts known for strong data use, teachers achieved only 48% accuracy when making data inferences involving basic statistical concepts. Thus educators are using data to inform decisions, but they do not always understand the data they are using. Since their data-misinformed decisions impact the students such decisions are meant to impact, this is a significant problem. Edtech products that present data to educators in an over-the-counter format - as opposed to simply "showing the data" and requiring educators to dig up resources to aid analyses - play an active role in improving educators' data use.

src: cdn-img.health.com

Over-the-Counter Data Study

Though numerous studies over the years have produced evidence on which the OTCD standards are based, one quantitative study in 2013 focused specifically on OTCD's direct impact on data analysis accuracy (as opposed to merely determining which edtech aspects educators prefer). 211 educators of varied backgrounds at nine schools in six different California school districts participated in the Over-the-Counter Data's Impact on Educators' Data Analysis Accuracy study. The study's premise was to determine the precise impact on analysis accuracy when data system reporting environments made data "over-the-counter," giving educators embedded supports like the kind over-the-counter medication provides for users in the form of labeling and supplemental documentation. Key findings were significant and hold implications for educators, educational technology and/or data system vendors, and anyone else involved in communicating data to educators:

Relating to primary research questions

• When a footer was present on education data reports viewed by study participants, educators' analyses of displayed data were 307% more accurate. When participants specifically indicated having used the footer (73% of the time), their data analyses were 336% more accurate.
• When a reference sheet (following templates) accompanied data reports viewed by study participants, educators' analyses of displayed data were 205% more accurate. When participants specifically indicated having used the sheet (50% of the time), their data analyses were 300% more accurate.
• When a reference guide (following templates) accompanied data reports viewed by study participants, educators' analyses of displayed data were 273% more accurate. When participants specifically indicated having used the guide (52% of the time), their data analyses were 436% more accurate.
• Overall, when any OTCD support accompanied data reports viewed by study participants, educators' analyses of displayed data were 264% more accurate. When participants specifically indicated having used the support (58% of the time), their data analyses were 355% more accurate.
• 87% of the study participants who received no supports indicated they would have used supports - like footers, reference sheets, or reference guides - if the supports had been available.
• Participants who received no supports averaged 11% data analysis accuracy (i.e., a score of 11% correct when answering varied data analysis questions such as, according to the viewed data, which area is most likely a site strength, which area is most likely a site weakness, which students did not score Proficient on the test, and which areas caused students to not score Proficient on the test).

Relating to Secondary Research Questions

• Educators' school site demographics (secondary independent variables: school level type, school level, academic performance, EL population, Socioeconomically Disadvantaged population, and Students with Disabilities population) had no significant impact on data analysis accuracy or support use.
• Educator demographics (secondary independent variables: veteran status, current professional role, perception of own data analysis proficiency, data analysis professional development time, and number of graduate-level educational measurement courses) had no significant impact on data analysis accuracy or support use.
• Minor variations in each support's framing/format (mainly in terms of length and color usage) had no significant impact on data analysis accuracy or support use.

src: jdh.adha.org

Over-the-Counter Data (OTCD) Standards

OTCD Standards involve embedding data analysis supports directly within reporting environments and adhering to best practices concerning design. OTCD Standards were designed to be used by anyone communicating data to educators and to be reflected in the tool(s) through which the data is communicated (e.g., data report, data system, or other edtech product with a data component). Their purpose is to foster optimal educator ("user") understanding, analysis, and use of the data being provided.

src: bjgp.org

Mentions of OTCD

Organizations' and publications' mentions of OTCD include:

• California Council on Teacher Education (CCTE) (conference poster presentation 18 in the 2013 conference program) and also "Remedying Educators' Data Analysis Errors with Over-the-Counter Data" article, pages 14-21 in CCNews: Newsletter of the California Council on Teacher Education, 24(4)
• California Educational Research Association (CERA) (presentation explaining OTCD components; presentation is described on page 27 of the conference program)
• Classroom 2.0 Learning 2.0 (click link for OTCD conference presentation)
• Connect 2013: Canada's Learning & Technology Conference (click Tuesday tab/link to view OTCD presentation mention)
• Connected Educators (Connected Educators Community Directory)
• Connected Educators (Collective Twitter List)
• CrunchBase Technology Database (featured)
• EdSurge (article); also, Bio; Newsletter #114 (April 17, 2013), Newsletter #115 (April 24, 2013), and Newsletter #116 (May 1, 2013) referenced and linked to OTCD study; and Newsletter #121 (June 5, 2013) referenced OTCD article
• EdTech Women (article)
• Edukwest (article)
• K-12 Online (conference presentation)
• Learning Forward (formerly National Staff Development Council) (scroll to presentation G09 and see slides 30 and 35)
• National Center for Education Statistics (NCES) Summer Forum at STATS-DC (mentioned when presentation described on page 49 was presented)
• The Network for Public Education (click the Friends & Allies page #/bubble in Southern CA until you're able to click the pin on Laguna Beach, CA)
• Startup America Partnership (featured)
• Teacher Tech (article)
• Technology Information Center for Administrative Leadership (TICAL) (click the link for the website/URL associated with the School Leadership Summit session description to view the presentation in which OTCD is discussed)

src: finrazor.cdn.prismic.io

References

Source of article : Wikipedia