Cognitive Labs

A series of cognitive labs has helped inform the development of both science and math test items. Issues tested during the cognitive labs include age appropriateness and ease of use of the computer interface, accessibility of the language, the effectiveness of various forms of supports, and cognitive processing of test items. Interviews were conducted in Spanish and English, while the test items appeared in English.
Here are some more details about the ONPAR cognitive lab sessions:

Dates: ongoing.

Number of students: Approximately 10-20 students, interviewed one at a time, are tested in each round of cognitive labs.  Ideally, 50% beginning ELLs and 50% native English or exited students are tested. Participants with strong science and math ability are ideal to test the validity of assessment items.

Time Requirements: Each session lasts approximately 45 minutes. The scheduling, of course, is determined by the school with regard for what causes the least amount of disruption to instructional time. The lab consists of students viewing a tutorial to orient them to the help features and item types, followed by 5-10 science/ math questions.  During each item, students are asked to think aloud and share their comments.

Space Requirements: Access to a quiet room in which to conduct the interviews is required.  The test is administered on a laptop computer provided by CAL, so the room must also have an accessible electrical outlet.

Parental Permission: Bilingual permission forms are available and must be returned before the interview may be conducted. Each interview is videotaped and all information is kept confidential and used only to assist with item development.

Staff requirements: Two to three trained CAL staff members, one of whom is a Spanish speaker participate in the interviewing process.  A local contact at the school is needed to assist with identifying students, and locating them for each of the interviews. Additionally, information about the students’ English and science/math proficiency will be requested.  An incentive is provided to the local school or teacher assisting with the study and completing a demographic survey about the student participants.   

Contact:
e-mail: onpar@cal.org

Back to top