4 Assessment Delivery

Chapter 4 presents the processes and procedures used to deliver the Dynamic Learning Maps^® (DLM^®) Alternate Assessment System in 2021–2022. As described in earlier chapters, the DLM System uses computer-delivered alternate assessments that provide the opportunity for students with the most significant cognitive disabilities to show what they know and can do in English language arts (ELA) and mathematics. DLM assessments are administered in small groups of items called testlets. The DLM assessment system incorporates accessibility by design and is guided by the core beliefs that all students should have access to challenging, grade-level content and that educators adhere to the highest levels of integrity in providing instruction and administering assessments based on this challenging content.

This chapter begins with an overview of the general features of assessment administration, including the Kite^® Suite used to assign and deliver assessments, testlet formats (computer-delivered and educator-administered), and accessibility features. Next, we describe the key features of the Instructionally Embedded assessment model. We explain how a student’s First Contact survey is used to recommend testlet linkage level for each Essential Element (EE) in the Instruction and Assessment Planner. We also describe administration resources and materials available to test administrators and district users, followed by test administrator responsibilities and procedures and test security. We then provide evidence from the DLM System, including administration time, device usage, linkage level selection, evaluation of blueprint coverage, and accessibility support selections. We also present evidence from assessment administration monitoring, including test administration observations, formative monitoring, and data forensics reports. Finally, we present evidence from test administrators, including user experience with the DLM System, students’ opportunity to learn, ratings of items on the First Contact survey, and educator cognitive labs.

4.1 Overview of General Administration Features

Based on students’ support needs, DLM assessments are designed to be administered in a one-on-one, student/test administrator format. Most test administrators are the special education educators of the students, as they are best equipped to provide the most conducive conditions to elicit valid and reliable results. Assessment administration processes and procedures also reflect the priorities of fairness and validity through a broad array of accessibility tools and features that are designed to provide access to assessment content and materials as well as limit construct-irrelevant variance.

This section describes the key, overarching features of DLM assessment administration, including the online testing platform, the Kite Suite, the two assessment delivery modes, and accessibility features.

4.1.1 The Kite Suite

The DLM alternate assessments are managed and delivered using the Kite Suite, which was designed and developed to meet the needs of the next generation of large-scale assessments for students with significant cognitive disabilities. Educators and students use the following applications: Kite Educator Portal and Kite Student Portal. The Kite Suite was developed with IMS Global Question and Test Interoperability item structures and Accessible Portable Item Protocol tagging on assessment content to support students’ Personal Needs and Preferences (PNP) Profiles (see the Accessibility section below) and World Wide Web Consortium Web Content Accessibility Guidelines. Kite Student Portal and supported browsers for Kite Educator Portal are published on the DLM website and in the Technology Specifications Manual (Dynamic Learning Maps Consortium, 2022b) linked on each state’s DLM webpage.

4.1.1.1 Kite Educator Portal

Kite Educator Portal is the administrative application where district staff and educators manage student data, assign instructionally embedded assessments, access resources needed for each assigned testlet, and retrieve reports.

• Assessment administrators, who are usually educators, use Kite Educator Portal to manage all student data. They are responsible for checking class rosters of the students who are assigned to take DLM testlets and for completing the PNP and First Contact surveys for each student (see the respective Accessibility and Linkage Level sections below for more information on the PNP and First Contact surveys, respectively).
• Educators create instructional plans in the system by choosing which EEs and linkage levels they intend to teach and recording those plans in a part of Kite Educator Portal called the Instruction and Assessment Planner (see section 4.2.2.3 for more information). After assigning the EE and linkage level, the test administrator retrieves information to support instruction on the associated nodes. When test administrators decide a student is ready to assess, they return to the Instruction and Assessment Planner and confirm the testlet assignment. See section 4.2 on key administration features of the Instructionally Embedded model for more information on testlet selection.
• After each testlet is assigned to a student, the system delivers a Testlet Information Page (TIP) through Kite Educator Portal. The TIP, which is unique to the assigned testlet, is a PDF that contains any instructions necessary to prepare for testlet administration. See section 4.3.1.2.1 of this chapter for more information.
• During instructionally embedded assessments, the Instruction and Assessment Planner displays information about student mastery for assessed EEs and linkage levels. Educators can also download or print reports on demand, including the student’s history of instructional plans created in the Instruction and Assessment Planner as well as a report that shows the EEs and linkage levels for which the student has completed a testlet or a testlet assignment is pending.

4.1.1.2 Kite Student Portal

Kite Student Portal is the platform that allows students to log in and complete assigned testlets. Practice activities and released testlets are also available to students and test administrators through Kite Student Portal (see Chapter 3 of this manual for more information). Kite Student Portal prevents students from accessing unauthorized content or software while taking assessments. Kite Student Portal is supported on devices running Windows or macOS (OSX), on Chromebooks, and on iPads.

Kite Student Portal provides students with a simple, web-based interface with student-friendly and intuitive graphics. The student interface used to administer the DLM assessments was designed specifically for students with the most significant cognitive disabilities. It maximizes space available to display content, decreases space devoted to tool-activation buttons (i.e., read aloud), and minimizes the cognitive load related to test navigation and response entry. An example of a screen used in an ELA testlet is shown in Figure 4.1. The blue BACK and green NEXT buttons are used to navigate between screens. The octagonal EXIT DOES NOT SAVE button allows the user to exit the testlet without recording any responses. The READ button plays an audio file of synthetic speech for the content on screen. Synthetic read aloud is the only accessibility feature with a tool directly enabled through each screen in the testlet. Further information regarding accessibility is provided in section 4.1.3 of this chapter.

Figure 4.1: An Example Screen From the Student Interface in Kite Student Portal

4.1.1.3 Local Caching Server

During DLM assessment administration, schools with unreliable network connections have the option to use the Local Caching Server (LCS). The LCS is a specially configured machine that resides on the local network and communicates between the testing machines at the testing location and the main testing servers for the DLM System. The LCS stores testing data from Kite Student Portal in an internal database; if the upstream network connection becomes unreliable or variable during testing, students can still continue testing, and their responses are transmitted to the Kite servers as bandwidth allows. The LCS submits and receives data to and from the DLM servers while the students are taking tests. The LCS must be connected to the internet between testlets to deliver the next testlet correctly.

4.1.2 Assessment Delivery Modes

The DLM System includes testlets designed to be delivered via computer directly to the student and testlets designed for the test administrator to administer outside the system and record responses in the system. The majority of testlets were developed for the computer-delivered mode because evidence suggested the majority of students with the most significant cognitive disabilities are able to interact directly with the computer or are able to access the content of the assessment on the computer with navigation assistance from a test administrator (Nash et al., 2016). Educator-administered testlets include all testlets at the Initial Precursor linkage level, some higher linkage level mathematics testlets requiring manipulatives, some alternate forms for students who are blind or who have visual impairments, and all writing testlets. A brief overview of the two types of testlets is included in the following sections. See Chapter 3 of this manual for a complete description of DLM testlets.

4.1.2.1 Computer-Delivered Assessments

Most DLM alternate assessments are delivered directly to students by computer through the Kite Suite. Computer-delivered assessments were designed so students can interact independently with the computer, using special assistive technology devices such as alternate keyboards, touch screens, or switches as necessary.

The computer-delivered testlets include various item types, including single-select multiple choice with three response options and text or images as response options, multiple choice multi-select with text or images as response options, matching items from two lists, sorting objects into categories, and highlighting selected text.

4.1.2.2 Educator-Administered Assessments

Some testlets were designed to be administered directly by the test administrator outside the Kite Suite. The Kite Suite delivers the testlet, but the test administrator is responsible for setting up the assessment, delivering it to the student, and recording student responses in Kite.

There are three general categories of educator-administered testlets.

1. Testlets with content designed for students who are developing symbolic understanding or who may not yet demonstrate symbolic understanding (Initial Precursor and some Distal Precursor).
2. Some mathematics testlets at higher linkage levels for which representing the content online would make the task too abstract and introduce unnecessary complexity to the item. Manipulatives are often used in this case, especially for students with blindness or visual impairment.
3. All writing assessments.

All three types of educator-administered testlets have some common features, which are described in Chapter 3 of this manual.

4.1.3 Accessibility

The DLM System was designed to be optimally accessible to diverse learners through accessible content (see Chapter 3 of this manual) as well as through updating the recommended linkage level driven by the First Contact survey and prior performance (see section 4.2 of this chapter for details). The interface in the Kite Suite was also designed to be easy to use to support accessibility. Consistent with the DLM learning map and item and test development practices described in earlier chapters (see Chapter 2 and Chapter 3, respectively), principles of universal design for assessment were applied to administration procedures and platforms. Decisions were largely guided by universal design for assessment principles of flexibility of use and equitability of use through multiple means of engagement, multiple means of representation, and multiple means of action and expression.

In addition to these considerations, a variety of accessibility supports are made available in the DLM assessment system. The Accessibility Manual (Dynamic Learning Maps Consortium, 2021a) outlines a six-step process for test administrators and Individualized Education Program (IEP) teams to use in making decisions about accessibility supports. This process begins with confirming the student meets the DLM participation guidelines and continues with the selection, administration, and evaluation of the effectiveness of accessibility supports. Test administrators select supports for each student in the PNP. The PNP can be completed any time before beginning testing. It can also be changed during testing as a student’s needs change. Once updated, the changes appear the next time the student is logged in to the Kite Suite. All test administrators are trained in the use and management of these features.¹³ See Chapter 9 for a complete description of test administrator training.

4.1.3.1 Overview of Accessibility Supports

Accessibility supports considered appropriate to use during administration of computer-delivered and educator-administered testlets are listed in the Accessibility Manual (Dynamic Learning Maps Consortium, 2021a). A brief description of the supports is provided here (see the Accessibility Manual for a full description of each support and its appropriate use). Supports are grouped into three categories: those provided through the PNP, those requiring additional tools or materials, and those provided outside the system. Additional techniques that are traditionally thought of as accommodations are considered allowable practices in the DLM assessment system. These are described in a separate section below.

4.1.3.1.1 Category 1: Supports Provided Within the DLM System via the PNP

Online supports include magnification, invert color choice, color contrast, and overlay color. Educators can test these options in advance to make sure they are compatible and provide the best access for students. Test administrators can adjust the PNP-driven accessibility during the assessment, and the selected options are then available the next time the student logs in to Kite Student Portal.

• Magnification. Magnification allows educators to choose the amount of screen magnification during testing.
• Invert color choice. In invert color choice, the background is black and the font is white.
• Color contrast. The color contrast allows educators to choose from several background and lettering color schemes.
• Overlay color. The overlay color is the background color of the test.

4.1.3.1.2 Category 2: Supports Requiring Additional Tools or Materials

These supports include braille, switch system preferences, iPad administration, and use of special equipment and materials. These supports are all recorded in the PNP even though the one-switch system is the only option actually activated by the PNP.

• Uncontracted braille. Uncontracted braille testlets are available during the testing window for grades 3–5 at the Target and Successor levels and for grades 6 through high school at the Proximal Precursor, Target, and Successor levels. The standard delivery method is to deliver braille-ready files electronically to the school or district for local embossing as each testlet is assigned. The Kite Suite also delivers the identical general testlet form. After the student takes the testlet in its embossed form, the test administrator transfers the student’s answers into Kite Student Portal.
• Single-switch system. Single-switch scanning is activated using a switch set up to emulate the Enter key on the keyboard. Scan speed, cycles, and initial delay may be configured.
• Two-switch system. Two-switch scanning does not require any activation in the PNP. Kite Student Portal automatically supports two-switch step scanning.
• Administration via iPad. Students may take the assessment via iPad.
• Adaptive equipment used by student. Test administrators may use any familiar adaptive equipment needed for the student.
• Individualized manipulatives. Individualized manipulatives are suggested for use with students rather than requiring educators to have a standard materials kit. Recommended materials and rules governing materials selection or substitution are described in the TIP (see section 4.3.1.2.1 of this chapter for more information on TIPs). Having a familiar concrete representation ensures that students are not disadvantaged by objects that are unfamiliar or that present a barrier to accessing the content.
• BVI forms. Alternate forms for students who are blind or have visual impairments (BVI) but do not read braille were developed for certain EEs and linkage levels. BVI testlets are educator-administered, requiring the test administrator to engage in an activity outside the system and enter responses into Kite Student Portal. The general procedures for administering these forms are the same as with other educator-administered testlets. Additional instructions include the use of several other supports (e.g., human read aloud, test administrator response entry, individualized manipulatives) as needed. When onscreen materials are being read aloud, test administrators are instructed to (1) present objects to the student to represent images shown on the screen, and (2) change the object language in the testlet to match the objects being used. Objects are used instead of tactile graphics, which are too abstract for the majority of students with the most significant cognitive disabilities who are also blind. However, test administrators have the option to use tactile graphics if their student can use them fluently.

4.1.3.1.3 Category 3: Supports Provided Outside the DLM System

These supports require actions by the test administrator, such as reading the test, signing or translating, and assisting the student with entering responses.

• Human read aloud. The test administrator may read the assessment to the student. Test administrators are trained to follow guidance to ensure fidelity in the delivery of the assessment. This guidance includes the typical tone and rate of speech, as well as avoiding emphasizing the correct response or important information that would lead the student to the correct response. Test administrators are trained to avoid facial expressions and body language that may cue the correct response and to use exactly the words on screen, with limited exceptions to this guideline, such as the use of shared reading strategies on the first read in ELA testlets. Finally, guidance includes ensuring that answer choices are always read in the same order as presented on the screen, with comprehensive examples of all item types. For example, when answer choices are in a triangle order, they are read in the order of top center, bottom left, and bottom right. In most cases, test administrators are allowed to describe graphics or images to students who need those described. Typically, this additional support is provided to students who are blind or have visual impairments. Alternate text for graphics and images in each testlet is included in the TIP as an attachment after the main TIP information. Test administrators who need to read alternate text have the Kite Suite open and the TIP in front of them while testing so they can accurately read the alternate text provided on the TIP with the corresponding screen. Human read aloud is allowed in either subject. The reading EEs included in the blueprints focus on comprehension of narratives and informational texts, not decoding. The read aloud support is available to any student who can benefit from decoding support in order to demonstrate the comprehension skills in the tested EEs.
• Sign interpretation of text. If the student requires sign language to understand the text, items, or instructions, the test administrator is allowed to use the words and images on the screen as a guide while signing for the student using American Sign Language, Signed Exact English, or any individualized signs familiar to the student. The test administrator is also allowed to spell unfamiliar words when the student does not know a sign for that word and accept responses in the student’s sign language system. Sign is not provided via human or avatar video because of the unique sign systems used by students with the most significant cognitive disabilities who are also deaf/hard of hearing.
• Language translation of text. The DLM assessment system does not provide translated forms of testlets because of the unique cognitive and communication challenges for students taking DLM alternate assessments and because students who are English learners speak such a wide variety of languages; providing translated forms appropriate for all DLM-eligible students to cover the entire blueprint would be nearly impossible. Instead, test administrators are supplied with instructions regarding supports they can provide based on (1) each student’s unique combination of language-related and disability-related needs, and (2) the specific construct measured by a particular testlet. For students who are English learners or who respond best to a language other than English, test administrators are allowed to translate the text for the student. The TIP includes information about exceptions to the general rule of allowable translation. For example, when an item assesses knowledge of vocabulary, the TIP includes a note that the test administrator may not define terms for the student on that testlet. Unless exceptions are noted, test administrators are allowed to translate the text for the student, simplify test instructions, translate words on demand, provide synonyms or definitions, and accept responses in either English of the student’s native language.
• Test administrator enters responses for student. During computer-delivered assessments, if students are unable to physically select their answer choices themselves due to a gap between their accessibility needs/supports and the Kite Suite, they are allowed to indicate their selected responses to the test administrator through their typical communication modes (e.g., eye gaze, verbal). The test administrator then enters the response. The Test Administration Manual provides guidance on the appropriate use of this support to avoid prompting or misadministration. For example, the test administrator is instructed not to change tone, inflection, or body language to cue the desired response or to repeat certain response options after an answer is provided. The test administrator is also instructed to ensure the student continues to interact with the content on the screen.
• Partner-assisted scanning. Partner-assisted scanning is a commonly used strategy for students who do not have access to or familiarity with an augmentative or communication device or other communication system. These students do not have verbal expressive communication and are limited to response modes that allow them to indicate selections using responses such as eye gaze. In partner-assisted scanning, the communication partner (the test administrator in this case) “scans” or lists the choices that are available to the student, presenting them in a visual, auditory, tactual, or combined format. For test items, the test administrator might read the stem of an item to the student and then read the answer choices aloud in order. In this example, the student could use a variety of response modes to indicate a response. Test administrators may repeat the presentation of choices until the student indicates a response.

4.1.3.2 Additional Allowable Practices

The Kite Student Portal user interface was specially designed for students with the most significant cognitive disabilities. Testlets delivered directly to students via computer were designed to facilitate students’ independent interaction with the computer, using special devices such as alternate keyboards, touch screens, or switches as necessary. However, because computerized testing was new to many students using the DLM alternate assessment, the DLM Governance Board recognized that students would need various levels of support to interact with the computer. Test administrators are provided general principles for the allowable practices when the supports built into the system do support a student’s completely independent interaction with the system.

To help make decisions about additional supports for computer-delivered testlets, test administrators receive training to follow two general principles. First, students are expected to respond to the content of the assessment independently. No matter which additional supports IEP teams and test administrators selected, all should be chosen with the primary goal of student independence at the forefront. Even if more supports are needed to provide physical access to the computer-based system, students should be able to interact with the assessment content and use their normal response modes to indicate a selection for each item. Second, test administrators are to ensure that students are familiar with the chosen supports. Ideally, any supports used during assessment are also used consistently during routine instruction. Students who have never received a support prior to the testing day are unlikely to know how to make the best use of the support.

In order to select the most appropriate supports during testing, test administrators are encouraged to use their best professional judgment and to be flexible while administering the assessment. Test administrators are allowed to use additional supports beyond PNP options. The supports detailed below in Table 4.1 are allowed in all computer-delivered and educator-administered testlets unless exceptions are noted in the TIP.

Table 4.1: Additional Allowable Practices

Practice	Explanation
Breaks as needed	Students can take breaks during or between testlets. Test administrators are encouraged to use their best judgment about the use of breaks. The goal should be to complete a testlet in a single session, but breaks are allowed if the student is fatigued, disengaged, or having behavioral problems that can interfere with the assessment. Kite Student Portal allows for up to 90 minutes of inactivity without timing out so that test administrators and students can pause for breaks during testlet administration. In cases in which administration begins but a short break is not sufficient for the student, the EXIT DOES NOT SAVE button can be used to exit the testlet (see Figure 4.1). The test administrator and student can then return to it and start over at another time.
Individualized student response mode†	The nodes assessed in the educator-administered testlets do not limit responses to certain types of expressive communication; therefore, all response modes are allowed. Test administrators can represent answer choices outside the system to maximize the student’s ability to respond. For example, for students who use eye gaze to communicate, test administrators can represent the answer choices in an alternate format or layout to ensure the student can indicate a clear response.
Use of special equipment for positioning	For students who need special equipment to access the test material such as a slant board for positioning or Velcro objects on a communication board, test administrators are encouraged to use the equipment to maximize the student’s ability to provide a clear response.
Navigation across screens	For students who have limited experience with, motor skills for, and/or devices for interacting directly with the computer, the test administrator can assist students to navigate across screens or enter the responses.
Use of interactive whiteboard	If the student has a severe visual impairment and needs larger presentation of content than the highest magnification setting provides, the test administrator can use an interactive whiteboard or projector or a magnification device that works with the computer screen to enlarge the assessment to the needed size.
Represent the answer options in an alternate format	Representing the answer options in an alternate format is allowed as long as the representation does not favor one answer choice over another. For instance, if the test administrator is presenting the answer choices to a student on a communication board or using objects to represent the answer choices, the correct answer choice cannot always be closest to the student or in the same position each time.
Use of graphic organizers	If the student is accustomed to using specific graphic organizers, manipulatives, or other tools during instruction, the use of those tools is allowable during the DLM alternate assessment.
Use of blank paper	If the student requires blank, lined, or unlined paper, this can be provided. Once there is any writing on the paper, it becomes a secure testing document and needs to be disposed of and shredded at the conclusion of the testing session.
Generic definitions	If the student does not understand the meaning of a word used in the assessment, the test administrator can define the term generically and allow the student to apply that definition to the problem or question in which the term is used. Exceptions to this general rule are noted in the TIP for specific testlets.
† Allowed using speech, sign, or language translation unless prohibited for a specific testlet.

Although there are many supports and practices allowable for computer-delivered and educator-administered testlets, there are also practices that test administrators are trained to avoid, including the following:

• Repeating the item activity again after a student has responded or in any other way prompting the student to choose a different answer
• Using physical prompts or hand-over-hand guidance to the correct answer
• Removing answer choices or giving hints to the student
• Rearranging objects to prompt the correct answer—for example, putting the correct answer closer to the student

Test administrators are encouraged to ask any questions regarding whether a support is allowable via the DLM Service Desk or through their state education agency.

4.2 Key Features of the Instructionally Embedded Assessment Model

As briefly described in Chapter 1, the DLM assessment system has two available models. This manual describes the Instructionally Embedded assessment model. Consistent with the DLM Theory of Action described in Chapter 1, the DLM assessment administration features reflect multidimensional, non-linear, and diverse ways that students learn and demonstrate their learning. Test administration procedures therefore use multiple sources of information to assign testlets, including student characteristics, prior performance, and educator judgment.

In the Instructionally Embedded model, the DLM System is designed to assess student learning throughout the year and features flexibility in the choice of assessment content and in assessments to support the timely use of data to inform instructional planning. Each testlet is administered after instruction in fall and spring testing windows so that testing informs teaching and benefits students’ learning. This assessment model yields summative results based on all instructionally embedded assessments across both windows.

With the exception of writing testlets, each testlet contains items for one EE and one linkage level. In reading and mathematics, items in a testlet are aligned to nodes at one of five linkage levels for a single EE. Writing testlets cover multiple EEs and are delivered at one of two levels: emergent (which corresponds with Initial Precursor and Distal Precursor linkage levels) or conventional (which corresponds with Proximal Precursor, Target, and Successor linkage levels).

This section describes the features of the Instructionally Embedded assessment model, including the Instruction and Assessment Planner, EE selection, linkage level selection, and test administration windows.

4.2.1 Instruction and Assessment Planner

The Instruction and Assessment Planner, housed in Educator Portal, is designed to facilitate instructionally embedded assessment administration. Students with the most significant cognitive disabilities are best able to demonstrate what they know and can do using a cyclical approach to their instruction, assessment, and evaluation, as opposed to being assessed at the end of a semester or school year on a mass of instruction they must recall from prior weeks and months (Brookhart & Lazarus, 2017). The instructionally embedded model of the DLM System encourages this approach by having test administrators choose an EE and linkage level, develop and deliver instruction for the chosen EE, and then assess the student when the educator determines the student is ready. The Instruction and Assessment Planner is the tools test administrators use to choose EEs and linkage levels for assessment. The planner was designed with feedback collected from educator cadres and focus groups (e.g., Clark et al., 2022) to ensure that the interface included the information that would be most informative for instruction and assessment administration. Additional information on instructionally embedded assessment can be found in the Test Administration Manual (DLM Consortium, 2021). Furthermore, a short video about how to use the Instruction and Assessment Planner is provided on the DLM website.

4.2.2 Testlet Assignment

This section describes how test administrators choose the EEs and linkage levels each student is assessed on. Educators complete the First Contact survey, which is used to recommended linkage levels for assessment. Test administrators take into account blueprint coverage requirements to guide the EEs that are selected for assessment, and the system recommendations for selecting the appropriate linkage level for each EE.

4.2.2.1 First Contact Survey

The First Contact survey is a survey of learner characteristics that covers a variety of areas, including communication, academic skills, attention, and sensory and motor characteristics. A completed First Contact survey is required for each student prior to the assignment of testlets.

The items on the First Contact survey are categorized into the following sections:

• Special Education
• Sensory Capabilities
• Motor Capabilities and Health
• Computer Instruction
• Communication (Expressive and Receptive)
• Language
• Academics

Four sections of the First Contact survey are used to assign students to complexity bands in reading, mathematics, and writing: Expressive Communication, Reading Skills, Mathematics Skills, and Writing Skills. For expressive communiction, reading, and mathematics, there are four complexity bands (from lowest to highest): Foundational, Band 1, Band 2, and Band 3. In writing, there are two complexity bands (from lowest to highest): Emergent and Conventional. First Contact survey items used for determining complexity bands are included in Appendix D.1. Based on the educator’s responses, the student’s assigned complexity band is automatically calculated and stored in the system.

• For the ELA reading testlets, Kite Suite uses the responses from the Expressive Communication and Reading Skills questions to assign a student to one of four complexity bands.
• For the mathematics testlets, Kite Suite uses the responses from the Expressive Communication and Math Skills questions to assign a student to one of four complexity bands.
• For writing testlets, Kite Suite uses the responses from the Writing Skills question to assign a student to one of two complexity bands.

For reading and mathematics, if a different complexity band is indicated between the two sets of questions (Expressive Communication and the subject area questions), the system selects the lower band. The goal is to present a testlet that is approximately matched to a student’s knowledge, skills, and understandings. That is, within reason, the system should recommend a testlet that is neither too easy nor too difficult and that provides a positive experience for the student entering the assessment. The correspondence among common student characteristics indicated on the First Contact survey, the corresponding First Contact complexity bands, and the recommended linkage levels are shown in Table 4.2.¹⁴ For a description of linkage levels, see Chapter 2 of this manual.

Table 4.2: Correspondence Among Student Characteristics Recorded on First Contact Survey, Complexity Bands, and Linkage Levels

Common First Contact survey responses about the student	First Contact complexity band	Linkage level
Does not use speech, sign, or augmentative and alternative communication; does not read any words when presented in print (reading); or does not sort objects (math)	Foundational	Initial Precursor
Uses one word, sign, or symbol to communicate; recognizes symbols (reading); or sorts symbols (math)	Band 1	Distal Precursor
Uses two words, signs, or symbols to communicate; reads at the primer to second grade level (reading); or adds/subtracts up to 80% of the time (math)	Band 2	Proximal Precursor
Regularly combines three or more spoken words to communicate for a variety of purposes; able to read print at the third-grade level or above (reading); or regularly add/subtract and form groups of objects (math)	Band 3	Target

The writing First Contact item is used to recommend the two types of writing testlets: emergent and conventional. Students whose educators indicated they wrote by scribbling, copying or using word bands, or writing words corresponding to some sounds are recommended an emergent-level testlet. Students whose educator indicated they wrote words or simple phrases, sentences or complete ideas, or paragraph-length text without copying and using spelling are recommended the conventional writing testlet.

4.2.2.2 Essential Element Selection

The Instructionally Embedded model assessments blueprints are unique in that they specify a pool of EEs that are available for assessment. Test administrators are responsible for choosing the EEs for assessment from the pool that meet a pre-specified set of criteria (e.g., “Choose three EEs from within Claim 1.”) to achieve blueprint coverage. The same criteria apply to both the fall and spring windows.

Test administrators can also test beyond what is required by the blueprint to support instruction if they choose. Exceeding the blueprint requirements is acceptable but not expected. The test blueprints are available on the DLM website as well as in the Instruction and Assessment Planner in Kite Educator Portal.

Blueprint requirements and the number of EEs available vary by grade, subject, and claim/conceptual area. Figure 4.2 shows the portion of the grade 3 mathematics blueprint that describes the coverage requirements for Claim 1. Four EEs are nested within two conceptual areas. To meet the blueprint coverage requirements for Claim 1, test administrators chose one of the three available EEs in Conceptual Area 1 (M.C1.1) and the EE available in Conceptual Area 3 (M.C1.3).

Figure 4.2: Excerpt from Grade 3 Mathematics Blueprint

Note. EE = Essential Element. The full grade 3 blueprint also includes selection criteria for claims 2, 3, and 4. For information on the development of the blueprints, see Chapter 2 of this manual.

4.2.2.3 Linkage Level Selection

In the fall window, the Kite Suite uses the subject-specific complexity band, as determined by responses to the First Contact survey, to recommend a linkage level for each EE for the student in the Instruction and Assessment Planner. The test administrator can either select an EE at the recommended linkage level or select a different linkage level. Test administrators are encouraged to use the system’s recommended linkage level if they are unsure of the best linkage level for the student. The test administrator can change the linkage level at any time before a testlet is assigned.

Test administrators are encouraged to choose levels that provide an appropriate balance of challenge and access for the student. The choice should be a level that represents a good instructional target for the student. Choosing a linkage level that is too low, such as one the student has already mastered, is not advisable. States may also provide additional guidance to test administrators on choosing the best linkage level for students.

In the spring window, the Kite Suite uses fall performance on assessed EEs to recommend the linkage level.

• The spring recommended linkage level was one linkage level higher than the linkage level assessed in fall if the student responded correctly to at least 80% of items. If the assessed fall linkage level was at the highest linkage level (i.e., Successor), the recommendation remained at that level.
• The spring recommended linkage level was one linkage level lower than the linkage level assessed in fall if the student responded correctly to less than 35% of items. If the assessed fall linkage level was at the lowest linkage level (i.e., Initial Precursor), the recommendation remained at that level.
• The spring recommended linkage level was at the same linkage level assessed during the fall window if the student responded correctly to between 35% and 80% of items.
• If a student did not test on a given EE in the fall, the spring recommended linkage level was based on the First Contact complexity band, as defined in Table 4.2.

For EEs that were not tested in the fall, the system uses the First Contact survey to recommend the spring linkage level.

4.2.3 Assessment Administration Windows

Instructionally embedded assessments are administered in testlets, and, with the exception of writing, each testlet contains items for one EE and one linkage level.

4.2.3.1 Fall Window

Test administrators use blueprint coverage criteria to decide which EEs and linkage levels to assess for each student throughout the fall window. In 2021–2022, the fall window occurred between September 13, 2021, and December 17, 2021. States were given the option of using the entire window or setting their own dates within the larger window. All states chose to use the full fall window in 2021–2022.

4.2.3.2 Spring Window

Test administrators use the same blueprint coverage criteria to make EE and linkage level selections for the spring window. They can choose, teach, and assess the same EEs and linkage levels as the fall window, or they can choose different EEs and/or linkage levels. In 2021–2022, the spring window occurred between February 7, 2022, and May 20, 2022. States were given the option of using the entire window or setting their own dates within the larger window. Across all states, the spring window ranged from 12–15 weeks.

4.2.4 Summary of the Instructionally Embedded Assessment Process

The instructionally embedded assessment process is summarized here:

1. Choose one or more EEs and linkage levels for instruction.
2. Record the choice of the EEs and linkage levels to create an instructional plan. Save the plan.
3. Retrieve instructional resources and deliver instruction.
4. Confirm the EE and linkage level to assign a testlet.
5. Review the TIP and prepare for testlet administration.
6. Administer the testlet to the student.

The test administrator first consults the blueprint and available mini-maps on the DLM website to aid in decision-making. Each testlet is packaged separately, so the test administrator can select multiple EEs at once and manage assessment sessions within the larger testing window. Test administrators also have the option to assess more than once on the same or different EEs, as long as additional testlets were available. The Instruction and Assessment Planner does not allow a student to be assigned the same testlet twice.

Next, the test administrator has the option to accept the system’s linkage level recommendation in the Instruction and Assessment Planner or choose another linkage level.

In the Instruction and Assessment Planner, the recommended linkage level is indicated with a bookmark icon. As shown in Figure 4.3, the blueprint requirement for Claim 1 mathematics for grade 3 was satisfied because the test administrator chose and assessed the student on at least one EE from conceptual area M.C1.1 and one EE from conceptual area M.C1.3. The test administrator assessed the student on the Distal Precursor linkage level for M.EE.3.NBT.2, which was the recommended linkage level. The student mastered the skills of the linkage level, as indicated by the star and the words “Complete: Mastered” along with the date the student was assessed. Then, for M.EE.3.OA.4, the Distal Precursor linkage level was again the recommended level, but the test administrator chose instead to assess the student on the Proximal Precursor linkage level. Again, the student mastered the skills of the linkage level as indicated.

Figure 4.3: Linkage Level Selection in Educator Portal

4.3 Resources and Materials

Test administrators, school staff, district staff, and IEP teams are provided with multiple resources to support the assessment administration process.

Resources are provided on the DLM website and in the Kite Suite. Some states provide additional materials on their own customized landing page (i.e., dynamiclearningmaps.org/{statename}) of the DLM website and on their own department of education website. Test administrators are made aware of their state-specific webpage through training, manuals, webinars, and replies from Service Desk inquiries. The About DLM tab of the website includes information about topics related to the DLM System as a whole and may be of interest to a variety of audiences. To provide updates and reminders to all participating states, the DLM website also features a Test Updates section of the homepage. This is a newsfeed-style area that addresses timely topics such as assessment deadlines, resource updates, and system status. Additionally, the Test Updates page offers educators the option to subscribe to an electronic mailing list to automatically receive the same message via email without visiting the website. The DLM website also provides resources that cover assessment administration training information; student and roster data management; test delivery protocols and setup; and accessibility features, protocols, and documentation.

This section provides an overview of resources and materials available for test administrators and district-level staff.

4.3.1 Test Administrator Resources

While some resources for test administrators are available in the Kite Suite, the majority of DLM resources are available on the DLM website.

4.3.1.1 Test Administrator Resources Provided on the DLM Website

The DLM website provides specific resources designed for test administrators. These resources are available to all states (Table 4.3) to promote consistent assessment administration practices.

Table 4.3: DLM Resources for Test Administrators and States

Resource	Description
About Testlet Information Pages	Provides guidance for test administrators on the types and uses of information in the Testlet Information Pages provided for each testlet.
Accessibility Manual (PDF)	Provides guidance to state leaders, districts, educators, and Individualized Education Program (IEP) teams on the selection and use of accessibility supports available in the DLM System.
Guide to DLM Required Test Administrator Training (PDF)	Helps users access DLM Required Test Administrator Training in Moodle.
Guide to Practice Activities and Released Testlets (PDF)	Supports the test administrator in accessing practice activities and released testlets in Kite Student Portal.
Instructional Resources on the DLM Website	Provides links to additional resources for test administrators, including lists of EEs, a list of materials commonly needed for testlets, professional development modules supporting EEs, guidance on using mini-maps to plan instruction, accessing and using familiar texts, and released testlets and sample Testlet Information Pages.
Test Administration Manual (PDF)	Supports the test administrator in preparing themselves and students for testing.
Test Updates Page (webpage)	Breaking news on assessment administration activities. Users can sign up to receive alerts when new resources become available.
Training Video Transcripts (PDF)	Links to transcripts (narrator notes) for the DLM Required Test Administrator Training modules.

In addition, there are several helplet videos available on the DLM website to support assessment administration:

• Accessibility in DLM Assessments
• Completing the First Contact Survey and PNP Profile
• DLM Instructionally Embedded Assessments
• DLM Writing Testlets
• Getting Started in Educator Portal
• Monitoring the Assessment Using Extracts
• More About Initial Precursor Items
• Overview of DLM ELA Testlets
• Overview of DLM Mathematics Testlets
• Using Kite Student Portal
• Using the DLM Instruction and Assessment Planner
• Verifying Rosters for Teachers
• Verifying Student Data for Teachers

4.3.1.2 Test Administrator Resources Provided in Kite Suite

The resources for test administrators that are provided in the Kite Suite include the TIPs as well as the practice activities and released testlets.

4.3.1.2.1 Testlet Information Pages

TIPs provide test administrators with information specific to each testlet. Test administrators receive a TIP in Educator Portal for each testlet after it is assigned to a student, and they are instructed to review the TIP before beginning the student’s assessment.

Each TIP states whether a testlet is computer-delivered or educator-administered and indicates the number of items on the testlet. The TIP also provides information for each testlet regarding the materials needed, including substitute materials allowed.

The TIP also provides information on the exceptions to allowable supports. While a test administrator typically uses all appropriate PNP features and other flexibility tools described in the Allowable Practices section of the Test Administration Manual, the TIP indicates when it is not appropriate to use a support on a specific testlet. This may include limits on the use of definitions, translation, read aloud, calculators (for mathematics testlets), or other supports.

If there are further unique instructions for a given testlet, they are provided in the TIP. For test administrators who deliver human read aloud that includes descriptions of graphics, alternate text descriptions of images are provided.

TIPs for ELA testlets also provide the name of the text used in the testlet, identify the text as informational or literature, and label the text as familiar or unfamiliar. They also include the name of the grade-level text that the DLM text is associated with and note if assessment administration time is expected to be longer than usual because the linkage level requires a comparison between two texts. TIPs for mathematics testlets also provide information on specific mathematics terminology.

Testlets that require special setup before assessment administration begins, such as mathematics testlets designed for students with blindness or visual impairments, have additional instructions.

4.3.1.2.2 Practice Activities and Released Testlets

Practice activities and released testlets are available to support test administrators and students as they prepare for testing.

• The educator practice activity is designed to teach test administrators how to deliver educator-administered testlets, while the student practice activity is designed to teach students about the testlets and item features in the Kite Suite.
• The released testlets are similar to operational DLM testlets in content and format and are designed to be used for practice.

For more information on practice activities and released testlets, see Chapter 3 of this manual.

4.3.2 District-Level Staff Resources

Resources are available for three district-level supporting roles: Assessment Coordinator, Data Manager, and Technology Personnel. The Assessment Coordinator oversees the assessment process, which includes managing staff roles and responsibilities, developing and implementing a comprehensive training plan, developing a schedule for test implementation, monitoring and supporting test preparations and administration, and developing a plan to facilitate communication with parents or guardians and staff. The Data Manager manages educator, student, and roster data. Technology Personnel verify that network and testing devices are prepared for assessment administration.

Resources for each of these roles are made available on the state’s customized DLM webpage. Each role has its own manual. A prerecorded training addressing each role and a FAQ compiled from Q&A sessions are also provided. Each role is also guided to supporting resources for other roles where responsibilities overlap. For example, Data Managers are guided to the Test Administration Manual to support data-related activities that are assigned to the test administrator and connect to troubleshooting data issues experienced by the test administrator. Technology Personnel are also guided to the Kite and Educator Portal webpage for information and documents connected to Kite Student Portal, Local Caching Server use, supported browsers, and bandwidth requirements. Assessment Coordinators are also guided to resources developed for the Data Manager, Technology Personnel, and test administrators for specific information and supplemental knowledge of the responsibilities of each of those roles. Some of those resources include the Guide to DLM Required Test Administrator Training, the Test Administration Manual, the Test Updates webpage, and electronic mailing lists.

Descriptions of training for district-level roles are provided in Chapter 9 of this manual.

4.4 Test Administrator Responsibilities and Procedures

The Test Administration Manual (DLM Consortium, 2021) describes procedures for test administrators, which are organized into four sets of tasks for different parts of the school year: (1) before assessments, (2) during the fall window, (3) during the spring window, and (4) while preparing for the next year.

4.4.1 Before Beginning Assessments

Test administrators are directed to perform multiple steps to prepare for student testing, including confirming student eligibility to participate in the DLM alternate assessment and sharing information about the assessment with parents to prepare them for their child’s testing experience. Test administrators are also directed to review the Test Administration Manual and become familiar with available resources, including state webpages, practice activities and released testlets, and procedures for preparing to give the assessment.

1. The manual directs test administrators to prepare for the computer-delivered aspects of the assessment system. Test administrators must activate their Kite Educator Portal account, complete the Security Agreement in Kite Educator Portal, and complete the DLM Required Test Administrator Training (see Chapter 9 of this manual). Test administrators review their state’s guidance on required and recommended professional development modules.
2. Test administrators are also directed to review the Accessibility Manual (Dynamic Learning Maps Consortium, 2021a) and work with IEP teams to determine what accessibility supports should be provided for each student taking the DLM assessments. Test administrators record the chosen supports in the PNP in Kite Educator Portal. Test administrators are also directed to review their state’s requirements for documentation of DLM accessibility supports as testing accommodations and adjust the testing accommodations in the IEP as necessary.
3. Test administrators are also tasked with reviewing student data, including student demographic information and roster data in Kite Educator Portal, for accuracy. Test administrators also must ensure that the PNP and the First Contact survey are updated and complete in Kite Educator Portal. Test administrators must ensure that the Kite Student Portal is installed on testing devices. They must also make sure that they are familiar with their role as test administrator and the students are familiar with DLM testlets by utilizing the practice activities and released testlets. Finally, test administrators must check student devices for compatibility with Kite Student Portal.

4.4.2 Administration in the Fall and Spring Windows

Steps for administration are the same in both the fall and spring windows. Test administrators are trained to follow all state guidance and choose appropriate EEs for instruction by using the blueprint coverage criteria provided in the assessment blueprint documents. Test administrators then retrieve instructional information for the EE, selecting the EE and linkage level for the student in the Instruction and Assessment Planner. They follow this step for each EE and linkage level.

Test administrators deliver instruction until they determine the student is ready for assessment. Test administrators then confirm test assignment in the Instruction and Assessment Planner, retrieve the TIP, and gather necessary materials before beginning testing. Student usernames and passwords are checked so that the students can access the assessments in Kite Student Portal.

Finally, test administrators assess the student on the testlet. While testing, users can go forward and backward within a testlet as much as needed before submitting answers. After completing a testlet, test administrators have the option to choose additional content for instruction (i.e., a new EE or a linkage level, depending on the student’s overall instructional program for the year).

4.4.3 Preparing for Next Year

Educators are directed to prepare for the following year by evaluating students’ accessibility supports (PNP settings) with IEP teams and making decisions about supports and tools for next school year. They are also directed to review the blueprint for the next grade as a source of information to plan academic IEP goals.

4.5 Security

This section describes secure assessment administration, including test administrator training, security during administration, and the Kite Suite; secure storage and transfer of data; and plans for forensic analyses for the investigation of potential security issues. Test security procedures during item development and review are described in Chapter 3.

4.5.1 Training and Certification

Test security is promoted through the DLM Required Test Administrator Training and certification requirements for test administrators. Test administrators are expected to deliver DLM assessments with integrity and maintain the security of testlets. The training for assessment administration details test security measures. Each year, test administrators must renew their DLM Security Agreement through Kite Educator Portal (Figure 4.4). Test administrators are not granted access to Kite Educator Portal if they have not completed the Security Agreement.

Figure 4.4: Test Security Agreement Text

Although each state may have additional security expectations and security-related training requirements, all test administrators in each state are required to meet these minimum training and certification requirements.

4.5.2 Maintaining Security During Test Administration

Several aspects of the DLM System support test security and test administrator integrity during use of the system. For example, the Instructionally Embedded model test blueprints allow for educators to choose EEs and linkage levels. Allowing test administrators to make choices about content that is appropriate for individual students promotes use of the system as intended and minimizes incentives to cheat due to a perception that the content of the assessment is too difficult or inaccessible for the student. Because TIPs are the only printed material, there is limited risk of exposure. Guidance is provided in the Test Administration Manual and on TIPs regarding allowable and not allowable practices. This guidance is intended to promote implementation fidelity and reduce the risk of cheating or other types of misadministration. For a description of fidelity to intended practices, see the description of test administration observations in section 4.7.1 of this chapter.

Agile Technology Solutions, the organization that develops and maintains the Kite Suite and provides DLM Service Desk support to test administrators in the field, has procedures in place to handle alleged security breaches (e.g., test content is made public). Any reported test security incident is assumed to be a breach and is handled accordingly. In the event of a test security incident, access is disabled at the appropriate level. Depending on the situation, the testing window could be suspended, or test sessions could be removed. Test forms could also be removed if exposed or if data is exposed by a form. If necessary, passwords would be changed for users at the appropriate level.

4.5.3 Security in the Kite Suite

The Kite Suite prioritizes security to ensure confidentiality, integrity, and availability for all application data. All Kite Suite data is housed within the United States, including application backups and recovery data. Kite Suite runs in Amazon Web Services (AWS) that implements a “Shared Responsibility” model as it pertains to security controls. AWS is responsible for the security of the cloud, which protects all the infrastructure and services that AWS offers. This is composed of the hardware, software, networking, and physical access to the facilities, and all of the security controls associated with those, including environmental and physical controls. Just as the responsibility to operate the IT environment is shared between AWS and its customers, so is the management, operation, and verification of the IT controls. AWS runs an extensive compliance program reflecting the depth and breadth of their security controls. AWS is NIST 800-53 and FedRAMP compliant. For the controls that are not covered by AWS, the Kite team aligns with NIST standards.

Application access and support access to Kite Suite data follows the principle of least privilege. Access to Kite Suite data is provided through role-based access control systems that limit data to be available to those individuals that require access to perform their jobs. Access is regularly audited by our documented daily, weekly, and monthly security checkout processes.

All Kite Suite network transmissions are encrypted to prevent interception, disruption of reception, communications deception, and/or derivation of intelligence by analysis of transmission characteristics such as signal parameters or message externals. All client web traffic is HTTPS encrypted, with support limited to modern, secure algorithms within the TLS 1.2 or greater protocol. This secures all communication during the session, including the authentication and authorization stages. Support sessions and data transfers are protected using Secure Shell (SSH), an encrypted protocol designed to give a secure connection over an insecure network, such as the internet. All internal network traffic is also encrypted to protect data in transit between network tiers and application components.

Intrusion prevention is a critical component of the Kite Suite security implementation. The Kite Suite implementation in AWS, Kite Suite security processes and procedures, and the Kite Suite development lifecycle all contribute to intrusion prevention.

All Kite Suite Windows Servers utilize Microsoft tools for antivirus, anti-malware, and software firewalls. All laptops and desktops for project staff are fully managed with current antivirus, anti-malware, and storage encryption.

To protect the integrity of test items and scoring materials, the Kite Test Security Agreement lists the security standards that all educators involved with administering tests must follow to protect both the student’s privacy as well as test items and scoring materials.

4.5.4 Secure Test Content

Test content is stored in Kite Content Builder. All items used for released testlets exist in a separate pool from items used for operational testing purposes, ensuring that no items are shared among secure and non-secure pools. Only authorized users of the Kite assessment system have access to view items. Testlet assignment logic prevents a student from being assigned the same testlet more than once, except in cases of manual override for test reset purposes.

4.5.5 Data Security

Project staff collect personally identifiable information (PII) protocols and usage rules from states. Project staff document any applicable state laws regarding PII, state PII handling rules, and state-specific PII breach procedures. The information is housed in the shared resources where Service Desk agents and other project staff can access the information as needed. The protocols are followed with precision due to the sensitive nature of PII and the significant consequences tied to breaches of the data.

The procedures that are implemented in the case of a security incident, privacy incident, or data breach that involve PII or sensitive personal information are implemented by an investigation team that focuses first on mitigation of immediate risk, followed by identification of solutions to identified problems and communication with the DLM Governance Board.

4.5.6 State-Specific Policies and Practices

Some states also adopt more stringent requirements for access to test content and for the handling of secure data, above and beyond those for the overall DLM System. Each DLM agreement with a state education agency (SEA) includes a Data Use Agreement. The Data Use Agreement addresses the data security responsibilities of DLM project staff in regard to the Family Educational Rights and Privacy Act (FERPA, Family Educational Rights and Privacy Act, 1974). The agreement details the role of Accessible Teaching, Learning, and Assessment Systems (ATLAS) as the holder of the data and the rights of the SEA as the owner of the data. In many cases, the standard Data Use Agreement is modified to include state-specific data security requirements. Project staff document these requirements for each state, and the Implementation and Service Desk teams implement the requirements.

The Implementation team collects state education authorities’ policy guidance on a range of state policy issues such as individual student test resets, district testing window extensions, and allowable sharing of PII. In all cases, the needed policy information is collected on a state summary sheet and recorded in a software program jointly accessed by Service Desk agents and the Implementation team.

The Implementation team reviews the state testing policies during Service Desk agent training and provides updates during the state testing windows to supervisors of the Service Desk agents. As part of the training, the Service Desk agents are directed to contact the Implementation team with any questions that require state input or the state to develop or amend a policy.

4.6 Evidence from the DLM System

This section describes evidence collected by the DLM System during the 2021–2022 operational administration of the DLM alternate assessment. The categories of evidence include data relating to administration time, device usage, test administrator selection of linkage levels, blueprint coverage, and accessibility support selections.

4.6.1 Administration Time

Estimated administration time varies by student and subject. Total time varies depending on the number of EEs a test administrator chooses and the number of times a student is assessed on each EE. Testlets can be administered separately across multiple testing sessions as long as they are all completed within the testing window.

The published estimated total testing time per testlet is around 5–10 minutes in mathematics, 10–15 minutes in reading, and 10–20 minutes for writing. The estimated total testing time is 60–75 minutes per student in ELA and 35–50 minutes in mathematics in each of the fall and spring windows. Published estimates are slightly longer than anticipated real testing times because of the assumption that test administrators need time for setup. Actual testing time per testlet varies depending on each student’s unique characteristics.

Kite Student Portal captured start dates, end dates, and time stamps for every testlet. The difference between these start and end times was calculated for each completed testlet. Table 4.4 summarizes the distribution of test times per testlet. The distribution of test times in Table 4.4 is consistent with the distribution observed in prior years. Most testlets took around seven minutes or less to complete, with mathematics testlets generally taking less time than ELA testlets. Time per testlet may have been impacted by student breaks during the assessment (for more information about breaks, see the Accessibility section above). Testlets with shorter than expected administration times are included in an extract made available to each state. States can use this information to monitor assessment administration and address as necessary. For a description of the administration time monitoring extract, see section 4.7.4 of this chapter.

Table 4.4: Distribution of Response Times per Testlet in Minutes

Grade	Min	Median	Mean	Max	25Q	75Q	IQR
English language arts
3	.150	3.75	4.72	84.08	2.32	5.82	3.50
4	.217	4.05	5.08	89.07	2.60	6.27	3.67
5	.167	4.08	5.06	89.08	2.60	6.30	3.70
6	.200	4.05	5.11	88.58	2.60	6.32	3.72
7	.200	4.63	5.70	85.95	2.90	7.02	4.12
8	.167	4.42	5.44	89.70	2.85	6.72	3.87
9	.317	4.80	5.99	89.18	3.07	7.37	4.30
10	.250	4.63	5.93	86.62	2.90	7.35	4.45
11	.250	5.03	6.42	89.93	3.18	7.75	4.57
12	.333	3.78	5.00	77.92	1.98	6.08	4.10
Mathematics
3	.067	1.78	2.73	81.83	1.02	3.23	2.22
4	.083	1.68	2.50	85.23	0.98	2.97	1.98
5	.083	1.77	2.55	84.93	1.08	2.98	1.90
6	.067	1.78	2.50	69.88	1.08	3.00	1.92
7	.083	1.43	2.17	88.72	0.85	2.55	1.70
8	.083	1.67	2.48	70.97	1.00	2.88	1.88
9	.150	1.72	2.51	72.87	0.97	3.02	2.05
10	.067	1.83	2.57	89.22	1.08	3.10	2.02
11	.083	1.77	2.56	88.28	1.05	3.05	2.00
12	.133	1.32	2.17	46.35	0.62	2.68	2.07
Note. Min = minimum, Max = maximum, 25Q = lower quartile, 75Q = upper quartile, IQR = interquartile range.

4.6.2 Device Usage

Testlets may be administered on a variety of devices. Kite Student Portal captured the operating system used for each testlet completed. Although these data do not capture specific devices used to complete each testlet (e.g., SMART Board, switch system, etc.), they provide high-level information about how students access assessment content. For example, we can identify how often an iPad is used relative to a Chromebook or traditional PC. Figure 4.5 shows the number of testlets completed on each operating system by subject and linkage level for 2021–2022. Overall, 40% of testlets were completed on a Chromebook, 28% were completed on an iPad, 26% were completed on a PC, and 7% were completed on a Mac.

Figure 4.5: Distribution of Devices Used for Completed Testlets

4.6.3 Blueprint Coverage

Test administrators selected the EEs for their students to test on from among those available on the ELA and mathematics blueprints in both the fall and spring windows. Table 4.5 summarizes the expected number of EEs required to meet blueprint coverage and the total number of EEs available for instructionally embedded assessments for each grade and subject. A total of 255 EEs (148 in ELA and 107 in mathematics) for grades 3 through high school were available; 12,742 students in those grades participated in the fall window, and 12,891 students participated in the spring window. Histograms in Appendix D.2 summarize the distribution of total unique EEs assessed per student in each grade and subject.

Table 4.5: Essential Elements (EEs) Expected for Blueprint Coverage and Total Available, by Grade and Subject

	English language arts		Mathematics
Grade	Expected n	Available N	Expected n	Available N
3	8	17	6	11
4	9	17	8	16
5	8	19	7	15
6	9	19	6	11
7	11	18	7	14
8	11	20	7	14
9–10	10	19	6	26
11–12	10	19	—	—
Note. High school mathematics is reported in the 9–10 row. There were 26 EEs available for the 9–11 band. While EEs were assigned to specific grades in mathematics blueprint (eight EEs in grade 9, nine EEs in grade 10, and nine EEs in grade 11), a test administrator could choose to test on any of the high school EEs, as all were available in the system.

Figure 4.6 summarizes the percentage of students, for each window and overall for the year, in three categories: students who did not meet all blueprint requirements, students who met all blueprint requirements exactly, and students who exceeded the blueprint requirements. Across both subjects and windows, 97% of students in ELA and 96% of students in mathematics met or exceeded blueprint coverage requirements. The coverage rates for the fall and spring windows were similar. For the full year, the proportion of students exceeding blueprint requirements increases if students are assessed on different EEs in the fall and spring windows (i.e., a student may exactly meet requirements in both the fall and spring but exceed requirements overall if different EEs are selected in each window).

Figure 4.6: Student Blueprint Coverage Status

Figure 4.7 summarizes the percentage of students in each blueprint coverage category based on their complexity band for each subject for each window. When comparing complexity band distributions in ELA and mathematics by blueprint coverage category, there was a slightly larger percentage of Foundational and Band 3 students not meeting requirements.

Figure 4.7: Student Blueprint Coverage Status, by Complexity Band

4.6.4 Linkage Level Selection

Figure 4.8 shows the percentage of testlets that were administered at the system-recommended linkage level or adjusted from the recommended level. Test administrators may choose to administer multiple testlets for a single EE at multiple linkage levels. Because the linkage level for subsequent testlets does not change within each window, we only examined adjustments for the first testlets administered for each student on each EE. Across both windows, 68% of ELA testlets and 63% of mathematics testlets were administered at the recommended linkage level. The most common adjustment was to administer a linkage level below the recommended level. This adjustment was observed for 23% of ELA testlets and 27% of mathematics testlets.

Figure 4.8: Educator Adjustment of Recommended Linkage Levels

Based on the linkage level selections that were made by test administrators, Table 4.6 shows the total number of testlets that were administered at each linkage level by subject and window. Because test administrators do not select a specific linkage level for writing testlets, those testlets are not included in Table 4.6. For both subjects and windows, the majority of testlets were administered at the Initial Precursor or Distal Precursor linkage level.

Table 4.6: Distribution of Linkage Levels Selected for Assessment

	Fall window		Spring window
Linkage level	n	%	n	%
English language arts
Initial Precursor	27,019	35.9	26,329	33.6
Distal Precursor	28,227	37.6	25,634	32.7
Proximal Precursor	15,500	20.6	17,491	22.3
Target	3,985	5.3	7,531	9.6
Successor	429	0.6	1,488	1.9
Mathematics
Initial Precursor	36,814	41.0	37,176	39.9
Distal Precursor	31,353	34.9	29,680	31.9
Proximal Precursor	16,811	18.7	17,351	18.6
Target	4,255	4.7	7,458	8.0
Successor	522	0.6	1,454	1.6

4.6.5 Administration Incidents

DLM staff annually evaluates testlet assignment to ensure students are correctly assigned to testlets. Administration incidents that have the potential to affect scoring are reported to state education agencies in a supplemental Incident File. No incidents were observed during the 2021–2022 operational assessment windows. Assignment of testlets will continue to be monitored in subsequent years to track any potential incidents and report them to state education agencies.

4.6.6 Accessibility Support Selections

Table 4.7 shows selection rates for the three categories of accessibility supports. Each of the support categories are discussed in detail above in the Accessibility section. Overall, 13,058 students (92%) had at least one support selected. The most commonly selected supports in 2021–2022 were human read aloud, spoken audio, and test administrator enters responses for student. Additionally, educators reported in the First Contact survey (see section 4.2.2.1 of this chapter) that 30% of students were able to access a computer independently, with or without assistive technology.

Table 4.7: Accessibility Supports Selected for Students (N = 14,222)

Support	n	%
Supports provided in Kite Student Portal
Spoken audio	9,144	64.3
Magnification	1,988	14.0
Color contrast	1,425	10.0
Overlay color	524	3.7
Invert color choice	335	2.4
Supports requiring additional tools/materials
Individualized manipulatives	5,375	37.8
Calculator	2,915	20.5
Single-switch system	534	3.8
Alternate form - visual impairment	409	2.9
Two-switch system	167	1.2
Uncontracted braille	19	0.1
Supports provided outside the system
Human read aloud	11,347	79.8
Test administrator enters responses for student	8,560	60.2
Partner-assisted scanning	910	6.4
Sign interpretation of text	216	1.5
Language translation of text	93	0.7

4.7 Evidence From Monitoring Assessment Administration

Monitoring of assessment administration was conducted using various materials and strategies. DLM project staff developed an assessment administration monitoring protocol for use by DLM staff, state education agency staff, and local education agency staff. Project staff also reviewed Service Desk contacts and hosted regular check-in calls to monitor common issues and concerns during the assessment window. This section provides an overview of all resources and supports as well as more detail regarding the assessment administration observation protocol and its use, check-in calls with states, and methods for monitoring testlet delivery.

4.7.1 Test Administration Observations

DLM project staff developed an assessment administration observation protocol to standardize data collection across observers and locations. This assessment administration protocol is available for use by state and local education agencies; however, participation in the test administration observations is not required. The majority of items in the protocol are based on direct recording of what is observed and require little inference or background knowledge. Information from the protocol is used to evaluate several assumptions in the validity argument, addressed in the Test Administration Observation Results section of this chapter.

One observation form is completed per testlet administered. Some items are differentiated for computer-delivered and educator-administered testlets. The four main sections include Preparation/Set Up, Administration, Accessibility, and Observer Evaluation. The Preparation/Set Up section includes documentation of the testing location, testing conditions, the testing device used for the testing session, and documentation of the test administrator’s preparation for the session. The Administration section is provided for the documentation of the student’s response mode, general test administrator behaviors during the session, subject-specific test administrator behaviors, any technical problems experienced with the Kite Suite, and documentation of student completion of the testlet. The Accessibility section focuses on the use of accessibility features, any difficulty the student encountered with the accessibility features, and any additional devices the student uses during the testing session. Finally, Observer Evaluation requires that the observer rate overall student engagement during the session and provide any additional relevant comments.

The protocol is available as an online survey (optimized for mobile devices and with branching logic) administered through Kite Survey Solutions, a survey platform within the Kite Suite.

Training resources are provided to state education agency staff to support fidelity of use of the assessment administration protocol and increase the reliability of data collected (see Table 4.8). State education agency staff have access to the Test Administration Observation Training video on the use of the Test Administration Observation Protocol. The links to this video, the Guidance for Local Observers, and the Test Administrator Observation Protocol are provided on the state side of the DLM website, and state education agencies are encouraged to use this information in their state monitoring efforts. State education agencies are able to use these training resources to encourage use of the protocol among local education agency staff. States are also cautioned that the protocol is only to be used to document observations for the purpose of describing the administration process. It is not to be used for evaluating or coaching test administrators or gauging student academic performance. This caution, as well as general instructions for completing and submitting the protocol, are provided in the form itself.

Table 4.8: DLM Resources for Test Administration Monitoring Efforts

Resource	Description
DLM Test Administration Observation Research Protocol (PDF)	Provides observers with a standardized way to describe the assessment administration.
Guide to Test Administration Observations: Guidance for Local Observers (PDF)	Provides observers with the purpose and use of the observation protocol as well as general instructions for use.
Test Administration Observation Training Video (Vimeo video)	Provides training on the use of the Test Administration Observation Protocol.

During 2021–2022, there were 157 assessment administration observations collected in six states. Table 4.9 shows the number of observations collected by state. Of the observations, 115 (73%) were of computer-delivered assessments and 42 (27%) were of educator-administered testlets. The observations consisted of 84 (54%) ELA reading testlets, 8 (5%) ELA writing testlets, and 65 (41%) mathematics testlets.

Table 4.9: Educator Observations by State (N = 157)

State	n	%
Arkansas	70	44.6
Iowa	15	9.6
Kansas	4	2.5
Missouri	13	8.3
North Dakota	1	0.6
West Virginia	54	34.4

To investigate the assumptions that underlie the claims of the validity argument, several parts of the test administration observation protocol were designed to provide information corresponding to the assumptions. One assumption addressed is that educators allow students to engage with the system as independently as they are able. For computer-delivered testlets, related evidence is summarized in Table 4.10; behaviors were identified as supporting, neutral, or nonsupporting. For example, clarifying directions (51.3% of observations) removes student confusion about the task demands as a source of construct-irrelevant variance and supports the student’s meaningful, construct-related engagement with the item. In contrast, using physical prompts (e.g., hand-over-hand guidance) indicates that the test administrator directly influenced the student’s answer choice. Overall, 60% of observed behaviors were classified as supporting, with 2% of observed behaviors reflecting nonsupporting actions.

Table 4.10: Test Administrator Actions During Computer-Delivered Testlets (n = 115)

Action	n	%
Supporting
Read one or more screens aloud to the student	76	66.1
Navigated one or more screens for the student	60	52.2
Clarified directions or expectations for the student	59	51.3
Repeated question(s) before student responded	33	28.7
Neutral
Used pointing or gestures to direct student attention or engagement	42	36.5
Used verbal prompts to direct the student’s attention or engagement (e.g., “look at this.”)	39	33.9
Entered one or more responses for the student	21	18.3
Used materials or manipulatives during the administration process	17	14.8
Asked the student to clarify or confirm one or more responses	11	9.6
Repeated question(s) after student responded (gave a second trial at the same item)	10	8.7
Allowed student to take a break during the testlet	6	5.2
Nonsupporting
Physically guided the student to a response	3	2.6
Reduced the number of answer choices available to the student	3	2.6
Note. Respondents could select multiple responses to this question.

For DLM assessments, interaction with the system includes interaction with the assessment content as well as physical access to the testing device and platform. The fact that educators navigated one or more screens in 52% of the observations does not necessarily indicate the student was prevented from engaging with the assessment content as independently as possible. Depending on the student, test administrator navigation may either support or minimize students’ independent, physical interaction with the assessment system. While not the same as interfering with students’ interaction with the content of the assessment, navigating for students who are able to do so independently conflicts with the assumption that students are able to interact with the system as intended. The observation protocol did not capture why the test administrator chose to navigate, and the reason was not always obvious.

A related assumption is that students are able to interact with the system as intended. Evidence for this assumption was gathered by observing students taking computer-delivered testlets, as shown in Table 4.11. Independent response selection was observed in 48% of the cases. Non-independent response selection may include allowable practices, such as test administrators entering responses for the student. The use of materials outside of Kite Student Portal was seen in 10% of the observations. Verbal prompts for navigation and response selection are strategies within the realm of allowable flexibility during test administration. These strategies, which are commonly used during direct instruction for students with the most significant cognitive disabilities, are used to maximize student engagement with the system and promote the type of student-item interaction needed for a construct-relevant response. However, they also indicate that students were not able to sustain independent interaction with the system throughout the entire testlet.

Table 4.11: Student Actions During Computer-Delivered Testlets (n = 115)

Action	n	%
Selected answers independently	55	47.8
Navigated screens independently	43	37.4
Selected answers after verbal prompts	30	26.1
Navigated screens after test administrator pointed or gestured	27	23.5
Navigated screens after verbal prompts	25	21.7
Used materials outside of Kite Student Portal to indicate responses to testlet items	12	10.4
Asked the test administrator a question	6	5.2
Revisited one or more questions after verbal prompt(s)	6	5.2
Independently revisited a question after answering it	3	2.6
Skipped one or more items	1	0.9
Note. Respondents could select multiple responses to this question.

Another assumption in the validity argument is that students are able to respond to tasks irrespective of sensory, mobility, health, communication, or behavioral constraints. This assumption was evaluated by having observers note whether there was difficulty with accessibility supports (including lack of appropriate available supports) during observations of educator-administered testlets. Of the 42 observations of educator-administered testlets, observers noted difficulty in 1 case (2%). For computer-delivered testlets, evidence to evaluate the assumption was collected by noting students who indicated responses to items using varied response modes such as gesturing (25%) and using manipulatives or materials outside of Kite (10%). Additional evidence for this assumption was gathered by observing whether students were able to complete testlets. Of the 157 test administration observations collected, students completed the testlet in 113 cases (72%).¹⁵ In all instances where the testlet was not completed, no reason was provided by the observer.

Finally, the test administration observations allow for an evaluation of the assumption that test administrators enter student responses with fidelity. To record student responses with fidelity, test administrators needed to observe multiple modes of communication, such as verbal, gesture, and eye gaze. Table 4.12 summarizes students’ response modes for educator-administered testlets. The most frequently observed behavior was verbally indicated response to test administrator who selected answers.

Table 4.12: Primary Response Mode for Educator-Administered Testlets (n = 42)

Response mode	n	%
Verbally indicated response to test administrator who selected answers	24	57.1
Gestured to indicate response to test administrator who selected answers	20	47.6
Eye gaze system indication to test administrator who selected answers	3	7.1
No observable response mode	2	4.8
Note. Respondents could select multiple responses to this question.

Computer-delivered testlets provided another opportunity to confirm fidelity of response entry when test administrators entered responses on behalf of students. This support is recorded on the PNP Profile and is recommended for a variety of situations (e.g., students who have limited motor skills and cannot interact directly with the testing device even though they can cognitively interact with the onscreen content). Observers recorded whether the response entered by the test administrator matched the student’s response. In 21 of 115 (18%) observations of computer-delivered testlets, the test administrator entered responses on the student’s behalf. In 18 (86%) of those cases, observers indicated that the entered response matched the student’s response, while the remaining 3 observers either responded that they could not tell if the entered response matched the student’s response, or they left the item blank.

4.7.2 Formative Monitoring Techniques

Several techniques for formative monitoring purposes are available for the DLM System. First, because DLM assessments are delivered as a series of testlets, an assessment administration monitoring extract was available on demand in Kite Educator Portal. This extract allowed state and local staff to check each student’s progress toward completion of all required testlets. For each student, the extract listed the number of testlets completed and expected for each subject. To support local capacity for monitoring, webinars were delivered before the testing window opened. These webinars targeted district and school personnel who monitor assessments and had not yet been involved in DLM assessments.

Formative monitoring also occurred through regular calls with DLM staff and state education agencies. Throughout most of the year, these calls were scheduled twice per month. Topics related to monitoring that appeared on agendas for partner calls included assessment window preparation, anticipated high-frequency questions from the field, and an opportunity for state education agency-driven discussion. Particular attention was paid to questions from the field concerning sources of confusion among test administrators that could compromise assessment results. During the spring window, check-in calls were hosted on the weeks between the regularly scheduled partner calls. The purpose of the check-in calls is to keep the DLM Governance Board apprised of any issues or concerns that arise during the testing window, which allows them to provide timely information to districts. States are provided with a description of the issues as well as actions that are in place to remedy the situation. During these meetings, partner states are encouraged to share any concerns that have arisen during the week from the field and to provide feedback on implemented fixes, if any were necessary.

4.7.3 Monitoring Testlet Delivery

Prior to the opening of a testing window, Agile Technology Solutions staff initiated an automated enrollment process that works in conjunction with test administrator EE and linkage level selection to assign the first testlet. Students who had missing or incorrect information in Kite Educator Portal were included in error logs that detail which information was missing (e.g., First Contact survey is not submitted) or incorrect (e.g., student is enrolled in a grade that is not tested). These error logs were accessed and evaluated by Agile Technology Solutions staff. When testlets could not be assigned for large numbers of students in a state due to missing or incorrect data, DLM staff worked with relevant state education agencies to either communicate general reminders to the field or solve problems regarding specific students.

Once the student completed the first testlet, test administrator EE and linkage level selection drove the remaining testlet assignments. During each operational window, the DLM psychometric team monitored test delivery to ensure students received testlets according to auto-enrollment specifications. This included running basic frequency statistics to verify counts appeared as expected by grade, state, and testing model and verifying correct assignment to initial testlet-based rules that govern that process.

4.7.4 Data Forensics Monitoring

Two data forensics monitoring reports are available in Educator Portal. The first report includes information about testlets completed outside of normal business hours. The second report includes information about testlets that were completed within a short period of time.

The Testing Outside of Hours report allows state education agencies to specify days and hours within a day that testlets are expected to be completed. Each state can select its own days and hours for setting expectations. For example, a state could elect to flag any testlet completed outside of Monday through Friday from 6:00 a.m. to 5:00 p.m. local time. The Testing Outside of Hours report then identifies students who completed assessments outside of the defined expected hours. Overall, 3,774 (1%) ELA and mathematics testlets were completed outside of the expected hours by 2,745 (20%) students.

The Testing Completed in a Short Period of Time report identifies students who completed a testlet within an unexpectedly short period of time. The threshold for inclusion in the report was testlet completion time of less than 30 seconds in mathematics and 60 seconds in ELA. The report is intended for state users to identify potentially aberrant response patterns; however there are many legitimate reasons a testlet may be submitted in a short time period. Overall, 14,190 (4%) testlets were completed in a short period of time by 3,866 (28%) students.

4.8 Evidence From Test Administrators

This section first describes evidence collected from the spring 2022 test administrator survey. Data on user experience with the DLM System as well as student opportunity to learn is evaluated annually through a survey that test administrators are invited to complete after administration of the spring assessment. Test administrators receive one survey per rostered DLM student, which collects information about that student’s assessment experience. As in previous years, the survey was distributed to test administrators in Kite Student Portal, where students completed assessments. The survey consisted of four blocks. Blocks 1 and 4 were administered in every survey. Block 1 included questions about the test administrator’s perceptions of the assessments and the student’s interaction with the content, and Block 4 included questions about the test administrator’s background. Block 2 was spiraled, so test administrators received one randomly assigned section. In these sections, test administrators were asked about one of the following topics per survey: instructionally embedded assessments, relationship to ELA instruction, relationship to mathematics instruction, or relationship to science instruction. Block 3 was added in 2021 and remained in the survey in 2022 to gather information about educational experiences during the COVID-19 pandemic. This section also presents evidence collected from an educator focus group on the use of instructionally embedded assessments, First Contact survey responses, and educator cognitive labs.

4.8.1 User Experience With the DLM System

A total of 3,168 test administrators responded to the survey (67%) about 6,875 students’ experiences. Test administrators are instructed to respond to the survey separately for each of their students. Participating test administrators responded to surveys for a median of one student. Test administrators reported having an average of 12 years of experience in ELA, 12 years in mathematics, and 10 years with students with significant cognitive disabilities.

The following sections summarize responses regarding both educator and student experience with the system.

4.8.1.1 Educator Experience

Test administrators were asked to reflect on their own experience with the assessments as well as their comfort level and knowledge administering them. Most of the questions required test administrators to respond on a 4-point scale: strongly disagree, disagree, agree, or strongly agree. Responses are summarized in Table 4.13.

Nearly all test administrators (95%) agreed or strongly agreed that they were confident administering DLM testlets. Most respondents (89%) agreed or strongly agreed that the required test administrator training prepared them for their responsibilities as test administrators. Most test administrators also responded that they had access to curriculum aligned with the content that was measured by the assessments (86%) and that they used the manuals and the Educator Resources page (90%).

Table 4.13: Test Administrator Responses Regarding Test Administration

	SD		D		A		SA		A+SA
Statement	n	%	n	%	n	%	n	%	n	%
I was confident in my ability to deliver DLM testlets.	39	1.5	98	3.8	1,186	45.9	1,259	48.8	2,445	94.7
Required test administrator training prepared me for the responsibilities of a test administrator.	68	2.6	220	8.6	1,342	52.3	938	36.5	2,280	88.8
I have access to curriculum aligned with the content measured by DLM assessments.	89	3.5	271	10.5	1,382	53.8	828	32.2	2,210	86.0
I used manuals and/or the DLM Educator Resource Page materials.	41	1.6	229	8.9	1,490	57.8	816	31.7	2,306	89.5
Note. SD = strongly disagree; D = disagree; A = agree; SA = strongly agree; A+SA = agree and strongly agree.

4.8.1.2 Instructionally Embedded Administration Experience

In November 2021, DLM staff conducted 11 focus groups with 30 educators from seven states to collect feedback on their administration and use of instructionally embedded assessments (Clark et al., 2022). Educators expressed a strong preference for instructionally embedded assessments over previous portfolio-based models, and had positive views of the flexibility of the model, including the ability to select EEs and linkage levels and determine the most appropriate time to assess students.

One spiraled block of questions on the test administrator survey asked test administrators to answer several questions about their experiences with administering instructionally embedded assessments. As seen in Table 4.14 the majority of test administrators agreed or strongly agreed that instructionally embedded assessments were useful to their instructional practice (84%); the Instruction and Assessment Planner in Kite Educator Portal helped them find the instructional level of best fit for each subject (87%); it was easy to create instructional plans in Kite Educator Portal (86%); and they preferred instructionally embedded assessments to traditional end-of-year assessments (85%).

Table 4.14: Test Administrator Responses Regarding Instructionally Embedded Test Administration

	SD		D		A		SA		A+SA
Statement	n	%	n	%	n	%	n	%	n	%
Instructionally embedded assessments are useful to my instructional practice.	143	4.5	364	11.5	2,094	66.0	572	18.0	2,666	84.0
The Instruction and Assessment Planner in Educator Portal helped me to find the instructional level of best fit for each student.	108	3.4	303	9.6	2,116	67.0	629	19.9	2,745	86.9
I found it easy to create instructional plans in Educator Portal.	90	2.9	352	11.2	2,059	65.3	652	20.7	2,711	86.0
I preferred instructionally embedded assessments to traditional end-of-year assessments.	130	4.1	336	10.7	1,963	62.3	721	22.9	2,684	85.2
Note. SD = strongly disagree; D = disagree; A = agree; SA = strongly agree; A+SA = agree and strongly agree.

Test administrators were asked to what level they agreed with the statement “I use result in the Instruction and Assessment Planner to make instructional decisions.” Approximately 40% of test administrators agreed or strongly agreed that they used the Instruction and Assessment Planner in making instructional decisions.

Test administrators were then asked to report all ways that they used instructionally embedded results. As shown in Table 4.15, the most commonly reported uses for instructionally embedded results were to identify individual student strengths and weaknesses, determine the need to provide additional instruction, and evaluate student progress.

Table 4.15: Test Administrator Reported Common Uses for Instructionally Embedded Results

Use	n	%
Identify individual student strengths and weaknesses	2,006	18.6
Determine whether I need to provide additional instruction	1,630	15.1
Evaluate student progress	1,606	14.9
Adjust planning	1,493	13.8
Inform the preparation of IEPs	1,319	12.2
Discuss with parents about student performance/progress	1,130	10.5
Group students within class	644	6.0
Discuss with students about their performance/progress	518	4.8
I do not use results from instructionally embedded assessments	264	2.4
Other	196	1.8

4.8.1.3 Student Experience

The spring 2022 test administrator survey included three items about how students responded to test items. Test administrators were asked to rate statements from strongly disagree to strongly agree. Results are presented in Table 4.16. The majority of test administrators agreed or strongly agreed that their students responded to items to the best of their knowledge, skills, and understandings; were able to respond regardless of disability, behavior, or health concerns; and had access to all necessary supports to participate.

Table 4.16: Test Administrator Perceptions of Student Experience with Testlets

	SD		D		A		SA		A+SA
Statement	n	%	n	%	n	%	n	%	n	%
Student responded to items to the best of his/her knowledge, skills, and understanding	239	3.7	558	8.6	3,492	53.6	2,227	34.2	5,719	87.8
Student was able to respond regardless of his/her disability, behavior, or health concerns	410	6.3	658	10.1	3,415	52.4	2,038	31.3	5,453	83.7
Student had access to all necessary supports to participate	207	3.2	324	5.0	3,574	55.0	2,392	36.8	5,966	91.8
Note. SD = strongly disagree; D = disagree; A = agree; SA = strongly agree; A+SA = agree and strongly agree.

Annual survey results show that a small percentage of test administrators disagree that their student was able to respond regardless of disability, behavior, or health concerns; had access to all necessary supports; and was able to effectively use supports. In spring 2020, DLM staff conducted educator focus groups with educators who disagreed with one or more of these survey items to learn about potential accessibility gaps in the DLM System (Kobrin et al., 2022). A total of 18 educators from 11 states participated in six focus groups. The findings revealed that many of the challenges educators described were documented in existing materials (e.g., wanting clarification about allowable practices that are described in the Test Administration Manual, such as substituting materials; desired use of not-allowed practices like hand-over-hand that are used during instruction). DLM staff are using the focus group findings to review existing materials and develop new resources that better communicate information about allowable practices to educators.

4.8.2 Opportunity to Learn

Table 4.17 reports the opportunity to learn results. Approximately 75% of responses (n = 4,900) reported that most or all ELA testlets matched instruction, compared to 72% (n = 4,678) for mathematics. More specific measures of instructional alignment are planned to better understand the extent that content measured by DLM assessments matches students’ academic instruction.

Table 4.17: Educator Ratings of Portion of Testlets That Matched Instruction

	None		Some (< half)		Most (> half)		All		Not applicable
Subject	n	%	n	%	n	%	n	%	n	%
English language arts	298	4.6	1,255	19.2	2,699	41.3	2,201	33.7	81	1.2
Mathematics	306	4.7	1,403	21.6	2,717	41.8	1,961	30.2	116	1.8

A subset of test administrators was asked to indicate the approximate number of hours spent instructing students on each of the conceptual areas by subject (i.e., ELA, mathematics). Test administrators responded using a 6-point scale: 0 hours, 0–5 hours, 6–10 hours, 11–15 hours, 16–20 hours, or more than 20 hours. Table 4.18 and Table 4.19 indicate the amount of instructional time spent on conceptual areas for ELA and mathematics, respectively. Using 11 or more hours per conceptual area as a criterion for instruction, 49% of the test administrators provided this amount of instruction to their students in ELA, and 41% did so in mathematics.

Table 4.18: Instructional Time Spent on ELA Conceptual Areas

	Number of hours
		0		0–5		6–10		11–15		16–20		>20
Conceptual area	Median	n	%	n	%	n	%	n	%	n	%	n	%
Determine critical elements of text	6–10	142	11.3	311	24.7	192	15.2	159	12.6	161	12.8	296	23.5
Construct understandings of text	11–15	112	8.9	289	23.0	176	14.0	171	13.6	178	14.2	331	26.3
Integrate ideas and information from text	11–15	112	9.0	311	25.0	186	15.0	184	14.8	177	14.2	273	22.0
Use writing to communicate	6–10	166	13.3	317	25.3	183	14.6	153	12.2	153	12.2	279	22.3
Integrate ideas and information in writing	6–10	213	17.1	321	25.7	177	14.2	164	13.2	151	12.1	221	17.7
Use language to communicate with others	16–20	57	4.5	195	15.6	161	12.8	145	11.6	180	14.4	516	41.1
Clarify and contribute in discussion	11–15	122	9.8	259	20.7	191	15.3	180	14.4	187	15.0	311	24.9
Use sources and information	6–10	263	21.0	321	25.7	185	14.8	163	13.0	136	10.9	183	14.6
Collaborate and present ideas	6–10	246	19.7	313	25.1	189	15.2	177	14.2	127	10.2	194	15.6

Table 4.19: Instructional Time Spent on Mathematics Conceptual Areas

	Number of hours
		0		0–5		6–10		11–15		16–20		>20
Conceptual area	Median	n	%	n	%	n	%	n	%	n	%	n	%
Understand number structures (counting, place value, fraction)	11–15	82	6.3	263	20.2	181	13.9	161	12.4	200	15.4	414	31.8
Compare, compose, and decompose numbers and steps	6–10	189	14.6	304	23.5	221	17.1	181	14.0	185	14.3	212	16.4
Calculate accurately and efficiently using simple arithmetic operations	11–15	208	16.1	240	18.6	184	14.3	162	12.6	194	15.1	301	23.4
Understand and use geometric properties of two- and three-dimensional shapes	6–10	202	15.6	389	30.1	224	17.3	186	14.4	146	11.3	145	11.2
Solve problems involving area, perimeter, and volume	1–5	511	39.5	294	22.7	157	12.1	143	11.1	105	8.1	84	6.5
Understand and use measurement principles and units of measure	1–5	298	23.1	379	29.4	202	15.7	179	13.9	127	9.9	103	8.0
Represent and interpret data displays	1–5	308	23.9	337	26.1	197	15.3	175	13.6	147	11.4	125	9.7
Use operations and models to solve problems	6–10	264	20.5	285	22.1	214	16.6	167	13.0	158	12.3	201	15.6
Understand patterns and functional thinking	6–10	164	12.7	359	27.8	233	18.0	188	14.6	164	12.7	184	14.2

Results from the test administrator survey were also correlated with total linkage levels mastered by conceptual area, as reported on individual student score reports.¹⁶ See Chapter 7 of this manual for a description of results and reporting. In mathematics, results were reported at the claim level rather than conceptual area, due to the blueprint structure. The median instructional time was calculated for each mathematics claim from test administrator responses at the conceptual area level. While a direct relationship between amount of instructional time and number of linkage levels mastered in the area is not expected, as some students may spend a large amount of time on an area and demonstrate mastery at the lowest linkage level for each EE, we generally expect that students who mastered more linkage levels in the area would also have spent more instructional time in the area. More evidence is needed to evaluate this assumption.

Table 4.20 summarizes the Spearman rank-order correlations between ELA conceptual area instructional time and linkage levels mastered in the conceptual area as well as between mathematics claim instructional time and linkage levels mastered in the claim. Correlations ranged from 0.20 to 0.37, with the strongest correlations observed for writing conceptual areas (ELA.C2.1 and ELA.C2.2) in ELA, and geometric principles (M.C2) in mathematics.

Table 4.20: Correlation Between Instructional Time and Linkage Levels Mastered by Conceptual Area or Claim

Conceptual area	Correlation with instruction time
English language arts
ELA.C1.1: Determine critical elements of text	.20
ELA.C1.2: Construct understandings of text	.26
ELA.C1.3: Integrate ideas and information from text	.29
ELA.C2.1: Use writing to communicate	.37
ELA.C2.2: Integrate ideas and information in writing	.29
Mathematics
M.C1: Demonstrate increasingly complex understanding of number sense	.25
M.C2: Demonstrate increasingly complex spatial reasoning and understanding of geometric principles	.27
M.C3: Demonstrate increasingly complex understanding of measurement, data, and analytics procedures	.25
M.C4: Solve increasingly complex mathematical problems, making productive use of algebra and functions	.23

Another dimension of opportunity to learn is student engagement with instruction. The First Contact survey (see section 4.2.2.1 of this chapter) contains two questions about student engagement during computer- and educator-directed instruction. Table 4.21 shows the percentage of students who demonstrated different levels of attention by instruction type. Overall, 85% of students demonstrated fleeting or sustained attention to computer-directed instruction and 83% of students demonstrated fleeting or sustained attention to educator-directed instruction.

Table 4.21: Student Attention Levels During Instruction

	Demonstrates little or no attention		Demonstrates fleeting attention		Generally sustains attention
Type of instruction	n	%	n	%	n	%
Computer-directed (n = 11,799)	1,787	15.1	6,977	59.1	3,035	25.7
Educator-directed (n = 13,241)	2,262	17.1	8,743	66.0	2,236	16.9

4.8.3 Educator Ratings on First Contact Survey

Before administering testlets, educators complete the First Contact survey, which is a survey of learner characteristics (see section 4.2.2.1 of this chapter for more details). Because ratings on the First Contact survey are distinct from the DLM assessment (which uses only a subset of items to calculate the student complexity band for each subject), they can serve as one source of external evidence regarding the construct being measured. The First Contact survey includes academic skill items: nine in the reading section and 13 in the mathematics section.

For each academic item on the First Contact survey, test development teams reviewed the learning maps to identify tested nodes that measured the same skill. Not all First Contact items directly corresponded to nodes in the map. Tested nodes were identified for two of the reading items and nine of the mathematics items. A summary of the First Contact academic items and the number of nodes identified in the learning maps is provided in Table 4.22.

Table 4.22: First Contact Items With Nodes Identified in the Learning Maps

First Contact item	Number of assessed nodes	Number of linkage levels measuring the nodes
Reading
Recognizes single symbols presented visually or tactually	1	1
Identifies individual words without symbol support	1	10
Mathematics
Creates or matches patterns of objects or images	3	6
Identifies simple shapes in 2 or 3 dimensions	8	4
Sorts objects by common properties (e.g., color, size, shape)	1	17
Adds or subtracts by joining or separating groups of objects	2	10
Adds and/or subtracts using numerals	15	13
Forms groups of objects for multiplication or division	2	12
Multiplies and/or divides using numerals	19	9
Tells time using an analog or digital clock	4	5
Uses common measuring tools (e.g., ruler or measuring cup)	5	3

For each tested node identified by the test development teams, all EEs and linkage levels measuring the node were identified. A dataset was created that included student mastery of the EE and linkage level measuring the node, as well as First Contact survey responses.¹⁷ See Chapter 7 of this manual for a description of linkage level mastery and scoring rules. The First Contact items asked educators to use a 4-point scale to indicate how consistently students demonstrated each skill: almost never (0%–20% of the time), occasionally (21%–50% of the time), frequently (51%–80% of the time), or consistently (81%–100% of the time).

Polychoric correlations for reading and mathematics were calculated to determine the relationship between the educator’s First Contact rating and the student’s reported mastery of the linkage level measuring nodes associated with the First Contact items.

Moderate but positive correlations are expected between First Contact ratings and student mastery of the linkage level for several reasons. The First Contact items were not originally designed to align with assessment items or linkage level statements. Also, educators are required to complete the First Contact survey before testlet administration; some educators complete it at the beginning of the school year. Educators may choose to update survey responses during the year but do not have to. Therefore, First Contact ratings may reflect student knowledge or understandings before instruction, while linkage level mastery represents summative performance. However, in general, higher First Contact ratings are expected to be associated with student mastery of the linkage level measuring the same skill.

Correlations for First Contact items with linkage level mastery are summarized in Table 4.23.

Table 4.23: Correlations of First Contact Item Responses to Linkage Level Mastery

		Correlation			Standard Error
First Contact section	Linkage levels (n)	Min	Max	Median	Min	Max	Median
Reading	11	.07	.57	.38	0.03	0.15	0.07
Mathematics	79	−.42	.70	.28	0.02	0.32	0.07

Mathematics First Contact items varied most in their relationship to linkage level mastery. Because mathematics nodes represent finer-grained skills, and test development teams identified more nodes in mathematics, more correlations were calculated (n = 79) than for reading (n = 11). Mathematics results were also likely affected by sample size. As few as 22 student data points were available for some linkage levels, compared to at least 80 in reading. The decreased sample size is likely attributable to fewer students testing at the Target and Successor linkage levels (see section 4.6.4 of this chapter). Furthermore, a negative relationship between mathematics First Contact rating and linkage level mastery was observed in six instances. An example is seen in the relationship between the Target level of the grade 4 EE M.EE.4.NBT.4 and the First Contact item “Adds and/or subtracts using numerals.” The linkage level statement for this EE and level is “Add and subtract within 100.” Although the linkage level measures the nodes “Add within 100” and “Subtract within 100,” it also measures other nodes that are not aligned to any First Contact item; this combination likely contributed to the negative relationship observed. However, small sample size is associated with increased standard errors (Moinester & Gottfried, 2014), and therefore these negative correlations should be interpreted with caution.

Overall, 93% (n = 84) of the correlations were positive, indicating generally positive associations between linkage level mastery and First Contact ratings. Results for all correlations are summarized in Figure 4.9.

Figure 4.9: Relationship of First Contact Responses to Linkage Level Mastery

4.8.4 Educator Cognitive Labs

Educator cognitive labs have been recommended as a potential source of response process evidence for alternate assessments based on alternate achievement standards, in which educator ratings are the items (Goldstein & Behuniak, 2010). This approach was used for DLM educator-administered testlets because educators interpret student behavior and respond to items about the student’s response. Most of these testlets involve test administrator interpretation of the responses of students who are working on consistent, intentional communication and who are working on foundational skills that promote their access to grade-level content. Writing testlets are also educator-administered at all linkage levels.

Cognitive labs were conducted in spring 2015 with 15 educators in five schools across two states. Educators completed think-aloud procedures while preparing for and administering educator-administered testlets in reading, writing, and math. They were first presented with the TIP, which is a short document that provides background information needed to prepare to administer the testlet (see section 4.3.1.2.1 of this chapter).

Educators were asked to think out loud as they read through the TIP. Next, the educator gathered the materials needed for the assessment and administered the testlet. Probes were sometimes used during the process to ask about educator interpretation of the on-screen instructions and the rationale behind decisions they made during administration. When the testlet was finished, educators also completed post-hoc interviews about the contents of test-administration instructions, use of materials, clarity of procedures, and interpretation of student behaviors. All labs were video recorded and an observer took notes during the administration. The initial phase of analysis involved recording evidence of intended administration and sources of challenge to intended administration at each of the following stages: (1) preparation for administration, (2) interpretation of educator directions within the testlet, (3) testlet administration, (4) interpretation of student behaviors, and (5) recording student responses. Through this lens, we were able to look for evidence related to fidelity (1, 2, 3, and 5) as well as response process (4). These 15 labs were the first phase of data collection using this protocol. Preliminary evidence on interpretation of student behaviors indicates that the ease of determining student intent depended in part on the student’s response mode.

• Educators were easily able to understand student intent when the student indicated a response by picking up objects and handing them to the educator.
• In a case where the student touched the object rather than handing it to the educator, the educator accepted that response and entered it, but speculated as to whether the student was just choosing the closest object.
• When a student briefly touched one object and then another, the educator entered the response associated with the second object but commented that she was not certain if the student intended that choice.
• When a student used eye gaze, the educator held objects within the student’s field of vision and put the correct response away from the current gaze point so that a correct response required intentional eye movement to the correct object.
• When a student’s gesture did not exactly match one of the response options, the educator was able to verbalize the process of deciding how to select the option that most closely matched the student’s behavior. Her process was consistent with the expectations in the Test Administration Manual.
• In one case, the educator moved objects to prepare for the next item, which took her attention away from the student and caused her to miss his eye gaze that indicated a response. She recorded no response. However, this was observed for a student whose communication and academic skills were far beyond what was being assessed. The testlet was not appropriate for this student and his typical response mode for DLM testlets was verbal.

4.9 Conclusion

Delivery of the DLM System was designed to align with instructional practice and be responsive to individual student needs. Assessment delivery options allow for necessary flexibility to reflect student needs while also including constraints to maximize comparability and support valid interpretation of results. The dynamic nature of DLM assessment administration is reflected in the initial input through the First Contact survey, as well as the linkage level and the EE selections made by test administrators. Evidence collected from the DLM System, test administration monitoring, and test administrators indicates that students are able to successfully interact with the system to demonstrate their knowledge, skills, and understandings.