This document discusses implications for assessment design based on research on learning. It makes three key points:
1) Classroom assessments need to change to better support learning by reflecting current research and understanding of learning progressions. Assessment should be an ongoing part of the learning process.
2) For assessments to effectively promote learning, teachers need a clear understanding of learning goals, information about students' current knowledge, and ways to help close gaps.
3) Large-scale assessments could also benefit learning if designed based on cognitive theories of learning, but are currently constrained by needs for standardized testing.
This document discusses implications for assessment design based on research on learning. It makes three key points:
1) Classroom assessments need to change to better support learning by reflecting current research and understanding of learning progressions. Assessment should be an ongoing part of the learning process.
2) For assessments to effectively promote learning, teachers need a clear understanding of learning goals, information about students' current knowledge, and ways to help close gaps.
3) Large-scale assessments could also benefit learning if designed based on cognitive theories of learning, but are currently constrained by needs for standardized testing.
This document discusses implications for assessment design based on research on learning. It makes three key points:
1) Classroom assessments need to change to better support learning by reflecting current research and understanding of learning progressions. Assessment should be an ongoing part of the learning process.
2) For assessments to effectively promote learning, teachers need a clear understanding of learning goals, information about students' current knowledge, and ways to help close gaps.
3) Large-scale assessments could also benefit learning if designed based on cognitive theories of learning, but are currently constrained by needs for standardized testing.
The Design and Use of Classroom Assessment Learning scientists generally argue that classroom assessment practices need to change to better support learning (also see Shepard, 2000). The content and character of assessments need to be significantly improved to reflect the latest empirical research on learning and, given what we now know about learning progressions, the gathering and use of assessment information and insights should become a part of the ongoing learning process. This latter point further suggests that teacher education programs should provide teachers with a deep understanding of how to use assessment in their instruction. Many educational assessment experts believe that if assessment, curriculum, and instruction were more integrally connected, student learning would improve (e.g., Pellegrino et al., 1999; Stiggins, 1997). According to Sadler (1989), three elements are required if teachers are to successfully use assessment to promote learning: (1) A clear view of the learning goals (derived from the curriculum) (2) Information about the present state of the learner (derived from assessment) (3) Action to close the gap (taken through instruction) Each of these three elements informs the other. For instance, formulating assessment procedures for classroom use can spur a teacher to think more specifically about learning goals, thus leading to modification of curriculum and instruction. These modifications can, in turn, lead to refined assessment procedures, and so on. The mere existence of classroom assessment along the lines discussed here will not ensure effective learning. The clarity and appropriateness of the curriculum goals, the validity of the assessments in relationship to these goals, the interpretation of the assessment evidence, and the relevance and quality of the instruction that ensues are all critical determinants of the outcome. Effective teaching must start with a model of cognition and learning in the domain. For most teachers, the ultimate goals for learning are established by the curriculum, which is usually mandated externally (e.g., by state curriculum standards). But the externally mandated curriculum does not specify the empirically based cognition and learning outcomes that are necessary for assessment to be effective. As a result, teachers (and others responsible for designing curriculum, instruction, and assessment) must fashion intermediate goals that can serve as an effective route to achieving the externally mandated goals and, to do so effectively, they must have an understanding of how students represent knowledge and develop competence in the domain. Formative assessment should be based in cognitive theories about how people learn particular subject matter to ensure that instruction centers on what is most important for the next stage of learning, given a learner’s current state of understanding. Pre-service and professional development are needed to help teachers formulate models of learning progressions so they can identify students’ naïve or initial sense-making strategies and build on those to move students toward more sophisticated understandings. This will increase teachers’ diagnostic expertise so they can make informed decisions about next steps for student learning. Several cognitively-based approaches to instruction and assessment have been shown to have a positive impact on student learning, including the Cognitively Guided Instruction program (Carpenter, Fennema, & Franke, 1996) and others (Cobb et al., 1991; Griffin & Case, 1997). The Design and Use of Large-Scale Assessment Large-scale assessments are further removed from instruction but can still benefit learning if well designed and properly used. If the principles of design identified above were applied, substantially more valid, useful, and fair information would be gained from largescale assessments. However, before schools, districts, states, or nations can fully capitalize on contemporary theory and research, they may need to substantially change how they approach largescale assessment. Specifically, they must relax some of the constraints that currently drive many large-scale assessment practices, as follows. Large-scale summative assessments should focus on the most critical and central aspects of learning in a domain – as identified by curriculum standards and informed by cognitive research and theory. Large-scale assessments typically are based on models of learning that are less detailed than classroom assessments. For summative purposes, one might need to know whether a student has mastered the more complex aspects of multicolumn subtraction, including borrowing from and across zero, whereas a teacher needs to know exactly which procedural errors lead to mistakes. Although policymakers and parents may not need all the diagnostic detail that would be useful to a teacher and student during the course of instruction, large-scale summative assessments should be based on a model of learning that is compatible with and derived from the same set of knowledge and assumptions about learning as classroom assessment. Research on cognition and learning suggests a broad range of competencies that should be assessed when measuring student achievement, many of which are essentially untapped by current assessments. Examples are knowledge organization, problem representation, strategy use, metacognition, and participatory activities (e.g., formulating questions, constructing and evaluating arguments, contributing to group problem-solving). These are important elements of contemporary theory and research on the acquisition of competence and expertise and are discussed and illustrated in detail in the various references mentioned earlier in the section on learning progressions. Large-scale assessments should not ignore these aspects of competency and should provide information about these aspects of the nature of student understanding, rather than simply ranking students according to general proficiency estimates. If tests are based on a research-grounded theory of cognition and learning, those tests can provide positive direction for instruction, making “teaching to the test” productive for learning rather than destructive (this point is discussed further below) Unfortunately, given current constraints of standardized test administration, only limited improvements in large-scale assessments are possible. These constraints include the need to provide reliable and comparable scores for individuals as well as groups, the need to sample a broad set of curriculum standards within a limited testing time per student, and the need to offer cost-efficiency in terms of development, scoring, and administration. To meet these kinds of demands, designers typically create assessments that are given at a specified time, with all students being given the same (or parallel) tests under strictly standardized conditions (often referred to as on-demand assessment). Tasks are generally of the kind that can be presented in paper-and-pencil format that students can respond to quickly, and that can be scored reliably and efficiently. As a result, learning outcomes that lend themselves to being assessed in these ways are assessed, but aspects of learning that cannot be observed under such constrained conditions are not. Designing new assessments that capture the complexity of cognition and learning will require examining the assumptions and values that currently drive assessment design choices and breaking out of the current paradigm to explore alternative approaches to large-scale assessment, including innovative uses of technology (see e.g., Quellmalz & Pellegrino, 2009; Pellegrino et al., 2014). V. Balanced Assessment Systems Many different assessments are used in schools, with each serving varying needs and different audiences. Perhaps the biggest divide is between external, large-scale assessments for purposes of summative evaluation and comparison by policy makers, and classroom assessments designed to assist teachers in their instructional work. One result of this variety is that users can become frustrated when different assessments have conflicting achievement goals and results. Sometimes such discrepancies can be meaningful and useful, such as when assessments are explicitly aimed at measuring different school outcomes. More often, however, conflicting assessment goals and feedback cause much confusion for educators, students, and parents. In this section we describe a vision for coordinated systems of multiple assessments that work together, along with curriculum and instruction, to promote learning. In many education systems worldwide, assessment is focused on classroom activities designed to provide information about the progress of learning and external, large-scale standardized assessments play a relatively minor or secondary role in the educational system (see National Research Council, 2003). In the United States, however, the resources invested in producing and using large-scale tests – in terms of money, instructional time, research, and development – far outweigh the investment in the design and use of effective classroom assessment (see e.g., Kaestle, 2013). And unfortunately, there is ample evidence that the largescale assessments in use today in the U.S. and elsewhere negatively impact classroom instruction and assessment. For instance, as discussed earlier, teachers feel pressure to teach to the test, which (given the focus of today’s assessments on disconnected facts and skills) results in a narrowing of instruction. This would not necessarily be a problem if the assessments found on such tests were of higher quality and represented the full range of levels of thinking and reasoning that we desire for students to attain. Then we would have tests worth teaching towards and the tasks would be much closer to those that are useful in the context of classroom instruction to promote student learning and engagement. They would be tasks and performances that merit the time and attention of teachers and students. If that was true, then we would not have the problem that exists now because teachers model their own classroom tests after the highly limiting and less-than-ideal tasks found on typical standardized tests (Koretz, 2009; Linn, 2000; Shepard, 2000). Given that they will engage in such a modeling exercise when the external tests matter for purposes such as accountability, it would be far better if what they were modeling constituted high quality and valid assessments of student achievement. So, in addition to the need to strike a better balance between classroom and large-scale assessment, we also need to coordinate systems of assessments that collectively support a common set of learning and teaching goals, rather than work at cross-purposes. To this end, an assessment system should exhibit three properties: comprehensiveness, coherence, and continuity. By comprehensiveness, I mean that a range of measurement approaches should be used to provide a variety of evidence to support educational decision-making. No single test score can be considered a definitive measure of a student’s competence. Multiple measures enhance the validity and fairness of the inferences drawn by giving students various ways and opportunities to demonstrate their competence. Multiple measures can also be used to provide evidence that improvements in test scores represent real gains in learning, as opposed to score inflation due to teaching narrowly to one particular test (e.g., Koretz, 2009). By coherence, I mean that the models of student learning underlying the various external and classroom assessments within a system should be compatible. While a large-scale assessment might be based on a model of learning that is coarser than that underlying the assessments used in classrooms, the conceptual base for the large-scale assessment should be a broader version of one that makes sense at the finer-grained level (Mislevy, 1996). In this way, the external assessment results will be consistent with the more detailed understanding of learning underlying classroom instruction and assessment. As one moves up and down the levels of the system, from the classroom through the school, district, and state, assessments along this vertical dimension should align. As long as the underlying models of learning are consistent, the assessments will complement each other rather than present conflicting goals for learning. Finally, an ideal assessment system would be designed to be continuous. That is, assessments should measure student progress over time, akin more to a videotape record rather than to the snapshots provided by most current tests. To provide such pictures of progress, multiple sets of observations over time must be linked conceptually so that change can be observed and interpreted. Models of student progress in learning should underlie the assessment system, and tests should be designed to provide information that maps back to the progression. Figure 3 provides a graphical illustration of what an assessment system might look and some of the factors that would serve to achieve balance and support these three principles. Figure 3 demonstrates that such a system would be (a) coordinated across levels, (b) unified by common learning goals, and (c) synchronized by unifying progress variables. No existing system of assessments has these design features and meets all three criteria of comprehensiveness, coherence, and continuity, but there are examples of assessments that represent steps toward these goals. For instance, Australia’s Developmental Assessment program (Forster & Masters, 2001; Masters & Forster, 1996) and the BEAR assessment system (Wilson, Draney, & Kennedy, 2001; Wilson & Sloane, 2000) show how progress maps can be used to achieve coherence between formative and summative assessment, as well as among curriculum, instruction, and assessments. Progress maps also enable the measurement of growth (thus meeting the continuity criterion). The Australian Council for Educational Research has produced an excellent set of resource materials for teachers to support their use of a wide range of assessment strategies – from written tests to portfolios to projects at the classroom level – that can all be designed to link back to the progress maps (thus meeting the criterion of comprehensiveness) VI. Moving Forward: Necessity and Opportunity Because assessments are tied to claims we would like to substantiate about students’ competencies, new approaches to assessment must be treated as a process of gathering evidence to confirm or disconfirm particular claims (Gorin, 2013). That evidence, which in a system of assessments can come from multiple sources, can be used to improve both how they are taught and how and what students are learning. The evidence might include a range of activities ranging from simple to complex performance tasks pursued within classrooms as well as assessments external to regular classroom activities (Bennett, 2013). Pellegrino et al. (2014) have described in some detail such a systems approach for science assessment. The description they provide is designed to promote the vision of science learning and teaching associated with the U.S. National Research Council’s Framework for K-12 science education (National Research Council, 2012) and the derivative Next Generation Science Standards (Achieve, 2013). Digital technologies hold great promise for helping to bring about many of the changes in assessment that many believe are necessary. Technologies available today and innovations on the immediate horizon can be used to access information, create simulations and scenarios, allow students to engage in learning games and other activities, and enable collaboration among students. Such activities make it possible to observe, document, and assess students’ work as they are engaged in natural activities – perhaps reducing the need to separate formal assessment for accountability from learning in the moment (e.g., Behrens & DiCerbo, 2013). Technologies will certainly make possible the greater use of formative assessment that in turn has been shown to significantly impact student achievement. Digital activities may also provide information about non-cognitive abilities, such as persistence, creativity, and teamwork that current testing approaches cannot. Juxtaposed with the promise is the need for considerable work to be done on issues of scoring and interpretation of evidence before such embedded assessment can be useful for these varied purposes. Many issues, including some alluded to above, have been discussed and debated among educators and assessment experts for many years. As part of those discussions it is now widely recognized that large-scale standardized testing has exerted a greater and greater influence over American schooling (Kaestle, 2013; Linn, 2013). At the same time, it has been shown repeatedly that teachers have the largest impact on education of any in-school factor. And it is what teachers do and what they teach and how they assess in classrooms that give teachers that influence. If teachers and schools are to enable the kind of transferable learning required of young people in contemporary society, assessments will need to support curriculum and teaching focused on such learning, along with traditional basic skills. New assessment systems, grounded in new standards, should include the features described earlier in this paper. Criteria for such assessment systems should be rigorous and ambitious, while taking account in the near-term of what is achievable financially, logistically, technologically, and scientifically. The path to reaching more ambitious education goals is likely to traverse distinct phases rather than occurring in one giant leap. Given where we are today and what should be feasible in the nearterm, the following set of indicators has been suggested for use in evaluating whether assessment systems and their components meet the five criteria discussed in Section I (see Darling-Hammond et al., 2013 for additional details) Educational entities – nations, states, provinces etc. – should evaluate the sets of assessments they currently have and/or develop against these criteria, and they should use their assessments in ways for which they have been appropriately validated. Doing so will help ensure positive consequences of assessment for instruction and student learning. To return to a quote from the Gordon Commission (2013b) mentioned earlier in this paper: “The best assessments can accelerate the acquisition of 21st century knowledge and competencies if they guide the actions of teachers and enable students to gauge their progress. To do so, the tasks and activities in the assessments must be models worthy of the attention and energy of teachers and students.” Transforming educational assessment in the ways proposed depends on a systems approach that includes multiple factors. Among these are advances in cognitive theory and research and applications of technology combined with investments in teacher knowledge and accompanying changes in educational policies. Policy makers at all levels need to actively promote this much-needed transformation of current assessment practice. An open question is whether such a systems approach is attainable across the levels of educational policy and practice that are typically operative and at scales ranging from local districts, to states, nations, and even at the international assessment level. Indicators of Quality in a System of Next Generation Assessments (1) Assessment of higher-order cognitive skills A large majority of items and tasks (at least two-thirds) evaluate the conceptual knowledge and applied abilities that support transfer (e.g., depth of knowledge levels 2, 3, or 4 in Webb’s (1997) taxonomy or the equivalent). At least one-third of the assessment content in mathematics and at least one-half in English language arts should evaluate higher-order skills that allow students to become independent thinkers and learners (DOK levels 3 or 4). (2) High-fidelity assessment of critical abilities. Critical abilities outlined in the standards are evaluated using highfidelity tasks that use the skills in authentic applications: Research, including analysis and synthesis of Information Experimentation and evaluation Oral communications – speaking and listening Written communications – reading and writing Use of technology for accessing, analyzing, and communicating information Collaboration Modeling, design, and problem solving using quantitative tools (3) Standards that are internationally benchmarked Calibration to PISA, international baccalaureate, or other internationally comparable assessment (based on evaluation of content comparability, performance standards, and analysis of student performance on embedded items). (4) Items that are instructionally sensitive and educationally valuable Research that confirms instructional sensitivity Rich feedback on student learning and performance Tasks that reflect and can guide valuable instructional activities (5) Assessments that are valid, reliable, and fair Evidence that the intended knowledge and skills are well measured Evidence that scores are related to the abilities they are meant to predict Evidence that the assessments are well-designed and valid for each intended use – and that uses are appropriate to the test purposes and validity evidence. Evidence that the assessments are unbiased and fairly measure the knowledge and skills of students from different language, cultural, and income backgrounds, as well as students with learning differences. Evidence that the assessments measure students learning accurately along a continuum of achievement, consistent with the purposes the assessments are intended to serve.