NSF PI Conference: Technology and Assessment Group
What is the role of new Technologies in Student Assessment?

Jim Minstrell

I. Overview: Some possible reasons for using technology in assessment.

A. Some reasons are obvious. Using technology can provide a quick turnaround from data collection to data analysis and implications. Assessment may be implemented more consistently from one site to another. Data can be collected in electronic form, reducing errors in data transformation. There is an opportunity to link to other electronically available sources of information or representations that already exist.

B. Some reasons suggest promise for changing learning environments. Pellegrino describes his vision of assessment in schools of the future being something like what has happened to taking inventory in stores. We wont need to shut down the learning for five days while inventory (state test) is taking place. Assessment will be an ongoing process informing decisions about learning and teaching. Technology can make assessment available on client (learner/teacher) demand. Intelligent assessment can probe learning based on learner's prior responses. Research on learning may even be conducted on-line.

II. Discussion Questions

A. Under what conditions is it appropriate to use technology in assessment?

1. When and how can technology in assessment be used to inform learning and teaching in the classroom; to inform summative grading/certification; to inform program evaluation and accountability? How would the technological assistance be different for the different purposes?
2. How can the technology be used to conduct research and to implement research implications into schools? How can we use the technology in assessment to capture student thinking in process? Is there a way to more accurately reflect thinking in different disciplines and to assist learners in moving their thinking to be more like experts? How do we use assessment technology to implement principles of best practice learned from research on learning and teaching?

B. How do we scale up in the use of assessment technology?

1. A lot of research is needed up front to develop intelligent/expert systems. Where does that funding come from? How do we develop assessment for the technology available in the future and yet test and use the system in systems that exist in schools today? Perhaps we need some sort of orderly process for review and funding of proposals that is itself a scaling up. A first stage looks for good programatic ideas. A second might involve running the program as an experiment. Next, could be "perfecting" and testing a prototype, preparing it for exportability. Finally, disseminating and testing the prototype in multiple sites. Funding for each stage would depend on the success of the previous stage.
2. How do we test/evaluate our technological assessment systems in school systems? Controlled experiments are hard to do with all the variables. Running "real world" evaluations of these tools is complex, is encumbered with details of collaborations among teachers, administrators, learners, parents, state offices, districts, researchers, etc. Different entities have different goals and different modes of operation. Attribution of effects is difficult because there are many possibilities of interactions from competing variables. One can not plan on schools (teachers and students) to have uniform technology (browsers and platforms) to use in the assessment. Is the lack of evidence for good effects because the information is not getting to the field or is it because many effects of use of technology in assessment are not good? If the later, we need to know that too.
3. WWW based instruction has proven to be very useful in government and industrial training and assessment. For instance, virtually all of the US Navy weather forecasting school is WWW based. Would this approach work for schools as well? Why or why not?

C. How do we maintain security and avoid misuse of assessment information?

1. Balancing collection of lots of data on an individual with the possibility of misusing that information. How do we address concerns of Human Subjects (no names attached) with need to provide useful information (names associated with data)? Presently it appears that Human Subjects committees focus on the worst case scenario rather than simply addressing a reasonable level of concern for participants in an educational setting.
2. What are the concerns associated with validity and reliability when technology is used in assessment across several school systems with different goal sets and administrators, teachers, parents, and students fearful of "testing"? How can technology help in motivating students and teachers to become engaged in doing their best on assessments? How do we guard against assessment designed for one purpose being used for other purposes?

III. References:

A. Different levels of involvement of teachers:

1. There are some systems that present information with minimum interaction with the student (e.g. a NASA website showing photos from Hubble.) The technology gets information out to learners, but what are the opportunities for assessment here?
2. There are systems that require student interaction for instruction and/or assessment, but operate virtually independently of the teacher. Example: Falmagne's ALEK system. To a lesser extent, the Anderson inspired systems. Falmagne Doignon, J.-P. and Falmagne, J.-C. (1985). Spaces for the assessment of knowledge. International Journal of Man-Machine Studies, 23, 175-196. Anderson-Corbett-Koedinger An Algebra tutor. (www-2.cs.cmu.edu/~pact/Documents/IntelligentCogTutors.doc).
3. There are systems that are designed to operate in co-operation with the teacher. Nice to do in theory, but requires more commitment from the teacher. Examples:, BEGUILE- Reiser, et. al. Integrating project-based inquiry into the middle school (tigersystem.net/aera2002/viewproposaltext.asp?propID=2732) DIAGNOSER- Hunt and Minstrell Diagnostic assessment tools for teachers and students to monitor learning of science and mathematics content. (tutor.psych.washington.edu) Alternaive assessment and electronic portfolios (transition.alaska.edu/www/portfolios.html) MODELIT- Supporting scientific modeling and simulation (hi-ce.eecs.umich.edu/sciencelaboratory/modelit/) WISE- Applying science to real world surroundings (interactiveu.berkeley.edu:8000/DLMindex/heat).

B. Other web-sites related to assessment using technology

Baker, Niemi Assessment in complex Learning Environments Use of concept maps and related scoring schemes. (www.cse.ucla.edu/CRESST/pages/about.htm)

DiscoverySchool.com A resource for teachers. (school.discovery.com/schrockguide/assess.html)

Mislevy- Applying Basian statistical analyses for diagnosing student thinking. (www.sri.com/policy/designkt/mislevy2.pdf)

NCREL, Use of technology to engage learning (www.ncrel.org/sdrs/edtalk/policyis.htm)

Quellmalz and SRI Performance assessment tasks on-line (pals.sri.com/)

Tanimoto- Tools for collecting students' responses to open-ended questions for teachers to read and capture students' problematic thinking. (www.ciltkn.org/cilt2000/abstracts/2101.html)

VanLehn, Graesser- Simulating human tutoring with feedback and next directions reflecting what is learned from prior responses. (www.press.umich.edu/jep/06-01/schulze.html)

C. Relevant technology and assessment sources available from National Academy Press, National Academy of Sciences

• "Grading the Nation's Report Card: Research from the Evaluation of NAEP."
• "Classroom Assessment and the National Science Education Standards".
• "Knowing What Students Know" has a section on use of technology for assessment.
• "Technology and Assessment" See especially Drew Gitomer at ETS, Barbara Means, and J.D. Fletcher that are pretty relevant to our concerns.