Some Thoughts on Design Studies in Education and Points for PI Discussion--Eamonn Kelly, George Mason University

The design experiment/study methodology and its goals, purposes and audiences in educational research are currently under specified, but the term is applied, typically, when describing the iterative refinement of some innovation (often involving technology) in teaching and learning environments.

Session Goal:

Working against current tendencies to establish randomized trials as the sine qua non of scientific methods is a quiet revolution in design-based research methods. By whatever label, these emerging methods provide a "working space" in which methodological innovations can grow. It is the goal of this session to provide a positive atmosphere in which we can expand and strengthen this emerging methodology.

The use of the word design allows us a broad canvas for productive thought and conversation. It expands the people from which we can draw for powerful ideas and interventions, including engineers, architects, computer scientists, knowledge managers, experts on diffusion of innovation, philosophers, anthropologists, cognitive scientists, cognitive neuroscientists, complexity theorists, game theorists, and others.

For example, the work on product development at http://www.ulrich-eppinger.net/ may be of value.

If we envision educational research as ultimately a service to teachers and students (to improve teaching and learning in the "real world"), then it would appear that design studies may be developed with a range of goals in mind.

For example, we may ask, how can we design Design Research so that the legitimate needs of those proposing randomized trials can be met and surpassed?

How can we design Design Research so that the legitimate needs of those on the diffusion or scaling end can be met?

For those interested in "what works?" questions, we may ask, "What is the best way to design the what that you want to know how it works, and what design standards and benchmarks do you have in mind for this what?"

The following list of questions is meant to spur thought in anticipation of a positive and creative session during the PI meeting.

Some challenges for some variants of current design studies, include:

1. How can we retain both the learning about the artifact (broadly conceived) and the learning about learning that the design of the artifact exposes on the part of students, teachers and researchers?
2. Paper-and-pencil measures are often, politically, the "gold standard" of learning. How to we demonstrate learning on these measures in addition to learning during the innovation? How do we best address "transfer of learning" questions, methodologically?
3. How can we help researchers measure learning and cognitive change objectively as well as qualitatively (subjectively) while students are actively involved in the design study?
4. How can we design studies that have features to allow them to be used later by others, particularly those who will have fewer resources? How can we design for adoption, adaptation, rebuilding, replication?
5. How can we design sets of studies on innovation that allow for a more aggregative science across sites, across time, across populations, across content areas?
6. What tools and methods allow us to better "data-mine" and more richly report the learning of students, teachers, and researchers?
7. What other data representation tools can we use that will make the rich learning during and after the design experiment available to many audiences?
8. How can we conduct design studies that take systemic factors (and systems thinking, generally) into account?
9. How can we design more effective, multi-tiered studies so that learning at each tier (teacher, student, researcher) is captured and so that the input of each participant is valued?
10. Design experiments typically unfold somewhat haphazardly with little guiding protocol, and often without the "lab testing" component suggested by Brown What models of principled design can guide us making these experiments more systematic and explicit?
11. Design experiments occur on many time scales (e.g., learning during the experiment on the innovation, related learning during a larger instruction unit, and related learning over longer time scales). How can we monitor and report on each time scale for learning? What implications can we draw for teaching? For research?
12. Other questions . . .

General Themes to Explore In Preparation for the Workshop

• How can we make design studies richer and more powerful?
• How can we make design studies more scientific along the lines of the NRC report?
• How can we make design studies more responsive to practice?
• How can we design studies that prompt innovations in instructional practices, assessment practices, and learning?