Design Experiments: Promise vs. Delivery

I. Overview and Current State of Research

Current State of the Research

Design:

"Design is the process of identifying and achieving preferred outcomes, of solving problems and responding to human needs; and of managing change. The design process involves many ways of knowing. While basic sciences rely on the scientific method and testing of hypothesis, and the arts depend primarily on intuition, design is somewhere in between; it borrows from both disciplines, developing methods and predispositions for acquiring knowledge, skills and attitudes that respond to the complexity of real life situations.

Design surrounds us, influences us, enables or hinders us, because it determines the products and systems we increasingly rely upon to accomplish our intentions. A simple inventory of one's daily encounters with design products and environments should demonstrate the overwhelming presence of design in our lives and will start to suggest design's impact upon how and what we think, feel and do" (Anna Sanko, Architecture Resource Center, New Haven, CT). http://www.nyfa.org/educate_by_design/ed_design.htm

As noted in the above quotation, design approaches to research find themselves straddling intuition/judgments of "best practice" and a scientific approach directed at disambiguating hypotheses.

Working against current tendencies to establish randomized field trials as the sine qua non of scientific methods is a quiet revolution in design-based research methods. By whatever label, these emerging methods provide a "working space" in which methodological innovations can grow. It was the goal of this session to provide a positive atmosphere in which we can expand and strengthen this emerging methodology.

The use of the word design allows us a broad canvas for productive thought and conversation. It expands the people from which we can draw for powerful ideas and interventions, including engineers, architects, computer scientists, knowledge managers, experts on diffusion of innovation, philosophers, anthropologists, cognitive scientists, cognitive neuroscientists, complexity theorists, game theorists, and others. For example, the work on product development at http://www.ulrich-eppinger.net/ may be of value.

If we envision educational research as ultimately a service to teachers and students (to improve teaching and learning in the "real world"), then it would appear that design studies may be developed with a range of goals in mind.

For example, we may ask, how can we design Design Research so that the legitimate needs of those proposing randomized trials can be met and surpassed?

How can we design Design Research so that the legitimate needs of those on the diffusion or scaling end can be met?

For those interested in "what works?" questions, we may ask, "What is the best way to design the what that you want to know how it works, and what design standards and benchmarks do you have in mind for this what?"

A number of researchers have begun to explore these issues in a serious way. A collection of related papers may be found at http://gse.gmu.edu/research/de/. In addition to these papers, manuscripts are under review for a special issue of the Educational Researcher on design experimentation, and these papers will be posted.

Another current source is the Journal of the Learning Sciences (http://www.cc.gatech.edu/lst/jls/index.html). In particular, see Edelson (2002) for both the points that are made and for the reference list.

The following list of questions was provided at the PI meeting to spur thought in anticipation of a positive and creative session.

Some challenges for some variants of current design studies, include:

• How can we retain both the learning about the artifact (broadly conceived) and the learning about learning that the design of the artifact exposes on the part of students, teachers and researchers?
• Paper-and-pencil measures are often, politically, the "gold standard" of learning. How to we demonstrate learning on these measures in addition to learning during the innovation? How do we best address "transfer of learning" questions, methodologically?
• How can we help researchers measure learning and cognitive change objectively as well as qualitatively (subjectively) while students are actively involved in the design study?
• How can we design studies that have features to allow them to be used later by others, particularly those who will have fewer resources? How can we design for adoption, adaptation, rebuilding, replication?
• How can we design sets of studies on innovation that allow for a more aggregative science across sites, across time, across populations, across content areas?
• What tools and methods allow us to better "data-mine" and more richly report the learning of students, teachers, and researchers?
• What other data representation tools can we use that will make the rich learning during and after the design experiment available to many audiences?
• How can we conduct design studies that take systemic factors (and systems thinking, generally) into account?
• How can we design more effective, multi-tiered studies so that learning at each tier (teacher, student, researcher) is captured and so that the input of each participant is valued?
• Design experiments typically unfold somewhat haphazardly with little guiding protocol, and often without the "lab testing" component suggested by Brown What models of principled design can guide us making these experiments more systematic and explicit?
• Design experiments occur on many time scales (e.g., learning during the experiment on the innovation, related learning during a larger instruction unit, and related learning over longer time scales). How can we monitor and report on each time scale for learning? What implications can we draw for teaching? For research?

General Themes that were Explored in the Workshop:

• How can we make design studies richer and more powerful?
• How can we make design studies more scientific along the lines of the NRC report?
• How can we make design studies more responsive to practice?
• How can we design studies that prompt innovations in instructional practices, assessment practices, and learning?

A related session studied similar issues and should be reviewed in conjunction with this one [Dissemination and the Integration of Research into Practice; http://www.prospectassoc.com/NSF/dis_integ.htm]. Note that the items in their Table under "Term" relate closely to the expanding view of design studies, which goes beyond standard constructivist teaching experiments using grounded theory.

TABLE I: Different Styles of Classroom Research on Innovation and Practice and implications for partnerships

Discussions

A number of issues were raised in the discussions reflecting the diverse backgrounds of the attendees.

One group focused on the "dissemination" or "adoption" aspects, particularly the likelihood that teachers will adopt innovative measures given the push for standardized testing. Current thinking on design experimentation does not address this issue in any serious manner, but the design experiment project does direct the reader to the seminal work of Everett Rogers [gse.gmu.edu/researcher/de].

A second group, reported on by Nemirovsky, focused more directly on the idea of an artifact. What they felt was missing was a framework in which the artifact was seen as fitting. This framework would include the student. In other words, can an artifact "cause" learning or is some level of interpretation and use relevant here? What of social theory? Of culture? Don't artifacts and local conditions "take on a life of their own"?

This group also asked what measures should be used to determine the effects of designed artifacts and whether they would be acceptable to a larger audience. Standardized tests? Case Studies? Which methods: qualitative or quantitative?

A third group, reported on by O'Bannan, looked at design experiments as not involving the development of an artifact per se. Rather, that the design experiment should focus on the process of how instruction is taking place and how to communicate this back to the teacher. Within this model, the design experiment looks at the creation, development and changes of conjectures.

A fourth group, reported on by Pinkard, explored the concepts more directly related to the relationship to science. Where does sampling fit in? What about generalization? Statistical significance? Do we ever move beyond prototyping? Whose job is it to do so? How do we deal with "lethal mutations" of our work once it is outside our control? Ultimately, this group felt that design experiment work ran the risk of being local and undisciplined, that it lacked strategy.

Overall, the comments of Baumgartner helped summarize: The label design experiment is loaded with meanings and connotations that have not yet been fully articulated. Design-based methods appear to be important and to contribute some information and insight that does not come from traditional methods. The work of the future should be greater articulation and a push for clearer and more rigorous standards. When this happens, we can be in a place to address issues of going to scale. The issues discussed in one group, written up by Dan Burke, capture the sense of flux characteristic of all of the groups (see Appendix).

Summary

In summary, the diverse nature of the group and their sustained interest in what was clearly a cloudy and unclear methodology was compelling evidence in favor of greater attention to the articulation of design-based research methods in education. Some researchers are drawn to a method that allows for innovation. Others wish to exploit the framework to study unfolding, iterative processes in classrooms. Still others see the emerging set of methods as necessary Development (D) work in a Research and Development (R&D) view of educational research. This view is in contrast to the recent NRC report that moves the field more forcibly to the Research (R) emphasis. What is clear from this meeting is that the engineering metaphor in educational research strikes many as one that deserves greater attention not only for its potential role in improving theory, but also in its impact on scaling.

Appendix

• How do the roles of "designer" and that of "researcher" differ and intersect?
  • Researchers can bias the process by projecting their representations.
  • Conversation analysis (for example) can be used to "distance" researcher and subject?
    • To answer "What do they (subjects) think they are doing"
    • To achieve the same epistemological form in a different "game"
  • Triangulation of interview data can be used to collate designer/teacher/student perspectives
  • Technical Artifacts can be designed to be "adaptive"
    • The context of social interactions changes their use
    • The nature of artifacts determines a number of factors
• Two big issues that need to be addressed:
  • When, why, and how do researchers involve users/novices (teachers and students) in the design process?
    • e.g. making use of "Design Rationale" e.g. "claims analysis"
  • How do we empathize with users (e.g. in using ethnographic methods)?
    • By describing the users' interpretive framework and allowing for multiple entries and outcomes in such a way as to discourage stratification
• Artifacts play a crucial role in determining interactions/use: Artifact Determinism.
• In what should the relationship between "Experiment" and "Design" consist?
  • Should design experiments seek to produce generalized or reproducible results?
  • There are differences between aspects of production and dissemination (expert knowledge) in research.
  • The local/contextual/situated-ness of knowledge presents challenges to educational/classroom research. (see e.g. Solomon's notion of innovations as "packages").
• In attempting to introduce technologies/artifacts for adoption, researchers are faced with changing communities/culture, which introduces those same challenges and opportunities associated with social movements.
  • In designing new artifacts for adoption, researchers are attempting to "cultivate and design culture" in the process.
• Given the unknowns and variation of classroom contexts and social/cultural situations, one challenge involves how to incorporate "designed ambiguity."
• What can count as "evidence" in design experiments?
  • e.g. how does one identify salient "episodes" in video analysis
  • how can researchers distinguish "critical" factors from others?
• What methods should and are considered "scientific" or "legitimate"?
  • Quantitative are often considered easier to justify (as producing warrantable "claims") than are qualitative claims.
  • Explanatory results and methods are often considered more "scientific" than descriptive results and methods.
  • Methods are hypothesis and theory laden.
• What counts as "GOOD" design and experimentation?

Reference

Edelson, D. C. (2002). Design research: What we learn when we engage in design. Journal of the Learning Sciences, 11(1), 105-121.