Integration of Research and Practice

"We may need to start a project by understanding the characteristics of the intact system before we introduce the innovation; we need to know much more about the nature of the system since we know as researchers the nature of the intervention"—Patricio Herbst, University of Michigan, Session participant.

The evolving goal of the session was how do we look at our own research from the lens of integrating it with practice, and what does this focus implies for how we understand our roles as researchers and builders of knowledge. An important theme was not simply how to make research useful for practice, but how do we change our discovery processes, research structures, and notions of boundary points to increase the use of research.

Use should be seen as an outcome of integration of research and practice-the "embeddedness" of research in practice should result in a union, rather than the intersection, of the two. In addressing integration, research should not only be centered on the classroom but on the classroom and its environment (the "system"). Participants noted that research of this type is being conducted, but the full meaning of "integration" is not widely recognized. And this recognition is important for expanding both the usefulness and the use of research in classroom practice.

Though the discussion reported here is directly applicable to ROLE quadrant IV (i.e., systems related research), we believe it has important implications for the other three quadrants as well. However, it was also recognized that not all research funded by ROLE will or should address these factors. There are different kinds of research with different intended ends and participants embraced a portfolio of projects perspective. However, all research should have idiographic or methodological relevancy for some significant question of interest to a specified audience.

Summary Findings

Dissemination Versus Integration

The group's discussion illuminated what sets "integration with practice" notions of embedded knowledge creation and exchange apart from more traditional (and familiar) notions of "dissemination." The distinguishing features of integration become clear when juxtaposed with dissemination thinking—its focus on considering the information's use as opposed to its possible usefulness. The underlying assumptions behind dissemination are no longer tenable, if they ever were:

• Dissemination of the research results is an add-on that comes after the work.
• Each research project must be disseminated, without regard to its goals, results, or audience.
• Incentives for use of research results are not needed: supplying information is sufficient.
• Dissemination is a passive activity that does not require researcher know-how.

Not all research should be disseminated to practice. Education research needs to be "engineered"—controlled for quality, aggregated, compared, and synthesized into a menu of options with clear conditions of applicability. This makes clear why a portfolio approach is needed to simultaneously explore the use of interventions or innovations in practice and the creation and study of interventions or innovations themselves.

"Disciplinary research questions drive the research in disciplines...but for us (education as applied to Pasteur-like science), practice should drive the research questions"—Jeanne Century, EDC, session participant

The "research question" for considering dissemination

It could be argued that the field has clear three types of education research: (1) Innovation—what could be, (2) Survey—what exists, and (3) Disciplinary—sociology of education; history of education, etc are examples. But it does not yet have such a clear view of what "experimental research" outside a disciplinary perspective (for the most part, based on psychology) could be.

"How to" research must be based on understanding "what exists" locally and on the comparative analysis of multiple "what could be" options—Nora Sabelli, SRI International, session chair.

Those that call for randomized controlled studies make this point, but do so in a limited and narrow way, as discussed in this year's NRC report on The Nature of Evidence on Education Research. In essence, education practice tries—and has imposed upon it—many "experimental interventions" with no support for or expectations of learning from the experiment's outcome. The experiment either "succeeds" or "fails" in a short timeframe, based on indirect measures of changes that take place in longer timeframes. If the experiment fails, a new, independent one is tried. This is why NSF should consider a special call for proposals that can explore the boundaries of "education experiments"—that is to say, what is learned, and how, from small failures.

True large-scale replication studies that could accomplish this bridge between "experiments" and general learning are much needed but expensive. Ideally, there should be a sufficient number of such experiments that allow understanding the boundaries of effectiveness of a types of interventions, out of a taxonomy of possible intervention approaches. The dire need for such studies, and thus for less costly mechanisms for studying the outcome of "experiments", is a crucial stumbling block to the credibility of education research. In fact, and upon reflection, the session suggested that proactive dissemination cum "technical assistance," seen as a research study that focuses on conditions, could be one way to gain a better understanding of possible solutions to this important research validation problem. Conceptually, this may in time lead to more effective—or less reductionist—ways to tackle causal attribution in very complex and messy environments where may variables and activities interfere with each other.

Not all practitioner questions lead to research, and not all research informs practice directly. Integration with practice is not a single step, nor is it the dissemination of final results. We should think in terms of an agricultural extension model (levels of research going from general to local). For the success of any intervention (or prescription), how it is implemented matters as much as the nature of the prescription itself. Research transmitted only by artifacts (manuscripts, manuals, one-time-only TPD, etc.) is limited in its dissemination power. One should introduce artifacts as part of a planned apprenticeship period.

One should not forget that understanding in gaps in knowledge is a likely outcome of an "engineering" task of aggregation and studies of the conditions of practice. This process is unlikely to end; the questions id raises are likely to evolve. "Engineering" practices are driven in important ways by clients' needs; "disciplinary research" does not have the same relation to the problems it studies. NSF has had traditionally, a clearer understanding of 'disciplinary research" than of "engineering research". Both are needed, within the field but also within NSF (HER) as well.

The demand and supply for quality research

A fundamental missing piece in both the use of and the support for education research is "classroom demand" for quality, useable, aggregated research. It has been documented that teachers and to some degree administrators limit their consideration of research to very specific questions. There is "demand" from some level of policy makers, but the demand is not coupled with clear and informed ideas of what is appropriate or good, quality research for their particular question. And, at high levels, research is neither trusted nor used in any direct sense of the term.

If you think about engineering, agriculture, or other technical fields, everyone in that field has a fundamental understanding of what role research plays in advancing their field. And some basic idea of how to separate good from bad research. Students in these fields go to work in industry and marketing as well as into research. A parallel group does not exist at the same level in education practice—most administrators started as teachers not as policymakers, researchers, or even educators with some exposure to research methods.

Likewise, businesses often have R&D units in order for the organization to have an absorptive capacity for innovation—schools have little, and often no, absorptive capacity. Thus, there is an need for creating mechanisms for aggregation, summarization, quality control, and the provision of hands-on, classroom-based, content-rich, "technical assistance". And, a need for schools to develop their own forms of absorptive capacity in ways that work for their unique mission.

Education as a socio-political community often acts in a "bookend" model: standards and accountability at one end; punishments and rewards at the other. But the crucial element is the process by which the demands are interpreted and implemented, which is where research-savvy support needs to be made available. Districts and schools use their limited resources to do this without easy access to comparative reference to what is known to work best. But even if they did have this capacity, they would be hard pressed to find appropriate summary information within the resources at their disposal.

"In our own research, it was helpful to articulate a model-a logic model allowed us to discuss with practitioners what counts as evidence at each part of the model. Practitioners helped us build mentoring into the model and ways to deal with personnel turnover-two elements of equal value to our research and their practice."—Jere Confrey, University of Texas Austin, session participant.

Researchers incentive structure is also oriented towards research and publications, not practice. NSF and ED should pay attention to the need for a new sub-culture of research, one perhaps better described as the equivalent of the clinical faculty in teaching hospitals. Appropriate incentives and rewards could be explored as part of funded programs and projects.

Reconceptualization of the research project, its purpose and boundaries

Implementation research of this type works smoothly when both perspectives (research and practice) are integrated into one person or group of people with a common history. Otherwise, time for developing trust and collaboration mechanisms can be a formidable obstacle. Teachers are not convinced by research with one teacher and five students. Even if they agree that it worked as research they want to be shown how it works in their own classroom—the "technical assistance" component of work that is mutually beneficial.

This is an important consideration for funding programs, since the requirement almost always indicate either the need for an existing strong history of collaboration, or a much longer timeframe for the work to show results.

"With our research [on undergraduate teaching and learning], the researchers and practitioners are the same group of physiologists working together on their own practice. The model has an output state—what should the student be able to do-but this is not the same as what content knowledge should they have. We need to start where our input state is—what students know. Research questions are generated through classroom experience. Then this faculty makes other faculty aware of their results in a setting where they can practice the results and implement them. Look at your classroom as a lab in engaging your research." Harold Model, University of Washington and Physiological Education Research Consortium, session participant.

"We are insiders and can make this bridging work happening because we are doing implementation research. We define as research questions on these insider topics and treat them as researchable questions like changing us…study the ways that universities treat collaborations with schools." Barry Fishman, University of Michigan, session participant.

"We are not necessarily insiders in the research context---how do we connect what we know with what teachers know? What lessons can they pull out from our own teaching that can illuminate our research at the k-12 level, where their teaching takes place"? Susan Goldman, University of Illinois at Chicago, session participant.

"We must differentiate research from technical assistance. Is it the evidence that they want or is it you in their classroom? If the latter, can implementation research do both?" Pam Aschbacher, California Institute of Technology, session participant.

Summary Recommendations

The session had a marked tone of thinking across projects, not on how each project can solve the manifold problems on its own. In brief, three major needs were identified that stand in the way of broader success in the integration of research and practice:

• Synthesis of research on practice-related topics that summarizes, compares, and makes clear the critical conditions for success of each intervention or practice considered in the synthesis.
• Focusing dissemination on coherent views about classroom needs for balance between disparate interventions, not on a single intervention or on the nature of the interventions themselves.
• Fostering field-wide research and dissemination infrastructures, perhaps by undertaking studies to understand what informational or expertise resources teachers and schools use and how they make use of them (for example, Eric Digests, professional society newsletters, multiple Internet portals, books, newspapers and periodicals).
• A better understanding of the implications of fostering a "research cum technical assistance" methodology.

It is the case that we cannot solve "the" big problems with a single project. We must envision a conglomeration of studies with longevity and several stages of support. Filtering, winnowing, synthesizing and molding work must take place in a way that controls for quality and considers factoring affecting use and usability. Although a single project cannot address all the facets of a problem, it is still true that it can conduct its work in ways that facilitate ROLE's role of creating and synthesizing a portfolio.

The areas of need suggest the following recommendations.

Recommendation: ROLE should request proposals for research on the mechanisms for aggregating and comparing research studies on important topics that take a classroom or school stance (i.e. a broad enough lens to encompass different interventions for the same problem, different strategies, and so on).

Recommendation: Establish expectations for considering the balance between and different and comparable approaches to a problem within a subset of projects. Encourage the use of comparisons among approaches when the problem to be solved, the questions to be asked, or the intervention to be studied, could be refined by a conceptual understanding of other approaches

Recommendation: Propose and accept the identification of intermediate measures of success. The program should consciously explore how to take into consideration the pace of change: the pace of research is quick while the pace of change in practice is slow. The pace of intervention research is probably comparable to that in practice.

These would be different types of research study that may have aspects in common with meta-studies, but should go beyond them to include, for example, surveys to fill holes and suggestions for future studies. Most importantly, these studies could be conducted in collaboration with practice as away of redefining what we mean by "technical assistance." This in turn would open the doors to closer working ties with non-research intermediary groups that can help scale-up research outcomes, as recommended by participants. Such studies could provide a means of establishing collaborations with strong opportunity of working with teacher professional societies in a more formative way, and with organizations of local service providers that work with schools. Some studies could fund "apprenticeship" periods for staff of these organizations in assessing scale-up, sustainability, and fidelity issues in adoption, replication, and adaptation.

Studies should consider how the work will or will not inform three different audiences: 1) scholars—what are the important holes that need to be studied, 2) practitioners—what do we know of a topic that is ready for use, and how to choose between the options identified for that topic, 3) policy community—what are the expectations to which reformed practice should be held, and what is the impact of different policies on such expectations. Again, informing different audiences may be a result of post-facto synthesis work.