Is Education Research Inherently Value-Laden? Working with messy epistemologies and conflicting implicit goals

Session leader: Wendy C. Newstetter
Participants: Doris Ash, Davis Carrejo, Hal Eden, James Kaput, Timothy Koschmann, Laura Lising, Elizabeth Birr Moje, Ricardo Nemirovsky, Janet Robinson, Ann Rosebery, Leona Schauble, Kusum Singh, Brian Smith, KennethTobin, David Uttal, Ken Whang, Karen Whitehead, Jennifer Wiley, Vickie Williamson

Current state of research

Philosophers of science, feminists, and a number of sociologists have considered the question of scientific research and the possibility of any sort of objective research even in the case of the natural sciences. These considerations appear to have fallen into two distinct and often polarized camps. One perspective championed by science studies aims to demonstrate through both historical and contemporary research that science is inherently value laden, its methodologies and foundational assumptions inevitably reflect forms of knowledge that are culturally sanctioned (e.g. the individual as the model of the knower, certain formulations of causality, the ways in which the subject/object relationship is imagined within science.). Along these same lines, there is ample evidence that scientific research does not always follow the norms of neutrality either in how it conceptualizes its data or in how it recruits and supports its practitioners---science seems to somehow foster a skewed presence of white males in its most prestigious disciplines, universities, and research projects. Whether this historical exclusion of women and minorities at the level of research formulation or researcher recruitment is part and parcel of the scientific project is still a question that is up for grabs. Lastly, industrial, political, and other economic pressures that may affect science at various levels of its practice further complicate the implication of the social within scientific work. The other camps holds a diametrically opposed perspective, one which appears to be the view of the everyday researcher and of many who try to understand the foundations of science (philosophers of science). This second perspective maintains that there is an equally obvious objectivity to science, an objectivity supported by the history of science and its successes. On this view, the norms that have ruled science merely need more self-reflection and further articulation, but science refers to some reality that overrides the social and defines research. Further, its methods are, if compromised in their execution, ideally able to insure a rational and objective understanding of the data, further hypothesis testing and so forth. The above views on the relationship between science and values may be more acute in arenas such as education, given that it is an applied field, a veritable magnet of social interests, and has itself been studied by scholars for its own ideological complicity with science and the scientific method. Thus scientific study into science education requires a degree of critical reflexivity that is almost self-subverting.

If we do grant a degree of objectivity to scientific research, i.e. research that is grounded in the empirical tradition, and we do accede numerous counts of scientific practices as value-laden, we are in a contradictory position. Neither history nor data can answer this question for us. Thus we must set out to "reconcile the claim that scientific inquiry is value or ideology laden and that it is productive of knowledge" (Longino, 1993) Helen Longino's careful division of the process of scientific work into those moments that are ruled by constitutive values and those that are ruled by contextual values begins to move us toward the possibility of reconciling the two camps described in the previous paragraphs. Longino posits that some values operate to define what are the norms that govern good scientific practice. These values refer to the priority of experiential/empirical information/data to inform one's ultimate conclusions, norms of theoretical coherence, forms of experimentation, and elegance in formulation. These constitutive values allow research to posses a degree of autonomy in its findings, but Longino believes that challenges to this autonomy are sufficiently cogent to disallow the idea that there are ultimate foundational rules that exceed the social organization of science. What she does posit is that certain social communities are capable of abiding by rules of interaction, reflection, and criticism that maximize the "objectivity" of their results. That is, a community of thinkers can be devoted to the constitutive values of science and thus adhere to norms that allow that community to enjoy agreed upon forms of knowledge making, including the relationship between data and hypotheses, the relevance of explanatory models, and other procedures that are usually lumped in the camp of those who believe such procedures are "value-free" and "free-standing." Longino clearly believes that the forms of science, except for its most base reliance on the empirical, may change and that different sub-fields of science may have different constitutive values and new ones may emerge. For Longino, diversity in a scientific community seems imperative to inoculate a scientific community against its own indifference to other social values that may affect research in less positive and "objective" ways.

Longino also elaborates the role for contextual values in science. These values "contaminate" science in a way but are unavoidable. Contextual values refer to the "social and cultural environment in which science operates" (Longino, 1990). It is absurd to think that science can operate without contextual values. These values help determine problem choice in that some problems are seen as cultural priorities while other languish. Their effects can be seen in explanatory models. For example, many current ideas on brain function and gender refer to models of causality that answer to gender anxieties as well as to the data that is collected by biologists, neurobiologists, and others. The problem relates to the necessary effect of background assumptions. As Longino insists, one can never eliminate these backgrounds. Although such assumptions enter into every level of scientific thinking and research, all efforts, such as logical positivism, have failed to eliminate such social influence. They never will in that one can never fully account for the under-determination of data. Data will never be able to tell us how to solicit it or which theory it must confirm. Induction, in sum, cannot answer all the questions of value that are raised by scientific research. Background assumptions, informed by contextual values inevitable enter the breach. In response, one can become attentive to such assumption through cultivating critical reflection within a research community. Such reflection requires a commitment to norms that fall into the category of constitutive values (knowing that these might change but are in force at this time). It also requires a diverse community that will see the taken for granted assumptions of its own cognitive efforts. Even when those efforts are framed as scientific research.

Discussion

Although the days of all out warfare that marked the debates between Phillips (1983) and Eisner (1983) in Educational Researcher in the early 1980's are gone, what constitutes "good" educational research methodology is still a contentious topic. The group that gathered for this implications discussion represented the wide range of value systems and views on this topic -- experimental psychologists, critical ethnographers, feminist theorists and conversation analysts, to name a few. Nevertheless everyone demonstrated a genuine sensitivity to the differing values, assumptions, goals, and methodologies that co-inhabit education and, perhaps more importantly, a strong desire to figure out how former warring camps can find ways to co-exist while undertaking quality research. From the beginning, it was clear that all felt it would be unwise to dictate acceptable research methodology by spelling out rules or scripts. A follow-up email stated: "I think it's worth highlighting that educational researchers are trying to move past the paradigm wars, even as we recognize and struggle with the fact that there are a number of differing perspectives on what counts as reality in teaching, learning, and general human interaction"(Moje).

This suggests that not any old kind of research is valuable or credible. Education as a field seems to be suffering from a lack of consensus concerning good problems to work on, reasonable ways of conducting inquiry, and what counts as acceptable evidence or explanation (Schauble). The group had difficulty coming to consensus on criteria for judging "good" from "bad" science. Falsifiability works well in a positivist world of immutable, stable elements, but fails when reality is understood as contingent and on-goingly co-constructed, a view held by ethnographers. Or in the research areas like that reported on by Sternberg in his talk, where phenomena are still poorly understood. Although these two concerns of being more sensitive to other kinds or research while demanding rigor seem paradoxical, some in the group pointed out that until very recently the natural sciences and applied sciences (like medicine) struggled under the same conditions (Nemirovsky). It is not clear that consensus can be legislated, and it is certainly clear that it shouldn't. A better understanding of what counts as quality research in a wide variety of approaches seems like an important conceptual tool for researchers to acquire. In moving towards such an understanding the group identified three focal areas where work needs to be done.

Individuals

In reviewing the spectrum of research approaches passed out at the beginning of the session, a participant asked whether assumptions about ontology and human nature are "rational". He clarified by querying whether people are actually "aware of and intentional about" (Tobin) their views of reality and, further, really understand how these assumptions direct the framing of questions, collection of data, and explanation. Without this self-awareness, it is difficult to have fruitful discussions about methods because the axioms fundamental to a research design are not fully understood by both parties in a discussion. One participant questioned whether a hybrid ontology was possible and how this might play out. A question that went unanswered.

Another concern was that individual researchers on the objective/realist end of the spectrum are "less accepting of other views than the other way around" (Tobin). Put another way, relativist researchers are more tolerant towards experimentalists. Perhaps this is because they have already made the personal journey from positivism/realism to relativism/constructivism and are better able to appreciate a set of values, which they have moved beyond. It may also have to do with the hegemonic position that positivistic notions of research still enjoy. Researchers in this tradition do not experience the strong pressures to accept other research paradigms. Or alternatively, perhaps they feel a threat to their power to decide research agendas. As a counter point, it was noted that relativists can be just as "close-minded (Rosebury) about other relativist approaches as realists are towards relativist approaches. This discussion raised the issue of whether, when and where students and young researchers are explicitly forced to confront the implicit/naturalized ontology that undergirds the set of epistemic practices they are learning and using. It also clarified the need to help students and young researchers understand explicitly the value of different research perspectives and methodologies. A perhaps more unsettling question is whether researchers who subscribe to radically different accounts of reality and human nature can ever come to consensus given the inherent axiomatic conflict that exists at the bedrock of their belief systems.

Communities

Like individuals, communities can be unaware of their naturalized, tacit assumptions about ontology and human nature. An important question raised in the session was how the practice of education itself is value-laden. Attempts to lump large and diverse groups into a single entity such as "African-American" or "disabled student" suggest a. positivistic bent to education. On the other hand, educational research has no "truth machine" (Koschmann) that definitively legitimates one fact for all times while dismissing others. That is because the education community does not "oversimplify" (Lising) its problems like other scientific fields like physics (…imagine a sphere). Learning and teaching in classrooms is a messy, uncontrolled and highly variable enterprise that requires thoughtful research approaches because to simplify is to misrepresent. By choosing to tackle complexity, the research community cannot always expect neat and predictable protocols and outcomes.

A major problem for a community in deciding which research paradigm to embrace has to do with the function of that research. The educational research community has floundered when trying to determine what constitutes good evidence and explanation. This is because learning and teaching are not like engineering where the bridge either stands or falls down (Schauble). Satisfactory explanations can take the form of regression models or descriptions of mechanisms at the micro level. If a community insists on privileging one over the other, an impasse and a developing sense of hostility can ensue. Communities need to find ways to appreciate both kinds of explanations and develop an understanding of when might be of more value than another. Determining the value (short-term focus and fix) or strategic use (part of a long-term process of anticipated discovery) is a complex problem in its own right. Where should the community focus its attention? This might be another way of articulating the applied Vs. theoretical conflict. Or the pure science towards understanding vs. the engineering of a solution to an identified problem. As a community, if we have not identified the big problems to be solved, should we be trying to engineer solutions?

At many universities, teams of cross-disciplinary researchers are collaborating more as they work on complex problems. These collaborations are not easy, however, and take a great deal of work. Disciplinary-specific glosses for many words like "discourse" make translation across the borders difficult and even contentious. A lot of time gets spent arguing about definitions, which can be unsettling to graduate students (Moje). When diverse research groups come to consensus there is the danger that the more interesting questions, the outliers, might get overlooked (Tobin). So there's a tension between keeping the research agenda lively, not stale, and developing mixed methods groups that have developed complimentary research agendas.

NSF panels

In the review process, panelists from different paradigms have the opportunity to learn more about and even develop an appreciation for alternative forms of knowledge making. One participant commented that sitting on the panels had been an important educational experience for him (Uttal). There is the danger, however, that panelists will be dogmatic and close-minded towards other types of research practices. When asked if there was a formal educational process for sensitizing new panelists, the NSF program officer explained that the panelists are not formally trained, but rather the directors look for people who demonstrate the ability to rationally evaluate research proposals for their merit rather than their epistemological orthodoxy. That does not insure, however, that all panels will be unbiased or narrow in judgement and some panelists are not asked back.

Recommendations for policy and program development

While the group did not explicitly develop recommendations for policy and program development, several can be inferred from the overall discussion.

• The panel review process could be improved if potential panelists were asked to develop personal profiles of their own values and epistemic practices. This requirement could help promote greater self-awareness of implicit ontologies and encourage a greater sensitivity to alternative knowledge-making practices while aiding the program directors in gathering balanced panels. Further, a NSF questionnaire concerning a panelists prior experiences with alternative methods and attitudes towards them could alert the directors to unsatisfactory candidates.
• While sitting on a panel can be an educational process, perhaps developing understanding of and sensitivity to other forms of research should not be left to the last minute or to chance. Prior to sitting on a panel, panelists could be "certified" either through an on-line course or an on-site seminar where various types of research in the educational arena were explained and their value made clear. This would at least insure the possibility of a common shared vocabulary, which could be applied to all kinds of proposals. A NSF manual that could be used at the panelists meeting if questions or concerns could compliment this process.
• If the NSF wants to foster a dynamic research environment without privileging one form of research over another, perhaps a roadmap or chronology of research methodologies could be developed. This would assist the educational research community in thinking about how to bring different ends of the spectrum together around a problem. As pointed out in the discussion (Whang), there is a need for both "good" research and "important" research. The former falls into the category of research that can be well controlled because it is focusing down very narrowly on specific phenomena that are fairly well understood. The latter, in contrast, refers to knowledge making in arenas that are very poorly understood and much harder initially to get positive results. One is much riskier than the other. Yet, both kinds of research are important and necessary. Perhaps a sheet that makes explicit categories of research would be helpful in situating various proposals on the good/important spectrum.
• If funding patterns encourage mixed method approaches and cross-disciplinary investigation, then the community will respond in kind. Longitudinal studies that enlist different methods at different phases might be helpful in bridging the methodological schisms. Of course, mixed methods approaches must demonstrate as much rigor in their design and execution as a single method.
• NSF has an opportunity to make sure funding goes to educational "science" and not to educational "engineering". While developing improved designs of educational environments and improved learning for all children is highly desirable, this should not be our primary goal. Too little is really known about learning-in-the-wild where context, identity and motivation can easily confound the best research in the lab. To engineer better solutions to problems, fundamental axioms, laws and assumptions are necessary. Unfortunately, educational researchers do not have these available, so to promise huge gains or improvements is talking more like an engineer and less like a scientist. Scientists identify the hard questions and go after answers there. Unfortunately, the political pressures on NSF to deliver and the mistake often made of making educational research out to be more like engineering than science, has put pressure on researchers to narrow down rather than opening up.

Eisner, E. W. (1983). Anastasia might be alive, but the monarchy is dead. Educational Researcher (May).

Longino, H. (1990). Science as social knowledge. Princeton, NJ: Princeton University Press.

Longino, H. (1993). Essential tensions- Phase Two: Feminists, Philosophical, and Social Studies of Science. In Antony & Witt (Eds.), A Mind of One's Own: Feminist Essays on Reason & Objectivity: Westview Press.

Phillips, D. C. (1983). After the Wake: Postpositivist Educational Thought. Educational Researcher (May).