NSF Logo and link Learning and Education:  Building Knowledge, Understanding Its Implications, May 15-17, 2002, Arlington, VA
Skip navigation and go to content
    
 

Simulations and Visualizations:
Issues for REC

Paul Horwitz
The Concord Consortium

This paper arose out of a discussion held at the REC PI meeting on May 16, 2002. The ideas and opinions are derived from those expressed at that meeting, but have been integrated and edited by the author so that attribution of them to individuals would be both difficult and misleading.


Some definitions

This is one of those fields where the descriptive terms seem to be used differently by different practitioners. Accordingly, let us start by giving some working definitions of three such terms: visualization, simulation, and model. To introduce these definitions we remind ourselves that there are only two things that a computer can do for its users: it can show them things and it can let them do things (Horwitz, 1999). That's really all there is to it - every computer application, no matter how complex, is simply a set of choices of what to show and what to let the user do. Of course there is a wealth of complexity hidden in that simple statement. It isn't just a question of what to show - different representations of what is shown can have very different consequences; similarly, different user interfaces can dramatically affect how easy it is to do one thing or another. So, specializing now to software intended to teach something, what is learned will depend, among other things, on one's choice of representations and interfaces.

Basically, visualizations are what we choose to show users, simulations are what we let them do. And models are what link the two. In the case of educational software, the design of the visualizations, simulations, and models will depend critically on what is being taught, for what purpose, and to whom.

An Example

Say we are building a simulator of a nuclear reactor for the purpose of training people who will later be running the real thing. Our choice of visualizations will no doubt include a representation of a control room, complete with all, or an important subset of, the dials, switches, buttons, digital and graphic readouts, and other displays and controls present in the real control room. We will make an effort, presumably, to assure that the simulated displays and controls look the same and are in the same relative locations as their real-world counterparts. In fact, given the lack of standardization in the industry (Frogner and Meijer, 1980), if we do our job well our simulator may strictly apply only to one particular reactor control room in the whole world!

The controls of our simulated control room will, of course, be "active" in the sense that they can be manipulated by the learner to set various parameters of an underlying model of nuclear reactor dynamics. They constitute the interface to the reactor simulation - they encompass, and consequently constrain, what the user can do.

Now imagine that our reactor simulator has been built to teach nuclear physics, say at the undergraduate level. In this case the visualizations will be considerably less realistic. The control room, in fact, will probably disappear completely, to be replaced, perhaps, by a visualization of uranium (or possibly plutonium) nuclei emitting and absorbing neutrons, fissioning into faster-moving fragments, and sometimes decaying spontaneously. The time scale of the simulation, which may have been sped up in the previous example, will here have to be slowed dramatically. The controls will also be quite unrealistic, offering students the ability to add or delete nuclei, change reaction cross-sections, or alter the speed of the neutrons - operations they could not perform in "real life."

The underlying models of our two contrasting examples will be quite different as well. The simulation intended for reactor operator training will most likely be run by a top-level model consisting of rules that govern the macroscopic behavior of the principal components. Thus, activity levels will depend on core temperature according to the dictates of a table derived from experimental data; the effects of water flow on reactor cooling will be similarly determined. The model for the nuclear physics simulation, in contrast, will pay much less attention to reactor-scale parameters, concentrating instead on the details of sub-atomic processes hidden to the real-world observer.

The differences between the models, visualizations, and simulations in the two applications arise from their different learning goals. The training of reactor operators is designed to give them a wealth of procedural knowledge that will guide them in carrying out their functions. Physics or engineering students, in contrast, seek to acquire a deep understanding of the physical principals that govern how the reactor works. To the operator, the nuclear reaction taking place in the reactor core can largely be treated as a "black box" heat source. The physicist can equally well ignore the details of how that reaction is monitored and controlled.

The criteria by which to judge the quality of the models will vary as well. In the training example one would expect the model to be faithful to the reactor itself - to reproduce, in other words, as accurately as possible the behavior of its real-world counterpart. In the case of the physics model, since the goal is to teach students how to interpret the macroscopic behavior of a large ensemble of unstable nuclei by extrapolating from the statistically deterministic behavior of a few hundred, accuracy is less critical than explanatory power. Inclusion of extraneous detail, in addition to restricting the generality of the model, might actually hinder understanding. The model will therefore be purposely simplified, glossing over, perhaps, such practical considerations as the exact geometry of the reactor core, in order to clarify basic concepts common to all situations.

Issues for educational simulations

The REC panel discussed a number of issues and questions associated with the creation of simulations for teaching.

Accuracy of models

The nuclear reactor simulation example above illustrates a key question: how realistic should the model underlying a simulation be? Although there was general agreement that the simple answer is, "it all depends," in fact, the consensus of the REC simulation and visualization panel was that in most instances scientific accuracy of a model is not as important, for educational purposes, as simplicity and explanatory power. A chess-playing program designed to teach the game to novices is likely to be a weaker player than one that is optimized for performance. Similarly, a simplified model of a complex phenomenon is often better suited to teach beginners than a more sophisticated model that matches experimental data more precisely at the cost of being more difficult to understand. The ideal situation, it would seem, would be to have a model that is more sophisticated than strictly necessary for beginners, but to simplify the simulations and visualizations associated with the model so that naïve learners will not become confused. To do this effectively, however, calls for scripting the model so that it can alter its mode of presentation to fit changing circumstances. This possibility is discussed further below.

Revising models

Aside from precision, another criterion for judging a model is the range of situations to which it can be successfully applied. The process of continual refinement and generalization of models is central to the scientific enterprise, and a valuable process for students to observe. Once again, an example may serve to illustrate the point1.

A special-purpose rule for predicting the results of a head-on, elastic collision between two equal-mass objects is a far cry from the universally applicable law of conservation of momentum. A strategy for helping students appreciate the power of such general laws is to guide them through a succession of what White and Frederiksen (1990) have termed "upwardly compatible causal models." Thus, one might start by modeling head-on collisions between identical particles, one moving and one stationary. Once the students are familiar with this case, one can first allow both masses to move, then generalize to glancing collisions. At each stage, students are encouraged to derive a sequence of ever more general rules, such as "When one ball is at rest, the moving ball stops dead and gives all its velocity to the stationary one," or "When both balls are moving before the collision, they exchange their velocities," to "The total velocity of the system is the same before and after the collision." Once we allow the masses to be different, the velocity conservation rule no longer works and must be generalized to the familiar law of conservation of momentum: "The product of mass times velocity is conserved in the collision."

As they move from one of these rules to the next, students are not only coming closer to an accepted scientific law governing collisions - they are also learning to test their models against experiments - either real or computer-based - revise them if need be, and generalize them to cover more and more situations. Finally, the sequence of rules becomes a genuine scientific model when the students are able to link it back to Newton's Third Law, and to recognize that momentum conservation is actually a consequence of the equality of forces between the two colliding masses.

In helping students to follow the path we have laid out above, we can make use of technology in a variety of ways - data collection probes can measure forces over very short times, computer simulations make the experiments easy to perform and noise-free. Finally, visualization techniques enable us to represent abstract concepts such as forces and velocities - and, if we choose, to make them, rather than the masses themselves, the logical objects of student investigation. A straightforward extrapolation of this latter idea would be to make the physical objects that represent the real-world situation disappear entirely! This possibility is taken up in the next section.

Hiding information

Visualization enables us to show information that would not normally be available; it also gives us the ability to hide information that would normally be present. Though this may seem at first glance counterproductive, in fact it makes it possible to present challenges to students that mirror those that real scientists confront. Science, after all, is largely a process of trying to figure out what is going on, in the absence of full information. Here is an example, once again based on the situation described in the previous section.

Let us add to the two colliding balls a third, invisible one. There are various ways of doing this, but to be specific, imagine that the three balls are lined up in a row, stationary, with the invisible one in the middle. The student makes one of the visible balls move toward the other visible one, but before it gets there it collides with the invisible ball; that ball in turn will collide, after a short interval, with the second visible one. By observing the behavior of the visible balls, the student can easily guess, that there must be an invisible one between them. It is a little harder to figure out whether the invisible ball is more or less massive than the visible ones, but it is possible. If one is very clever, one can determine whether the invisible ball is moving at the end of the experiment simply by checking to see whether the total momentum of the visible balls is the same as that originally imparted to the first one. The presence of missing momentum is an unambiguous indication that the invisible ball is still in motion. Regardless of how well students perform on this task, it is an authentic one - the reasoning required mirrors very well how physicists think2.

Having students create their own models

A recurrent question in the use of computer simulations and visualizations is whether to allow students to create, as well as manipulate, the underlying models that link and control them. Simple but powerful programming languages have been created that make it possible for students with limited expertise to program their own models of complex systems. Languages such as NetLogo and StarLogo are especially applicable in domains that involve interactions among many identical components, whose statistically deterministic "emergent behavior" mimics that of such real-world analogs as insect behavior, molecular dynamics, and traffic flow (Wilensky and Resnick, 1999). Since the components usually follow simple rules (e.g., "maintain a minimum distance between your car and the car ahead of you"), such complex dynamical models can often be implemented even by novice programmers (Wilensky, 1999).

The power and flexibility that comes with being able to create one's own model is a distinct advantage of this approach, and the associated pedagogy complements and empowers a constructivist strategy, but these advantages can sometimes come at the price of entailing a rather steep learning curve. It is not necessary for students to recapitulate the work of Galileo, Kepler, and Newton, for instance, in order to "discover" the models underlying classical mechanics. Nor do they need to experiment with "triploid" and "tetraploid" organisms to find that Mendel's diploid model for transmission genetics offers a better fit to the inheritance data. The research base does not yet exist that would enable us to determine, in a rigorous way, how much leeway to give students in the creation of their own models, and under which circumstances. There is growing evidence, however, that if students are given ready-made models, rather than being challenged to generate them, one must ensure that the tasks that accompany the models elicit reasoning and understanding, rather than requiring only rote learning (Gobert, 2000).

Instructivism

More general than the issue of when to let students create their own models, is the question of how - and how much - to "scaffold" students' use of simulations and visualizations. New delivery vehicles, such as the WISE platform (c.f. Linn and Slotta, 2000), and new software development environments, such as Pedagogica™ (Horwitz and Christie, 1999), are making it easier for researchers and educational technology developers to embed simulations (and by implication their underlying models) in broader educational contexts. This can serve several purposes (Buckley, Gobert, and Christie, 2002):

  • it can situate the simulation in a real-world context, making it more likely that students' expertise in manipulating a model will transfer to other situations and contribute to deeper understanding not only of the scientific domain under investigation, but of others as well;
  • it can monitor students' actions and react to them in the context of a specific challenge or problem;
  • by logging and reporting on students' actions and responses to questions, it offers teachers and researchers a unique and powerful tool for fine-grained, performance-based assessment.

The embedding of simulations and visualizations within more traditional curricular materials has created a new kind of interactive curriculum tool called a "hypermodel" (Horwitz, 1995; Horwitz and Christie, 1999). Hypermodels integrate stored information in the form of multimedia materials, experimental data, and text, with a manipulable model of the subject domain. Just as hypertext enables one to navigate through textual materials by clicking on individual words and phrases, with hypermodels students navigate through a learning activity by manipulating a computer-based model. The activity typically presents a more or less open-ended challenge (e.g., "Breed these organisms as efficiently as possible, trying to get all the offspring to look like this.") and then leaves the students alone and monitors them (Which organisms do they choose to breed? How do they react to the outcome?) as they try to accomplish the goal.

At various points throughout an activity the computer may "pop up" and offer help, either as metacognitive prompts (e.g., "Why did you do that?" "Did it accomplish what you expected?" "What is your immediate goal?"), or in the form of hints ("What happens when you breed two recessives?") The trick in programming such interventions is not to overwhelm the student with "helpful" comments (the infamous Microsoft Word paperclip comes to mind!), but at the same time to avoid having students become discouraged or "turned off."

Thus, one of the central questions that the Modeling Center at the Concord Consortium is studying is how much structure to build into a hypermodel activity. The two extremes are clearly suboptimal: too much structure (think of the "drill and kill" examples from computer-aided instruction) and an activity becomes boring, too little ("just give them some open-ended software and leave them alone") and most students are likely to give up in despair. Moreover, students' responses to questions asked in situ constitute powerful probes of their thinking. They can be useful both for research purposes and in configuring an activity in real time to meet individual students' needs.

There is also the problem of transfer. If an activity is not explicitly tied to specific teaching goals, it runs the risk of becoming a meaningless videogame - the students may become quite expert at winning the game without developing the understanding of the domain that we seek - and that the standards call for. The eternal quest for the happy medium between linear instruction and radical constructivism is what we have dubbed "instructivist," to convey that it seeks a Hegelian synthesis of two contrasting educational philosophies. Once again, the research literature does not (yet!) provide a clear guide.

Communication and collaboration

The use of models as communication devices was also brought up during the REC panel discussion that was the impetus for this paper. Awareness of the importance of communication and the social context of learning has been growing for decades (Rogoff and Lave, 1999). The use of models as a means of communication, however, is relatively new (Black, 1972; Gilbert, 1980). A number of recent projects have demonstrated the extraordinary value of having students share their models, comment on them, and view each other's comments (cf. Gobert et al, 2002). Not only does this appear to have a beneficial effect on students' learning and retention, it also teaches social and team-building skills, and places students in situations that resemble the world of real scientists.

But without simulations that enable one to observe the behavior of a model, and visualizations that convey their effects, it is difficult to share a model with someone else. Static language and mathematical equations cannot capture the dynamic aspects of a time-evolving process. The implications of recent advances, not only in the technology for creating and viewing models but also in means of sharing them with others, offer a fertile field for research in social cognition and learning, and are likely to have significant implications for the use of simulations and visualizations in classrooms, and for teacher preparation programs.

Teacher professional development

The panel also discussed the question, "What is the proper role of the teacher in a classroom that uses simulations and visualizations? The answer, and its implications for teacher professional development, will surely be influenced by the progress of the instructivism debate referred to above. No computer activity, no matter how carefully constructed, is likely to "stand alone" - now, or in the foreseeable future. If for no other reason than the obvious fact that the curriculum cannot - and should not! - be delivered entirely on a computer, the teacher will always be a role model and mentor, and will play an essential role in coordinating student's activities, managing their interactions and discussions, keeping them on task, and motivating them to learn. Nonetheless, in a class that uses simulations there will be times when the students are working on the computers, either singly or in pairs or small groups. In this situation, the teacher's role is to support them in that activity. The situation is not unique in teachers' prior experience - other group activities such as laboratories are similar in some respects. However, it is harder to determine what students are doing on a computer than in a "wet lab," and consequently it is more difficult to detect those "teachable moments" when a small intervention can help a student make sense of something or discover a connection that might otherwise remain obscure. Improvements in the technology can help, but clearly teachers are going to have to become thoroughly familiar with each computer activity their students will undertake, the common problems they are likely to face, and strategies for helping what will in fact be a more idiosyncratic and student-centered learning process than they are accustomed to dealing with.

Recommendations for REC Research

The description above has identified several areas in which more research is needed. Accordingly, the following recommendations for REC flow directly from the discussions of the simulations and visualizations panel, and all of them, in fact, were raised at some point by the panel.

Importance of visualization research

The connection between the two terms used in describing the panel - simulation and visualization - and the third term, modeling, needs additional teasing out. For example, there has been extensive research work on information visualization, but most of it deals with visualization of data, not models3. Concept maps (Novak, 1980) constitute an interesting and fruitful attempt to visualize and represent relationships between concepts, but they are static, propositional representations that are purely verbal and inherently paper-based. The ability to create manipulable models in the form of simulations and visualizations is uniquely a product of the computer age. We must learn how to adopt this new tool to literally "use vision to think" - to externalize that critical aspect of the mental process that so augments human reasoning and memory. The educational and learning sciences can and should contribute to that work.

Instructivist research

The issue brought up above under the heading "Instructivism" harks back to the tension between teaching content and teaching process, and is probably as old as education itself. In science education, specifically, the tension often arises in the form of arguments over whether to teach science as a series of stories about the workings of the natural world, or to concentrate instead on teaching the scientific reasoning behind the discovery, validation, and refinement of those stories. In reality, of course, either extreme is counterproductive - science is clearly much more than a collection of facts, but it is pointless to teach students how to reason without giving them anything substantive to reason about.

In the context of educational software, the "product-process" controversy can be characterized as a tension between computer-aided instruction (CAI) and the "microworlds" most closely identified with Logo. More research is needed to fully explore the continuum between the two extremes, the area that we have termed "instructivism," and to bridge the gap between them. Many questions remain to be answered. Do different students, as experience and anecdotal data would suggest, require different "treatments" along the instructivist continuum? Do students pass through that continuum in stages, perhaps requiring less and less support as their understanding deepens and their inquiry skills improve? Can we build software that responds differently to different students, and to the same students at different points, altering the amount and type of scaffolding as circumstances indicate?

Technology-assisted assessment

One of the main differences between the use of simulations and visualizations to teach science and more traditional methods is in the use of, and reliance on, language4. Prior to the advent of the computer, science teaching perforce relied heavily on language to describe abstract ideas. This mode of learning works well with the subset of the population that possesses the requisite linguistic skills, but very poorly with those students who do not use language and other abstract representations effectively. More work needs to be done to explore the extent to which simulations can help such "language deficient" students5 achieve and demonstrate understanding commensurate with their ability. Can the computer help them by bridging the gap between the simulation and the linguistic or mathematical representations of knowledge?

Currently, assessment is almost entirely a language-intensive activity. Much more research is required to enable us to use simulations as assessment tools, but the initial experience is promising (Buckley, Gobert, and Christie, 2002). If a student cannot read or write very well, can't manipulate algebraic symbols with facility, but can solve a challenging problem using a manipulable computer model, doesn't that imply that the student knows something? And shouldn't we be able to assess such students by giving them a simulation-situated problem to solve? As computers become more and more ubiquitous in schools, the time is fast approaching when it will be cost effective to deliver the SAT and other high-stakes tests via computer, rather than shipping stacks of sealed (and ferociously guarded!) paper assessments around the country. We need to prepare for that time by learning how to make the radical improvements in assessment that truly interactive technology potentially affords.

Research on pedagogy

At the moment, most of our information on how to use simulations and visualizations in the classroom is based on anecdotal evidence. In particular, we face fundamental questions about how to evaluate students' work with simulations. In an age of increasing "accountability" and high-stakes testing, how can we reconcile an approach that encourages collaborative problem solving with the requirement to assign individual grades? How can the use of interactive simulations be adapted to individual learning styles and skill levels? What depth of content knowledge is required to enable teachers to feel comfortable in assigning open-ended exploratory activities? To answer questions such as these on a scientific basis will require much broader projects than the current norm. It is possible that improvements in technology may make such projects more affordable in the near future.

Support for large scale, scientific education research

With the internet becoming ubiquitous in the Nation's schools (Cuban, 2001), it is now feasible to introduce technology and acquire data over the World Wide Web. The WISE project (Linn and Slotta, 2000) provides an authoring and curriculum development environment that enables teachers to write and disseminate their own curriculum material, and to obtain data for evaluation of educational effectiveness. The Concord Consortium's Pedagogica™ (Horwitz and Christie, 1999; Horwitz and Tinker, 2001) is a suite of tools with similar capabilities. Other research groups are working on the same problem. With respect, specifically, to research into the use of simulations and visualizations, it will soon be possible for thousands of schools, and hundreds of thousands of students, to use the same simulations for the same purposes6. As the students work through the activities, they will generate a wealth of data that will be made available instantly to teachers and researchers. Such an infrastructure will make possible large-scale, quantitative or qualitative research with enough statistical power to enable us to make scientifically valid statements about the relative strengths and weaknesses of different pedagogical approaches.

To pursue the thought a step further: if one were trying to determine the differences between different types of scaffolding, one could develop multiple versions of the same activity and randomly assign different versions to different students within the same class. This would avoid most of the complications inherent with the use of comparison "control groups" that are never as close to the "treatment group" as one would ideally like. Just such a research protocol, using Pedagogica, is being employed by the Concord Consortium, on an IERI grant (Concord Consortium, 2001).

Need for economies of scale

At present, research using simulations and visualizations is still very expensive. Simulation technology, particularly if it involves complex models and sophisticated pedagogy, is expensive to produce, and the obstacles to deploying such technology in schools, and receiving data back from it reliably, are formidable indeed7. The problem is particularly daunting for younger researchers who cannot command the level of resources required to make a contribution to this promising field. The suggestion has therefore been made (Concord Consortium, 2002) that the NSF consider funding a national "Educational Accelerator," similar to the National Accelerator Laboratory (aka "FermiLab") in Illinois, that would take on many common technology-intensive tasks for "customers" in the education research and K-12 school communities. Such a national laboratory would:

  • create new technology for educational researchers and help them adapt it to their particular purposes;
  • recruit schools and teachers who wish to participate in research projects, link them to researchers, help them to install and maintain the technology, and offer professional development services to help them use it effectively;
  • handle the acquisition of data from the schools and transmit it to the researchers under appropriate guarantees of privacy and security;
  • support the analysis of school-based data by researchers and communicate the results of such analyses back to teachers, students, and parents, as desired;
  • offer internships and summer courses for Ph.D. students in education research;
  • provide technology to schools of education for pre-service and in-service training, as well as courses in instructional design and technology research; and
  • host national conferences and symposia on issues related to educational technology.

Natural scientists have learned take advantage of the vast economies of scale associated with technology-intensive research. It makes sense for the education research community to do the same. The formation of a National Educational Accelerator would turn the notion of "scalability" on its head - making it possible for many research groups to take advantage of a common technological infrastructure. Such an initiative could catalyze a revolution in science education, and bring the elusive national goal of universal science literacy within our grasp.

Acknowledgements

It is a pleasure to thank Dr. Barbara Buckley and Dr. Janice Gobert for their insightful comments on earlier versions of this paper, and their suggestions for its improvement.

Footnotes

1 This example is taken from a physical sciences curriculum under development at the Concord Consortium.

2 A more complicated variant of the "missing momentum" technique led the physicist Wolfgang Pauli to hypothesize the existence of an unseen, zero-charge, zero-mass particle (subsequently named the "neutrino" by Fermi) that is emitted in the process called beta decay. Recently, more delicate experiments have demonstrated that neutrinos actually have a very small mass.

3 The addition of animation to one's arsenal of visualization tools does not necessarily address this problem. An animated weather map does not convey any insight into why that thunderstorm developed where and when it did, or what is likely to happen to it next.

4 Our use of the word "language" is quite general here, and is intended here to include mathematical symbolism as well as text.

5 Native speakers of languages other than English form an important subset of these students.

6 Concord Consortium's Modeling Across the Curriculum will achieve this goal in the near future.

7 In order to avoid the problems inherent with students' use of the school computers, many schools have developed elaborate "firewalls" designed to inhibit indiscriminate downloading of software, and in many cases, the saving of files in general. This makes it difficult both to install software in such schools, and to find a way to save students' work.

References

Black, M. (1972) Models and Metaphors, Cornell University Press.

Buckley, B.C., Gobert, J., and Christie, M.T (2002) Model-based Teaching and Learning with Hypermodels: What do they learn? How do they learn? How do we know? Paper presented at the Annual Meeting of the American Education Research Association, New Orleans, LA.

Concord Consortium (2002) Center for Educational Research with Technology, proposal submitted to the Centers for Learning and Teaching Program, National Science Foundation.

Concord Consortium (2001) Modeling Across the Curriculum, proposal submitted to the Interagency Education Research Initiative, National Science Foundation.

Cuban, L., (2001) Oversold and Underused: Computers in Classrooms, Harvard University Press

Frogner, B. and Meijer, J. (1980) On-Line Power Plant Alarm and Disturbance Analysis Systems, EPRI Final Report NP-1379.

Gilbert, J.K. (1980) The Use of Models in Science Teaching, European Journal of Science Education, Vol. 2(1), 1 - 13.

Gobert, J. (2000). A typology of models for plate tectonics: Inferential power and barriers to understanding. International Journal of Science Education, 22(9), 937-977.

Gobert, J., Snyder, J., & Houghton, C. (2002, April 1-5). The influence of students' understanding of models on model-based reasoning. Paper presented at the Annual Meeting of the American Educational Research Association, New Orleans, LA.

Horwitz, P. (1995) Linking Models to Data: Hypermodels for Science Education, The High School Journal, Vol. 79, No. 2, pp. 148 - 156.

Horwitz, P. (1999) Designing Computer Models that Teach, in Computer Modeling and Simulation in Pre-College Science Education, pp. 179 - 196, Nancy Roberts and Wallace Feurzeig, eds., Springer Verlag.

Horwitz, P. and Christie M. T. (1999) Hypermodels: Embedding Curriculum and Assessment in Computer-Based Manipulatives, Journal of Education, Volume 181, Number 2, pp. 1 - 23 (with Mary Ann Christie)

Horwitz, P. and Tinker, R. (2001) Pedagogica to the rescue: A short history of hypermodels, @ Concord, the Concord Consortium Newsletter, Vol. 5(1), pp. 1-4.

Linn, M.C. & Slotta, J.D. (2000). WISE science. Educational Leadership. Alexandria: Association for Supervision and Curriculum Development.

Novak, J. D. (1990). Concept maps and Vee diagrams: Two metacognitive tools for science and mathematics education. Instructional Science, 19, 29-52.

Rogoff, B., and Lave, J., Editors (1999) Everyday Cognition: Its Development in Social Context, iUniverse.com Publishers.

White, B., & Frederiksen, J. (1990) Causal Model Progressions as a Foundation for Intelligent Learning Environments. Artificial Intelligence, 24, pp. 99-157.

Wilensky, U. (1999). GasLab--an Extensible Modeling Toolkit for Exploring Micro- and Macro- Views of Gases. In Roberts, N. , Feurzeig, W. & Hunter, B. (Eds.) Computer Modeling and Simulation in Science Education. Berlin: Springer Verlag.

Wilensky, U. & Resnick, M. (1999). Thinking in Levels: A Dynamic Systems Perspective to Making Sense of the World. Journal of Science Education and Technology. Vol. 8 No. 1. pp. 3 - 18.

   
    
 
Division of Research, Evaluation and Communication
National Science Foundation
4201 Wilson Boulevard • Arlington, Virginia • (703)292-8650