Technology Assessment in Software Applications -- Book by Eva L. Baker, Harold F. O'Neil Jr.; Lawrence Erlbaum Associates, 1994

Preface
The term technology conveys the idea of hardware, the silicon and plastic computer platforms. Our usage is more general, technology defined as a "systematic treatment" or "applied science." For a process to be labeled a technology, it must be replicable, or able to be used repeatedly with the same consequences ( Lumsdaine, 1963). We count as technology replicable procedures, with or without hardware. For example, empirically validated procedures used to organize teams of students for learning tasks are a form of technology, even though no hardware is required. Labeled soft technology, any computer software would fit this definition of technology.

This volume represents an expansion of ideas presented at the conference "Technology Assessment: Estimating the Future" held at the University of California, Los Angeles. The goal for the conference was to identify the major strategies and more promising practices for assessing technology. Perspectives were shared by leaders in computer science, cognitive and military psychology, and education. The speakers, representing government, business, and university sectors, helped to set the boundaries of present technology assessment. The conference proved to be very much work-in-progress in its focus and underscored the relatively little systematic thought given to the assessment (and evaluation) of technologies of all sorts. A subset of the conference is provided in this book. The chapters were updated, situated in the context of existing research, and extensively reviewed and edited. Two volumes resulted -- this one concerning technology assessments of software systems, and another dealing with assessments in education and training (see Baker & O'Neil, 1994).

Technology Assessment is designed for professionals in the assessment community, broadly defined to include psychology, computer science, and engineer- ing disciplines. It explores both the use of techniques to assess technology and the use of technology to facilitate the assessment process. In summary, the purpose of this book is to portray the state of the art in technology assessment and to provide conceptual options to help us understand the power of technology. Technological innovation will continue to develop its own standards of practice and effectiveness. To the extent that these are empirically based, designers, supporters, and consumers will be given better information for their decisions.

The impact of technology on all aspects of contemporary life is an unchallenged fact. We alternately suffer with and revel in the side effects of technology: from toxic waste, the social effects of too much television, and carpal tunnel syndrome, to the wonders of spreadsheets, tiny video cameras, and WD-40. Yet technology effects, both good and bad, seem always to take us by surprise. Why don't we systematically look at technology and better anticipate its consequences? In fact, many scholars do study technology, but most of our methods are weak and our attention at best happenstance. Limited in many ways by a geewhiz view of technology, we greet every new "advance" as unalloyed good. It is only later we feel we may have been tricked. Paradoxically, the task of technology development itself affects the assessment of its utility. Technology serves to codify and automate procedures and to extend our sphere of action. Unlike scientific findings, which are subject to interpretation, we act as if technology development is binary: either it works or it doesn't, it runs or it crashes, it delivers or fails. Thus for many designers and developers, technological innovation is an existence proof. Look at what we made! And the novelty either does or doesn't capture our imagination.

Assessing a technology involves making an estimate or judgment of its current state with regard to its effectiveness in a foreseeable range of applications. Technology assessment involves, therefore, at least two levels of prediction: first, predicting from the current status of technology to future utilities (e.g., U.S. Department of Commerce, 1990); second, generalizing from a specific case or cases to unknown applications to be invented in the future. These leaps in generalizability typically do not worry experimentalists, who engage in their predictions using conventions of science. Scientific safeguards include the cre-ation of sampling distributions of populations to which results will generalize, and the conception of what is studied as variables rather than as objects. In the case of assessment of individual technologies, these safeguards are not in place. Thus, it is entirely probable that an evaluation of a single technological enterprise, such as a computer program, might be misleading in the prediction of its utility to future contexts or to other cases or instances of application. On the one hand, the particular case of technology might be rapidly superseded by another approach, vitiating its potential contribution. For example, the creation of video cameras destroyed any predictions one might have wished to make about handheld motion picture cameras. On the other hand, the example of the technology application assessed may only bear limited relationship to its successive versions -- for instance, in text editing software.

Thus, the methods available for technology assessment appear to be relatively weak. One prominent method focuses on demonstration. A developer creates a prototype or demonstration model of an innovation, for instance, a networking protocol. This protocol is demonstrated with mock or real objects and data to a potential set of sponsors or users. Based on their reactions, the technology is assessed in terms of its utility in meeting stated requirements, its identification and satisfaction of new requirements, or its ease and efficiency compared to an existing option. A stronger version of this demonstration involves actual operational versions of the technology. For example, the Department of Defense demonstrated the utility of its new iteration of distributed interactive simulation ( Alluisi, 1991; Berry, 1991) by connecting up actual air, sea, and land resources with simulated resources for a nationally televised congressional hearing. Demonstrations are existence proofs. They translate concepts into a version of reality that may be especially convincing to novices.

A second class of technology assessment enterprises is based on explicit examples of the evaluation of technology. In such instances, an innovation, such as a new approach to teaching mathematics, or a user interface, is tested empirically. Beyond the ability to demonstrate its processes -- that is, "to run" -- the system or innovation is judged on the extent to which it meets its stated goals. Users are typically assessed, and their judgments of the ease and efficiency of operation may be used as data for the study ( Chorafas, 1990).

To get an entire picture of a class of technology, rather than making inferences from piecemeal evaluations using a one-innovation-at-a-time approach, formal meta-analytical approaches may be used, where studies or empirical evaluations of individual technologies are statistically combined to give the status of the population ( Kulik, 1994). Such studies permit one to understand the reliability or consistency of findings and to learn something about the generalizability of the results to settings or requirements differing in specific dimensions, such as age of user, degree of support, and so on.