William S. Verplanck, Ph.D.
This brief manual is prepared to enable its user to apply the methods of OPERATION-ANALYSIS for the summary and evaluation of reports of systematic observations and of experimental research in both psychology and behavior science. Users can summarize their analyses by entering pertinent information on previously prepared forms, from which pertinent information can be readily abstracted.
The analysis has been developed to further several purposes:
- to permit the editorial evaluation of each experimental paper by providing for systematic report on each decision that the experimenter has made or, according to our present understanding of the variables affecting behavior, should have made in designing and carrying out the experiment.
When the analysis of a paper is complete, the summary should enable precise replication of the experiment. The analyst will learn with experience that many papers omit information on the procedures carried out, sometimes because the variable is not considered pertinent, sometimes for brevity, and sometimes for inscrutable reasons. Missing information may be critical for the results.
- eventually to permit the rapid retrieval of information on the effects of various variables on a given behavior or set of behaviors and to enable the developmemt of software that can use such information to produce data on all previously reported research in which a given operation was carried out.
- The long-range goal of the analysis is a more ambitious one: It is the recategorization of psychological knowledge on the basis of the operations carried out by the experimenter independent of his or her specific purpose in performing an experiment to confirm or fail to confirm some hypothesis developed from theoretical context.
Analysis may show that an effect presumed to be due to one variable is in fact attributable to another. More importantly, a single operation may prove to have been used in the “operational definition” of a variety of concepts.
- A final function of Operation-Analysis lies in planning an experiment. Use of the guide and the analysis sheet through the designing phase of an experiment enables the experimenter to confront and consider each of the kinds of decisions he or she will need to make.
A GUIDE TO AID IN THE OPERATION-ANALYSIS
This guide is designed to be used in filling out the “Operation-Analysis Sheets.” These instruments are designed to give authors and readers a complete and concise statement of the experimental procedures described as carried out.
Information on each should be stated completely enough to permit evaluation and to make replication possible. The analysis should permit evaluation of a paper from the following points of view:
- completeness of report
- adequacy of selection methods
- choice and range of specific events to manipulate in carrying out the experiment
- precision of methodology
- complexity of method, and appropriateness as a basis for generalization
- completeness/selection of data; method of analysis
The analysis sheet leaves its users to their own devices in evaluating (a) the relationship of the experiment and its results to the problem as it was stated by the experimenter, (b) the appropriateness of the experiment in providing a solution to a problem or as test of a theory, and (c) the adequacy of the presentation of previous findings and discussion.
The following are general rules to be followed throughout:
- Brevity, clearness, and completeness are basic requirements. These demand legibility.
- The purpose of the analyst is to be complete; omit no information needed to report this experiment. Fill every space, even if only with “not reported”.
- Complete sentences need not be used.
- Many variables can and must be considered under more than heading. In these cases, the statement is entered under the entry considered major and cross-referenced to the other entry. If neither is major, put full statement in firstentry. Then cross-refer back to the full entry from the second one.
- Where information is lacking, the missing information may fall into any of four categories:
- Does not apply
- Not relevant
- Not reported: Information is required, but is not found in the report.
- Report is unclear; author’s statements can be interpreted in more than one way.
The entries for the first three are “n.r.” — not reported. The analyst should leave to the reader the decision whether the information is needed or appropriate.
For the fourth entry: “not clear”.
Each sheet is designed for analysis of a single experiment. Papers reporting on multi-experiments will require as many sheets as there are experiments (cross referencing where applicable).
One size fits all:
Some of the categoric operation statements on which are required are not appropriate in particular fields of research; only a minority of research reports need descriptions of surgical interventions.
Note: The numbering system in this guide corresponds with the analysis sheet. The completed analysis should leave no section blank.
What is the full APA style citation? (Authors should choose a title that could serve as an abbreviated abstract of the paper). What are the key words under which this paper would be found indexed in the subject index of abstracts?
1.2 Traditional references
What are the traditional theoretical reference headings relating the problems this experiment investigates to similar attempts in the literature? How should it be indexed?
1.3 Who designed the experiment
Who carried out data analysis and reporting?
Who ran the subjects? (author? research assistant? graduate student? undergraduate? staff members?) Cases ranging from carelessness, incompetence, through assistants eager to get data that will please the boss and on to downright fraud have filled the literature with “unreproducible results.” Hence, responsibility must be identified…. can you trust your assistants?
1.4 Exploratory research:
2. SUBJECT SELECTION
2.1 Age and Cohort
Some of the most egregious errors in psychology have arisen from the failure to consider cohort, and on the identification of people by their cohort occurs more frequently in everyday divisions (e.g., “slacker,” “babyboom,” “yuppy.”)
What was the age of the subject? (report as accurately as possible; for groups, mean and variablitiy)
Year of Birth (for groups, means and variability)
Species-Genetic identification of nonhuman subjects, including breed as appropriate (e.g., Cebus capucinus); Social or other available categorization of humans subjects (e.g., College Sophomores taking Introductory Psychology)
How many subjects started in the experiment? How many completed it? Describe any procedures related to incomplete data. (In research related to medical status or surgical procedures report on recovery or death.)
2.4 Recruitment of Subjects
How, when, and where were the subjects obtained? (e.g., Patients on ward, clients, volunteers from class? Course requirement? Extra credit? Paid? Purchased from dealer? Bred in dept. colony?)
How many groups were there? How many subjects per group? Males and Females in each? What was the number of subjects per group on whom data are reported?
How many member of each sex per group? Treatment? Did these totals change during the experiment?
2.7 Behavioral Testing
What tests or behavioral criteria were used to select or to assign subjects to groups? What were the criterion scores or the reliability/validity data on such tests?
How were subjects assigned to groups? Random? Matched?
2.9 Experimental history:
How many subjects, received what treatment, have previously served as subjects in similar (or other) research?
3. OPERATIONS BEFORE THE EXPERIMENT PROPER BEGINS
3.1 Bio-cyclical Events
What were the chemical, anatomical, imposed or natural environmental (e.g., diurnal, oestral, seasonal) cycles under which the subjects existed before and during the experiment? At what point in such cycles were data collected?
[N.B. Season of the year]
For non-human subjects:
What was the dietary and feeding routine of the subjects before the experiment began?
In what milieu did the subjects live before the experiment began?
Humans: home; dormitory; hospital; detention centers, etc.
Other animals: single cages? group cages?
3.4 Subject History
What specific background did the individual subjects bring to the experiment that might affect results? (Include problems or other pertinent features. e.g. institutionalization)
3.5 Specific Experiment-Related History
What prior experience did the subjects have with the specific materials, apparatus, procedures, and conditions of the experiment? Include all experience given to the subjects by the experimenter prior to the commencement of the experiment proper such as warm-ups, “adaptation”, practice trials, shaping, etc.
How were the subjects enabled to carry out the specific procedures of the experiment? What said or done to make it likely that the subjects would perform according to the experiments plan or expectations? (shaping, encouragement, deprivation, outcome contingencies). When instructions were stated, clearly indicate their completeness, their accuracy, and their truth or falsity, if they are misleading, deliberately or otherwise.
4. MAINTENANCE UNDER EXPERIMENTAL CONDITIONS
When exactly was the experiment conducted? (dates, hours)
4.2 Bio-cyclical Events
What changes in cyclical relationships (3.1) were made or occurred once the experiment had begun?
4.3 Feeding and Maintenance
What changes were made in the dietary and feeding routine of the subjects at the beginning of or during the experiment? What changes were made in the living milieu of the subjects at the beginning of the experiment and continued while the experiment was carried out?
4.4 Experimental Space
What were all the physical characteristics of the microclimate and microsituation in which the experiment was carried out? (This relates to the immediate space where the experiment was carried out be it a hospital ward, an office, an experimental cubicle, or a Skinner box).
What “things” or “gear” were used to conduct the experiment? Who made them? Were they standard, modified, or home-brew? (give the complete details of all pieces of equipment used and how they were with one another).
5. PHYSICAL STATUS, INTERVENTION AND EXAMINATION
5.1 Physical Description
What were the pertinent physical characteristics of the subjects? (Information as to size, weight, eye color, etc.; especially any specific anatomical of physiological characteristics which distinguish the subjects not already recorded in 2.1 or 2.2).
5.2 Nutritional Considerations
What manipulation of nutritional variables occurred? (If any: e.g., dietary composition).
5.3 Experimental Surgery
What were the precise details of all surgical procedures performed? (If any: histological data).
5.4 Biochemical (Hormonal-Pharmacological) Interferences
What usage of biochemical substances occurred? Who was the supplier? What were the dosages, rate, and method of administration? What was the routine of administration? (blind, double-blind, “this will…”)
What were the precise details of all diagnostics, pathological or histological analysis, X-ray examinations, or other detailed examinations of the subjects?
6. EXPERIMENTER CONTRIBUTIONS
6.1 Pilot Experimentation
What were the details of previously unreported (or unsuccessful) attempts to obtain these experimental findings? What was done to find the appropriate method that would give these clear-cut results?
6.2 Non-Subject Involvement
Who was involved in the experiment besides the subjects? What did they do? (Who ran the subjects, did the statistics, interacted with the subjects and wrote their report?)
6.3 Experimental Design
How does the report describe the overall scheme, plan, or blue-print of the experiment? (Latin squares, split plot, case history). Diagram if possible, a la Campbell and Stanley, or Sidman using only key descriptors (details to be furnished in later sections).
7. IDENTIFICATION OF STIMULI, RESPONSES AND DEPENDENT EVENTS
Identify the presumed behavior-controlling objects or events empirically controlled or manipulated by fully describing their nature (physical, psychological, physiological) and specification? (Include manipulanda and variable events; e.g., programmed presentation of lights, lowering of a platform, opening of a door, distribution of forms, etc.).
What are the identities of all the events in the activity of the subjects presumed to result from the experimental manipulation? (Include all criteria used to identify, report, and measure manifestations of the same). Include all that is measure to be subsequently analyzed, or that could be analyzed, whether or not data on it is later reported in the paper.
7.3 Dependent Events
What are the consequences and means of delivery of the consequences as a result of a subjects activity on a trial or during the experiment? (payoffs, reinforcers, punishers, avoidings, all contingencies).
8.1 Pretreatment or Baseline Measures
What were the techniques and procedures used to obtain these (pretreatment or baseline) measure? (Identify what they were by groups and subgroups. These measures are the basis for within subject or within group analysis).
What are the orderly relationships (conjuncture and sequential) of stimuli, responses and dependent events as they were presented to the subjects (by individuals, by groups, group by group)? (This is the usual “procedure” on each trial or operant cycle. Parametric values will be reported on in Category 10).
9. MEASUREMENT AND ANALYSIS
9.1 Techniques used to Obtain Data (Probes)
What were the techniques (probes) used to enable the experimenter to obtain data that yielded measures. (questions, stimulus presentations, manipulanda, puzzles, observers, etc.)
What measures, and measured behaviors did the probes of subjects behavior produce? (counts of presses, counts of answers, counts of eyeblinks) What did the data measure?
9.3 Data Selection
Which available measures were made and analyzed? Which measures were transformed? How? Why? Which measures were taken but not analyzed? Why? Which available measures were not taken? Why?
9.4 Data Analysis Techniques
What were the names (and references) of all data analysis techniques used?
9.5 Tabular and Graphic Displays
Which measures or results of analysis were displayed? How were they displayed?
10. SPECIFIC VALUES OF ALL VARIABLES
10.1 Independent Variables
What are the independent variables (and their values) for each group and treatment?
10.2 Experimenter Determined Probabilities of Occurrence
What were the probabilities (percentages) of presentations on activities, trial by trial, cycle by cycle, session by session, for groups and treatments? (With what regularity did events controlled by the experimenter occur sequentially or in just a position?)
10.3 Sequencing Rules
What were the rules the experimenter used to determine which particular events, procedures, or placements occurred when they did? When were they used? (Randomizing, “staircase” ordering, haphazard placement, counter-balancing, etc.)
10.4 Procedure Parameters of Trials or Operant Cycles
What were the number and duration of: trial per block, blocks per session, intertrial intervals, interblock intervals, intersession intervals, cycles per session, and sessions?
11.1 Intra Subject or Group Treatment
What were the results when subjects or groups were compared the subject’s or group’s own baseline data or other reference base? (within subjects or groups results)
What were the outcomes of between group or between treatment level comparisons? (between subjects, groups or treatment result) Specify.
Did outcomes permit (allow) new statements (qualitative results as distinct from quantitative) or new terminological usage? what new statements or terminologies did the experimenter make or introduce?
11.4 Side Effects
What was observed to have occurred (even if not measured) that the experimenter did not expect? what was different (and noticed) from pre-experimental states of activity that did not fall within the scope of the experimenter’s plans? (These are the serendipitous results of an open-eyed observer on what was happening).
What was the hypothesis (or hypotheses) stated for the experiment? Against what theoretical background was it conceived and designed?
How did the outcomes support (deny) the hypothesis(es)? What new questions were raised?
Briefly encapsulate the qualitative findings, how they were obtained, and how they relate to theory.
13. EVALUATIVE COMMENT
13.1 Critique of the Experiment
Was the report complete? What was missing? what was reported or missing that could effect the experimenter’s explanations of the results? (Use this guide sheet as your instrument of critique).
13.2 Reviewer’s Choice
What are your constructive criticisms? What are your suggestions? What is your overall evaluation?
1 The earliest antecedent to this 1996 Manual is dated 1970. With accruing experience, the manual has been progressively revised to facilitate the precision of analyses.
2 This manual and the analysis sheets have been developed through several years of experience in analyzing reports of experiments published in a wide variety of journals and fields, and using various methods. The analysis sheet permits problems to be identified that often have eluded the writers, editors, or readers of a paper who sometimes fail to note the omission of significant information on details.