Can independent variables be manipulated

It would be wrong to conclude that if the post-dependent variable manipulation check showed no effects, the treatment failed to create the predicted mental state. The mental state might have been there when it was supposed to be, but disappeared by the time it was measured. The person may infer their state not by remembering what they felt at the time of the treatment, but by remembering how they responded to the dependent variable measure. For example, if they punished the confederate, they may infer that they must have been angry.

Postponing IMCs until the end of the study is also problematic: as long studies wear on, participants often pay less attention and resort to satisficing Krosnick, ; Galesic and Bosnjak, , so they may fail IMCs at the end of the study even if they had been attentive earlier. This would make end-of-study IMCs overly-conservative measures of attention. Additionally, attentiveness ebbs and flows throughout the course of an experiment.

Berinsky et al. It is unrealistic to assume that attention remains constant throughout a study or that a single IMC is a reliable measure of attention. Participants may be inattentive during a crucial measure but pass a postponed IMC or vice versa. There are many ways to increase attentiveness besides using IMCs. The design of the survey should be considered; shorter surveys, engaging surveys, and surveys administered in person hold attention well Krosnick, ; Galesic and Bosnjak, ; Oppenheimer et al.

However, research shows that this concern may be misplaced. MTurkers have been incentivized to be equally or more attentive than college subject pool students Hauser and Schwarz, , especially MTurkers who meet certain worker restrictions that are easy to implement Peer et al. The only way to find out whether a manipulation check affected the outcome of a study is by an experiment: run the study with and without the manipulation check.

If the results are the same, we can conclude that the manipulation check did not interfere with the process we are studying. The logic is the same as the logic of checking for pretest sensitization: If we ask participants about their attitudes toward an outgroup right before we show them our anti-prejudice film, we cannot tell whether the effects are due to the film as we hypothesize , to the pretest, or to the combination of the pretest and the film.

Campbell and Stanley recommended the Solomon four-group design to deal with this problem. If the experiment involves two conditions — pretest-treatment-posttest and pretest-no treatment-posttest , then two more conditions must be added to discover or rule out the effects of the pretest — a no pretest-treatment-posttest condition and a no pretest-no treatment-posttest condition.

In the case of manipulation checks, the four conditions would be treatment-manipulation check-dependent variable measure, no treatment-manipulation check-dependent variable measure, treatment-no manipulation check-dependent variable measure , and no treatment-no manipulation check-dependent variable measure , or systematically varying the placement of the manipulation check to come before or after the measurement of the dependent variable. However, this design is extremely rare see Table 1.

Another method is to find a manipulation check that is not an event that the participant can notice. Webb et al. Examples include some observational measures, some behavioral measures, some physiological measures, and even some analyses of verbal measures. Observational measures, common in animal behavior and developmental psychology, involve an observer recording the behavior of an individual or a group, either directly, or usually less obtrusively, on a videotape.

Do the participants in the anger condition frown more? Or do observers, blind to condition, rate these people as angrier, more confident, or more certain than people in the other conditions?

If the video camera in the registered replication of the Strack pen-in-mouth study had been concealed from the participants, rather than explicitly brought to their attention, it would have likely been unobtrusive. Of course this sort of manipulation check can typically only be used when the participant is doing something other than filling out forms or sitting at a computer.

Sometimes aspects of a behavior can be measured without the need for human observers. Reaction time, used to measure speed of mental processing, is the most common example. Although typically used as a dependent variable measure, there is no reason that reaction time could not be used as a manipulation check, for example if the researcher wanted to vary whether problems were easy or difficult, or stimuli were clear or ambiguous.

Variability of responses can also be used by researchers as a check on how closely the participants are paying attention. Other aspects of behavior besides speed and variability are potentially measurable. For example, researchers have examined chosen distance from a confederate or speech errors as measures of discomfort e. Physiological measures such as heart rate, blood pressure, GSR, pupil dilation, or respiration rate are often used as measures of stress or arousal; facial expressions as measures of particular emotions; and measures of brain activity are increasingly capable of measuring specific mental processes Cacioppo et al.

However, not all behavioral are unobtrusive. We generally assume that physiological and brain processes cannot be consciously controlled, but behavioral responses might be. Or if she has been told that the person she is about to talk to is warm and similar to herself e. Even verbal measures can sometimes be used unobtrusively. Although these measures are verbal, they are likely to be far less obtrusive than the usual rating scales used as manipulation checks.

The participant generally has no idea what the experimenter will be looking for in coding the essay, or even whether it will be coded at all. Continuous unconscious online measures that begin before the independent variables are introduced and continue until the end of the study may be safe.

The experimenter looks for changes that occur from before to after the independent variable manipulation in brain or autonomic nervous system activity, in the rate of speech hesitations, in the number of times the participant looks away from the stimulus, etc. The experimenter hopes that the scanner or the videotape camera or any other measuring device will become a constant background factor of the experiment, so that the participant soon stops paying attention to it and responds only to the experimental events.

The experimenter also hopes that there are no complicated interactions between the presence of the manipulation check device and the key dependent variables.

A common objection to the use of behavioral measures, either as dependent variables or as manipulation checks, is that they have multiple meanings. Many behaviors do not have one-to-one correspondence with underlying states, and it is important for the researcher to consider other possible meanings of the behavior and design the experimental context so that the intended meaning is the one that is plausible in that context see Ellsworth and Gonzalez, , for a discussion of this issue.

Researchers have to make assumptions about the meaning of the behavior in using these measures. But the multiple meanings of verbal measures, and their sensitivity to small contextual factors have been known for decades: there is no reason to believe that a rating scale provides a pure and unambiguous measure of the mental states corresponding to the words on the scale, and there are dozens of well-documented reasons to believe that it does not Schwarz, ; Kagan, Finally, one can conduct the manipulation check before actually running the experiment, in a pilot study whose whole purpose is to find out whether the treatment successfully produces the intended psychological state.

The researcher creates or selects events or films or vignettes or some other stimuli designed to create different states — contempt and compassion, joy and relief, high credibility and low credibility and so on, presents them to the participants, and then asks what they meant to the participants i. Sometimes there are no significant differences — both treatments produce similar states, or there is so much variability in responses to one or both treatments that the noise drowns out the signal, or a treatment means something quite different from what the researcher thought it would.

This is disappointing and frustrating and means that we have to try again, guided by our mistakes. It takes time, but in the long run it is better to find out that your treatments are ineffective and to come up with better ones before running the whole experiment.

A manipulation check within the context of the real experiment may be a less trustworthy method of discovering the mistake, for all the reasons we have described. If the researcher has shown through a pilot study that the independent variable produces the expected belief, emotion, or other state in people like the participants to be run in the actual experiment, then there is no need to clutter up the actual experiment with an intrusive measure that may disrupt the flow of events and have an independent or interactive effect on the dependent variable measures that the researcher cares about.

This is the same logic that underlies Spencer et al. This approach essentially establishes the validity of the manipulation i. Of course, the pilot test should be run on people like the actual participants pretty soon before the actual study is run, in order to be confident that the participants in the real experiment will respond in the same way.

Even words change in their frequency and their meanings over time Ramscar, The purpose of pilot testing is to discover whether a treatment is effective for a particular group of participants at a particular time — people like the people who will be in the actual experiment. The strategy of testing the meaning of a manipulation in a pilot study is also useful for researchers attempting to conduct a direct replication, as it allows them to check on the effectiveness of the manipulation for the new sample while leaving the replication itself intact.

Validating the meaning of manipulations is important, and we are not advocating that manipulation checks be abandoned. We are arguing against the mindless inclusion of obtrusive measures — manipulation checks, measures of mediating variables, or any other measures between the manipulation and the dependent variable measure — that may influence the thoughts and behaviors of participants. The addition of a manipulation check in the service of testing validity can introduce new problems that threaten validity.

By adding additional measures the researcher may change the internal psychological processes. There is more than one way that a manipulation can be validated, and researchers should give the same careful consideration to their choice of a manipulation check as they do to their choice of dependent variable measures. Authors should justify including a manipulation check with an experiment if they chose to do so, explaining why it is necessary and why it is unlikely to affect their conclusions.

Often the best choice may be to forego including a manipulation check in the actual study by establishing the effectiveness of the manipulation through other means such as in pilot work. Editors and reviewers should evaluate whether a particular manipulation check improves or impairs the quality of any given study rather than assuming that using a manipulation check automatically improves it. The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Angrist, J. Google Scholar. Aronson, E. Baron, R. The moderator—mediator variable distinction in social psychological research: conceptual, strategic, and statistical considerations. Baumeister, R. Psychology as the science of self-reports and finger movements: whatever happened to actual behavior? Beek, T. Instructions - Experimenters.

Bem, D. Testing the self-perception explanation of dissonance phenomena: on the salience of premanipulation attitudes. Berinsky, A. Separating the shirkers from the workers? Making sure respondents pay attention on self-administered surveys. Cacioppo, J. Handbook of Psychophysiology. Cambridge: Cambridge University Press. Campbell, D. Experimental and Quasi-Experimental Designs for Research. Carlsmith, J. Methods of Research in Social Psychology. Reading, MA: Addison-Wesley. Cohen, J. Hove: Psychological Press.

Creswell, J. Neural correlates of dispositional mindfulness during affect labeling. Culpepper, S. Methods 16, — Curran, P. Methods for the detection of carelessly invalid responses in survey data. Dehghani, M. TACIT: an open-source text analysis, crawling, and interpretation tool. Methods 49, — Ellsworth, P. Gonzalez, C. Tavris, and J. Hogg and J. Fayant, M. On the limitations of manipulation checks: an obstacle toward cumulative science. Festinger, L. Festinger and D. Galesic, M.

Effects of questionnaire length on participation and indicators of response quality in a web survey. Public Opin. Hauser, D. SAGE Open 5, 1—6. CrossRef Full Text. Attentive Turkers: mTurk participants perform better on online attention checks than do subject pool participants.

For a new researcher, it is easy to confuse these terms by believing there are three independent variables in this situation: one, two, or five students involved in the discussion, but there is actually only one independent variable number of witnesses with three different levels or conditions one, two or five students.

The second fundamental feature of an experiment is that the researcher controls, or minimizes the variability in, variables other than the independent and dependent variable. These other variables are called extraneous variables.

They also randomly assigned their participants to conditions so that the three groups would be similar to each other to begin with. Notice that although the words manipulation and control have similar meanings in everyday language, researchers make a clear distinction between them. They manipulate the independent variable by systematically changing its levels and control other variables by holding them constant. Again, to manipulate an independent variable means to change its level systematically so that different groups of participants are exposed to different levels of that variable, or the same group of participants is exposed to different levels at different times.

As discussed earlier in this chapter, the different levels of the independent variable are referred to as conditions , and researchers often give the conditions short descriptive names to make it easy to talk and write about them.

Notice that the manipulation of an independent variable must involve the active intervention of the researcher. Comparing groups of people who differ on the independent variable before the study begins is not the same as manipulating that variable. For example, a researcher who compares the health of people who already keep a journal with the health of people who do not keep a journal has not manipulated this variable and therefore has not conducted an experiment.

This distinction is important because groups that already differ in one way at the beginning of a study are likely to differ in other ways too.

For example, people who choose to keep journals might also be more conscientious, more introverted, or less stressed than people who do not. Therefore, any observed difference between the two groups in terms of their health might have been caused by whether or not they keep a journal, or it might have been caused by any of the other differences between people who do and do not keep journals.

Thus the active manipulation of the independent variable is crucial for eliminating potential alternative explanations for the results. Of course, there are many situations in which the independent variable cannot be manipulated for practical or ethical reasons and therefore an experiment is not possible. For example, whether or not people have a significant early illness experience cannot be manipulated, making it impossible to conduct an experiment on the effect of early illness experiences on the development of hypochondriasis.

This caveat does not mean it is impossible to study the relationship between early illness experiences and hypochondriasis—only that it must be done using nonexperimental approaches. We will discuss this type of methodology in detail later in the book. Independent variables can be manipulated to create two conditions and experiments involving a single independent variable with two conditions is often referred to as a single factor two-level design.

However, sometimes greater insights can be gained by adding more conditions to an experiment. When an experiment has one independent variable that is manipulated to produce more than two conditions it is referred to as a single factor multi level design.

As we have seen previously in the chapter, an extraneous variable is anything that varies in the context of a study other than the independent and dependent variables.

In an experiment on the effect of expressive writing on health, for example, extraneous variables would include participant variables individual differences such as their writing ability, their diet, and their gender. They would also include situational or task variables such as the time of day when participants write, whether they write by hand or on a computer, and the weather. Extraneous variables pose a problem because many of them are likely to have some effect on the dependent variable.

This influencing factor can make it difficult to separate the effect of the independent variable from the effects of the extraneous variables, which is why it is important to control extraneous variables by holding them constant.

Extraneous variables make it difficult to detect the effect of the independent variable in two ways. They found that the women in their study, but not the men, performed worse on the math test when they were wearing swimsuits. They argued, furthermore, that this process of self-objectification and its effect on attention is likely to operate in a variety of women and situations—even if none of them ever finds herself taking a math test in her swimsuit. This conversion from research question to experiment design is called operationalization see Chapter 2 for more information about the operational definition.

Consider if there were only two conditions: one student involved in the discussion or two. Even though we may see a decrease in helping by adding another person, it may not be a clear demonstration of diffusion of responsibility, just merely the presence of others. The construct validity would be lower.

However, had there been five conditions, perhaps we would see the decrease continue with more people in the discussion or perhaps it would plateau after a certain number of people. In that situation, we may not necessarily be learning more about diffusion of responsibility or it may become a different phenomenon. By adding more conditions, the construct validity may not get higher. When designing your own experiment, consider how well the research question is operationalized your study. A common critique of experiments is that a study did not have enough participants.

The main reason for this criticism is that it is difficult to generalize about a population from a small sample.

At the outset, it seems as though this critique is about external validity but there are studies where small sample sizes are not a problem Chapter 10 will discuss how small samples, even of only 1 person, are still very illuminating for psychology research. Therefore, small sample sizes are actually a critique of statistical validity. The statistical validity speaks to whether the statistics conducted in the study support the conclusions that are made.

Proper statistical analysis should be conducted on the data to determine whether the difference or relationship that was predicted was found. The number of conditions and the number of total participants will determine the overall size of the effect. With this information, a power analysis can be conducted to ascertain whether you are likely to find a real difference. When designing a study, it is best to think about the power analysis so that the appropriate number of participants can be recruited and tested more on effect sizes in Chapter To design a statistically valid experiment, thinking about the statistical tests at the beginning of the design will help ensure the results can be believed.

These four big validities—internal, external, construct, and statistical—are useful to keep in mind when both reading about other experiments and designing your own.

However, researchers must prioritize and often it is not possible to have high validity in all four areas. Morling points out that most psychology studies have high internal and construct validity but sometimes sacrifice external validity.

Again, to manipulate an independent variable means to change its level systematically so that different groups of participants are exposed to different levels of that variable, or the same group of participants is exposed to different levels at different times.

As discussed earlier in this chapter, the different levels of the independent variable are referred to as conditions , and researchers often give the conditions short descriptive names to make it easy to talk and write about them. Notice that the manipulation of an independent variable must involve the active intervention of the researcher.

Comparing groups of people who differ on the independent variable before the study begins is not the same as manipulating that variable. For example, a researcher who compares the health of people who already keep a journal with the health of people who do not keep a journal has not manipulated this variable and therefore not conducted an experiment. This distinction is important because groups that already differ in one way at the beginning of a study are likely to differ in other ways too.

For example, people who choose to keep journals might also be more conscientious, more introverted, or less stressed than people who do not. Therefore, any observed difference between the two groups in terms of their health might have been caused by whether or not they keep a journal, or it might have been caused by any of the other differences between people who do and do not keep journals.

Thus the active manipulation of the independent variable is crucial for eliminating the third-variable problem. Of course, there are many situations in which the independent variable cannot be manipulated for practical or ethical reasons and therefore an experiment is not possible. For example, whether or not people have a significant early illness experience cannot be manipulated, making it impossible to conduct an experiment on the effect of early illness experiences on the development of hypochondriasis.

This caveat does not mean it is impossible to study the relationship between early illness experiences and hypochondriasis—only that it must be done using nonexperimental approaches. We will discuss this type of methodology in detail later in the book.

In many experiments, the independent variable is a construct that can only be manipulated indirectly. In such situations, researchers often include a manipulation check in their procedure.

A manipulation check is a separate measure of the construct the researcher is trying to manipulate. As we have seen previously in the chapter, an extraneous variable is anything that varies in the context of a study other than the independent and dependent variables. In an experiment on the effect of expressive writing on health, for example, extraneous variables would include participant variables individual differences such as their writing ability, their diet, and their shoe size.

They would also include situational or task variables such as the time of day when participants write, whether they write by hand or on a computer, and the weather. Extraneous variables pose a problem because many of them are likely to have some effect on the dependent variable. This influencing factor can make it difficult to separate the effect of the independent variable from the effects of the extraneous variables, which is why it is important to control extraneous variables by holding them constant.

Extraneous variables make it difficult to detect the effect of the independent variable in two ways. Imagine a simple experiment on the effect of mood happy vs. Participants are put into a negative or positive mood by showing them a happy or sad video clip and then asked to recall as many happy childhood events as they can.

Table 6. Every participant in the happy mood condition recalled exactly four happy childhood events, and every participant in the sad mood condition recalled exactly three. The effect of mood here is quite obvious. In reality, however, the data would probably look more like those Table 6. Even in the happy mood condition, some participants would recall fewer happy memories because they have fewer to draw on, use less effective recall strategies, or are less motivated.