Description
Research Hypothesis
This week’s topic is the development of your research hypotheses. The word hypothesis comes from the Greek word meaning ‘foundation.’ Indeed, the hypotheses that you formulate for your research will often determine its success and impact. Your hypotheses are the means of achieving the purpose and goal of your work. Getting your hypotheses correct is the keystone that holds up the rest of your work.
In quantitative research, your evidence is contained within the experiments and data collection that you perform in order to test your hypotheses and answer your research questions. The analysis applied to your data will require that it be reduced to numeric values. Even if you are using survey data or categorical data, it will have to be coded into a form that must, at the very least, be able to be counted, sorted, and number crunched.
Quantitative research, since it is about the data, involves statistics. You do not have to be a statistician, but you do need to understand what kind of data and how to collect it to be able to test your hypotheses. Testing a hypothesis entails computing statistics from your data, and then applying the appropriate statistical test to the statistics.
Having valid test results require that you know the assumptions on which the statistical test you plan to use is based and whether or not the population from which your data was sampled falls within the bounds of validity for the test. If you are not confident in your statistical skills, it is suggested that you plan for some statistical consultation as you plan your research.
Usually, there is a comparison or determination that you want to make that will involve comparing the statistics of two groups of data— the control and experimental groups. As an example, suppose that you have what you think is a modification to a commonly used algorithm that produces a significant increase in the speed, memory utilization, and accuracy of the problem that the algorithm solves. This will lead to the postulation of three hypotheses.
Hypotheses come in pairs, the members of the pairs are what is termed the null hypothesis, denoted H0, and the alternative hypothesis, denoted HA. These notations may vary from source to source, but what they are called is fairly standard. The results of a statistical test provide evidence for rejecting the null hypothesis in favor of the alternative hypothesis. When the test results do not provide enough evidence, then the null hypothesis is accepted or fails to be rejected.
Using the algorithm example, one of the hypotheses can be stated:
H0: The mean execution time of the modified algorithm is not less than the mean execution time of the original algorithm.
HA: The mean execution time of the modified algorithm is less than the mean execution time of the original algorithm.
The other two hypotheses can be stated by replacing ‘execution time’ with ‘memory utilization’ and ‘accuracy.’
Many statistical tests come down to calculating a statistic and then determining how likely the observed value is. If it has a small probability of being observed, then this is statistical evidence for rejecting the null hypothesis. If not, then the null hypothesis fails to be rejected. Note that it is the experimenter who determines what is significant and that rejecting a null hypothesis is never proof of the alternative hypothesis being true. It is simply that the statistical evidence supports the alternative hypothesis.
As you can see, a hypothesis must be based on something that is measurable, that is, quantitative, a statistic. The crux of quantitative research is knowing what data makes the valid testing of your hypotheses possible and can you get the data in an ethical and affordable way. Data that is obtained unethically or that is unethical to collect by its nature will most likely not be publishable. Or, it may be that the data can be obtained ethically, but it may come with a cost that is not acceptable.
Your problem statement, purpose statement, hypotheses, and research questions all form a single fabric of work. Each individual component must be consistent with and supportive of the others. You need to have data that has the possibility of answering the questions posed to it by the hypotheses. Carefully consider your work as a whole and make sure you understand what you are proposing to do.
Assignment: Develop a Research Hypothesis
Instructions
For this assignment, you must develop a workable research hypothesis based on an ideal dataset that you specify. You will need to consider what statistical test to use to test your hypothesis, given your ideal data.
Your hypothesis should incorporate the following:
A brief introduction and then state your null and alternative hypotheses.
A clear description of the ideal dataset that you will use for testing your hypothesis. Describe each element of the dataset along with its unit of measure. For example, age in months, distance in kilometers, dosage in milligrams, etc. Include how you will distinguish between two samples in your dataset?
A discussion of the feasibility of collecting your ideal dataset. How will you collect it? Can it be collected ethically? Is it affordable?
A statement of the statistical test you will use to test your hypothesis with a justification for its selection. Include what level of significance you will use and why.
Add your hypothesis to the end of the paper you submitted in the previous assignment. Be sure to check for alignment between all of the research components.
Length: 2-3 page paper
References: Include a minimum of 2 scholarly references.
The completed assignment should address all of the assignment requirements, exhibit evidence of concept knowledge, and demonstrate thoughtful consideration of the content presented in the course. The writing should integrate scholarly resources, reflect academic expectations and current APA standards.
Unformatted Attachment Preview
Sage Research Methods
Doing Quantitative Research in Education with SPSS
For the most optimal reading experience we recommend using our website.
A free-to-view version of this content is available by clicking on this link, which
includes an easy-to-navigate-and-search-entry, and may also include videos,
embedded datasets, downloadable datasets, interactive questions, audio
content, and downloadable tables and resources.
Author: Daniel Muijs
Pub. Date: 2013
Product: Sage Research Methods
DOI: https://doi.org/10.4135/9781446287989
Methods: Secondary data analysis, Quantitative data collection, Dependent variables
Keywords: teaching, pupils, students, achievement, educational settings, working memory
Disciplines: Education
Access Date: January 8, 2024
Publishing Company: SAGE Publications Ltd
City: London
Online ISBN: 9781446287989
© 2013 SAGE Publications Ltd All Rights Reserved.
Sage
Sage Research Methods
© Daniel Muijs 2011
Experimental and Quasi-Experimental Research
2.1 Types of Quantitative Research
Once we have taken the decision to do a quantitative study, we have to design it. There are two main types
of quantitative research designs, experimental designs and non-experimental designs. Experimental designs
are sometimes known as ‘the scientific method’ due to their popularity in the scientific research where they
originated. Non-experimental research is sometimes (wrongly, as we will see in the next chapter) equated
with survey research, and is very common in the social sciences.
When hearing the term ‘experimental designs’, most of us think back to school experiments in science. Experimental research in the social sciences follows the same basic pattern as those (natural) science experiments.
The basis of the experimental method is the experiment, which can be defined as a test under controlled conditions that is made to demonstrate a known truth, or examine the validity of a hypothesis. The key element
of this definition is control, and that is where experimental research differs from non-experimental quantitative research. When doing an experiment, we want to control the environment as much as possible, and only
concentrate on those variables that we want to study. This is why experiments traditionally take place in laboratories, environments where all extraneous influences can be shut out. In non-experimental research, we will
not be able to control out extraneous influences. Control is also increased by the fact that in an experiment
the researcher manipulates the predictor variable, while in non-experimental research, we have to use the
variable ‘as it appears’ in practice.
Example 2.1: Violent Attitudes and Deferred Academic Aspirations: Deleterious Effects of Exposure to Rap Music
In this study, a team from the University of North Carolina (Johnson, Jackson and Gatto, 1995) sought to look
at the effects of watching violent rap music videos (experimental group), compared to non-violent rap music
videos (control group 1) or no music videos (control group 2) on adolescents’ attitudes to violence and deviant
behaviour. Forty-six 11–16-year-old boys from a boys’ club in Wilmington, North Carolina, were randomly as-
Page 2 of 21
Doing Quantitative Research in Education with SPSS
Sage
Sage Research Methods
© Daniel Muijs 2011
signed to one of the three conditions. In the violent condition, subjects were shown eight videos containing violent images. Those in the non-violent condition were shown eight non-violent videos. Following the viewing,
subjects were asked to read a text passage in which a boy hit another boy who kissed the first boy’s girlfriend,
and respond to a number of questions on the acceptability of this behaviour. After that, they read another
text passage, featuring a discussion between two characters. One had acquired a ‘nice car’ and ‘nice clothes’
through dodgy activities, while the other was completing college. Subjects were then asked to respond to a
number of questions probing their views of these alternative career choices. The control (non-video) group
also participated in this activity. Subjects were told that the videos were part of a memory test and had been
randomly chosen. Results showed that the subjects exposed to the videos were significantly more likely to
approve of violent behaviour and of a deviant career path than those who viewed the non-violent video. The
controls were least likely to approve of violence or a deviant career path.
This experiment suggests a deleterious effect of watching violent videos. However, a number of caveats need
to be taken into account. Firstly, as mentioned above, this is clearly a somewhat contrived situation. It is not
clear from this experiment how strong this effect is, and, therefore, whether it is practically significant within a
real-life context in which many other factors may affect attitudes to violence. The sample is from a very specific group (black boys from a boys’ club), and the extent to which these findings generalise to other populations
needs to be examined. The authors also did not provide any information on prior factors that could differ between the groups (e.g. age), notwithstanding random assignment to groups. This study therefore would need
replication in further experiments before we could say anything definitive, although the findings are clearly of
great interest.
2.2 How to Design an Experimental Study
There is a number of steps to go through in doing experimental research.
2.2.1 Define Your Research Objectives
Any research design starts with formulating research objectives. This step needs to be taken before you decide whether or not to do experimental research, as the research objectives will determine what kind of rePage 3 of 21
Doing Quantitative Research in Education with SPSS
Sage
Sage Research Methods
© Daniel Muijs 2011
search to do. Your research objectives describe what you want to study and how. You need to spell out clearly
what the aims of your research are. Research objectives need to be realistic. It is important to realise that
you can’t do everything. We have to limit ourselves to what is actually researchable. For example, let’s say
we want to look at the effects of different test conditions on performance. When we think this through, there is
an almost unlimited number of conditions that could vary slightly and affect test performance, such as lighting
levels, how many adults are present, seating arrangements, temperature and so on. To look at all of these
in one study would be impractical, and all but impossible. So we will need to set ourselves a more limited
goal, by thinking about which aspects might really make a difference and choosing just one (or a small number), such as seating arrangements. Our research objective would then be to look at whether or not seating
arrangements affect exam performance.
We also need to be clear on what our population is. The population is the group of people we want to generalise to. For example, if we were to do this experiment, we would use, say, 40 students in two different seating
arrangements, and see what effects we can find. Usually, we don’t just want to draw conclusions that are only
applicable to that group of 40 students. What we want to do is say something about seating arrangements
among students more generally. Many statistical methods that we will discuss in the following chapters have
been designed to allow us to do just that. But before we can do this, we must be clear about which population
we actually want to generalise to. All students of 18 and over? First years only? This is important, because
it will affect who we get to take part in our experiment. If I did a study using only secondary school kids, I
couldn’t then generalise to primary age kids.
2.2.2 Formulate Hypotheses
The research objectives you have developed now need to be refined into the form of a number of specific
research hypotheses you want to test. A research hypothesis can be defined as ‘a tentative explanation that
accounts for a set of facts and can be tested by further investigation’, as we mentioned earlier. In experimental research, we traditionally look at two distinct types of hypotheses: the null hypothesis and the alternative
hypothesis. The alternative hypothesis is the one we want to be true; the null hypothesis is the opposite. I
might, for example, want to know whether adding moving pictures to a presentation will improve pupils’ memory of the key content of the presentation. I would have two hypotheses:
Page 4 of 21
Doing Quantitative Research in Education with SPSS
Sage
Sage Research Methods
© Daniel Muijs 2011
• Null hypothesis (H0): adding moving pictures will not improve pupils’ retention of content.
• Alternative hypothesis (H1): adding moving pictures will improve pupils’ retention of the content.
This example presents the most simple case, where there is only one hypothesis to be tested. In many studies, there will be several hypotheses, and one can also hypothesise mediating factors that influence the relationship between the variables. An additional hypothesis that includes as a mediating factor whether or not
moving pictures are aligned to content could be:
• H1: adding moving pictures will improve pupils’ retention of content if the moving pictures are closely
aligned to the content.
• H0: adding moving pictures will not improve pupils’ retention of content if the moving pictures are
closely aligned to the content.
While the terminology refers to a ‘null hypothesis’, this does not necessarily mean that the null hypothesis
always has to specify that there is not going to be any effect, while the alternative hypothesis specifies that
there will be an effect. The null hypothesis can itself predict a specific value, for example:
• H1: the difference between boys and girls on a word retention test will be more than 20%.
• H0: the difference between boys and girls on a language test will be less than 20%.
or,
• H1: the mean score on a self-esteem inventory will be between 20 and 30.
• H0: the mean score on a self-esteem inventory will be between 10 and 20.
In practice, most researchers test a null hypothesis of no difference because standard statistical tests are
usually designed to test just that hypothesis. However, it is important to remember that other types of null
hypotheses are possible, as a value or difference of zero might not be realistic for the research question you
are looking at.
Page 5 of 21
Doing Quantitative Research in Education with SPSS
Sage
Sage Research Methods
© Daniel Muijs 2011
Example 2.2: How Should Verbal Information Be Presented to Students to Enhance Learning from Animations: Auditorily as Speech, or Visually as On-Screen
Text?
This question was studied by Mayer and Moreno (1998), who conducted an experiment in which students
were asked to view an animation showing the process of lightning accompanied by either concurrent narration
or on-screen text. The theory they wanted to test was that visual and auditory learning are processed in two
different parts of the working memory: the visual working memory and the auditory working memory. That
would mean that if narration is given alongside the animation, students will represent the narration and animation in two different parts of the working memory, while if on-screen text is presented with animation, students
will try to represent both the animation and the text in the same part of memory (the visual auditory memory),
which may then become overloaded. Better performance was therefore hypothesised for the text group.
The experiment was conducted by randomly assigning students to the two groups, one viewing the narration
with on-screen text, and the other with narration. Following the presentation, students were given a retention,
matching and transfer test. It was found that students in the animation-narration group did significantly better
than those in the animation-text group on all three tests, supporting the experimenters’ hypothesis.
2.2.3 Set Up Your Research Design
Once one or more hypotheses have been set up, you need to decide how to test these hypotheses. If an experimental methodology is chosen (the advantages and disadvantages will be discussed in the next section
of this chapter), you will then have to decide which experimental design to use.
The traditional experimental design, known as the pre-test post-test control group design, works as follows.
Participants (often known as ‘subjects’ in experimental research) are placed into two groups, the experimental
and the control group. The experimental group will receive the ‘treatment’ (for example, watching a violent
music video as in example 2.1); the control group will not. Both groups will receive a pre-test on whatever
instrument is used to assess the effect of the experiment (e.g. a test) before the treatment is given, and a
Page 6 of 21
Doing Quantitative Research in Education with SPSS
Sage
Sage Research Methods
© Daniel Muijs 2011
post-test, usually on the same instrument, after the treatment has been given. The sequence therefore is as
shown in Figure 2.1.
Figure 2.1 The classic experimental design.
Following the post-test, statistical analyses are carried out to see whether the treatment has had an effect
(which we will look at later in this book).
There is a number of variations on this basic design. As we see in Example 2.2, it is often desirable to have
more than one treatment group. There can, for example, be variations in the treatment that we might want to
study. In Example 2.2, we have two treatment groups and one control group. More control groups and treatment groups are also possible. The pre-test post-test design is also not always followed, as we can see in
Example 2.1 where no pre-test is used. Usually, it is better to use both a pre-test and a post-test, though,
as without pre-testing we can never be sure that any difference we find on the post-test is the result of the
treatment, and not the result of differences that already existed between the two groups before the treatment.
Another decision you will have to take is whether or not to give the control group a placebo. This practice
comes from medical research, where it is well known that some patients show recovery as a result of a belief
in the treatment rather than as a result of the treatment itself. Because of this, it is common practice in medical
trials to provide the control group with a placebo treatment (for example, a sugar pill) rather than nothing at all.
Often, a percentage of the group given a sugar pill will show recovery as a result of their belief that they are
taking an effective pill. This obviously means that if no placebo was given, we couldn’t say for certain whether
any effect of the treatment was because it actually worked, or because some patients believed it worked. This
can be an issue in educational research as well.
That individual behaviours may be altered because participants in the study know they are being studied was
demonstrated in a research project (1927–32) which looked at raising worker productivity in a factory. This series of studies, first led by a Harvard Business School professor, Elton Mayo, along with associates F. RoethPage 7 of 21
Doing Quantitative Research in Education with SPSS
Sage
Sage Research Methods
© Daniel Muijs 2011
lisberger and William J. Dickson, started out by examining the physical and environmental influences of the
workplace (e.g. brightness of lights, humidity) and later moved on to the psychological aspects (e.g. breaks,
group pressure, working hours, managerial leadership). One of the main findings was that productivity increased regardless of the innovation introduced. One explanation is that this is the result of the extra attention
paid to the workers (by the researchers), which motivated them to work harder. The same effect could also
occur in educational settings. An intervention, such as a programme to help improve pupils’ reading skills,
could motivate pupils because of the additional attention they are receiving, leading to higher achievement.
Likewise, when teachers engage in a new project, they may work harder and be more motivated simply because they are doing something new, or because they know they are part of a research study.
Selecting a placebo can be hard in educational experiments, though. It is not as simple as giving patients a
sugar pill. Any placebo intervention has to be sufficiently plausible to have an effect, and therefore is often
likely to become an intervention in itself (another reading intervention, in our example!). This causes two problems: firstly, the additional cost and effort involved in developing a plausible placebo, and, secondly, the fact
that we are now measuring the effect of one treatment against that of another treatment, rather than against
a control. Therefore, in these cases, it can often be a good idea to have two control groups: a ‘placebo’ group
(which receives a placebo intervention) and a ‘real’ control group (which doesn’t receive any intervention). In
some cases, schools have been given money to buy any intervention they want rather than the researchers
developing a second intervention themselves.
2.2.4 Select Instruments
Once you have selected a suitable experimental design, you need to select or develop appropriate pre-and
post-test measures. This is crucially important, as neither a high-quality experimental design nor sophisticated
statistical analyses can make up for bad measurement. A carpenter also needs proper tools. Imagine trying
to build a car with a hammer, some nails and a plank of wood and you will know what I mean! The measurement instruments must first of all measure what we want them to. This is known as validity. Secondly, our
instruments must be reliable. Validity and reliability are discussed in Chapter 4.
Page 8 of 21
Doing Quantitative Research in Education with SPSS
Sage
Sage Research Methods
© Daniel Muijs 2011
2.2.5 Select Appropriate Levels at Which to Test Your Hypotheses
In an experimental design, you will have to think carefully about the right level of treatment at which to test
your hypothesis. The importance of this becomes clear when you think of the medicine paracetamol. The
right dosage can stop headaches and pains. Too little will not have any effect, too much will kill you. While
the consequences of too much educational intervention are usually less serious, getting the ‘dosage’ right
is nonetheless important. Think of a programme that provides extra support in reading to students who are
behind their reading age. If too little extra support is provided, it may not have the desired effect. If too much
support is provided, students may become bored and disaffected with the programme, or improvements in
reading may come at the expense of other subjects, such as maths.
In some cases you might want to test the effect of different levels of the treatment. In Example 2.2 (see page
15), would it make a difference how much text is added to the animation as to whether or not this treatment
leads to positive results? In that case, a series of experiments can be carried out, varying the level of treatment given to the experimental group.
2.2.6 Assign Persons to Groups
Assigning persons to groups is the next stage in the experimental design. As we mentioned above, in experimental research, we are always trying to minimise the influence of any external factors. This means that we
want to ensure that the experimental and control group differ as little as possible at the start of the experiment.
Otherwise, any effect we might find might be caused by differences between people in the groups rather than
by the treatment. Imagine, for example, that in Example 2.2 we had selected students from a high set class
to be in the animation-narration group, and students from a lower set to be in the animation-text group. The
differences found on the tests would then be likely to be the result of the fact that the animation-narration
group were academically higher performers, rather than being the result of narration being a more effective
accompaniment for animation than text. Therefore, we want there to be no bias in our assignment of people
to groups.
The best way to achieve this is through randomisation. This means that once we have selected subjects to
take part in our study, they are randomly assigned to either the control or experimental group – for example,
by giving everyone a number and then randomly selecting numbers to be part of either the experimental or the
Page 9 of 21
Doing Quantitative Research in Education with SPSS
Sage
Sage Research Methods
© Daniel Muijs 2011
control group. Randomisation is most likely to ensure that there is no bias, as everyone will have an exactly
equal chance to be in each group. The effect is essentially similar to playing a card game with two people. By
shuffling the cards and dividing them, we are ensuring that every card has an equal chance of ending up with
each player. Obviously, we do need to have a sufficiently large group of people to make randomisation work.
In order to test whether this has been successful, it is good practice to collect data on each participant on any
variable that you think might affect outcomes, such as gender, age or ability. Then we can check whether the
groups really are similar on all important variables.
2.2.7 Carry Out the Experiment Meticulously
Once everything is in place, the experiment needs to be carried out. When carrying out the experiment, that is,
administering the pre-test, then carrying out the treatment, and finally doing the post-test, we need to ensure
that we control extraneous factors as much as possible. As we have seen above, if we wish to say something
about what is cause (our treatment) and what is effect, we have to ensure that this control is maintained. This
means two things: firstly, we will want to control the environment. It would be hard to conduct an experiment in
an environment in which all kinds of other things are going on, and be sure that whatever outcome we find is
a result of the treatment (think of a classroom, for example). This is why many experiments are carried out in
a laboratory, where the researcher has complete control over the environment. A second factor that we need
to control is how the experiment is carried out. Every time we give the treatment to a subject, we must ensure
that this is done in the same way. We need to do this to make sure that we do not introduce experimenter
bias, the effect of the experimenter on the experiment. For example, if one experimenter giving our reading
programme to students was really enthusiastic about the programme, while another was very sceptical and
communicated this to the kids by saying things like, ‘Well, I’m not sure this will help you, it’s only an experiment,’ we might well find different effects between the two.
2.2.8 Analyse the Data
Once the experiment has been done, and the post-test administered, we have to analyse the results. Typically, methods such as t-tests and analysis of variance are used. We will discuss these methods in Chapters 7
and 10 respectively. The results will then tell us whether we can provisionally reject our null hypothesis (the
Page 10 of 21
Doing Quantitative Research in Education with SPSS
Sage
Sage Research Methods
© Daniel Muijs 2011
one we don’t want to be true) or not.
2.3 Advantages and Disadvantages of Experimental Research in Education
2.3.1 Advantages
The main advantage of experimental research is the control over external factors I have mentioned several
times in the previous section. Why do we want to control external factors and variables outside our experimental designs? We do this because this allows us to make a stronger claim to have determined causality.
One of the things we are often trying to do in quantitative research is determine what causes what, what is
cause, and what is effect. Often in talking about the results of research, the term ‘cause’ is used both frequently and loosely; for example, ‘An overly academic curriculum is a cause of pupil disaffection.’ Many studies
want to determine causes, and policymakers frequently want to address causes of perceived problems (e.g.
‘the causes of crime’).
Causality is, in fact, very hard to determine in practice. Three main elements need to be present before we
can say that one variable causes another:
1. There needs to be a relationship between the two variables. This relationship can be positive or
negative. In a positive relationship, higher values on one variable will go together with higher values on another variable. For example, higher levels of achievement in school tend to be associated
with higher levels of satisfaction with school. In a negative relationship, lower values on one variable
will be associated with higher variables on another. For example, in schools, higher percentages of
pupils with parents from low socio-economic status backgrounds will tend to be associated with lower
levels of achievement on standardised tests. If there is no relationship, there is no causality. Various
statistical methods exist to determine whether or not two or more variables are related, and I will discuss these in following chapters.
2. There needs to be a time order between the two variables. In order to be able to say that a variable
causes another, it must come before the other in time. Let’s look at the relationship between birth orPage 11 of 21
Doing Quantitative Research in Education with SPSS
Sage
Sage Research Methods
© Daniel Muijs 2011
der and achievement at school, for example. Some studies have found that there is a relationship between birth order and achievement in school (Muijs, 1997), with firstborns scoring higher than those
born later. There is possibly a causal effect here. It would clearly be nonsense to hypothesise that
school achievement causes birth order, as achievement follows birth order in time. In this case the
direction of causality is clear: birth order would have to cause an achievement effect. In many cases
in educational research, things are not quite that clear-cut, however. Think, for example, about the
relationship between pupils’ self-esteem and their achievement. Here it is not clear which comes first
in time. Do pupils with lower self-esteem start doing worse because of this? Or does low achievement affect pupils’ self-esteem negatively? Possibly the relationship is reciprocal, with both elements
influencing one another in a circular relationship, lower achievement leading to lower self-esteem,
which, in turn, affects achievement. But which came first? This is often a chicken-and-egg type question that is extremely hard to solve.
3. The relationship found must not be the result of confounding variables. This means that the relationship cannot be explained by a third variable. A well-known example of this is the relationship
between storks and births: in some European countries, the traditional answer when children ask
their parents where babies come from is to say that storks bring them. Some statisticians have found
strong evidence that this claim is in fact true: for example, Lowry (2002) reports that if one examines the records of the city of Copenhagen for the 10 or 12 years following World War II, there is a
strong positive correlation between the annual number of storks nesting in the city and the annual
number of human babies born in the city. Therefore, storks bring babies, or do they? There is, in
fact, a confounding variable here. During the 10 or 12 years following World War II, the population
of Copenhagen (like that of most European cities) grew. As a result of this, there were more people
of child-bearing age, and therefore more babies were born. As the population increased, there was
also an increase in construction to accommodate this, which in turn provided more nesting places for
storks, leading to increasing numbers of storks being present in the city.
All three of these factors (a relationship, a time sequence and no confounding variable) need to be present
before we can conclude that one variable causes another.
Why is experimental research better at determining causality than any other type of research? This follows
from the element of control I mentioned earlier. Factor one, establishing whether there is a relationship, can
be done through any type of quantitative research, and experiments are not necessarily better than non-experimental research designs at establishing this. However, the situation is different for the other two prerequiPage 12 of 21
Doing Quantitative Research in Education with SPSS
Sage
Sage Research Methods
© Daniel Muijs 2011
sites for establishing a causal relationship. In experimental studies, the researcher is manipulating the treatment, so we can be certain of the time sequence. Likewise, the problem of extraneous variables causing a
relationship is smaller in experimental research than in any other type of research, because the experimenter
can control the environment and ensure that as few extraneous factors are involved as possible, as we saw
in the section on how to design experiments.
Does this mean that when we do an experiment, and find a significant result, we can be certain of cause and
effect? This is clearly not the case, for the following reasons:
1. Results from a single experiment may be due to chance. Only if research is replicated, that is, the
findings are repeated in different studies using different participants, preferably in slightly different
settings, can we be certain of this.
2. It is always possible that findings are caused by an extraneous factor that we have not thought of
when setting up our experiment.
3. We are creating an artificial situation. Therefore, the question remains: do these effects occur in
real-life situations?
2.3.2 Disadvantages
This leads us to some of the weaknesses of the experimental approach. The laboratory set-up is always an
artificial one, and the correspondence to real-life situations can be questionable. How applicable are the results of experiments to real-life educational situations? Here, the control, which is an advantage of the experimental method, becomes a disadvantage. In everyday settings, any causal effect found in an experimental
setting is likely to be influenced by a whole load of contextual factors and influences, which will tend to make
the relationship far less predictable than in a laboratory setting. Remember, for example, the study on the
effect of violent video games given in Example 2.1. While in an experimental study we may find an effect of
watching these videos on children’s behaviour, it is rare that children will be in a situation in which the video
will be the only influence on their behaviour. When they are actually playing at school, for example, interactions with peers, school rules, weather, etc., will all influence their behaviour as well. If we look at the other
example about presentation of material in animated form, we would have to question whether this effect really matters in practice, or if it is so small that it makes no real difference to learning in classroom situations
compared to other factors (such as teacher interactions). Transferability is clearly an issue in educational exPage 13 of 21
Doing Quantitative Research in Education with SPSS
Sage
Sage Research Methods
© Daniel Muijs 2011
perimental research.
Another problem with experimental research is that it can be difficult to put into practice in educational settings. Consider, for example, the issue of evaluating educational programmes and initiatives. We might want
to do this by using an experimental design, because we want to see whether the intervention has caused an
improvement in the school. We might want to develop an intervention to improve the reading performance of
pupils, and test whether this intervention is successful. A pure experimental design would involve randomly
assigning pupils to the treatment and control groups in the school in which the experiment is taking place.
This is often problematic in practice. Teachers and parents are often not overly keen on this type of design,
and there are obvious ethical issues in allowing one group of pupils to receive an intervention that we think/
hope is effective while other pupils do not receive this intervention. Practically, realigning timetables and other measures to facilitate the experimental design is also difficult. The difficulties are even larger when one is
doing an experiment in a number of schools.
A further problem occurs when we are implementing an intervention that is specifically designed to take place
in a classroom, such
