Working with Your Tentative Quantitative Research
Question
This Section has Activities that require you to work with a
tentative research question. As you know, you will almost definitely change (or
at least refine) your research question(s) when you write your Concept Paper,
and you may not even choose to do a quantitative study (in fact, your research
question will probably change for Activity 8). These Activities will teach you
key concepts in research design that will serve you regardless of your later
choices.
Assignment 7 Samples, Power Analysis, and Design Sensitivity
Researchers are concerned with who they studytheir samplefor two very different basic
reasons:
(1) in order to be able to say something convincing about a population, i.e. to
be able generalize claims about a sample to a population;
(2) in order to be able to say something convincing and meaningful about
relationships between constructs.
Generalizing
In research that attempts to contribute to theory, as dissertation research
must, (1) is not usually of interest. If, for example, your literature review
suggests that gender or IQ scores or some other variable moderates a
relationship, you will, of course, need to be sure that you include
participants with the necessary characteristics (gender, IQ score, etc.) and
you will need to ensure that your sample has an identity (e.g., second grade
public school teachers in Madison, Wisconsin) but you do not need worry about
whether your participants are selected in sufficient numbers in a specific way
(e.g., randomly, stratified randomly) from a specific population to allow you
to say something about the population from which the sample is drawn.
On the other hand, an inadequate literature review or chance may lead you to
select a group in which a moderating variable is at work (so that you may not
find a relationship that really does exist, just not for your sample) or a
sample that does not meet the assumptions (even relaxed assumptions) of the
statistical test you planned to use (oops!). These will be matters to address
by using non-parametric statistics or to speculate about in Chapter Five of
your dissertation or for future researchers to discover.
As a researcher and a doctoral student in education, you must, though, know
about (1) and understand the concepts involved with it. The first part of this
assignment addresses this.
Power
Researchers who attempt to contribute to theory are primarily concerned with
exploring relationships among constructs, not with seeing if something that is
true of a group is true of a population. They are concerned with (2)having a sample that allows
you to find a relationship among constructs. This is a matter of both the size
of your sample and the sensitivity of your design. As you know, sample size and
other aspects of design are intimately related, and you must think about them
together when developing your research questions and planning your study.
Unless you are lucky enough to be able to obtain all the participants you need
in order to do your preferred study, you will have to work back and forth
between design and sample size, adding covariates or blocking to reduce error
variance or maybe even changing from a between subjects to a within subjects
design or making some other serious modification in your design until you
arrive at a viable design that has a good chance of answering your research
questions.
Unless you design your study adequately and select a sample of sufficient size,
your design may be a set-up for a Type II error
failing to find a difference or a relationship that is really thereand your study may be largely
a waste of time! You want to have a large enough N to find a relationship among
constructs that is really there and to be able to argue that the relationship
is meaningful.
Power Analysis
There are four factors involved in calculating sample size:
- Statistical
test – Your sample size is partly a function of the statistical test you
use. Some tests (e.g., Chi-squared) require larger sample to detect a
difference than others (e.g., ANCOVA). - Expected/estimated
Effect size – The effect size is potency of your intervention or the
strength of the relationship you are investigating. For example, a
psychedelic drug has a very potent effect on number and vividness of
hallucinations. You may only need a single subject design to detect them.
The effect of a traffic safety class taken in 2nd grade on a group of high
school students may take a very large sample to detect. In the language of
statistics, an effect size is the difference between the mean scores of
two groups divided by the pooled standard deviation. This is called Cohen�s d. The greater
difference between groups on a measure after you factor in how spread out
the scores are, the more potent the intervention. You will calculate an
effect size as part of the analysis of your data in order to determine
that you have found something meaningful (not merely statistically
significant), but in advance of doing your study, you must estimate the
effect size in your study. Lipsey and Hurley (2009) describe a way to
estimate effect size that many Learners will find helpful: Review the
literature on the same or similar relationships or interventions to find
the range of relevant effect sizes to estimate the effect size for your
study. - Alpha.
The alpha level is the probability of a Type I errorof rejecting the null,
no difference, hypothesis when it is true
that you are familiar with. By convention this is set at p=.05.
Convention may not be your best guide. As you know from the readings in
Activity 3, the null hypothesis is always false and can always be rejected
with a large enough sample, so a .05 level may unnecessarily require you
to have a larger sample than you need. Better to use the literature and
your judgment to justify an alpha level that makes sense for your study.
This justification will involve looking at the danger of a Type I error
versus the cost in resources of avoiding it. - Beta.
The beta level is the probability of a Type II errorof accepting the null,
no difference, hypothesis when it is false, in other words, of failing to
detect a difference when it is there. The main point of a power analysis
is to have enough subjects and no more to detect a difference. As with
alpha, you set beta based on a judgment. The convention is .2, which
yields a power of .8 (1-beta).
Activity Resources
- Trochim,
W. M. K., & Donnelly, J. P. (2008): Chapter 2 - Fritz, A. E., & Morgan, G. A. (2010).
- Houser,
J. (2007). - Acheson, A. (2010).
- Piasta, S. B., & Justice, L. M. (2010).
- McCready, W. (2006).
- G*Power 3 software and documentation
- Faul, F., Erdfelder, E., Lang, A. G., & Buchner, A.
(2007). - Mayr,
S., Erdfelder, E., Buchner, A. & Faul, F. (2007).
Warm-up Activity
Download G*Power and play around with it. See how changes in assumptions and
parameters affect sample size estimates.
Part 1
- Compare
and contrast internal and external validity. Describe and give examples of
research questions for which external validity is a primary concern.
Describe and give examples of research questions in which internal
validity is a primary concern. Discuss strategies researchers use in order
to make strong claims about the applicability of their findings to a
target population. - Compare
and contrast random selection and random assignment. Be sure to include a
discussion of when you would want to do one or the other and the possible
consequences of failing to do random selection or random assignment in
particular situations. - Explain
the relationship between sample size and the likelihood of a statistically
significant difference between measured values of two groups. In other
words, explain why, all else being equal, as sample size increases the
likelihood of finding a statistically significant relationship increases. - Compare
and contrast probability and non-probability sampling. What are the
advantages and disadvantages of each?
Part 2
You must estimate the sample size you will need in order to have a reasonable
chance of finding a relationship among the variables stated in your research
hypotheses (should one exist), given your statistical analysis(es) and
assumptions/calculations of factors 2-4 above. You must do this, even if you
plan to use a convenience sample (see below). There are a number of sample size
calculators available. Using G*Power, which is required in this Activity. You
will use G*Power’s a priori power analysis function to calculate a sample size. If it
yields an unrealistically large size sample, you will rethink your design and
assumptions and, perhaps, use G*Power’s
compromise power analysis to estimate a workable sample size that makes
sense. If you plan on using a convenience sample, you would use both analyses
as part of your argument that your convenience sample is large enough.
Main Task: Submit the Following
1.
- Calculate
the sample size needed given these factors: - one-tailed
t-test with two independent groups of equal size - small
effect size (see Piasta, S.B., & Justice, L.M., 2010) - alpha
=.05 - beta =
.2 - Assume
that the result is a sample size beyond what you can obtain. Use the
compromise function to compute alpha and beta for a sample half the size.
Indicate the resulting alpha and beta. Present an argument that your study
is worth doing with the smaller sample.
2.
- Calculate
the sample size needed given these factors: - ANOVA
(fixed effects, omnibus, one-way) - small
effect size - alpha
=.05 - beta =
.2 - 3
groups - Assume
that the result is a sample size beyond what you can obtain. Use the
compromise function to compute alpha and beta for a sample approximately
half the size. Give your rationale for your selected beta/alpha ratio.
Indicate the resulting alpha and beta. Give an argument that your study is
worth doing with the smaller sample.
3. In a few sentences, describe two designs that can address
your research question. The designs must involve two different statistical
analyses. For each design, specify and justify each of the four factors and
calculate the estimated sample size you’ll
need. Give reasons for any parameters you need to specify for G*Power.
Include peer-reviewed journal articles as needed to support your responses to
Part I.
Support your paper with a minimum of 5 resources. In addition to these
specified resources, other appropriate scholarly resources, including older
articles, may be included.
Length: 5-7 pages not including title and reference pages
Assignment Outcomes
Compare and contrast the roles of sampling technique,
convenience samples, and selection bias in quantitative designs.
Determine the appropriate sample size based on an a priori power analysis.
