1. Research design
1. The plan for selecting subjects (Ss), research sites, and data collection procedures (i.e., who, what, where, when, how, etc.)
1. The purpose is to provide a credible answer to a research question
2. Credibility is the extent to which the results approximate reality and are judged trustworthy and reasonable
2. Three sources of variability
1. Systematic
   1. Variability created by the researcher through the choice of specific treatments that result in differences (i.e., variability) in the dependent variable
   2. This is a desired type of variability that is directly controlled by the researcher
   3. The point of research design is to maximize systematic variability
2. Error
1. Variability that is created by sampling, measurement instruments, and other kinds of random events that make it difficult to show relationships between variables
2. This is NOT desirable
3. The purpose of research design is to minimize error variability
3. Extraneous
1. Variability caused by the direct effect of a variable not being studied on the dependent variable
2. This is NOT desirable
3. The purpose of research design is to control this type of variability
3. Controlling variability through design
1. MINMAXCON
1. Minimize error variation through reducing sampling and measurement error
2. Maximize experimental variation by maximizing the differences between the levels of treatment to which groups of subjects are exposed
1. Rogerian or behavioral counseling approaches
2. Didactic or co-operative group instructional approaches
3. Control extraneous variation that is caused by variables that are outside the experiment
1. The effect of gender on second grade math achievement can be eliminated by analyzing the data for differences between males and females
2. The effect of maturation can be controlled by selecting subjects of the same age
2. Controls
1. Random assignment of subjects to groups reduces the effects of selection and selection-related threats (e.g., selection-maturation, selection-history, etc.)
2. Eliminating variation by selecting only one level of a variable (e.g., working with males only, using the same class period for all treatments, etc.)
3. Use additional independent variables related to the dependent variable (e.g., investigating the effectiveness of co-operative group strategies on the math achievement of both girls and boys
4. Use sophisticated statistical analyses that adjust for differences between groups (e.g., ANCOVA)
2. Issues related to research design
1. Sampling (Ch. 6)
2. Data collection (Ch. 8)
3. Data analysis (Ch. 7 and Ch. 11)
4. Design validity (Ch. 9 and Ch. 10)
5. Ethical and legal responsibilities (Ch. 6)
3. Sampling
1. Population, sample, and subjects
1. Population: all members of a group possessing the characteristics that define the population (e.g., all third grade students in Orleans Parish, all graduate students at the University of New Orleans enrolled in the College of Education, etc.)
2. Sample: a subset or portion of a population
3. Subject: an individual in a sample
2. Target and accessible population
1. Target population - the larger population to which the researcher ultimately would like to generalize the results of the study
2. Accessible population - the population to which the researcher has access
3. Examples
1. Example 1
1. Target population - all third grade students in Louisiana public schools
2. Accessible population - only those third grade students in Orleans Parish public schools because these are the only third graders to which the researcher has access
2. Example 2
1. Target population - all institutions of higher education in the Southeastern United States
2. Accessible population - institutions of higher education in Louisiana and Mississippi because the researcher has contacts that will allow her access to these universities
4. Difficulty arises with the generalizability of the results when the accessible population is different - usually smaller and more restricted - than the target population (i.e., see external validity)
3. Sampling procedures: probability and non-probability
1. Probability sampling
1. Selecting a sample from a well defined population using procedures that allow the researcher to estimate the probability of a subject being included in the sample
2. Types
1. Simple random
2. Systematic
3. Stratified random
4. Cluster
2. Non-probability sampling
1. Samples that pre-exist or have been selected without the benefit of probability procedures (e.g., accessibility, representing specific characteristics important to the researcher, etc.)
2. Types
1. Convenience
2. Purposeful
3. Quota
3. A metaphor for understanding the differences between probability and non-probability sampling
1. Assume a whole apple pie represents a population. A small slice of that pie represents a sample of that population.
2. When using probability sampling we begin with the entire apple pie and cut a piece of it. We can be assured the piece (i.e., the sample) is representative of the pie (i.e., the population).
3. When using non-probability sampling we begin with a slice of pie (i.e., a sample). Unfortunately, we don't know what type of pie this slice comes from, so we have to describe the characteristics very carefully (e.g., taste, smell, form, consistency, color, etc). Based on these characteristics can we identify what it came from? Was it an apple pie? Some other fruit or combination of fruits? An apple cobbler? Obviously the task is to generalize from the slice (i.e., the sample we have) to the pie (i.e., the population from which the slice is likely to have come).
4. Limitations of non-probability sampling are focused on the inability to generalize to a population and the possibility of bias
4. Sample size
1. The major issue is one of being able to generalize to the population
2. Factors affecting sample size
1. Type of research
2. Research hypotheses
3. Financial constraints
4. Importance of the results
5. Number of variables being studied
6. Methods of data collection
7. Accuracy needed
8. Size of the population
3. General "rules of thumb"
1. Fifteen (15) subjects per group in a comparative study
2. Ten (10) to fifteen (15) subjects per variable in a relational study
3. One-hundred (100) subjects for every major subgroup (e.g., males and females; Republicans and Democrats, etc.) being investigated in a survey study
4. Data collection
1. Techniques
1. Types
1. Questionnaires
2. Structured interviews
3. Structured observations
4. Tests
5. Affective scales
6. Alternative assessments (e.g., portfolios, performance assessments, etc.)
2. See Chapter 8 for a discussion of each specific technique
2. Validity
1. The extent to which inferences made on the basis of scores from an instrument are appropriate, meaningful, and useful
1. Give an algebra class a physics exam and the scores are not going to give you appropriate, meaningful, or useful information about the students' achievement in algebra
2. Give a fifth grader the Graduate Record Exam (GRE) and his score is not going to predict his performance in school in a meaningful, appropriate, or useful manner
3. Give a student a self-esteem scale and the scores are unlikely to give you meaningful, appropriate, or useful information about her attitude toward math
2. Types of evidence
1. Content
2. Criterion-related
1. Predictive
2. Concurrent
3. Construct
3. Situationally specific
1. An exam for an introductory education class will contain items that are appropriate for those students in that class but inappropriate for students in a business class
2. Using the ACT to predict college performance is appropriate, but using it to predict performance in medical school is not appropriate
3. Reliability
1. Consistency of measurement
1. Can we be assured that John's true score on the English exam is a 94?
2. Can we be assured that Sally's true score on the science performance assessment is 87?
2. Types of reliability coefficients
1. Stability (i.e., test-retest)
2. Equivalence (i.e., parallel forms)
3. Internal consistency (artificially splitting one test into two)
1. Split-half
2. KR 20 and KR 21
3. Cronbach's alpha
4. Importance of validity and reliability
   1. Without validity and reliability the researcher is not likely to achieve credible results
   2. Reliability is a necessary but not sufficient test characteristic. That is, validity must be present regardless of whether an instrument is reliable or not.
5. Online sources of information about published tests
1. The American Psychological Association (APA)
2. Buro's Institute's Tests Reviews Online
3. ERIC Database
6. Other sources - typically print - of information about published tests
1. Tests in Print
2. Handbook of Research Design and Social Measurement (5^th edition)
3. Index to Tests Used in Education Dissertations
4. Directory of Unpublished Experimental Mental Measures
5. The ETS Test Collection
1. Test Collection Bibliographies
2. Tests in Microfiche
3. Six volumes of the ETS Test Collection Catalog
6. Tests: A Comprehensive Reference for Assessments in Psychology, Education, and Business (4^th edition)
7. Test Critiques, Volumes 1-10
8. Mental Measurements Yearbooks
9. Tests and Measurements in Child Development: Handbook I and II
10. A Sourcebook of Mental Health Measures
11. Measures for Psychological Assessment: A Guide to 3,000 Original Sources and Their Applications
12. Handbook of Family Measurement Techniques
13. Socioemotional Measures for Pre-School and Kindergarten Children: A Handbook
14. Handbook for Measurement and Evaluation in Early Childhood Education
15. Dictionary of Behavioral Assessment Techniques
7. Developing instruments - seek advice from an expert
1. Advantage - directly addresses the issues in which you are interested
2. Disadvantage - it is very difficult and time consuming to develop technically sound instruments
5. Design validity: the extent to which the results of an experiment match the reality of the world
1. Internal validity: the extent to which extraneous variables have been controlled so that the research can be assured the effect was produced by the cause
1. Related primarily to experimental designs
2. Rival hypotheses: alternative explanations for the results
3. Threats
1. History: extraneous incidents or events that affect the results
2. Selection
1. A difference between or among groups usually as a result of non-random assignment to groups
2. Relates to the manner by which subjects were assigned to groups, not how they were selected from the population
3. Statistical regression: movement of extremely unusual scores (high or low) to the average
4. Pretesting: having taken a pretest influences the results
5. Instrumentation: data collection procedures and/or instruments affect the results through low validity or reliability, floor or ceiling effects, etc.
6. Attrition: differential loss of subjects from each group
7. Maturation: changes in the subjects that affect performance on the dependent variable (e.g., preschool children's academic development will be affected by their maturation over the course of a year)
8. Diffusion of treatment: subjects in both the control and experimental groups are exposed to the experimental treatment
9. Experimenter effect: deliberate and unintentional influences (positive and negative) that the research has on the subjects (e.g., an outstanding teacher using a poorly designed unit of study)
10. Treatment replications
1. Independence of observations
2. A particular concern in education where "classes" rather than individuals receive treatments
11. Subject effects
1. Changes in the subjects behavior (positive or negative) in response to the research situation
2. Initiated by the subjects themselves
3. An example is the John Henry effect
4. See the Assignments for this chapter for practice assessing the presence of the threats to internal validity
2. External validity: generalizability of results to other people, settings, and times
1. Two types - population and ecological
2. Population: the extent to which the results are generalizable to and across populations
1. To means from one population to another population (e.g., fifth graders to sixth graders)
2. Across means from one subgroup in a population to another subgroup in that population (e.g., across males and females within the population)
3. Threats
1. Experimentally accessible vs. target population
2. Interaction of characteristics of the sample and treatment effects
3. Ecological: the extent to which the results are generalizable to different environments
1. Settings, environments, times
2. Threats
1. Lack of explicit description of the independent variable
2. Multiple treatment interference: the interaction of multiple treatments make it impossible to generalize the effects of the treatments independently of one another
3. Hawthorne effect: results from a study where subjects know they are taking part in a study cannot be generalized to situations where the research study and the subject's knowledge of it are absent
4. Novelty and disruption: results from studies with novel and/or disruptive environments cannot be generalized to environments where such novelty and/or disruption does not exist
5. Experimenter effect: results from a study in which an experimenter effect exists cannot be generalized to environments where a similar experimenter effect does not exist
6. Pretest and post-test sensitization: results form a study established in an experimental environment where pretesting and post-testing took place cannot be made when pretesting and post-testing are not present
7. Interaction of history and treatment effects: results from a study in which history played a part cannot be generalized to a situation where such an historical event does not take place
4. See the Assignments for this chapter for practice assessing the presence of the threats to external validity
3. Statistical conclusion validity
   1. The extent to which the calculated statistics accurately portray the actual relationships
   2. Statistical concerns related to using probabilistic statistical models (i.e., inferential statistics)
   3. Threats
   1. Low statistical power
   2. Violated assumptions of the statistical tests being used
   3. "Fishing" for an answer and the effect on error rates
   4. Lack of reliability of measuring instruments
   5. Restriction of the range of scores
   6. Lack of reliability of treatment implementation
   7. Extraneous variability in the experimental setting
4. Construct validity
1. Inferences that are made from the nature of the measurement and interventions used to for the constructs they purportedly represent
2. Threats
1. Inadequate explication of the constructs
2. Mono-operation bias
3. Mono-method bias
5. Ensuring internal, external, statistical, and construct validity leads to a more credible research study
6. Ethical and legal considerations
1. Research organizations and legal acts
1. American Psychological Association Ethical Principles
2. American Educational Research Association
3. Family Educational Rights and Privacy Act of 1974 (i.e., the Buckley Amendment)
4. The National Research Act
2. Ethical principles: beliefs about what is right or wrong, proper or improper, good or bad, etc.
1. The primary investigator is responsible for the ethical standard adherence
2. The investigator should inform subjects of all aspects of the research that might affect their willingness to participate
3. The investigator is open and honest with subjects
4. Subjects are protected from physical and mental discomfort
5. The investigator must secure informed consent from the subjects before they participate
6. Information obtained about subjects is held in confidence
7. Institutional approval from a committee on the use of human subjects must be obtained if the research is being conducted through a university or school system
8. The investigator must communicate results is such a manner as to minimize misinterpretation and or misunderstanding
9. The investigator has the responsibility of recognizing when potential benefits have been withheld from a control group
10. The investigator should provide subjects with the opportunity to see the results of the study in which they participated
3. Legal responsibilities
1. Protecting the welfare and rights of subjects
2. Consistent with the ethical principles mentioned above

Original outline prepared for Addison, Wesley, Longman by Jeffrey Oescher, University of New Orleans

---

Back to Top



E-Mail Jeffrey Oescher