Experimental Design - Basic Concepts

Terminology.    The main goal of any scientific research is to describe behavior, predict the properties, determine the main cause, and explain it as part of a scientific question. To answer a research question, we traditionally follow the following steps: begin with a hypothesis, identify what factors whose effects are to be studied, variable(s) that are manipulated in the study, carefully select a measure of effects, the observed variable; as much minimize the unwanted or irrelevant effects or confounds, design an appropriate experiment and do sampling. Lastly, we perform statistical analyses and draw a conclusion.

There are 4 types of variables according to their nature: nominal (non-numeric, category), ordinal (rank, or ordering), interval (no true zero, e.g. temperature), and ratio scale (zero means absence, e.g. weight). These types exist also according to the way we measure them, or their level of measurement. The first two are qualitative, i.e. kinds, types, categories. The last two are quantitative variables or quantity, amount.

From the perspective of an experiment, there are 3 main types of variables. We have come across the dependent and independent variables a few times, including when talking about linear regression. Independent variables are under the control of the experimenter, independent of the participants. Dependent variable(s) is usually measured. A design that has only one DV is called a univariate design, more than 1 DV simultaneously analyzed is called a multivariate design. The following table nicely summarizes the differences. The third type, the confounding variable, is undesirable because it seems to explain our research question, but the actual fact is irrelevant.

Independent variable (IV): a variable that you are manipulating, a.k.a - Explanatory variable - Regressor - Controlled or manipulated variable - Predictor variable - Input variable

Dependent variable (DV): a variable that is affected in the experiment, a.k.a - Explained variable - Regressand - Measured variable - Response variable - Outcome or output variable

We then design an appropriate experiment to test our hypothesis, determine how many groups of subjects, and wisely sample our subjects from a population of interest. There are a few ways to do sampling, e.g. random sampling, stratified sampling. The independent variables are manipulated to provide a different experimental procedure to samples or subjects. These different procedures are called treatments, and hence independent variables are also called treatment variables. In designing our experiment, there are two main ways we can assign the participant into. Refer to the table below.

Within-subject design, repeated-measure design = the same participant experiences multiple treatment or protocols in the study. - Lower error variance,  - Higher statistical power, - Risk of carryover effect or order effect, Counterbalancing and/or Latin square method is able to overcome this risk.

Between-subject design, independent-group design = participant is randomly assigned to only one treatment or one level of the independent variable of the study. - Higher error variance (inter-subject var.) - May require a larger sample, time & cost.

A simple example is here. Suppose you want to test the effect of marijuana on the final exam score. The independent variable here is the consumption amount. The dependent variable is the final exam score. Here, it is good to have 2 independent or treatment groups of subjects, i.e. no single subject goes through the experiment more than once. The obvious statistical comparison is between the group of students who take marijuana and another group who doesn't. In an ideal experiment, the only difference between the two groups is the treatment (marijuana consumption). In real life, this is hard to achieve. Irrelevant differences must then be controlled. A controlled experiment is taken exactly to minimize those differences.

What about confound variables? An obvious confound may be the student IQ. If this variable is able to explain the exam score well, this result is not directly useful to our study. In other words, our design should only reflect the influence of our independent variable on our dependent variable. Sometimes, a control group is used to minimize the confounding effects in our analyses. What we can do now is to compare control vs marijuana and control vs no marijuana. The presence of confounding effects and other subjective influences can also be minimized by performing randomization as a form of control during the experiment. For example, we can assign each participant to a group randomly regardless of their IQ level as the focus here is on marijuana consumption.

Another aspect to understand is the term fixed and random variable. A fixed variable whose different levels are have been arbitrarily, but rationally, chosen. Inferences taken about a certain fixed variable is limited to the selected members. A random variable has levels that are subsets of a larger population, bearing in mind that each level has an equal opportunity to be selected. It helps us to link our data to a population in general. Subjects are typically a random variable. There is always a risk that our experiment introduces a bias in our data. This may occur simply because of not selecting the subjects wisely, error in measurement, and finally, the confounding variables not accounted for. It is important that the dependent variable is sensitive and reliable to describe our hypothesis.

Cause and effect are important to explain certain behavior. Saying the average exam scores between these two groups are significantly different from each other does not directly say that drug consumption is the main cause of the effect. In addition, correlation and regression analyses do not imply causation. Traditionally, one has to prove that marijuana consumption is the necessary and sufficient factor for the effect to occur. But in reality, this is difficult to prove simply because there are many confounding factors that may explain an outcome or result. One way is to construct a chain of reaction and apply mediation analyses.

One critical factor in behavioural research is the validity of the variables being investigated. Validity means truth, how accurate a variable is to represent information. There are a few kinds of validity:

(1) Construct validity: adequacy of the operational definition of a variable. In social sciences, a construct is a measure we are interested in. Unlike in physics, constructs are often abstract or subjective. For example: if a social anxiety measure for teens is the one we are researching on, then that is our construct. Construct validity says that the measure has to describe or quantify social anxiety, not other things such as mood or temperament. 

(2) External validity: describes how a conclusion of a study can be generalized to other populations or similar studies. In our example using marijuana consumption on campus, how true does this happen across different schools in general? Hypothesis testing is performed to achieve this purpose. 

(3) Concurrent validity: when validity is tested against an established measure of the same or a related underlying construct assessed within a similar time frame, then we are looking for this type of validity. 

(4) Internal validity: refers to how well the effect in our experiment is attributed to our independent variable. A good experimental design must ensure this is the case. 

(5) Ecological validity: not only does it apply in the laboratory context, but it also has to accurately reflect what we want in the real-world setting. This is important as some measures, e.g. in sports or healthcare, require subjects to be tested outside of the laboratory. 

Finally, the results of our trials or experiments have to be reliable when we perform measurements or observations. A reliable observation is one that is consistent and repeatable. Consistency can be measured in terms of inter-rater reliability (using a few judges, measure once) or test-retest reliability (using a single judge, measure a few times).

Note: For more discussion, see Cozby (2011).

Factorial Design.     So far we only see one simple treatment, e.g. drug consumption, in our study. Let's introduce a new term: an independent variable is also called a factor, often a categorical variable. For example, when we talk about drug abuse, the drug type is a factor. Each substance: alcohol, marijuana, nicotine, and amphetamine is called the level of that factor.  In our experiment, if in addition to drug consumption, we introduce another independent variable in our example, e.g. criminal history and study hours we have two factors in the design. Proper design allows us to explain behavior in a multi-dimensional view.

Often, an experimental design can be a little complicated. Suppose we have 2 factors. If every level of factor-A occurs with every level of factor-B, it is called crossing. A study that has many levels or multiple crossing factors is termed factorial design. We can construct a factor A x B table. In general, a factorial experiment is typically the case. However, often factor-A may contain different level of factor-B. This is called nesting, factor-B is nested in factor-A.

Which statistical test should we use? Comparing two means in your data can be tested using t-test, be it paired or independent samples t-test. In a factorial design, when dealing with one factor with two levels only, the t-test is sufficient. When more levels are involved, however, the Analysis of Variance or ANOVA is used as an extension of t-test. ANOVA is based on F-test, which is a ratio of two variances (or mean-squares, MS). Refer to the equation below. When the null hypothesis is true that there is no difference in means, the F-value should be 1.0. A large F-value indicates some differences may exist among the means due to a systematic treatment or manipulation effect. Note that it doesn't tell which means are different! To find which means, we should do post-hoc tests.

There are two main types of ANOVA: (1) a 1-way analysis compares levels of a single factor based on a single continuous response variable; and (2) a 2-way analysis compares levels of two factors on a single continuous response variable; and so on.... Now, it seems that study hours, a continuous independent variable, is correlated with the exam score. To treat this in ANOVA, we use a variant analysis called Analysis of Covariance or ANCOVA.  ANCOVA compares the dependent variable by both a factor (categorical) and a continuous independent variable. ANOVA characterises between-group variations due to a treatment effect. In contrast, ANCOVA divides between-group variations to treatment and a continuous covariate.

Knowing what statistical test to use is paramount in research!! Several other hypothetical situations and the type of possible statistical tests used will be presented here:

To see the difference in exam score between Harvard and UPenn students in their "Intro to Statistics" class, I recruit a total of 30 first-year undergraduate students from each university. Then, I perform unpaired t-test (nondirectional or 2-tailed, with α = 0.05) as the samples come from two independent groups or origins.

To see whether there is a difference in exam score among Harvard, MIT, and Stanford students in their "Intro to Statistics" class, I recruit a total of 30 first-year undergraduate students from each university. Then, I perform 1-way ANOVA to test the main effect of university (2-tailed, with α = 0.05).

To see whether there is a difference in exam score among Harvard, MIT, and Stanford students in their "Intro to Statistics" class, I recruit a total of 30 first-year undergraduate students from each university. I introduce a new factor called gender (male and female). Then, I perform 2-way ANOVA to test the main effect of university and of gender. If the score of different university is also influenced by gender, I'm looking at the interaction between the two factors. Interpretation of the main effects can be different in the presence of an interaction!

To see whether there is a difference in exam score among Harvard, MIT, and Stanford students in their "Intro to Statistics" class, I recruit a total of 30 first-year undergraduate students from each university. It seems that the score is correlated with IQ level. If I treat the IQ level as a continuous independent variable or covariate, then, I perform 1-way ANCOVA to test the main effect of university (2-tailed, with α = 0.05) having controlled the IQ level as the covariate.

[Reference #1]: Paul Cozby, "Methods in Behavioral Research, 10 ed" (2011). 
[Reference #2]: JL Myers' "Fundamentals of Experimental Design" (1979).