In the previous section, we saw potential benefits to using random assignment – creating groups that we are willing to consider equivalent. However, there is still the chance that “luck of the draw” could lead to a higher success proportion for the response variable in one group than the other. In this section you will explore methods for determinng p-values that address how often random assignment could create a difference between the treatment groups at least as large as the one observed.

Goals: In this lab, you will

• Investigate a research question that involves comparing two groups on a categorical response through simple numerical and graphical summaries.
• Continue to explore the concept of "statistical significance" and p-values for the scenario of comparing two groups on a categorical variable, with random chance arising from the random assignment process.
• Use the results of a designed experiment and sample data to draw conclusions about a population or process.

 

Antonioli and Reveley (2005) investigated whether swimming with dolphins was therapeutic for patients suffering from clinical depression. The researchers recruited 30 subjects aged 18-65 with a clinical diagnosis of mild to moderate depression through announcements on the internet, radio, newspapers, and hospitals in the U.S. and Honduras. Subjects were required to discontinue use of any antidepressant drugs or psychotherapy four weeks prior to the experiment, and throughout the experiment. These 30 subjects went to an island off the coast of Honduras, where they were randomly assigned to one of two treatment groups. Both groups engaged in one hour of swimming and snorkeling each day, but one group (Dolphin Therapy) did so in the presence of bottlenose dolphins and the other group (Nature Group) did not. At the end of two weeks, each subject’s level of depression was evaluated, as it had been at the beginning of the study, and each subject was categorized as experiencing substantial improvement in their depression symptoms or not. (Afterwards, the control group had 1-day session with the dolphins.)

Research question: Does swimming with dolphins increase the probability of "substantianl improvement" in depression symptoms over swimming?

 

The following two-way table summarizes the results of this experiment:

	Dolphin Group	Nature Group	Total
Showed substantial improvement	10	3	13
Did not show substantial improvement	5	12	17
Total	15	15	30

We must ask the same questions we have asked before – is it plausible that this difference has arisen by random chance alone if there was no effect of the dolphin therapy? If so, how surprising would it be to observe such an extreme difference between the two groups?

What's different in this study is now the source of randomness is the random assignment of subjects to the treatment groups.  What if the subjects' responses had nothing to do with which group they were in, could the random assignment have turned out differently? What could have the results looked like instead?

In other words, we will "fix" the response outcomes, patients were going to improve or not regardless of the treatment group, and redistribute these 13 improvers and 17 non-improvers across the two groups.

So how surprising is the table from the actual study? How often might we get a table even more extreme than what we observed?  We will first turn to simulation to explore the random assignment process.  

• Gather 30 cards and designate 13 of them to represent improvements (e.g., face cards or red suited cards, blue colored or S-labeled index cards) and then mark 17 of them to represent non-improvements (e.g., non-face cards or black suited cards, green colored index cards or F-labeled index cards).
• Shuffle the cards well and randomly deal out 15 to be the Group A = dolphin therapy group and the rest to Group B = nature program.
• Construct the 2×2 table to show the number of improvers and non-improvers in each group (where clearly nothing different happened to those in “group A” and those in “group B” – any differences between the two groups that arise are due purely to the random assignment process).

 

We really need to do this simulated random assignment process hundreds, preferably thousands of times. This would be very tedious and time-consuming with cards, so let’s turn to technology

Open the Dolphin Study applet Confirm that the two-way table displayed by the applet matches that of the research study. Check Show Shuffle Options and confirm that there are 30 total “people”: 13 blue and 17 green. Press Shuffle. The applet repeats what you did: Shuffles the 30 cards and deals out 15 for the “Dolphin therapy” group, separating blue cards (successes) from green cards (failures), and 15 for the “Nature Group” (Control); creates the table of “could have been” simulated results; adds a dot to the dotplot for the simulated difference in conditional proportions. Now press Shuffle four more times. Does the difference in conditional proportions vary among the repetitions? Enter 995 for the Number of Shuffles. This produces a total of 1,000 repetitions of the simulated random assignment process under the null hypothesis (showing the “null distribution” or a “randomization distribution”).  Include a screen capture of (only) your null distribution.

	The previous question is pretty important, ask if you are not sure.

Check the Overylay normal distribution box.