SAT Coaching in Unlikely Places: Offering Achievement Test Preparation to Students with Academic and Economic Need

After studying the literature on the effects of SAT preparation programs, it was found that students who received coaching had more positive outcomes. No study considered the affective effects among students who received some kind of formal SAT preparation nor did the studies address how to increase access to preparation programs to serve the needs of lower-income populations. The authors of the present study designed and implemented a small-scale SAT program to deliver preparation to an underserved community and measured its impact on students’ achievement scores and attitude towards the SAT. This study confirmed the findings of other studies in that students who were coached exhibited higher achievement scores; additionally, it was found that students who were coached developed a more positive attitude towards the SAT and college.

Introduction

The purpose of this study was to analyze the effects of a SAT prep course on a group of students who would not normally seek out coaching due to socio-economic need. After considering the literature on SAT preparation, we developed our own program for SAT preparation and sought out an urban high school with high need and low socio-economic status. We delivered services free of charge to students who otherwise might not have received SAT preparation (due to cost and travel) and analyzed our program’s impact on students’ achievement and attitude towards the SAT. We asked the following questions:

Research Questions

Before creating our program, we went to the literature to try to get a sense of what research said about effective SAT coaching programs.

Literature Review

Background and Description

In theory, there are at least three ways to improve test scores. The first is the genuine improvements in the abilities measured by the test that would result in an increase in test scores (Peltier, 1989). The second is increased familiarity with the test format and pacing and therefore decreased anxiety about taking the test that would result in a more accurate assessment of ability and higher scores. The third way to improve test scores is to improve test-taking skills such as techniques to save time or techniques to quicken the process of answering certain questions.

There have been many studies that have been conducted analyzing the effects of various forms of test preparation on student performance on the SAT and other standardized exams. In a publication distributed to all secondary schools in 1968, the College Entrance Examination Board (CEEB) summarized the results of seven studies, and it was revealed that the magnitude of the gains resulting from coaching varied slightly but they were always small.

On the other hand, these beliefs do not reflect what the commercial coaching programs have to say about the effects of test prep. According to Peltier (1989) in 1983 Kaplan, the most popular commercial coaching school claimed that students who take their course would gain on average 120 to 180 points on the SAT. More recently, Kaplan was so confident in their success rate that they allowed any previous student to repeat the course at no charge if the student did not feel ready to take the test (Kaplan, n.d.).

The big commercial coaching programs must be helping some students improve, because so many people are willing to spend a great deal of money to send their children to these programs. It has been estimated that about 50,000 students spend approximately $10,000,000 annually on commercial coaching programs for all standardized examinations (Sesnowitz, Bernhardt, & Knain, 1982). But just how effective are these programs? In an exhaustive search through the literature on SAT preparation, there were three major studies that took up the issue of effectiveness, and the remainder of this literature review will discuss these in detail.

Coached and Un-coached Students

Powers and Rock’s (1999) study revealed major differences between the coached and un-coached students. Some of the main differences were due to grades (coached students had slightly higher grades), parents’ income (coached students came from more affluent families), and parents’ education (coached students’ parents had more formal education).

In addition to attending a coaching program, the coached students were twice as likely to have prepared for the SAT in a variety of other ways like using the College Board’s book of practice tests, actual samples of real SATs, as well as other test preparation books (Powers & Rock, 1999). The high cost of coaching was also a factor in coaching service access. Two of the most popular commercial test prep centers are Kaplan and Princeton Review. The Princeton Review test prep center has a $595 course and the creators of Princeton Review have claimed that students’ scores increased by an average of 140 points in both the mathematics and verbal sections combined (Peltier, 1989). This cost may be prohibitive to many families.

Coaching Quality

Although the type of coaching was mentioned in each of the studies’ discussions, the quality of coaching as well as the comparison of various coaching strategies and their effect on the student’s scores was not analyzed. In the study by Alderman and Powers (1980), the special preparation varied from school to school yet had some similarities. Most of the programs had an English teacher as the instructor with a student to teacher ratio of 25:1, and they used a commercial review book to review material. Some of the differences in the test prep courses ranged from a one-class workshop lasting less than an hour to extensive instruction lasting more than 100 hours. However, no study mentioned how the coaches were prepared, or how familiar they were with the SAT test. Another study by Sesnowitz, Bernhardt, and Knain (1982) only mentioned that the two major preparation courses were either a 10-week course with four hours of class per week or 24 hours of classroom instruction. With this in mind, one might wonder whether the effectiveness of the course had to do with the structure of it, what time of the day it was given, who the instructor was, how many students were in each class, and how often the class met.

Achievement

In a controlled study by Powers and Rock (1999), achievement levels among students in specific preparation classes were analyzed. It was determined that 12 percent of the coached students improved their verbal scores by 100 points and 16 percent of the coached students made equally large improvements in their math scores. On the other hand, 8 percent of the un-coached students improved their verbal score by 100 points or more upon retesting and another 8 percent improved just as much in their math score. Therefore, coached students were somewhat more likely than their un-coached counterparts to exhibit large score increases on both verbal and math sections of the SAT.

In Sesnowitz, Bernhardt, and Knain’s (1982) study, the achievements of the students attending the two major commercial prep schools were determined, and it was revealed that students attending School A (receiving services) increased approximately 28 points on the verbal section and 24 points on the math section, while School B (receiving no services) showed no significant effect. For the most part, the bulk of the studies conducted determined that SAT prep has a positive effect on the scores of students enrolled in them (Alderman, Powers, Rock, Sesnowitz, Berhardt, and Knain, 1982). Remaining unexamined are questions about students’ motivation to do well, including their affective disposition towards the SAT, as well as the issue of equity in access to coaching programs. The present study is an attempt to fill in this gap in the literature.

Methodology

After reviewing the literature, my faculty mentor and I decided to design a program to coach underserved students in high school for the SAT. We recruited three other Ozanam Scholars (a scholarship program that recruits students interested in serving in high need communities) at St. John’s University in Queens to help deliver SAT coaching services. We were committed to establishing a relationship with the students being prepared for the SAT and determining the students’ feelings about the course, as these feelings relate to motivation. We would also ask for a small food stipend from the University in order to help attract students to our program by serving pizza.

After working with a University professor and administrator, we gained IRB permission for our study and access to a low-income high school in New York City. We also received approval for our food stipend. We worked in the school to offer seven SAT coaching sessions for junior and senior high school students. Each session lasted 3 hours. The first day, students were given a pre-SAT practice test and a short, open-ended survey; at the end of the 7 sessions, students were given a post-SAT practice test and re-took the open-ended survey. Students were also interviewed about their experiences in the study. The interview followed a protocol (see Appendix) so that all 4 scholars could conduct the interviews. Specifically we asked students if and how their attitudes towards the SAT had changed. To do this, the interviews were taped and transcribed; under the supervision of our university professor, the 4 Ozanam Scholars looked for patterns and themes that emerged from the students’ interviews. We also triangulated with our field notes and students’ answers on the open-ended surveys.

Participants

The students involved in the research project were recruited through the university’s connection with the Gear Up Program, an early college readiness program for high school and junior high school students. There were about 15 students who participated in the program and five students who attended class every week. The students were a group of diverse, urban high school junior and senior students.

Procedure

Once the students were recruited, the class began. The SAT class met for a total of seven sessions over a four-week period to prepare the students for the December exam. The class met for three hours after school (on the high school site for the students’ convenience) from four to seven in the evening on November 4, November 8, November 15, November 18, November 23, November 30, and December 2 of 2010.

During the three hours of the course we reviewed both the critical reading and mathematics section of the SAT. The class consisted of both independent practice and group work. Usually, we would practice a section of the SAT, either critical reading or math, independently in the beginning of class under the timed conditions and then we would review it together as a group. We would sit in a circle and go over the questions and students that answered the question correct would volunteer to explain the problem to students who did not completely understand the question. The class also consisted of direct instruction time in which I would talk about a certain topic in front of the classroom and we would practice problems illustrating that topic together on the chalkboard. In addition, the coaching team would be helping any student with individual questions. We also discussed strategies in taking the SAT and how certain questions can be answered more quickly and efficiently.

During the class we ate dinner together and used the time to talk about the SAT. This was a great time to get to know the students and listen to their hopes and fears about the SAT. We also spoke about their plans for the future and possible career options, shared our own experiences, and advised them on various topics such as how to stay calm for test day.

Instruments

During the first session of the class the students took two critical reading sections of the SAT and two mathematics sections of the SAT, all under test-like timed conditions. This pre-preparation test was then graded. During the last session of the SAT course, a post-preparation test was administered with two critical reading sections and two math sections under the same timed conditions. These two scores were compared. Because only 5 students out of the 15 attended every class, only the scores of 5 students are presented here.

Qualitative analysis of students’ responses to questions dealing with their thoughts and feelings about the test was also conducted. At the beginning of the SAT course, the students were given surveys that asked questions about their feeling about the SAT and college. That same survey was then given to the students at the end of the course to determine if any changes had occurred due to the course. Individual interviews with each of the students were also conducted. The survey and interview questions can be found in the Appendix.

Results

The following results were found upon calculating and analyzing the pre-test and post-test scores of the students that were present throughout the entire SAT course. The names are pseudonyms that were assigned to each of the students.

When analyzing the pre-test and post-test scores, the average raw score for the first critical reading score between all five students was 6.85 for the pre-test and 14.82 on the post-test. The difference between these two scores when converted to the scaled score is a difference of 80 points on the SAT. As for the second critical reading section, the raw score for the pre-test was 3.5 and the post-test was 8.875. The difference was converted to a scaled score difference of 50. Therefore, the total scaled difference of scores between the pre- and post-tests in the critical reading sections is 130. In addition, we only administered two out of the three sections of the critical reading sections. If the same trend held true for the third section, the total difference in critical reading scores alone would be 195 points.

As for the mathematics sections, the same technique was used. The average raw score for the first pre-test mathematics was 6.9, and the post-test was 11.9. The difference translated to a scaled score difference of 50 points. The second mathematics section had an average of 6.15 and a post-test of 7.8, which translated to a scaled score difference of 20 points. The total difference of pre- and post-math scores was 70 points. If the trend held true with the third section, the students in the SAT course had increased their score by 105 points. On average, all of the students’ SAT scores increased by 300 points. An increase of 300 points in two of the three sections of the SAT (total of 1600 points) is a significant increase.

This study also looked into the students’ affective disposition toward taking the SAT and attending college. We analyzed surveys and interviews to study this effect and found three major themes running through each of the student’s responses: social interaction, focus, and commitment.

Social Interaction

The students liked how we would interact with them and work in groups. They liked studying together because it made practicing the SAT more fun and enjoyable. One student said, “I liked everything but my favorite part was probably the group work. When we all read together and discussed the answers.” Students appreciated the interaction because they felt more connected to us and understood how much we cared. Another student said, “I tend to get distracted and not study when I’m all alone.” And then when asked if the group work helped her stay focused she said yes. This social interaction theme also ties in with their desire to talk to teachers, friends, and family about the SAT and college. One student said, “I think I want their support. I want to ask what they think.” Another student commented on how she liked eating pizza together, which was a very social interaction.

Focus

Students felt that the SAT course was organized and focused on improving their SAT scores and helping them with specific problems. One student said, “I like how our lessons were planned out, ‘cause [sic] we took the test and then we’d do it and then go over it and then we’d just like discuss the details.” Another student said, “It was helpful because it was specific on the topic at hand.” That same student also said, “The course helped me study. I wasn’t applying myself before.” And when asked if the class and its group studying helped her study she said yes. Another student said, “I liked the people and how the lessons were planned out.”

Commitment

Students felt that the instructors were dedicated to the course and committed to helping the students learn and improve their scores. One student said, “I felt that I like learned more from your class, not like the other class I felt it didn’t really impact much.” This student explained to me that her last SAT preparation course had many teachers each week so nothing was consistent. On the other hand, she really enjoyed our SAT prep course, because the same people were present each session. We were dedicated to helping her progress and succeed on the SAT.

Conclusion

Upon analysis of the quantitative and qualitative data obtained from this research project, it was concluded that the SAT preparation course had a positive effect on students’ SAT scores as well as on their affective dispositions toward taking the SAT. The students’ scores increased with an average of 300 points and they felt more confident in taking the SAT and more willing to talk to their teachers, friends, and family about the SAT and college. The students also enjoyed the structure of the SAT course and how consistent we were in providing the service. The students also appreciated the social bond that was built throughout the course and the conversations during dinner.

Implications

Based upon the conclusions in this study, education administrators might consider offering more SAT preparation classes at their schools that incorporate both group work, independent practice, and direct instruction by a SAT preparation specialist. They may also like to incorporate a more personalized approach to SAT prep, including time for dinner and discussing college choices. Other schools may consider starting a SAT class if they do not have one already. These classes would benefit the students greatly, especially those that are motivated by others to study.

Suggestions for Further Research

This was a small-scale study: only a total of five students attended every class. Further research study should be conducted with more participants to see if the positive results are similar. A more extensive study would also allow for a better understanding of students’ feelings about which style of learning is best for them. This would help to eventually design the most beneficial SAT prep class.

References

Alderman, D., & Powers, D. (1980). The effects of special preparation on SAT-verbal scores. American Educational Research Association, 17(2), 239-251.
Kaplan Test Prep and Admissions. (n.d.). Retrieved March 2010 from Kaptest.com: http://www.kaptest.com
Peltier, G. (1989). Empowering students to improve their college admission test scores. The Clearing House, 63(4), 163-166.
Powers, D., & Rock, D. (1999). Effects of coaching on SAT I: reasoning test scores. Journal of Educational Measurement, 36(2), 93-118.
Sesnowitz, M., Bernhardt, K., & Knain, D. M. (1982). An analysis of the impact of commercial test preparation courses on SAT scores. American Educational Research Association, 19(3), 429-441.

The authors would like to acknowledge the contributions of the Ozanam Scholars’ SAT Coaching Team: Andrew Mello, Ellen Beebe, & Nell O’Connor

Appendix

Pre and post SAT Prep Survey

Interview Questions

Before Starting: Remind students that their participation is completely voluntary and there is no penalty for withdrawing from the study.

Ask for clarification if necessary. Transcribe interview. Give copy of the transcript to the student. Read through it together.

Ask: Is this accurate? Did you say everything that you meant to say? Do you want to change anything? What would you like to add?

SAT Coaching in Unlikely Places: Offering Achievement Test
Preparation to Students with Academic and Economic Need

Tracey Cannova
Mary Beth Schaefer*

St. John’s University

Abstract

Introduction

Research Questions

Literature Review

Background and Description

Coached and Un-coached Students

Coaching Quality

Achievement

Methodology

Participants

Procedure

Instruments

Results

Social Interaction

Focus

Commitment

Conclusion

Implications

Suggestions for Further Research

References

Appendix

Pre and post SAT Prep Survey

Interview Questions

SAT Coaching in Unlikely Places: Offering Achievement Test Preparation to Students with Academic and Economic Need

Tracey Cannova Mary Beth Schaefer*

St. John’s University

Abstract

Introduction

Research Questions

Literature Review

Background and Description

Coached and Un-coached Students

Coaching Quality

Achievement

Methodology

Participants

Procedure

Instruments

Results

Social Interaction

Focus

Commitment

Conclusion

Implications

Suggestions for Further Research

References

Appendix

Pre and post SAT Prep Survey

Interview Questions

SAT Coaching in Unlikely Places: Offering Achievement Test
Preparation to Students with Academic and Economic Need

Tracey Cannova
Mary Beth Schaefer*