Public Policy InvolvementVol. 20, No. 1
|
Interested in submitting an article to KON FORUM? Papers are now being accepted for review.
|
|
Kappa Omicron Nu
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Inputs |
Activities |
Outputs |
Outcomes |
||
Initial |
Intermediate |
Longer-term |
|||
|
|
|
|
|
|
Purpose and Scope of Evaluation
The purpose of this evaluation is to examine the value of the UCA Super Kids Program as evidenced by the knowledge and behavior skill development of the students in that program. The reasons for this evaluation are threefold: (a) to assist in future program planning and provide information to stakeholders, (b) to judge the overall value, worth, and merit of the program for participants, and (c) to determine if program goals or objectives are being met and to what extent for the stakeholders. This evaluation is formative as program improvement is the primary goal of the evaluation (McLaughlin, 2003; Fitzpatrick, et. al., 2004).
The scope of the evaluation is the UCA Super Kids Program. The evaluation will focus on three desired outcomes, which relate to the three program goals: (a) assessment of change in science knowledge gained during program participation, (b) assessment of positive behavior development and change due to program participation, and (c) examination of participation in Super Kids Program and subsequent enrollment at the University of Central Arkansas. An objectives-oriented approach is used to determine if and to what degree program goals and objectives are achieved (Fitzpatrick, et. al., 2004).
The primary target audience includes the Super Kids graduates and their parents, the local public and private elementary schools, UCA, the Program Director (Dr. Mark Cooper), and the Super Kids staff and teachers. The secondary target audience includes the Conway community and Faulkner County as the individuals and families involved in Super Kids live and function in this area.
Efficient program-level evaluation incorporates course-embedded assessments that link learning activities to program outcomes (Huba and Freed, 2000). For example, the student pre- and post-test evaluation is a course-embedded assessment that links the science content taught in the program to the program outcome of improvement in science knowledge. The parent survey is another program assessment tool used to link the character and friendship building components of the program to the program outcome of mastery of age-appropriate actions. The following guiding questions emerged from the identified program goals: (a) Does program participation affect the science knowledge of the participants and how? (b) Does program participation affect the behavior development of the participants and how? (c) Do Super Kids graduates attend UCA?
The evaluator is familiar with purposes to uphold the guiding principles for evaluations: systematic inquiry, competence, integrity/honesty, respect for people, and responsibilities for general and public welfare (American Evaluation Association, 2004). This evaluation plan outlines the technical standards and data based inquiry the evaluator will use. The evaluator will seek assistance from others to review statistics or other areas outside her scope of practice or area of comfort.
Integrity and honesty are crucial to the reputation of any evaluator, and the evaluator possesses these characteristics along with basic knowledge of program evaluation. Any potential conflict of interest must be identified prior to the start of the program evaluation along with the strategy the program evaluator will use to address the conflict of interest.
Another potential problem is resistance from the Super Kid’s staff at having the program evaluated. This program has never been formally evaluated and that process may be uncomfortable for those directly involved with the program. To maintain political viability it is vital to maintain open communication with Dr. Cooper and the Super Kids staff. Initial planning meetings will be conducted to allow stakeholders the opportunity to discuss their needs. The identified needs will be ranked to determine the value of each in light of the budget, time and personnel available for program evaluation. Identifying and agreeing on the value of the information obtained in the evaluation and how that information will be used or analyzed to meet the needs of the stakeholders will add value to the evaluation results. They will be informed that this evaluation is to provide the evaluator with experience in program evaluation and to provide them with formative information for program improvement and not summative information for program continuance.
Institutional Review Board (IRB) approval ensures protection of the rights of the program participants, and this approval will be obtained before performing the program evaluation. The management plan addresses dissemination of information to all interested parties. The identified boundary of the program evaluation is the UCA Super Kids Program objectives relative to program outcomes. The evaluator has some resources to assist in this evaluation including graduate assistants, photocopier access, office supplies, and a computer. Limitations include time constraints, as the program is in session only during the month of June each year, and financial support, as no part of the FACS budget is allocated for this project. Finding respondents in a timely manner, locating, and training support staff are other limitations.
Design
The data collection combines qualitative and quantitative methods, including teacher and staff interviews, student pre-tests/post-tests, and parent surveys. Review of the Mental Measurements Yearbook (2001) does not reveal instruments usable with this age group to assess science knowledge of the topics taught in Super Kids. Attention was given to item writing and format development in preparation of the instruments (Cox, 1996). Appendices C, D, and E contain the teacher and staff interview guide, student pre-and post-tests, and parent survey, respectively.
The teacher and staff interviews will be conducted at the end of the fourth and final week of Super Kids and hand-coded for trends. The student pre-test/post-test is on the second-grade reading level according to the Flesch-Kincaid scale. Five questions are included on this instrument to assess a representative sample of the content taught on each day of the program (Linn, Miller, & Gronlund, 2005). Descriptive statistics and paired sample t-tests for pre- and post-test variables will indicate the level of change in students’ science knowledge due to program participation.
The parent surveys will show no identification (e.g., “P1” for “Parent Number One”), and will be hand-tallied. The percentages of the total will be calculated for each of four possible responses: “definitely yes”, “generally yes”, “generally no”, and “definitely no” for each of the questions asked on the survey.
The University of Central Arkansas uses a program called Banner for academic advising, student registration, and housing demographic data. Line items can be added into the system to identify students who attended Super Kids and in which years they attended the program. The computer can then identify and tally raw numbers of students who attended the Super Kids Program. The raw number can be compared to the Super Kids graduation numbers for the appropriate year. Trends can be tracked starting with 2008-2009 school year because Super Kids is in its tenth year. Retention data are computed by UCA each year. Using the line items for Super Kids participation, we can determine if these students vary from non-participants in retention.
The evaluation agreement in Appendix F outlines the responsibilities of the evaluator and program director, timeline, and evaluator’s fee. The information sources are the student participants, the parents of the student participants, and the university’s admissions computer system. The students and parents will be selected based on their participation in the Super Kids Program and their willingness to participate by completing the consent form, surveys, pre-tests, and post-tests. The management plan outlines the procedure for administering these instruments. All data obtained will be coded with no names on the forms and stored in a locked file cabinet in Dr. Cooper’s office.
Content validity of the instruments will be reviewed by subject-matter experts, Writing Center employees, and graduate students. Recommended appropriate changes will be made. Reliability of the evaluation results will increase each time the evaluation is performed (Patten, 2007). The following table outlines the design of the evaluation.
Table 2. Evaluation Design
Evaluation Question
Information Required
Information source
Method for Collecting Data
Analysis Procedures
Interpretation Procedures & Criteria
1. Does program participation affect science knowledge of the participants and how?
Pre/Post test scores from 1 st-3 rd grade participants
Students pre/post tests
Pre/Post Tests
Descriptive statistics (means, standard deviation), paired sample t-tests using SPSS
Determine if student’s science knowledge levels changed to a practical degree after attending Super Kids.
2. Does program participation affect the behavior development of the participants and how?
Parent survey responses from parents of participants
Parent surveys
Survey tally form
Content analysis, coding by hand
Determine if student’s behavior development in the areas of confidence and character improved by attending Super Kids.
3. Do Super Kids graduates attend UCA?
Admissions data from UCA Admissions Department
Banner Admissions database
Access report in Banner system
Content analysis, tally raw numbers
Determine if students who participated in Super Kids attend UCA.
Plan for Reporting Results
Three groups of audience members will receive the evaluation results—students and parents, teachers and staff, and the UCA/Conway community. Each audience group is interested in different aspects of the evaluation (Fitzpatrick et al., 2004). Students and parents are interested in science content learned, confidence and friendship building skills acquired, access to quality summer activity, and continuance of similar programs for their children after third grade. Teachers and staff are interested in enrollment data, science content learned, and confidence and friendship building skills acquired. The UCA and Conway community are interested in continued enrollment in Super Kids and other programs at UCA and the effects this program has on high school students choosing UCA for college.
The evaluation report should include the program description, context, purposes, procedures, and findings. The Program Director is involved in the process from the beginning in order to increase the likelihood that the evaluation plan will be used. Dr. Stephanie Vanderslice, Assistant Professor of Writing at UCA, will serve as an impartial reviewer of the evaluation reports. Graduate students in the Statistics Department will review the data analysis for errors. The admissions personnel and the program evaluator will both access the Banner data to ensure accuracy. Table 3 outlines the plan for reporting results related to the identified evaluation questions.
Table 3. Reporting Plan
Evaluation Question
Audience
Content
Format
Schedule
Context
1. Does program participation affect science knowledge of the participants and how?
Super Kids students, parents, elementary schools, Program Director, teachers, staff
Student pre-and post-test scores and data analysis
Colorful bar graphs in evaluation report, slide show
At end of each summer session
Presented at fall staff meeting (Oct), interim & executive summary / full report, article in local and UCA newspapers
2. Does program participation affect the behavior development of the participants and how?
Super Kids students, parents, elementary schools, Program Director, teachers, staff
Parent survey
Results reported in presentation of evaluation results, use of color, simple format, easy to read
At end of each summer session
Mailed to participants, presented at staff meeting, included in executive summary and full report,
3. Do Super Kids graduates attend UCA?
UCA Depts: (Admissions, Retention, Early Childhood Educ.) Conway, Faulkner Co,. Program Dir.
Admissions reports and applications
Bar graph from Admission Department
Each fall, after day 10 of session
Presented at Super Kids, Admissions, & Recruitment staff meetings, interim, executive summary / full report, article in local and UCA newspapers
Management Plan/Schedule
Positive student outcomes are the focus of everyone involved in Super Kids. Periodic communication and monitoring is part of the process, so no one feels uninformed. Examples of how the evaluation results might be useful will be provided at the beginning of the evaluation and suggestions for implementation will be provided in the full report. Examination of the results will help identify the strengths as well as the problem areas so they can be addressed and thus strengthen the impact of the evaluation.
Full disclosure of all findings will be provided in writing with oral explanation to stakeholders. Dr. Cooper, the Super Kids staff, Admissions and Recruitment staff, and parents are encouraged to provide feedback on the interim reports. The following management plan outlines the tasks, time lines, personnel responsible, resources required, and the cost of each task. Appendix G contains the proposed timeline in the form of a Gantt chart.
Table 4. Management Plan / Schedule
Evaluation Question
Tasks
Estimated Task Beginning and Ending Dates
Personnel Involved and Estimated Costs
Other Resources Needed and Costs
Total Task Cost
1. Does program participation affect science knowledge of the participants and how?
1.a. Obtain IRB approval (Appendix I)
1.b. Administer student pre-tests on first day of Super Kids X 4 weeks.
1.c. Administer student post-tests to students on last day of Super Kids X 4 weeks; conduct teacher/staff interviews.
1.d. Collect and code student pre-test responses.
1.e. Enter data into SPSS and run data analysis.
1.f. Prepare report of results.
1.a. March
1.b. Each Monday in June during Super Kids program.
1.c. Each Friday in June during Super Kids program.
1.d. First week of July.
1.e. First and second weeks of July.
1.f. ~August-September
1.a. Evaluator, 2 hours paperwork time @ $100 per 8-hour day = $25.00
1.b. Student teachers and MSE candidates, 30 minutes @ no pay = $0.00
1.c. Student teachers and MSE candidates, 30 minutes @ no pay = $0.00; Evaluator, 4 hours @ $100 per day = $50.00
1.d. Evaluator, 1 day @ $100 per day = $100.00
1.e. Evaluator, 1 day @ $100 per day = $100.00; graduate students @ no pay = $0.00
1.f. Evaluator, 2 days @ $100 per day = $200.00
1.a. none
1.b. photocopier—part of Early Childhood Ed. Department at UCA
1.c. photocopier
1.d. none
1.e. SPSS software, evaluator already has
1.f. computer, Word, PowerPoint software, evaluator already has
1.a. $25.00
1.b. $0.00 for teacher/MSE candidates
1.c. $0.00 for teacher/MSE candidates; $50.00 for Evaluator
1.d. $100.00
1.e. $100.00
1.f. $200.00
2. Does program participation affect the behavior development of the participants and how?
2.a. Send parent survey to prospective parents with enrollment packet
2.b. Collect and code parent responses
2.c. Prepare report of results (teacher/staff interviews address in #1)
2.a. ~ Spring Break
2.b. First week of July
2.c. August-September
2.a. Super Kids Program secretary, 4 hours @ $10.50 an hour = $42.00 included in previous mail-out
2.b. Evaluator, 4 hours @ $100 per day = $50.00
2.c. Evaluator, 1 day @ $100 per day = $100.00
2. a. photocopier—part of Early Childhood Ed. Department at UCA
2.b. none
2.c. computer, Word, PowerPoint software
2. a. none additional
2.b.$50.00
2.c. $100.00
3. Do Super Kids graduates attend UCA?
3.a. Add line items to admissions information to designate Super Kids, UCA Challenge, and MSI Program attendance as a child
3.b. Access admissions information through Banner system
3.c. Prepare report of results
3.a. July (in anticipation of fall semester)
3.b. August, after 10 th day of class to ensure actual and accurate enrollment data
3.c. ~August-September
3.a. Admissions personnel and Evaluator, ½ day @ $10.50 per hour for admissions personnel; $100.00 per day for evaluator = $42.00 for admissions personnel; $50.00 for evaluator
3.b. Admissions personnel, 1 hour @ $10.50 an hour = $10.50
3.c. Evaluator, 1 day @ $100 per day = $100.00
3.a. Access to Banner System at UCA—allowed to Admissions Personnel and Evaluator
3.b. none
3.c. computer, Word, PowerPoint software
3.a. $92.00
3.b. $10.50
3.c. $100.00
4. Program Evaluation Reports
1.a. Prepare interim reports, executive summary, full report, and PowerPoint report
1.b. Present reports
1.c. Write articles
1.a. July-October
1.b. July & September interim reports; October—full report (fall planning meeting)
1.c. October
1.a. Evaluator, 2 days @ $100 per day = $200
1.b. Evaluator, 3 hours @ $100 per day = $37.50
1.c. Evaluator, 2 hours @$100 per day = $25.00
1.a. computer, Word, PowerPoint software
1.b. laptop, projector (UCA has)
1.c. computer, Word, PowerPoint software
1.a. $200.00
1.b. $37.50
1.c. $25.00
Proposed Evaluation Budget
The following budget (Table 5) will be reviewed and approved by Dr. Cooper before the program evaluation begins. Appendix H contains the narrative explanation of each budget item.
Table 5. Proposed Evaluation Budget
Category
Year 1 Budget
PERSONNEL/STAFFING
Salaries:
Evaluator
$ 1037.50
Super Kids Secretary
$ 42.00
Admissions Personnel
$ 52.50
Student teachers
$ 0.00
MSE candidates
$ 0.00
CONSULTANTS
Graduate Students
$ 0.00
Writing Center
$ 0.00
TRAVEL/PER DIEM
$ 0.00
COMMUNICATIONS
$ 50.00
OPERATIONS
Printing / Duplication
$ 50.00
Data Processing
$ 0.00
Printed Materials
$ 100.00
Supplies / Equipment
$ 20.00
OVERHEAD
$ 486.76
TOTAL
$ 1838.76
(Horn, 2001).
Adherence to Program Evaluation Standards
The Joint Committee on Standards for Educational Evaluation identifies 30 standards for program evaluation divided into four categories. The four categories are utility, feasibility, propriety, and accuracy (The Joint Committee on Standards for Educational Evaluation, 1994). Appendix I provides an outline of each standard addressed in the program evaluation. Using these standards clarify the purpose of the evaluation by maintaining context and focus on the stakeholders’ needs.
Meta-evaluation is important to increase the understanding of the evaluation plan and the credibility of the evaluation to stakeholders. It provides reassurance that the decisions made based on the program evaluation are credible (The Joint Committee on Standards for Educational Evaluation, 1994). The evaluator, program director, staff, and teachers should be involved in the meta-evaluation. Appendix J provides a checklist (Shepard, 1977) for conducting a meta-evaluation of the Super Kids Program evaluation plan.
Summary, Conclusion, and Reflection
The purpose of program evaluation is to prove the worth, value, or merit of a program. This evaluation of the UCA Super Kids Program utilized an objectives-based approach to determine achievement of program objectives. This approach was utilized to answer specific guiding questions about how well the program is working to achieve desired outcomes. This approach is also organized and systematic.
The objectives of the Super Kids Program serve as the standards or criteria for the program evaluation. Guiding questions emerge from the program objectives. Relevant information to be collected includes the student pre-tests and post-tests, teacher and staff interviews, parent surveys, and existing information. The evaluation plan facilitates application of the thirty Joint Committee’s Standards for Educational Evaluation and data analysis for documentation of achievement of program objectives.
For the conclusions of the program evaluation to be justified they must relate to the guiding questions and be fully reported to the stakeholders. Care must be taken for all conclusions to accurately, logically, and adequately reflect the results of the evaluation, including limitations.
One last thought to keep in mind is the concept of practical change may differ from statistically significant change. When data analysis is complete, all areas may not show statistically significant change, but may indicate practical change, which is appropriate for a formative evaluation with the purpose of program improvement. The important question here concerns stakeholders’ ability to use the program evaluation results, including the data analysis, to make program improvement decisions.
References
American Evaluation Association. (2004). Guiding principles for evaluators. Retrieved June 2, 2008, from http://www.eval.org/Publications/GuidingPrinciples.asp .
Cox, J. (1996). Your opinion please!: How to build the best questionnaires in the field of education. Thousand Oaks, CA: Sage Publications.
Fitzpatrick, J. Sanders, J., & Worthen, B. (2004). Program evaluation: Alternative approaches and practical guidelines (3 rd ed.). Boston: Pearson Education, Inc.
Horn, J. (2001). A checklist for development and evaluating evaluation budgets. Retrieved June 2, 2008, from www.wmich.edu/evalctr/checklists/ .
Huba, M., & Freed, J. (2000). Learner-centered assessment on college campuses. NeedhamHeights, MA: Pearson Company.
Kaleidoscope Program. Retrieved July 31, 2008, from http://www.century.edu/continuinged/subjects/enrichment.aspx .
Kettner, P., Moroney, R., & Martin, L. (1999). Designing and managing programs (2 nd ed.). Thousand Oaks, CA: Sage Publications.
Kids U. Retrieved July 31, 2008, from http://www.sou.edu/youth/summer/kids_college.html .
Linn, R., Miller, M. D., & Gronlund, N. (2005). Measurement and assessment in teaching (9 th ed.). Upper Saddle River, NJ: Pearson Education Incorporated.
Lyubomirsky, S., King, L., & Diener, E. (2005). The benefits of frequent positive affect: Does happiness lead to success? Psychological Bulletin, 131(6), 803-855.
McLaughlin, J. (2003). Planning useful evaluation systems: Worksheets. Retrieved June 3, 2008, from [email protected] .
McLaughlin, J., & Jordan, G. (1998). Logic models: A tool for telling your program’s performance story. Retrieved June 2, 2008, from http://www.pmn.net/library/Logic.htm .
Patten, M. (2007). Understanding research methods (6 th ed.). Glendale, CA: Pyrczak Publishing.
Personal Communication. Mark Cooper, Super Kids Program Director. June 8, 2008.
Shepard, L. (1977). A checklist for evaluating large-scale assessment programs. Retrieved July 22, 2008 from http://wmich.edu/evalctr/checklists/assessment_eval.pdf .
Stockburger, D. (2008). Introductory statistics: Concepts, models, and applications (2 nd ed.).Mason, OH: Thomson Corporation.
Super Kids Brochure. (2006). University of Central Arkansas, College of Education, Department of Early Childhood and Special Education.
Super Kids Brochure. (2007). University of Central Arkansas, College of Education, Department of Early Childhood and Special Education.
Super Kids Brochure. (2008). University of Central Arkansas, College of Education, Department of Early Childhood and Special Education.
The 14 th Annual Mental Measurements Yearbook. (2001). Psychological Reports, Retrieved July 24, 2008, from Academic Search Elite database.
The Children’s Creative Learning Center. Retrieved July 31, 2008, from http://www.ed.uab.edu/cclc/ .
The Joint Committee on Standards for Educational Evaluation. J. R. Sanders, Chair. (1994).The program evaluation standards: How to assess evaluation of educational programs. (2 nd ed.). Thousand Oaks, CA: Sage Publications.
Young Artists Summer Program. Retrieved July 20, 2008, from http://www.sl.edu/outreach.yes.htm .
Young Child Program. Retrieved July 20, 2008, from http://www.unco.edu/cebs/sep/young_child/index.html .
Appendix A
Description of Super Kids Content
Young Ornithologists: Birds of a Feather
Come join our flock! You will get more than a bird’s eye view of our feathered friends. With binoculars and guidebooks in hand, we will observe beaks and bills, legs and eggs, and feathers and feet. Our birds eat, move, communicate, and care for themselves. Our creativity will soar as we build feeders, design nests for nibbling, and experiment with flying. We are sure to hatch several feathered young ornithologists.
Young Medical Scientists
It’s never too early to discover the next Dr. Lee Salk. Scrub-up and get prepared for an exciting experience in a medical laboratory. Come and find out what it is like to be the doctor rather than the patient. Action packed, hands-on exploration in a wide variety of medical fields is a promise. Students will definitely feel like medical scientists!
Young Paleontologists: Dinosaur Galore!
Dinosaurs may be out-of-sight, but they are not out-of-mind! The exploration and investigation of dinosaurs is alive, well, and growing. New dinosaur discoveries are constantly changing the way we think about their appearance, movement, and behavior. Here is an opportunity for your young paleontologist to learn more about these “terrible-lizard friends” and have a “dino-mite” time!
Young Botanists and the Plant World
Come join our garden paradise while sprouting new ideas about the plant world. Budding botanists will get down and dirty in a classroom crawling with vines. As the sun shines on each new day, our thumbs will grow greener with every investigation about the friends and foes of the plan world. Our creativity will blossom as students’ plan their feet in a green house built for Super Kids!
Young Ecologists and the Rainforest
Welcome to the sounds and sights of the jungle! Come explore the rainforests around the world where the temperature hardly changes from day to day, season to season, and year to year. Learn how rainforests affect our air, weather, and lives. Grab your cameras and gear to come join the adventure. The rainforest will come alive with exotic creatures that beg to meet new friends.
Young Oceanographers and the Ocean
Super Kids will enjoy the sparkling water, exotic fish, colorful coral, sandy beach, and warm sun. Get ready to explore the underwater kingdom. The young oceanographers will investigate what makes the ocean salty, where the water comes from, ocean creatures, and the survival characteristics and habits of ocean creatures. Pack your scuba gear and prepare for the dive into an ocean of new ideas and excitement.
Young Astronauts and Outer Space
Blast off to an exciting study of the planets, space travels, space stations and many other space related topics. Students will step into a space laboratory once they enter the classroom. This futuristic investigation of space will spark the imagination of every student. It should come as no surprise when a future astronaut is discovered from this experience!
Young Biologists and Planet Earth
Come join our animal friends and have egg-ceptional fun discovering and witnessing the metamorphosis of all kind of creatures that hatch from eggs. Our young biologists will experience firsthand the sounds and sights of toads, frogs, birds, baby chicks, and other animal surprises. Students will get their feet wet and hands dirty exploring the egg-friendly habitats of our animal buddies.
Young Explorers and the Arctic
Cool off from the summer heat and join the young explorers to the vast regions of the North and South Poles. Students will meet new friends—polar bears, penguins, seals, and walruses. Our young explorers will learn how these animals, and even humans, survive the coldest parts of the world. Earmuffs and gloves are needed for this “chilly” experience!
Appendix B
Summary of Similar Programs
Name of Program
College or University
Content Taught
Age of Attendees
Cost
Children’s Creative Learning Center
University of Alabama at Birmingham
All curricular areas
Ages 7-9
$125 per week
Kaleidoscope
Century College
White Bear Lake, MN
Art, science, math
Ages 3-16
$55.00 per class
Kids U
Southern Oregon University, Ashland, OR
Art, science, tennis, fencing, swimming
Ages 8-12
$99 per class
Super Kids
University of Central Arkansas, Conway, AR
Science, character, confidence
Grades 1-3
(ages 6-10)
$145 per week
Young Artists Summer Program
Silver Lake College, Manitowoc, WI
Art
Grades 1-9
(ages 6-14)
$40 per class
Young Child Program
University of Northern Colorado
Visual and performing arts, math, science, language, literature
Ages 4 to 4 th grade
$185 for a two-week session
Appendix C
Teacher and Staff Interview Guide and Tally Form
Below is a list of guiding questions dealing with the preparation of teachers and staff for the UCA Super Kids Program. These questions deal with issues that are important to the organization and delivery of that program. Other questions may be pursued as the interview progresses.
- What is your responsibility or area of focus with the Super Kids Program?
- Describe your preparation for teaching or working with the Super Kids Program.
- Describe the science curriculum used in the program. How is it adapted to make it grade/age-appropriate?
- Describe the behavior development portion of the program. What is the theory base? Describe some of the behavior development activities.
- What else would you like to express concerning your involvement with the Super Kids program?
Interview Question
Response Code
Tally
1. Responsibility
Teacher—room
Staff—duties
Other
2. Preparation
Lesson plans
Activity preparation
Mail-outs
Room decorating/preparation
T-shirt orders
Copying
Reserve room for Closing Ceremony
Other
3. Science
Curriculum
Grade/age-appropriate
Other
4. Behavior Development
Theory base
Friendship Activities
Confidence Activities
Other
5. Other issues or topics identified
Other
Other
Appendix D
Example Student Pre- and Post-Test
Name: ________________________________________
Classroom: Young Ecologists—Biology Room, Oviparous Animals
Circle the answer which best answers the question:
- Which animal comes from an egg?
- Human
- Squirrel
- Chicken
- Bear
- Which of these choices describes a reptile?
- Warm-blooded
- Lay eggs on land
- Has fur
- Has long ears
- Which of these birds can fly for long distances?
- Penguin
- Ostrich
- Flamingo
- Cardinal
- How many weeks does it take a tadpole to turn into a frog?
- 2 weeks
- 8 weeks
- 12 weeks
- 16 weeks
- A trout is a
- Reptile
- Bird
- Fish
- Human
Student Pre-Test / Post-Test Data Analysis Procedure :
Create a data file in Notepad or Excel and imported into SPSS with the File à Read Text Data command. Follow these steps:
- Text file does not match a predetermined format
- Variable are delimited by a specific character
- Variable names are included at the top of each file, with a space, no text qualifier, and cache data locally
- Code each possible answer as correct (“1”) or not correct (“0”) and entered into the SPSS data file using the variable view.
- Add value labels as follows:
- Subjects: 1-20 for each student in the class
- Grade: entering first grade = 1, entering second grade = 2, entering third grade = 3
- Egg: human = 0, squirrel = 0, chicken = 1, bear = 0
- Reptile: warm = 0 eggsland = 1, fur = 0, ears = 0
- Fly: penguins = 0, ostrich = 0, flamingo = 0, cardinals = 1
- Frogweeks: 2=0, 8=0, 12=0, 16 = 1
- Trout: reptile = 0, bird = 0, fish = 1, human = 0
- Create Pretest and Posttest sum scores for items 5.c-5.g.
- Click Save.
- To analyze the data using SPSS version 15.0 click Analyze > Compare Means > Paired Samples T-Test (Stockburger, 2008).
Appendix E
Parent Survey
Parents: It is very important to the teachers and staff of UCA Super Kids that parents have a chance to tell how they feel about what is being taught to their children. Below are some statements to help us get needed information concerning the Super Kids Program. Please answer exactly the way you feel. You are not asked to put your name on this paper or on the return envelope that is provided. Your answers will be grouped with those of other parents to give an overall picture of how parents feel. Thank-you for taking your time to provide the needed information, and for helping to improve the Super Kids Program. Please return the survey by the end of the week that your child attends Super Kids.
Read each statement listed below and check the box that comes closest to your feelings.
Definitely
YesGenerally
YesGenerally
NoDefinitely
NoThe program blends well with science content taught in my child’s elementary school.
The character building lessons support what I teach my child at home.
I observed my child exhibiting good character during or after attending Super Kids.
The confidence building lessons support what I teach my child at home.
I observe my child exhibiting confidence in his/her abilities during or after attending Super Kids.
My child has made developed friendship building skills (e.g. self-control, problem-solving) at Super Kids.
My child feels comfortable at the UCA Child Study Center facility.
My child has expressed a desire to attend UCA Challenge.
My overall feeling of the Super Kids program is positive.
Please write any additional comments you would like the teachers and staff to know:
Appendix F
Evaluation Agreement
TO: Dr. Mark Cooper, Program Director of UCA Super Kids
FROM: Nina Roofe, Program Evaluator
RE: Evaluation of UCA Super Kids Program
DATE: _____________
I am happy to confirm acceptance of our agreement to conduct a program evaluation for the UCA Super Kids Program. As per our previous discussion, you (Program Director) will provide the previously requested documents and access to personnel during June and July, and I (Evaluator) will provide the evaluation as outlined in the management plan for the agreed on price of $1838.76 distributed as outlined in the budget.
This price includes presentation of the executive summary, slide show, and full report at the fall planning meeting in October 20__. Please indicate your formal acceptance of this plan by signing below. Retain one copy for yourself and return the original to me.
I look forward to working with you.
__________________________________ ________________
Dr. Mark Cooper, Program Director, UCA Super Kids Date
_________________________________ ________________
Nina Roofe, Program Evaluator Date
Appendix G
Gantt Timeline
Appendix H
Budget Narrative
Personnel/Staffing
- $1037.50 is requested for Nina Roofe to serve as the Program Evaluator; this is based on estimated hourly rate of pay and going consulting fees of $100.00 per 8-hour work day
- $42.00 is requested for the Super Kids Program secretary; this is based on estimated hourly rate of pay of $10.50 per hour X number hours estimated she will provide to the evaluation component of this program
- $52.50 is requested for Admissions Personnel; this is based on estimated hourly rate of pay of $10.50 per hour X number hours estimated he/she will provide to the evaluation component of this program
- $0.00 is requested for student teachers/MSE candidates as this provides a learning opportunity and/or required part of their educational program
Consultants
- $0.00 is requested for the graduate students in statistics as he/she is paid by the University
- $0.00 is requested for the Writing Center Director as she provided review as a service to the University
Travel/Per Diem
- $0.00 is requested for travel or per diem as all parties currently work at UCA and would not incur additional travel or meal costs as a result of participating in the evaluation of Super Kids
Communications
- $50.00 is requested for postage and phone use; this is based on an estimated mail out of 100 enrollment packets at the corporate rate of 0.40 cents per packet plus an additional $10 for estimated phone use
Operations
- $50 is requested for copying of parent surveys, student pre-and post-tests, and interview guides
- $0 is requested for data processing as this is already present with the evaluator
- $100 is requested for printing of the executive summary, full report, and PowerPoint presentation handouts
- $20 is requested for miscellaneous office supplies and equipment usage
Overhead
- $486.76 is requested for overhead, which is 43% of the salaries only; percentage set by UCA Budget Office, spoke with Terrie Camino who explained that indirect cost at UCA is calculated at 43% of salaries only, not including fringe benefits or other costs.
Total
- $1838.76 is requested for the evaluation of the UCA Super Kids Program
Appendix I
Evaluation Standards Checklist
Standard addressed
Standard partially addressed
Standard not addressed
Standard not applicable
Page(s)
U1 Stakeholder Identification
X
5
U2 Evaluator Credibility
X
8
U3 Information Scope and Selection
X
10-12
U4 Values Identification
X
9
U5 Report Clarity
X
13-14
U6 Report Timeliness and Dissemination
X
13-14
U7 Evaluation Impact
X
15-18
F1 Practical Procedures
X
15-18
F2 Political Viability
X
9
F3 Cost Effectiveness
X
19, 46-47
P1 Service Orientation
X
4
P2 Formal Agreements
X
44
P3 Rights of Human Subjects
X
27-35
P4 Human Interactions
X
27-35
P5 Complete and Fair Assessment
X
10-12,
15-18P6 Disclosure of Findings
X
15-18
P7 Conflict of Interest
X
8
P8 Fiscal Responsibility
X
19, 46-47
A1 Program Documentation
X
4-6
A2 Context Analysis
X
10-12
A3 Described Purposes and Procedures
X
7-12
A4 Defensible Information Sources
X
11, 13-14
A5 Valid Information
X
10-14
A6 Reliable Information
X
10-14
A7 Systematic Information
X
13-14
A8 Analysis of Quantitative Information
X
10-12
A9 Analysis of Qualitative Information
X
10-12
A10 Justified Conclusions
X
21
A11 Impartial Reporting
X
13-14
A12 Meta-evaluation
X
20, 50-51
(The Joint Committee on Standards for Educational Evaluation, 1994)
Appendix J
Meta-Evaluation Checklist
Evaluator’s Name: ____________________________________
Date: _____________
Questions to Ask
Answer
Comments
1. Is the purpose of the evaluation clearly defined and accurate?
2. Is the scope of the evaluation clearly defined and accurate?
3. Does the design and data analysis answer the question(s) it was intended to answer?
4. Do the assessment results have the desired generalizabilty?
5. Are the evaluation data accurate and consistent?
6. Would other competent assessors agree on the conclusion of the evaluation?
7. Are the stakeholders accurately identified?
8. Are the target audiences accurately identified?
9. Are the findings relevant to the audiences of the program?
10. Do the reports (executive summary, full report, newspaper articles) accurately convey the results of the program evaluation?
11. Have the most important and significant data been included in the assessment?
12. Do the audiences view the assessment as valid and unbiased?
13. Are the results provides to audiences in a timely manner?
14. Are the results disseminated to all intended audiences?
15. Is the program cost-effective in achieving the assessment results?
16. Is the management plan and schedule realistic and inclusive of all needed elements?
17. Is the budget accurate and appropriate?
18. Do the Joint Committee Standards apply to this evaluation?
(adapted from Shepard, 1977)
Kappa Omicron Nu Forum Volume 20 No. 1
Consumer Moral Ambiguity: The Gray Area of Consumption
Sue L. T. McGregor
Peer Review: A Filter for Quality
Dorothy I. Mitstifer
Mentoring Students in Cross-Specialization Teams
Dorothy I. Mitstifer
Consumerism as a Source of Structural Violence
Sue L. T. McGregor
Consumer Entitlement, Narcissism, and Immoral Consumption
Sue L. T. McGregor
A Satire: Confessions of Recovering Home Economists
Sue L. T. McGregor
The Nature of Transdisciplinary Research and Practice
Sue L. T. McGregor
Reflection Matters: Connecting Theory to Practice in Service Learning Courses
Mary E. Henry
What's It All About—Learning in the Human Sciences
Dorothy I. Mitstifer
Leadership Responsibilities of Professionals
Dorothy I. Mitstifer
Categories of Sexual Harassment: A Preliminary Analysis
Catherine Amoroso Leslie, William E. Hauck
Knowledge Management / Keeping the Edge
Dorothy I. Mitstifer
Super Kids Program Evaluation Plan
Nina L. Roofe
Nina L. Roofe
The Wilberian Integral Approach
Sue L. T. McGregor