ECUR 809 Final Assignment - Evaluation Proposal

The culminating activity for every good course is a project.

Evaluation Proposal for the Youth Educational Services Program at LA Matheson Secondary

YES Program Survey

Rational:

This survey was designed to be included in my YES program evaluation proposal.  I didn't want to make the survey too long as teachers time is very valuable and a survey that is too lengthy will reduce completion rates.  Throughout the revision process, the paper survey was kept to under two pages or one page printed front to back.      

The original survey was sent to four teachers to be pilot tested.  Two teachers responded (spring break may have been a factor) but I am still waiting for additional responses. As an interim solution, the survey was also pilot tested by one engineer and one home-maker. 

Some options for delivery of the survey include email, school mailbox, in person, or at a staff meeting. 

Teachers feel more open and able to give honest feedback if the survey is anonymous; therefore, the requested demographic information was kept minimal.  The question regarding number of teaching years will help the YES staff to see if early service teachers need more support in understanding the program and the benefits of collaboration. After testing the survey, this question was made categorical to make it easier to answer as some teachers forgot the number of years they have been in the profession and to further protect identity. This question was moved from the beginning of the survey as it did not catch the interest of the respondents. 

The final version is as follows:

How long have you been teaching?

 0-5 years

 5-10 years

10+ years

The question regarding the knowledge of the number of students in the classroom receiving support caused a number of participants to stumble.  The question required to much precision and one respondent answered that they did not know.  Another respondent suggested that this question excluded non-enrolling teachers and counsellors.  As the intent of the question was to target the awareness of students receiving support from the YES program and the communication that surrounds that awareness, the question was changed to a categorical multiple choice with the option of identifying as a non-enrolling teacher. 

The final version is as follows:

How many students, that you are aware of, in your classes this year, receive support from the YES program?

ÿ                                            None

ÿ                                           1-2

ÿ                                           3-5

ÿ                                           6+

ÿ                                           I don’t know

ÿ                                          I do not have any enrolling classes

In an effort to include the non-enrolling teachers’ responses to all questions in the survey, the wording was changed for questions that referred to “students in your class”

Question number ten in the survey, asking participants if they agree that the YES program helps students stay in school was poorly worded.  Some people viewed that as a fact, as it is the overarching goal of the program.  The intent behind the question was to see if there was a correlation between collaboration and the respondent’s perception of the efficacy of the program.  The question was reworded as follows:

I have seen the YES program help vulnerable students stay in school.  

   

A short thank you was added to the end of the paper version of the survey. 

The final copy can be viewed in a paper (pdf) or electronic format.

ECUR 809 Assignments #4

Logic Model: Youth Educational Support Program

In inner city schools, subgroups of the student population are “vulnerable” to failure or “at-risk” of dropping out of school.  It is recognized that students come to school with non-academic problems that affect their learning these include hunger, social emotional issues, and conflicts at home; many of these students lack the resources to address their non-academic needs.  The Youth Education Support Program (YES) was developed to help fulfil this need for students in the Surrey School District. YES provides services to students in the setting of the high school that they are attending.  The aim of the YES program is to keep vulnerable youth in school through relationship building.  
  

The four goals of the YES program as outlined in their program plan are as follows:

·         To make students aware of their responsibilities to others, themselves and their school

·         To facilitate meaningful connections with peers, adults, the school and the community

·         To keep vulnerable students in school and improve student behavior

·         To build cooperative partners with teaching, counselling and administration staff in order to improve student achievement and attitude in the classroom

The YES program began more than ten years ago out of a partnership between Henry D. Sheldon High School in Eugene, Oregon and Princess Margaret Secondary in Surrey, BC. At that time YES was a pilot of an interagency model that was designed to connect practicum students from local post-secondary institutions with a high school site for volunteer experience and field study.  The use of the practicum students in the program delivery helps to keep the cost of the program low.  The YES program has continued to evolve since its conception and is currently being offered at 5 inner city schools in Surrey. 

The YES program is delivered out of a portable situated on the school property.  Before school, at lunch, and after school students can drop in to the portable where they connect with the YES coordinator or practicum students.  Teachers, counsellors and administers can refer students to the program; the YES staff then initiates a connection with the student at some point in the school day and works to develop a relationship with the student through various activities over time.  The YES staff facilitates many activities at the school, lunch programs, special events, community referrals, drop in games, and one-to-one supports. 

This evaluation will focus on the effectiveness of the collaboration between the YES staff and teachers at the LA Matheson site.  Is this partnership helping students experience success and ultimately helping them to stay in school?  How can the collaborative partnership be improved?  
  

Logic Model (PDF file)

ECUR 809 Assignment #3

Youth Educational Support Program: Evaluation Planning Assessment

In inner city schools, subgroups of the student population are “vulnerable” to failure or “at-risk” of dropping out of school.  It is recognized that students come to school with non-academic problems that affect their learning these include hunger, social emotional issues, and conflicts at home; many of these students lack the resources to address their non-academic needs.  The Youth Education Support Program (YES) was developed to help fulfil this need for students in the Surrey School District. YES provides services to students in the setting of the high school that they are attending.  The aim of the YES program is to keep vulnerable youth in school through relationship building.    

The four goals of the YES program as outlined in their program plan are as follows:

·         To make students aware of their responsibilities to others, themselves and their school

·         To facilitate meaningful connections with peers, adults, the school and the community

·         To keep vulnerable students in school and improve student behaviour

·         To build cooperative partners with teaching, counselling and administration staff in order to improve student achievement and attitude in the classroom

The YES program began more than ten years ago out of a partnership between Henry D. Sheldon High School in Eugene, Oregon and Princess Margaret Secondary in Surrey, BC. At that time YES was a pilot of an inter-agency model that was designed to connect practicum students from local post-secondary institutions with a high school site for volunteer experience and field study.  The use of the practicum students in the program delivery helps to keep the cost of the program low.  The YES program has continued to evolve since its conception and is currently being offered at 5 inner city schools in Surrey. 

The YES program is delivered out of a portable situated on the school property.  Before school, at lunch, and after school students can drop in to the portable where they connect with the YES coordinator or practicum students.  Teachers, counsellors and administers can refer students to the program; the YES staff then initiates a connection with the student at some point in the school day and works to develop a relationship with the student through various activities over time.  The YES staff facilitates many activities at the school, lunch programs, special events, community referrals, drop in games, and one-to-one supports. 

This assignment will focus on developing an evaluation design for the LA Matheson site.  According to the program coordinator, the program has never been formally evaluated.   

Planning a Program Evaluation: Worksheet

University of Wisconsin-Extension • Cooperative Extension • Program Development & Evaluation

UW-Extension provides equal opportunities in employment and programming, including Title IX and ADA.

© 2006 by the Board of Regents of the University of Wisconsin System                    http://www.uwex.edu/ces/pdande

Steps in Program Evaluation

Manage the evaluation § Human Subjects Protection § Timeline § Responsibilities § Budget		1. Engage stakeholders
		2. Focus
		§ Describe program-logic model § Define purpose § Determine use/users § Determine key questions § Select indicators § Determine design	3. Collect data
			§ Identify sources § Select method(s) § Pilot test § Set schedule § Determine sample	4. Analyze & interpret
				§ Process data § Analyze § Interpret data § What did you learn? § What are the limitations?	5. Use
					§ Share findings and lessons learned § Use in decision making § Determine next steps
Standards of evaluation: · Utility     · Feasibility     · Propriety     ·Accuracy

          

Engage Stakeholders

Who should be involved?

Program Coordinator, Practicum students, Teachers/Counsellors, School Administration, Students, School Board Members

How might they be engaged?

The stakeholders, namely the program coordinator, practicum students, teachers, counsellors, and administration will be engaged early on in the evaluation through discussions, conversations and emails.  Working closely with the program coordinator will help to ensure that the evaluation targets the right areas and that the results will be utilized.  The teachers, counsellors and administration will be participants in the data collection process; they will help steer the evaluator to the means of data collection that will best suit their needs.

          

Focus the Evaluation

What are you going to evaluate?  Describe program (logic model).

See logic model (Assignment #4) – The evaluation will focus on the effectiveness of the cooperative relationships formed between YES support staff and other staff at the school in improving student achievement and attitude

What is the purpose of the evaluation?

The purpose of the evaluation is to provide formative feedback to improve the program delivery and increase the effectiveness of the program activities towards the goals set out.  As a part of the evaluation process, the evaluator will work with the program staff to develop systems for information management that can have long-term use.  This evaluation will examine the collaboration between the YES staff and other school staff. One of the secondary purposes of the evaluation process will be the development of a deeper understanding of the program by teachers and funding stakeholders such as the board members.

Who will use the evaluation? How will they use it?

Who/users	How will they use the information?
Program Coordinator	The information obtained will help the program coordinator improve the delivery of the program activities.
Teachers/Counsellors/Administrators	The information obtained will help these stakeholders make better use of the service.
School Board Members	The information obtained will help the funders make decisions about the allocation of funds to the program

What questions will the evaluation seek to answer?

How can the communication between YES staff and teachers be improved?

Sub questions: How do the YES staff communicate with teachers, counsellors and administration? Is the communication between the YES staff and the teacher, counsellors, and administration effective? What percentage of teachers is in collaboration with the YES staff?

Is the collaboration between teachers and YES staff helping to meet the at risk students’ needs?

Sub questions: Which students are participating in the YES program?  What are the characteristics of an “at risk” student? What are the students’ needs? Who else is supporting the students’ non-academic needs?

What information do you need to answer the questions?

What I wish to know	Indicators – How will I know it?
How can the communication between YES staff and teachers be improved?	Qualitative Data - Observations, Focus group discussions Quantitative Data - Surveys completed by the teachers Look for possible alternative/additional methods of communication
Is the collaboration between teachers and YES staff helping to meet the students’ needs?	Qualitative Data - Observations, Case studies, Interviews Quantitative Data - Surveys completed by the students, students marks pre-YES intervention and students marks post intervention  Define students’ needs and examine if the collaboration between teachers and YES staff provides for those needs Needs being met could be demonstrated by - increased attentiveness in class - improved academic standing - improved classroom behaviours Observations of students before YES support and after YES support

When is the evaluation needed?

This particular evaluation will need to be completed by the end of the current school year but one of the goals of the evaluation is to provide a means of building data collection for on-going formative evaluation into the program.  This might look like a school based review every two years, overseen by an outside evaluator in a supporting role.

What evaluation design will you use?

An interactive, participant oriented evaluation approach will be used.  Strategies and theories from both responsive and monitoring evaluation models will be applied.

          

Collect the information

 

What sources of information will you use?

Existing information:	Very little formal data exists.  Part of the purpose of the evaluation will be to help the program staff set up an information management system.  There is currently a collection of surveys filled out by former practicum students providing an evaluation of their supervision by the YES coordinator and their experience in the program.
People:	Interviews, surveys and focus groups will be the primary means of data collection.
Pictorial records and observations:	The evaluator will directly observe the program activities and perform case studies of the participants.

What data collection method(s) will you use?

x	Survey	x	Document review
x	Interview		Testimonials
x	Observation		Expert panel
	Group techniques		Simulated problems or situations
x	Case study		Journal, log, diary
	Tests		Unobtrusive measures
	Photos, videos		Other (list)

Instrumentation: What is needed to record the information?

Survey tools will need to be created

Information management system – track the students and list the services provided.

When will you collect data for each method you've chosen?

All data will be collected during the program, as the program does not have a definitive beginning or end. 

Will a sample be used? 

No	x
Yes		If yes, describe the procedure you will use.

          

Analyze and Interpret

How will the data be analysed?

Data analysis methods:	The majority of the data collected will be qualitative and based on observation a thematic model will be used.  Basic statistical analysis will be performed on any quantitative data.
Who responsible:	Evaluator in collaboration with the Program Coordinator

How will the information be interpreted — by whom?

The qualitative data will be summarized by the evaluator using a thematic model.

The information gained by the evaluation process will help the stakeholders to better understand the YES program, specifically, how the relationships developed between YES staff, teachers and counsellors helps students in the classroom.

What did you learn? What are the limitations?

As the evaluation progresses the evaluator will work closely with the stakeholders to determine their needs and steer the evaluation efforts; although this is one of the strengths of responsive evaluation it is also a limitation.

The qualitative data collected will provide an opportunity to gain new insights but it may also be viewed as subjective; the utility provided will outweigh this limitation.

The evaluation will rely heavily on qualitative data which has the potential to be time consuming.

               

Use the Information

How will the evaluation be communicated and shared?

To whom	When/where/how to present
YES coordinator, administration and counsellors	The evaluation results will be presented/discussed in a meeting.  The insights gained by the evaluator will be shared and a list of recommendations will be reviewed at this meeting.
Teachers	The evaluation recommendations pertaining to the teachers will be written in a brief report and emailed to teachers.
Board Members	Recommendations requiring funding approval will be communicated to the school board via the school administration – a written report including qualitative data will be provided.

Next steps?

YES coordinators, practicum students, administrators, teachers and counsellors will use the insights and recommendations to improve the quality of delivery for the program.

Manage the evaluation                                                Standards

x	Human subject’s protection	x	Utility – by design, the evaluation will encourage utility as it is participant oriented.  The involvement of the stakeholders will ensure the evaluation produces useful feedback.
x	Management chart	x	Feasibility – Valuable information can be obtained using a simple design. This will increase the feasibility of performing an evaluation.
x	Timeline	x	Propriety
x	Responsibilities – to be shared between the evaluator and program coordinator	x	Accuracy – Multiple data collection methods will ensure accuracy of the information obtained
x	Budget – The design will be simple to keep costs down.