Here is the Program Evaluation Plan for the iJob program at Nutana Collegiate. I would appreciate any feedback others might have for this evaluation plan - I would be happy to make some changes to improve it. Also, Jay suggested that I break up the participant survey into headings, which you will see I did for this evaluation plan. Please pass on any other suggestions. Thanks to everyone who has made this class a welcoming learning environment!
Sarah
CONTENTS
Introduction and Background ………………………………………………………………………………………… 3
Purpose………………………………………………………………………………………………………………………….. 3
Audiences………………………………………………………………………………………………………………………. 3
Decisions and Questions………………………………………………………………………………………………… 4
Methods…………………………………………………………………………………………………………………………. 5
Sample……………………………………………………………………………………………………………………………. 5
Instrumentation……………………………………………………………………………………………………………… 6
Limitations……………………………………………………………………………………………………………………… 7
Logistics and Timeline…………………………………………………………………………………………………….. 8
Budget……………………………………………………………………………………………………………………………. 9
Reference……………………………………………………………………………………………………………………….. 9
Appendix A: Pre-Training iJob Participant Survey……………………………………………………………. 10
Appendix B: Facilitator Interview Questions……………………………………………………………………. 12
Appendix C: Work Placement Host Interview Questions…………………………………………………. 13
Appendix D: iJob Participant Exit Survey………………………………………………………………………….. 14
Appendix E: iJob Post Program Interview………………………………………………………………………… 16
Introduction and Background
Client: iJob program facilitators
Evaluator: Sarah Loewen
This evaluation plan is for the iJob program at Nutana Collegiate. iJob is a program for youth seeking employment, yet who do not have the skills and experience to secure a job. The program is designed to provide training and an employment experience in order to give participants the skills and experience they require to be successful in the work world. Approximately 12-15 youth, ages 16-29 participate in quarterly sessions throughout the school year. The purpose of this evaluation is to evaluate whether or not the iJob program is meeting its objectives by successfully preparing youth for employment.
Purpose
The purpose of the evaluation is twofold. The first purpose is to determine if the program is successfully providing the participants with the skills and experience necessary to gain and maintain employment. This is the summative aspect of the evaluation. The question being answered is: does this program meet its goals? The second purpose of this evaluation is to determine what specific aspects of the program are successful and what aspects are in need of adjustment. This is the summative part of the evaluation and the information gathered will be used to inform improvements to the program.
Audiences
The primary audience of this evaluation are the 2.5 program facilitators so that they can determine what aspects of the program are effective and what aspects may require improvement. The program facilitators have a strong desire to see the iJob program be as successful as possible and therefore they are seeking external evaluation to help determine areas for improvement. The secondary audience, if the program facilitators choose to share the evaluation, is the program funders. The program is funded by CanSask, an employment branch of the government. The secondary audience may choose to use the evaluation to determine if the program is reaching its goals and ought to have funding renewed in subsequent years.
Decisions and Questions
Questions
The key question for this evaluation is: is the iJob program meeting its goal to prepare participant for the work world? In order to gain insight into whether or not the answer is yes, the following more detailed questions will be answered during this evaluation.
Formative:
1. What is the participants’ level of work-readiness and work knowledge prior to entering the program?
2. Are participants ready to work when they begin their work placements?
3. What are the specific areas of the program that are successful and what are specific areas of the program that could be improved?
4. How can these areas for improvement be changed to meet the needs of the participants more effectively?
Summative:
1. Do participants leave the program with a strong resume?
2. Do participants leave the program with strong interview skills?
3. Do participants leave the program with career goals?
4. Do participants leave the program with positive work habits?
5. Do participants leave the program with a stronger sense of self and future directions?
6. Do participants leave the program feeling ready to enter the work force?
7. Do participants leave the program informed about how to look for employment?
8. Do participants leave the program with knowledge on work safety?
9. Do participants leave the program with knowledge on their rights at work?
10. Does the level of knowledge and experience that the participants leave the program with represent a positive change from when they began the program, or do the participants who enter the program already have many of the skills that the iJob program seeks to teach prior to entering the program? In other words, what is the change in employment readiness that participants experience in the iJob program?
11. Do program participants benefit from their experience in iJob 3 months after the program?
12. Do participants have jobs 3 months after the program? Why or why not? (what barriers might they face?)
Decisions
Based on the program evaluation results, program facilitators may have to decide if there are areas of the program that they wish to change in order to improve the quality of the iJob program. Facilitators may also have to decide how to go about making changes in order to create the most effective model. It may also be determined that the program is most effective in its current format.
Methods
A Participatory, collaborative or empowerment style evaluation (Scriven, 2001) will be used for this evaluation. This means that the program facilitators will be engaged in the evaluation process, they are encouraged to participate fully and offer suggestions and feedback throughout the evaluation process.
The evaluation will be conducted using the following methods in this order:
1. Survey of all participants at start of program to determine level of skill and knowledge in employability skills upon entering the program.
2. Observation of the program overall and in particular how the participants respond to the training.
3. Interviews with program facilitators individually to determine areas of the program that facilitators feel are weak and areas they feel are strong and the reasons for their perceptions of these areas. Interviews are conducted individually to ensure confidentiality of facilitator comments.
4. Interviews with work placement hosts to determine if they feel the participants are ready for employment when they arrive to their work placements or if there are gaps in their readiness for the job.
5. Survey of participants at the end of the program to determine if the participants feel that they have acquired relevant skills and experience for entering the work force and to determine their level of satisfaction with the program overall.
6. Interview participants a few months after the program has ended to find out if the program has been beneficial to their employment status in the medium term.
This method will be attainable within the time constraints, budget and personnel available to conduct the evaluation.
Sample
Data will be collected from all fifteen participants in the fourth quarter of the 2010-2011 school year cohort. This quarter is from April 18 to June __. Data collection will occur on the following timelines:
Data Collected From | Method of Data Collection | Date | Sample Size |
iJob Participants | Initial Survey | April 19, 2011 | 15 participants |
iJob Participants | Observations | Various times near beginning and end of training | 15 participants |
Program Facilitators | Interviews | Week of May 2-6 | 3 Facilitators (2 full time and one part-time) |
Work Placement Hosts | Interviews | Week of May 24-27 | 3-5 Hosts |
iJob Participants | Exit Survey | June 20 | 15 participants (assuming all participants complete the program) |
iJob Participants | Interview | September 2011 | 2-3 participants |
Instrumentation
Instrument | Description | Rationale | Reliability/Validity? |
Initial Participant Survey Appendix A | Participants will be given an initial survey that will ask them about various aspects of work readiness. It is used to determine participant’s level of work readiness prior to beginning the iJob Program. | A survey is used initially because it is a simple way to collect clear data regarding participant skill and experience level at the start of the program. There is no complex feedback being sought, but rather a clear baseline of skills and experience is required at this stage. | There is a question of reliability for these surveys as participants are asked to self-assess. Participants may have divergent views on what is considered, for example, strong interview skills. This is why the survey will be used in conjunction with observations. |
Observation of Participant Work Readiness | Evaluator will observe participant demonstration of skills and knowledge in work readiness during program time. | This will serve as a means of gauging the accuracy of the participant self-identified initial survey. It will also provide deeper insight into participant needs and strengths in terms of being ready for the work world. | Evaluator must strive to observe in a biased free manner that will ensure stronger accuracy. There is always a risk of bias in observation. |
Facilitator Interviews Appendix B | Facilitators will be interviewed individually about their own observations on the program. | Facilitators have the most firsthand experience with the program and they may feel that certain aspects are working well and that certain aspects need improvement. This is their opportunity to express these thoughts. They are interviewed, rather than a focus group, to ensure confidentiality. | There is a possibility that the facilitators will have a motive to make the program appear to be something it is not. For example, they may have a vested interest in ensuring that the program appears successful, so this will guide their responses rather than their own true experience. |
Work Placement Host Interviews Appendix C | Work placement hosts will be interviewed at their business location individually. | Interviews are conducted in order to gain the perspective of work placement hosts on how well trained the participants are when they arrive for their work placements and if there are areas of training they feel is lacking. | The reliability is high because work place hosts invest a great deal of time into the participants and have a strong desire to host capable and work-ready participants. In addition, there is a high likelihood that they will appreciate having the opportunity to give feedback. |
Participant Exit Survey Appendix D | Participants will be surveyed at the end of the program. This survey will establish skills acquired as well as participant overall satisfaction with the program. | Survey is used rather than focus group in order to protect anonymity of participant feedback. It is determined that after spending several weeks with the same 12-15 participants, that individuals may not feel safe to express their thoughts on the program if they are negative because they will have established social bonds and relationships with fellow participants and may not wish to speak negatively about the program based on these relationships. | Reliability is high due to the anonymity of the survey and also because at the end of the program participants feel no pressure to impress the facilitators as they have completed the program and will no longer interact with them on a regular basis. |
Post-Program Interview of Participants Appendix E | A 3-5 participants will be interviewed 3 months after they have completed the program. | These interviews will seek to find out if iJob participants are experiencing success in the work world after the program has been completed. | High reliability because participants have no motivation not to be transparent in their responses. The challenge may be to connect with participants this long after the program. Also the sample size may not accurately reflect the post-training results of the program. Perhaps the few who will be interviewed are not reflective of the whole. |
Limitations
This evaluation may encounter a few limitations. First, because it is a small program, all the individuals who can offer feedback on the program are also heavily involved in the program. This may influence the overall results of the evaluation, which is also why observation time is important. Time for observation is the only external source of gathering data for this evaluation. Secondly, the population of participants are considered “high risk.” This means that many of their lives are complicated by challenges with addictions, domestic violence, gang involvement, poverty, low self-esteem and etcetera. These challenges mean that day to day, full participation may not be possible. Participants may miss days of the training or have to leave the program in order to deal with other life challenges. Thirdly, also due to the nature of the complicated lives of many participants, it may be difficult to locate 2-3 participants in September to conduct the post-training interviews. Many participants change their address and contact information on a regular basis. This may also mean that the sample size for the post-training is not large enough to accurately reflect the nature of the program’s results. Lastly, this evaluation will not assess the effectiveness of the program budget, but rather the focus is to evaluate if the program is meeting its objectives.
Logistics and Time Line
This evaluation will take place from the end of April, 2011 to the beginning of September, 2011. This will allow for the evaluation to capture a new set of participants entering the program at the end of April, establish a baseline of participant knowledge in employment skills prior to entering the program, observe their participation in the program, interview program facilitators and work placements hosts, survey the same sample group of participants at the end of the program as were surveyed at the beginning and to interview a 2-3 participants a few months after they have completed the program. The following is a detailed outline of the proposed timeline.
Program evaluator will drive to work placement businesses to conduct the interviews. This will be most effective for business owners who are busy in their day-to-day operations.
Evaluation Timeline: April 2011 to October 2011
April 19th (first day of new set of participants)
· Initial Survey of all iJob Participants
April 20-May 13
· Approximately 1 hour of observation per time
· 2-3 times over these three and a half weeks
May 16-May 20
· Facilitator interviews
· Three Facilitators
· Approximately 45 minutes each
May 24-May 27
· Work Placement Host Interviews
· Three to five interviews
· Approximately 30 minutes each
June 20 (last day of iJob program)
· Exit survey of all iJob participants
September 6-September 9
· Contact all iJob participants and schedule interviews
September 12-September 16
· Conduct post-iJob interviews
· 2-3 participants
· Approximately 30 minutes each
September 19-September 30
· Data analysis
· Compile final evaluation
October 3-October 7
· Find a suitable time for facilitators and evaluator to meet
· Disseminate results of evaluation
October 11-October 14
· Find a suitable final time to meet with program facilitators and evaluator together
· Having had time to look over the results, discuss any concerns and questions the facilitators may have regarding the evaluation
Budget
Item | Estimated Total |
Personnel: Evaluator Time ($30/hour @ 60 hours) | $1,800 |
Office Supplies (photocopies, pencils) | $20 |
Mileage ($0.48/km) | $40 |
Grand Total | $1,860 |
Note: space to conduct interviews and surveys will be provided at no charge by the iJob program. Interviews of work placement hosts will be conducted at their business locations.
Reference
Scriven, M (2001). An Overview of Evaluation Theories. Evaluation Journal of Australasia. 1(2) December.
APPENDIX A: PRE-TRAINING PARTICIPANT SURVEY
Please respond to how the following statements reflect you:
PART A: RESUME
I have a resume…………………………………………………….…….. never rarely sometimes often always N/A
A professional has given me feedback on my resume….. never rarely sometimes often always N/A
Others have complimented my resume……………………….. never rarely sometimes often always N/A
I always update my resume………………………………………….. never rarely sometimes often always N/A
I prepare a new resume specifically for each job …………. never rarely sometimes often always N/A
PART B: INTERVIEWS
I am successful at job interviews………………………………….. never rarely sometimes often always N/A
I have practiced job interviews with a professional………. never rarely sometimes often always N/A
I make eye contact during interviews…………………………… never rarely sometimes often always N/A
I prepare for an interview in advance…………………………… never rarely sometimes often always N/A
I research the company before an interview……………….. never rarely sometimes often always N/A
PART C: WORK HABBITS
I arrive to work shifts on time ………… …………………………. never rarely sometimes often always N/A
When I am at work, I keep busy with work tasks ………... never rarely sometimes often always N/A
I take initiative at work ……………………………………………….. never rarely sometimes often always N/A
I ask for help when necessary at work ………………………… never rarely sometimes often always N/A
I put in my best effort at work ………………………………….… never rarely sometimes often always N/A
I want to impress my supervisor at work ………………….… never rarely sometimes often always N/A
I have a strong work ethic ……………………………………….…. never rarely sometimes often always N/A
PART D: PERSONAL AWARENESS/KNOWLEDGE
I know my rights at work …………………………………………….. never rarely sometimes often always N/A
I know my personal learning style ……………………………….. never rarely sometimes often always N/A
I have written goals for my life ……………………………….…… never rarely sometimes often always N/A
I have career goals ……………………………………………………… never rarely sometimes often always N/A
I care about work safety………………………………………..…….. never rarely sometimes often always N/A
I know how to be safe at work …………………………………….. never rarely sometimes often always N/A
I know what I want to do for my career ………………………. never rarely sometimes often always N/A
I know that I have a positive future ahead of me .……….. never rarely sometimes often always N/A
I know how to get the career I want ……………………………. never rarely sometimes often always N/A
PART E: JOB SEARCH/READINESS
I know how to look for a job effectively ………………..…….. never rarely sometimes often always N/A
I feel ready to get a job ……………………………………………….. never rarely sometimes often always N/A
I am confident in my job skills………………………………………. never rarely sometimes often always N/A
PART F: SHORT RESPONSE
Do you feel ready to enter the work force today? Yes No
Why or why not?
What are some things you would like to know more about to get ready for the work world?
What are skills that you would like to acquire to be ready for the work world?
What do you feel is the one thing that would improve your ability to be successful on the job right now?
APPENDIX B: FACILITATOR INTERVIEW QUESTIONS
1. What do you see as the strengths of the iJob program? Why are these strong areas?
2. What do you see as the most necessary part of the iJob program in preparing the participants for the work world?
3. What are areas that you feel could be better? Why are these areas weaker?
4. What do you see as the most significant challenge for the iJob program and how do you manage this challenge?
5. Do you feel that the iJob program is reaching its goal of preparing participants for the work world? Why or why not?
APPENDIX C: WORK PLACEMENT HOST INTERVIEW QUESTIONS
1. Why do you host iJob participants at your business?
2. When participants arrive for work, do you feel they are well prepared?
3. Are there areas in which you feel they could be better prepared for the work world?
4. Are there any common challenges you experience with hosting the iJob participants? What are these challenges?
5. If had to make 2 suggestions on how to improve the iJob program based on what you know now, what would they be?
6. Any other comments you would like to share?
APPENDIX D: iJOB PARTICIPANT EXIT SURVEY
PART A: RESUME
I have a resume…………………………………………………….…….. never rarely sometimes often always N/A
A professional has given me feedback on my resume….. never rarely sometimes often always N/A
Others have complimented my resume……………………….. never rarely sometimes often always N/A
I always update my resume………………………………………….. never rarely sometimes often always N/A
I prepare a new resume specifically for each job …………. never rarely sometimes often always N/A
PART B: INTERVIEWS
I am successful at job interviews………………………………….. never rarely sometimes often always N/A
I have practiced job interviews with a professional………. never rarely sometimes often always N/A
I make eye contact during interviews…………………………… never rarely sometimes often always N/A
I prepare for an interview in advance…………………………… never rarely sometimes often always N/A
I research the company before an interview……………….. never rarely sometimes often always N/A
PART C: WORK HABBITS
I arrive to work shifts on time ………… …………………………. never rarely sometimes often always N/A
When I am at work, I keep busy with work tasks ………... never rarely sometimes often always N/A
I take initiative at work ……………………………………………….. never rarely sometimes often always N/A
I ask for help when necessary at work ………………………… never rarely sometimes often always N/A
I put in my best effort at work ………………………………….… never rarely sometimes often always N/A
I want to impress my supervisor at work ………………….… never rarely sometimes often always N/A
I have a strong work ethic ……………………………………….…. never rarely sometimes often always N/A
PART D: PERSONAL AWARENESS/KNOWLEDGE
I know my rights at work …………………………………………….. never rarely sometimes often always N/A
I know my personal learning style ……………………………….. never rarely sometimes often always N/A
I have written goals for my life ……………………………….…… never rarely sometimes often always N/A
I have career goals ……………………………………………………… never rarely sometimes often always N/A
I care about work safety………………………………………..…….. never rarely sometimes often always N/A
I know how to be safe at work …………………………………….. never rarely sometimes often always N/A
I know what I want to do for my career ………………………. never rarely sometimes often always N/A
I know that I have a positive future ahead of me .……….. never rarely sometimes often always N/A
I know how to get the career I want ……………………………. never rarely sometimes often always N/A
PART E: JOB SEARCH/READINESS
I know how to look for a job effectively ………………..…….. never rarely sometimes often always N/A
I feel ready to get a job ……………………………………………….. never rarely sometimes often always N/A
I am confident in my job skills………………………………………. never rarely sometimes often always N/A
PART F: SHORT RESPONSE
Do you feel ready to enter the work force today? Yes No
Why or why not?
How do you feel the iJob program has, or has not, prepared you for a successful future?
What is the most important skill or experience you gained during your time in the iJob program?
What is a barrier for you in entering the work force that you still experience presently?
Do you have a plan for the next 6 months of your life?
Do you feel that the program facilitators did all that they could to prepare you for work?
Is there anything you wish you had been able to learn or experience in this program to better equip you for working?
APPENDIX E: iJOB PARTICIPANT POST-PROGRAM INTERIVEW
(3 months after training)
1. Are you working right now? Why or why not?
2. Are you working towards the same goals you set when you were in the iJob program? Why or why not?
3. Looking back on your experience in the iJob program, do you feel that it has helped you? Why or why not?
4. Now that you have been out of the program for a few months, do you have any recommendations for improvement to the program? What are your recommendations?
5. What is the single most important skills or experience you gained during your time in the iJob program?
6. If there was one aspect of the program that you had to take away, what aspect do you feel is the least beneficial to you in the long-term?
