Logic Model for iJob Program (assignment #4)

To view logic model click:
https://docs.google.com/viewer?a=v&pid=explorer&chrome=true&srcid=0B7fgnm8YcwKQOGQzM2I2ZDYtMjZlMC00MGM1LTk3NTUtYzdjZjQyOTYyY2Ey&hl=en&authkey=CJy_zusC

Inputs

The inputs category shows the resources that are required to conduct the program.  These are all the people, materials and supplies that are necessary to run the iJob program. For example: Nutana school community is a resource because this collegiate offers a wide variety of programs and services to students who attend the school and all participants of the iJob program.  Nutana is a part of a “School Plus” model which means that services that students would normally have to access outside of the school are brought to the school in order to eliminate barriers to attending class (and the iJob program).  For example, there is a nurse practitioner, a day care, a social worker, an addictions counsellor, etc.  The school community is important because many referrals come from the school community.

Outputs

The outputs section shows the activities and who is participating in these activities.  There are only three main groups of participation: the facilitators, the participants and the industry partners.  The first activity listed, interview program candidates, is done by the program facilitators in order to ensure that they are admitting candidates who will benefit from the program.  The following twelve items listed under activities are the workshops or lessons that the facilitators lead in the pre-employment time of the program.  All these activities are geared towards achieving a successful work placement, which is phase two of the

Outcomes

The short term outcomes indicate the expected results from the activities listed under outputs.  These outcomes represent the desired skills and knowledge that the participants will gain during the pre-employment training program.

The medium term outcomes indicate the expected results at the end of the whole program.

The long term outcomes indicate the expected “big picture” results that will occur if the short and medium term outcomes are achieved.  The long term outcomes are difficult to measure, but indicate in what way this program expects to impact society at large.

Assumptions:

This program will successful based on a few assumptions.  These are factors that are assumed to be stable or present in order for the program to carry through on its objectives.  For example, it is assumed that participants are able to attend the program in order to benefit from the trainings being offered and therefore reach positive outcomes.

External Factors:

This program may be influenced by several external factors over which the facilitators have no, or very little, control.  For example, life circumstances of the participants.  Given the “at risk” population who primarily attend the program, it is a somewhat common occurrence that participants have incidents in their lives that prevent them from completing the program.  These incidents could be related to addictions, involvement with criminal activity, intimate partner violence, or other challenges/barriers.

Evaluation Assessment: Assignment #3

General Overview of Program

The iJob program is designed to prepare young people for the work world.  The program has two phases: phase one is pre-employment training which consists of approximately three weeks of activities (such as goal setting seminar, career exploration opportunities, work safety training, etc.).  Once the facilitators feel that the participant is ready for a work experience, the participant is placed with an industry partner for a six week paid work experience.  Participants are supported by the facilitators during this time with regular visits to the work place.  The long-term outcome for this program is to engage young people in the formal economy with careers that they feel good about.

Engage Stakeholders

Who should be involved?

The main stakeholders for the iJob program are the participants, the facilitators, the overall school community (where the program is hosted) and industry employers (who host students on work placement).  All four levels of stakeholders should be involved in the evaluation.

How might they be engaged?

The stakeholders might be engaged using a variety of methods depending on what best suits the stakeholder.  For example, the facilitators, participants, and members of the school community might be engaged using longer surveys and interviews.  The industry employers might be engaged using short interviews in order to be considerate of the time they are willing to spend engaged in an evaluation for the iJob program.

Focus the Evaluation

What will be evaluate?

The evaluation will focus on the program’s ability to meet its medium term outcomes as well as the overall satisfaction of the participants in the program.  Participant satisfaction is important for future recruitment.  This evaluation will not focus on the allocation of funds as this is set out by the funder and is a fixed element to the program.

What is the purpose of the evaluation?

The purpose of the evaluation is to access the program’s ability to meet its objective of providing pre-employment training and a paid work experience opportunity for participants.  The evaluation, if positive, will support the program’s application for continued funding.  Lastly, the purpose of the evaluation is to gain understanding about the areas of the program that experience success and to identify if there are areas that need improvement.  Furthermore, it is to understand why certain areas are successful and how certain areas, if any are identified, can be improved in order to strengthen the overall program delivery.

Who will use the evaluation? How will they use it?

Who? Users?	How will the use the information?
Program Facilitators	The information will be used by the facilitators for them to know what is currently working well for the overall program and to identify ways in which the program could be enhanced. Facilitators may also choose to use the evaluation to provide evidence of the success of the program to the program funders.
Program Funders	If the facilitators choose to use the evaluation in their proposal for continued funding for the 2011-2012 school year, then the funders will use the information in the evaluation to determine eligibility for continued funding.
Nutana School Community	The school community may choose to use the evaluation to enhance participation in the program or to promote the success of the in-school program or to promote creating similar programs in other schools.

What questions will the evaluation seek to answer?

The evaluation will seek to answer the following questions:

Pre-Employment Training

·         What strategies/approaches/activities are currently working well to prepare participants for a successful work experience?

·         What activities are currently working well to encourage students to gain a clearer understanding of their personality/interests and how this impacts career choices?

·         Do participants leave the pre-employment training feeling that they have the skills to apply for a job?

·         Are there important training aspects that participants need/desire to be incorporated into the program that are currently not incorporated?

·         Do participants feel prepared for a work placement after the pre-employment training?

·         Are there areas in which they feel underprepared?

Work Placement

·         Do participants experience successful work placements?

·         Do participants feel sufficiently supported on their work placement?

·         Of the students who do not complete the work placement, what were the challenges, barriers to completion?

·         How can the program enhance its ability to address these barriers/challenges to completing the work placement?

·         Do participants feel they can confide with the facilitators if they are experiencing challenges at their work placement?

General

·         After completing the program, do students either pursue further education or have employment?

·         If not, what are the barriers to achieving this success and how can the program seek to address/eliminate these barriers?

·         Do participants feel, at the end of the program, that they have a strong and positive vision of their future?

·         Do facilitators feel they are sufficiently prepared to be successful?  Do they feel that they are sufficiently supported by the school community?

·         Do facilitators experience any barriers to their successful delivery of the program?

Community Engagement

·         How does the school community perceive the iJob program?  Is this perception satisfactory or could it be improved upon?

·         From where does iJob receive most of their referrals?

·         Do businesses who host work placements students feel that this is a beneficial experience?  What are factors that contribute to feeling that this is a worthwhile activity and what are ways in which the involvement on industry can be improved upon?

·         Do the promotional materials accurately reflect the program?  How are these materials being used to attract more participants?

What information is needed to answer these questions?

What I wish to know	Indicators – how will I know it?
Pre-Employment Training
What activities are currently working well to prepare participants for a successful work experience?	Results from survey/interview questions will ask students to identify which activities prepared them for their work placements.  Students will rank activities as either high in preparation or low.
What activities are currently working well to encourage students to gain a clearer understanding of their personality/interests and how this impacts career choices?	Results from survey/interview questions will ask students to identify which activities helped them to gain clearer understanding of their interests and career choices.  Students will rank activities as either high in helping or low.
Are there important training aspects that participants need/desire to be incorporated into the program that are currently not incorporated?	Results from survey/interview questions will ask participants to identify if there are skills/information, etc that they feel are necessary but not currently offered in the program.  Industry partners will also be asked if participants arrive well prepared.  Industry partner responses will be one indicator if there are areas missing in the pre-employment training program.
Do participants leave the pre-employment training feeling that they have the skills to apply for a job?	Results from survey/interviews will indicate if participants feel they could go out and apply for a job when the pre-employment training is completed.
Do participants feel prepared for a work placement after the pre-employment training?	Survey/interview of participants will ask participants if they felt well prepared.  Their response will indicate if they were prepared or not.
Are there areas in which they feel underprepared?	If students respond that there are areas in which they feel underprepared, participants will be asked in what way they felt underprepared.  Their response will indicate which areas they need to enhance.
Work Placement
Do participants experience successful work placements?	Responses on survey/interview of industry partners, facilitators and participants will indicate.
Do participants feel sufficiently supported on their work placement?	Responses on survey/interviews of participants will indicate level of support participants feel.
Of the students who do not complete the work placement, what were the challenges, barriers to completion?	Responses on survey and interviews of industry partners, facilitators and participants will indicate the challenges and barriers.
How can the program enhance its ability to address these barriers/challenges to completing the work placement?	Observation and survey/interviews of all stakeholders will produce suggestions on how to address barriers.
Do participants feel they can confide with the facilitators if they are experiencing challenges at their work placement?	Interviews with participants will indicate level of comfort participants feel in confiding with facilitators.
General
After completing the program, do students either pursue further education or have employment?	Phone calls to participants after they have completed the training will indicate what they are doing after the program.
If not, what are the barriers to achieving this success and how can the program seek to address/eliminate these barriers?	Phone conversations with participants will seek to understand the barriers to being either employed or being a student after completing the program.
Do participants feel, at the end of the program, that they have a strong and positive vision of their future?	Questions during phone conversation with participants and survey/interview of participants who are still connected with school community will indicate their vision of their future.
Do facilitators feel they are sufficiently prepared to be successful?  Do they feel that they are sufficiently supported by the school community?	Interview with facilitators will indicate if they feel sufficiently prepared in addition to evaluator observations.  Facilitator responses and evidence of facilitators having the support and materials needed to be successful will indicate the response.  Also, survey of the staff within the school community will indicate to some degree the level of support iJob experiences from the school community.
Do facilitators experience any barriers to their successful delivery of the program?	Interview and observation of evaluator will indicate the response.
Community Engagement
How does the school community perceive the iJob program?  Is this perception satisfactory or could it be improved upon?	Results of discussion with members of the school community will indicate the perception of the iJob program by the school community.  Questions might be for example: do you know what iJob does?  Do you see the benefits of this work?
From where does iJob receive most of their referrals?	Results from looking at iJob records that are kept about each participant and how they heard about iJob will be analyzed in order to find out how most participants find out about the program.
Do businesses who host work placements students feel that this is a beneficial experience?  What are factors that contribute to feeling that this is a worthwhile activity and what are ways in which the involvement of industry can be improved upon?	Results from a short interview with industry partners will indicate if they feel this program is worthwhile.  Responses to specific questions about what does or does not make it worthwhile will indicate what factors contribute to their opinions.
Do the promotional materials accurately reflect the program?  How are these materials being used to attract more participants?	Program evaluator will take a critical look at the promotional materials to seek out consistency between program objectives and promotional materials.  In addition, materials will be shown to 3rd person (not involved in iJob program) and ask 3rd person: based on reading this brochure, what is your impression of the iJob program?  Would you recommend this program?

When is the evaluation needed?

There is no deadline.  However, new participants will begin the program on April 14^th and finish at the end of June.  It would be effective to begin the evaluation when the new participants arrive in order to take a baseline survey to measure change over time in the program.

What evaluation design will be used?

A simple approach to program evaluation will be used (as outlined by Scriven).  The evaluation will seek to primarily measure if the iJob program meets its objectives and to answer why it does or why it does not.  Although knowing if the program meets its objectives is not enough, it is also important to take a further step to know what activities of the program are directly enabling the program to meet its target objectives and which activities are not so that the program can be “fine tuned” and be most efficient.  The evaluation will use observation, interviews, survey’s and informal dialogue.

Collect the Information

What sources of information will be used?

Existing Information:

·         progress reports

·         financial reports

·         existing student feedback form

People:

·         facilitators

·         participants

·         industry partners

·         school staff

Pictorial Records and Observations:

·         evaluator will take notes of observations during time spent in the program

What data collection method will be used?

ÿ       Survey (baseline and post-participation)

ÿ       Interviews

ÿ       Observation

ÿ       Document Review

ÿ       Testimonials

What is needed to record the information?

·         Surveys (for participants, facilitators, industry partners, and school community)

·         Interview questions for all interviewees

·         Scheduled interview times

·         Contact information for participants and industry employers

·         Time spent in the program to observe

·         Transportation to get to various work placement locations

When will data be collected for each method that has been chosen?

Method	Before program	During program	Immediately after	Later
Survey of participants	Yes		Yes
Interviews with participants		Yes		Yes
Interviews with industry partners		Yes
Interviews with facilitators		Yes	Yes
Survey school community		Yes
Review of reports and finances	Yes

Will a sample be used?

Yes.  A sample survey will be tested in March or April prior to beginning the official evaluation.  A new set of participants will begin the program in mid-April and the final survey will be ready for the new start date.  It will be tested on the current participants in advance of that date.  The pilot survey will be tested in a time set aside during the program.

Analyze and Interpret

How will the data be analyzed?

Data analysis method: The data will be analyzed by charting the results of the surveys, interviews, informal dialogue and observations in order to draw conclusions as to the opinions, views, and satisfaction of the program by those who have been involved in the evaluation and most importantly to evaluate how well the program is meetings its overall objectives.

Who will analyze data: The data will be analyzed by both the program evaluator, Sarah Loewen, and a second independent expert in program evaluation, Reynolds Lauer. 

How will the information be interpreted?

Both individuals, named above, will first analyze the data independently and then secondly discuss their respective and independent conclusions in order to build in a “check and balance” system that will bring to light any bias in the analysis and interpretation of the data results.  If each individual comes to similar conclusions, this conclusion will be accepted as accurate.  If, however, the individuals come to differing conclusions, a third individual will be approached to analyze the data in order to confirm an accurate analysis.

What did I learn?  What are the limitations?

Limitations in this program evaluation may be the following: industry partners do not make the time to meet with me; participants feel an obligation to give positive feedback even if they do not feel this way about the program – this could be because of a variety of reasons, such as they do not want to disappoint the facilitators; also, as this is the first program I have evaluated, there is a strong possibility that I will miss an important element in the evaluation and finish with inaccurate results and feedback for the iJob program.

How will the information be used and communication?

To Whom	When/Where/How to present?
Jay Wilson	Evaluation Report will be posted on my blog
iJob program facilitators	Facilitators will be given a copy of the report and an opportunity to discuss the results with the evaluator
Program Funders	If the facilitators choose to do so, they have the opportunity to provide a copy of the evaluation to their funding agency.

Next steps?

Reflect on the process of evaluating the iJob program and take notes on successes and challenges in the process.  Have informal conversations with facilitators and program participants about the evaluation process for my own feedback on how I can improve my approach to program evaluation.