Assignment #2: Prenatal Exercise Program for Urban Aboriginal Women

The prenatal exercise program for urban Aboriginal women could be effectively evaluated by one of a variety of models.  However, I would recommend a mixed approach using a combination of Scriven’s simple approach to program evaluation and Lincoln and Guba’s Naturalistic approach.   

Scriven advocates looking closely at the goals of a program in order to determine its success.  Although the program goals are not clearly outlined in the description, there is enough information to deduce the key objectives.  The program is intended to, “function as both a means and an end to address health-related issues in the participants” (p.1).  The specific health issue mentioned is type 2 diabetes, which it also states is known to be reduced by an increase in physical activity by the individual at risk of developing type 2 diabetes.  Therefore, it is fair to evaluate the program based on a measured increase of physical activity by the participants.  If this key goal is not reached, then there are serious concerns with the success of the program.  Scriven’s approach will accurately guide an evaluator to look for evidence to the success or failure to reach the goal of increased physical activity among participants.

However, Scriven is not overly concerned with the details of how a program is reaching its success, but rather he emphasises the importance of simply observing whether or not the program is reaching its goals (Trochim, 1998).  This aspect of his approach is not desirable because it would be important to determine what exactly is motivating the participants to increase their physical activity in order to know what elements of the program to increase or maintain and what elements are not needed.  This is especially important because a number of elements for the program require a significant investment of time and money.  It would be key to understand the importance of each element because if one element is not contributing to the program success, then organizers can save money by eliminating it.  One might want to know if success of the program is due to the free access to transportation?  The free child care?  The weekly phone call reminding participants of the upcoming class?  The night of the week that was chosen?  Or the involvement of an Elder?  Or the nutritional snacks and social time at the end of the activity period?  If one of these elements is considered totally irrelevant to the participants, then it can be eliminated for future projects.  This might also make the program more affordable and therefore more likely to expand and reach further success.

Lastly, I would recommend incorporating aspects of the Naturalistic approach to program evaluation because for this type of social program, an evaluation that focuses on social interaction and is non-experimental in nature is more likely to put participants at ease.  Once participants are comfortable with the evaluation process the evaluator is in a better position to gain insight into what motivates participants in the program because they would be more likely engage in open and honest dialogue with the evaluator.  This dialogue would be key to understanding what motivates the participants to increase their physical activity and therefore the evaluator would be able to get to the heart of what inspires the participants, which is what ultimately determines the success of the program.

Sources:

Klomp, H., Dyck, R., Sheppard, S. (2003).  Description and evaluation of a prenatal exercise program for urban Aboriginal women. Canadian Journal of Diabetes, 27: 231-238.

Program Evaluation Course power point handout from Blackboad, “Background to Program Evaluation.  AKA: The exciting Theory that is PE”

Trochim, William M. K., “An Evaluation of Michael Scriven’s “Minimalist Theory: the Least Theory that Practice Requires.” (1998)  Retrieved on February 3, 2011 from: http://www.socialresearchmethods.net/research/An%20Evaluation%20of%20Michael%20Scriven.PDF

Program Evaluation Assignment #1

The Women’s Enterprise Management Training Outreach Program – India (WEMTOP) evaluation does not seem to fit into any of the models discussed in the Module One class.  I say this because it is difficult to know what approach they used in conducting the evaluation because it is not described.  What is known is simply that the evaluators interviewed representatives from 12 of 21 non-governmental organizations (NGOs) working on the project and fifteen of the women’s groups (out of how many groups?  It is not stated).  They also met with staff from the lead training partner and three other partner training institutions, and donors.  Based on this information, I cannot tell for example, if they approached the evaluation as two teams (i.e. Adversary Model) or if they were not told the goals in advance (i.e. Scriven, Goal-Free Model).  The report consists of background information, information about some of the challenges for the project, accomplishments of the project at the various levels, overall impact (which is very vague), some explanation of the cost of the project and ends with conclusions and recommendations.

There are some strengths to this program evaluation.  First of all, the evaluation is easy to read and clearly outlines a few challenges of the project.  More importantly, the evaluators made recommendations based on the challenges that they identified.  I see this as a positive approach because the evaluators are not saying that because the project faces certain challenges it is a failure.  Instead they are saying that given these challenges, we recommend trying these ideas in order to make the program better.  Secondly, the evaluators appear to have spoken with many people involved in the program at various levels.  For example, they spoke with women who are intended to benefit from the program and also to decision makers, such as the Executive Director of training.

Despite some strengths, there are many weaknesses to the program evaluation.  First, there is no information about the evaluators.  This is a World Bank program, and I wonder if the evaluators are World Bank staff members and if this is the case, there could be a conflict of interest.  For example, if they make the program appear highly successful it could increase their chances of receiving more funding. 

A second weakness is that it states that they interviewed fifteen of the women’s groups that were trained, however there is not information about how many women’s groups were trained in total.  This could be 5% of the groups of 90% of the groups.  It would make a difference to know this information because in my mind, the more participants they spoke with directly relates to the legitimacy of the statements they make about how the participants benefited and felt about the project.  

A third weakness is that it does not clearly state the goals and objectives of the project at the beginning of the report.  Throughout the evaluation there are indications of the overall objectives and goals of the program, and by reading the report one does get a sense of what was intended to be the goals, but they are not clearly outlined until close to the end when the report makes some statements about accomplishments.  An example of such a vague statement at the beginning is, “a defining design principle [of the project] was that it represented a ‘package completion approach;” (p. 3).  In other words the project is intended to provide a range of services.  This is too vague.  The finally when one gets close to the end of the evaluation, there seem to be some clearer statements about the objectives, but it would help to outline this much earlier.

Fourthly, the evaluators state accomplishments that do not fully address the objectives.  For example, the output objective states that, “1000 women would have improved their enterprise management and entrepreneurial skill; and enjoy increased self-awareness and consciousness of gender issues after participating in GMT” (p. 3).  Then in the same paragraph, when it comes to evaluating the success of this objective, the report states that, “1077 producer women received at least one model of Grassroots Management Training” (p. 3).  Although it appears successful in that 77 more women participated than the initial goal, there is no statement in the report to indicate if the women who did the training did or did not “enjoy increased self-awareness and consciousness of gender issues…”.  This type of disconnected linkage between apparent objectives and outcomes occurs more than once. 

Overall, this program evaluation seems to have more weaknesses than strengths, yet somehow after reading the report I feel that I have a decent understanding of the successes and challenges of the project.  In conclusion, maybe I am being too hard on the evaluators!

To read this evaluations, go to: http://info.worldbank.org/etools/docs/library/74184/winter2002/proceedings/pdfpapers/mod9cn.pdf

Program Evaluation

Hope this works!