The Women’s Enterprise Management Training Outreach Program – India (WEMTOP) evaluation does not seem to fit into any of the models discussed in the Module One class. I say this because it is difficult to know what approach they used in conducting the evaluation because it is not described. What is known is simply that the evaluators interviewed representatives from 12 of 21 non-governmental organizations (NGOs) working on the project and fifteen of the women’s groups (out of how many groups? It is not stated). They also met with staff from the lead training partner and three other partner training institutions, and donors. Based on this information, I cannot tell for example, if they approached the evaluation as two teams (i.e. Adversary Model) or if they were not told the goals in advance (i.e. Scriven, Goal-Free Model). The report consists of background information, information about some of the challenges for the project, accomplishments of the project at the various levels, overall impact (which is very vague), some explanation of the cost of the project and ends with conclusions and recommendations.
There are some strengths to this program evaluation. First of all, the evaluation is easy to read and clearly outlines a few challenges of the project. More importantly, the evaluators made recommendations based on the challenges that they identified. I see this as a positive approach because the evaluators are not saying that because the project faces certain challenges it is a failure. Instead they are saying that given these challenges, we recommend trying these ideas in order to make the program better. Secondly, the evaluators appear to have spoken with many people involved in the program at various levels. For example, they spoke with women who are intended to benefit from the program and also to decision makers, such as the Executive Director of training.
Despite some strengths, there are many weaknesses to the program evaluation. First, there is no information about the evaluators. This is a World Bank program, and I wonder if the evaluators are World Bank staff members and if this is the case, there could be a conflict of interest. For example, if they make the program appear highly successful it could increase their chances of receiving more funding.
A second weakness is that it states that they interviewed fifteen of the women’s groups that were trained, however there is not information about how many women’s groups were trained in total. This could be 5% of the groups of 90% of the groups. It would make a difference to know this information because in my mind, the more participants they spoke with directly relates to the legitimacy of the statements they make about how the participants benefited and felt about the project.
A third weakness is that it does not clearly state the goals and objectives of the project at the beginning of the report. Throughout the evaluation there are indications of the overall objectives and goals of the program, and by reading the report one does get a sense of what was intended to be the goals, but they are not clearly outlined until close to the end when the report makes some statements about accomplishments. An example of such a vague statement at the beginning is, “a defining design principle [of the project] was that it represented a ‘package completion approach;” (p. 3). In other words the project is intended to provide a range of services. This is too vague. The finally when one gets close to the end of the evaluation, there seem to be some clearer statements about the objectives, but it would help to outline this much earlier.
Fourthly, the evaluators state accomplishments that do not fully address the objectives. For example, the output objective states that, “1000 women would have improved their enterprise management and entrepreneurial skill; and enjoy increased self-awareness and consciousness of gender issues after participating in GMT” (p. 3). Then in the same paragraph, when it comes to evaluating the success of this objective, the report states that, “1077 producer women received at least one model of Grassroots Management Training” (p. 3). Although it appears successful in that 77 more women participated than the initial goal, there is no statement in the report to indicate if the women who did the training did or did not “enjoy increased self-awareness and consciousness of gender issues…”. This type of disconnected linkage between apparent objectives and outcomes occurs more than once.
Overall, this program evaluation seems to have more weaknesses than strengths, yet somehow after reading the report I feel that I have a decent understanding of the successes and challenges of the project. In conclusion, maybe I am being too hard on the evaluators!
To read this evaluations, go to: http://info.worldbank.org/etools/docs/library/74184/winter2002/proceedings/pdfpapers/mod9cn.pdf
