Certificate of Advanced Study in Evaluation

Examples of Our Graduates' Work

Below is a list of recent participants in our certificate program, and the titles of the work they completed as a portion of the program.  Project descriptions are coming for all participants, and will be posted when available.

 

Hazel Atuel
Preliminary Evaluation of the Freedom From Exploitation Programs

The word “prostitution” evokes images of what is considered the oldest profession in the world. From Ancient Greek vases to Manet’s Olympia, the face of prostitution is that of a beautiful and sensual woman enjoying the mundane pleasures of life. In recent times, prostitution has been reconceptualized as a form of sexual exploitation that reflects the economic power differential between men and women, and follows the straightforward law of supply (prostitutes) and demand (johns). To alleviate the prostitution problem, efforts must be directed at addressing both sides of the equation.

Recognizing the dual challenge posed by the prostitution dilemma as well as the critical role the legal system plays in mitigating the problem, Freedom From Exploitation (FFE) was created to provide informational and educational services to victims and perpetrators of prostitution within San Diego County. The overarching goals of FFE are to: (1) Assist women to leave a dangerous lifestyle and move toward a healthy secure, and meaningful life; (2) Provide education to initiate change in how prostitutes are treated by members of the community for the betterment of society; and (3) Invoke change in the US judicial system’s misconceptions of prostitution by advocating for fair and equal treatment of women prosecuted for prostitution.

Currently, FFE has four programs, which are implemented in collaboration with local government agencies or community-based organizations dealing with the prostitution issue from different angles. Adult Survivor of the Streets is in partnership with the City Attorney and City Court, and is a diversion program for first-time offending prostitutes. Teen Survivor of the Streets is in partnership with the Juvenile Court and the Girls Rehabilitation Facility, and targets female minors arrested for prostitution or who have a history of prostitution. Promoting Recovery through Outreach to At-Risk Populations is in partnership with the County’s Health and Human Services, and targets female minors at-risk for prostitution. Finally, Public Speaking is in partnership with the City Attorney’s Prostitution Impact Panel, and is a diversion program for first-time offending johns.

Following Donaldson (2003), the evaluation entailed developing a program theory unique to each FFE program. Each theoretical model offers a viable explanation on specific affective and cognitive factors that could potentially account for positive behavior change among the target population. Following Scriven (in Shadish et al., 1991), the evaluation involved collecting preliminary evidence of program value from the perspective of program partners.

 

 

Cindy Beckett
Even Start Program Analysis

The California Even Start Literacy Program in California currently consists of 154 programs that serve ESL learners and their families. The program consists of four components that are shown to enhance literacy and include Adult Education, Early Childhood Education, Parenting Education and Parent/Child Interaction (PCILA). The program is blanketed under the Federal “No Child Left Behind” legislation and serves to increase family literacy skills for both children and adults. The collection of data reflective of those indicators is required by programs on a state wide level in order to measure program outcomes. Indicators were developed at the national level and based on sound research in family literacy.

Cindy Beckett’s study investigated areas of functioning at the program level by identifying the differences in program processes in a variety of program areas, the evaluation activities that support these processes, and the dynamics of evaluator roles. During the project year, programs were required to complete a “self-study” in order to measure progress and identify strengths and weaknesses. The results of the self study along with information gleaned from telephone interviews with project directors were utilized to explore the areas of interest.

Cindy found that tools such as the self-study and the gathering of additional qualitative data help to identify areas of weakness that can help programs at the program level by creating efficient systems. As in Empowerment Evaluation, the self-study process helps programs to critically look at their processes and to strategize in order to change systems and create healthy functioning organizations.

 

 

Shannon Bettridge
Development of a Logic Model: Lessons Learned

The Law and Mental Health Program (LAMHP) is a distinct clinical program within a large, urban psychiatric hospital in Toronto, Ontario, Canada. The LAMHP is responsible for the assessment, treatment and supervision of a large number of individuals who have a mental illness and have come into contact with the criminal justice system. All patients are involuntary and the LAMHP itself has limited control over the referral or discharge of its own clients. At all times, staff and leadership in the LAMHP must balance the treatment needs of patients with the legal requirements and responsibilities bestowed upon forensic programs (including protection of the public) as outlined in the Criminal Code of Canada and other relevant legislation.

As Manager of Program Development and Evaluation at LAMHP, Shannon worked with the directors, managers, and senior clinicians of the program to develop a system-level LAMHP logic model, from which other more detailed program-level logic models and a clearly prioritized and developed evaluation plan can be developed.

 

Geeta Cheema
Evaluation Plan for the Health Promoting Schools Coordinator Initiative

Geeta created an evaluation plan for Interior Health’s Health Promoting Schools Coordinator Initiative (“the program”), one aspect of the health authority’s plan to support health in school settings. By funding the placement of Health Promoting Schools Coordinators in School Districts, this program is expected to establish or further develop the components of health promoting schools as defined in the health promotion literature. Given the innovative nature of the program, Interior Health is interested in knowing if it is ‘making a difference’ within the funded School Districts by the end of the first and third year of implementation.

Geeta facilitated a series of meetings with a stakeholder Working Group to develop the evaluation plan. Through application of Donaldson’s Program Theory-Driven Evaluation Science, including the incorporation of health promotion theory, the Working Group was assisted to develop a series of impact theories that conveyed their understanding of how the program was expected to ‘work’. By reflecting on these models, the Working Group identified key evaluation questions that will examine the process, initial outcomes and impacts of the program. A data collection strategy was devised to answer each of the evaluation questions, and the evaluation plan also included a communications strategy and implementation budget.

The practicum project highlighted the contribution of evaluation to program development, and the importance of stakeholder engagement in evaluation work.

 

 

Jaime Chua
Shifting Gear from Logical Framework to Theory Driven Evaluation: A Descriptive Study of Techno-Structural and Behavioral Change Management

Jaime Chua's evaluation study describes an international nonprofit organization committed to the development of a peaceful, prosperous, just, and open Asia-Pacific region. To realize this mission, the organization supports high-impact programs that help improve governance and economic reform, women's empowerment, and regional relations in collaboration with civil society, private sector, and public partners. Specific activities include leadership training, organizational development, policy research, and institutional reform.

In response to increasing expectations for performance accountability, Jaime evaluated existing evaluation policy at his workplace. Jaime designed his exploratory and descriptive evaluation study to: (1) assess the current program design and evaluation practice and (2) recommend and substantiate an alternative program design and evaluation methodology. Jaime used the CDC’s six-step evaluation framework as a guide and used in-depth interviews, document analysis, and participant observations for data collection. Based on his analysis, he also developed a set of technical, structural, and behavioral change management plan to convert the organization’s logframe-based evaluation model into a theory-driven evaluation model.

 

Sonja Evensen
Basics of Evaluation (Modules 1 through 5)

Sonja Evensen created a five-module evaluation training series for Pacific Resources for Education and Learning (PREL), an independent, nonprofit 501(c)(3) corporation that serves the educational community in the U.S.-affiliated Pacific islands, the continental United States, and countries throughout the world. PREL bridges the gap between research, theory, and practice in education and works collaboratively with schools and school systems to provide services that range from curriculum development to assessment and evaluation.

Sonja created the training series in response to the January 2005 Federal Register Notice, which called on educational researchers to apply “scientifically-based research” to evaluate the effectiveness of their programs. The modules are: (1) Defining Evaluation: Meanings, Uses, Types; (2) Steps in Program Evaluation: Logic Model; (3) “Evaluation Consumer” Basics; (4) Data Basics; and (5) Experimental Design. The modules can be used as part of a series or as a stand alone and have been used in different settings both in Hawaii and among project partners in the Pacific Region.

 

 

Leslie Fierro
What are we preparing for? A description of the status of evaluation coursework within master's level epidemiology programs

Leslie Fierro is interested in enhancing existing capacity to conduct evaluations in the discipline of public health. During her career as an epidemiologist, she has noticed an increase in the need for evaluation and the number of requests for epidemiologists to assist or lead these evaluations. Epidemiologists are trained to study the distribution and determinants of diseases in populations. As part of this training, epidemiologists acquire a number of quantitative skills that may be valuable in an evaluation setting. Although epidemiologists have some tools that are useful for conducting evaluations, recent surveys assessing the training needs of practicing epidemiologists indicate that there is a need for basic instruction in evaluation. For example, in a recent 2006 National Assessment of Epidemiology Capacity Survey conducted by the Council of State and Territorial Epidemiologists, 45% of respondents agreed that additional training was needed for epidemiologists in developing “program logic models and theories of action” (CSTE, 2006) (p. 20).

The goal of Leslie’s practicum project was to provide a description of the exposure master’s level epidemiologists get to introductory coursework in program and intervention evaluation as part of their academic training. Leslie systematically reviewed the websites for all schools of public health accredited by the Council of Education in Public Health between January 2007 and April 2007. Descriptions of required and elective coursework for students seeking a master’s level degree in epidemiology were extracted from 36 of the 38 schools of public health and classified based upon their emphasis on program or intervention evaluation. A total of 1,538 courses descriptions reviewed, few focused on program or intervention evaluation. Findings indicate that there is a need for additional coursework that emphasizes introductory concepts in program and intervention evaluation within master’s level epidemiology curricula.

 

 

Charles Gasper
Evaluation of Nursing Perceptions Surrounding the Medication Delivery Process Affected by Bridge Bar Coding Implementation

As noted in the evaluation study by Charles Gasper, a minimum of 44,000 deaths occur each year due to adverse drug events (Kohn, Corrigan & Donaldson, 1999). Nearly 78% of these errors, according to recent national data, are preventable using bar coding technology (Hicks, Santell, Cousins, & Williams, 2004). In 2004, the Sisters of Mercy Health System embarked upon the implementation of a bar coding and electric medication record to help alleviate this problem – Bridge.

Charles Gasper conducted an evaluation of nurse perceptions related to the Bridge medication bar code scanning system at nine hospitals within the Mercy network.  His evaluation looked, in part, at the change in perceptions in patient safety pre- and post- implementation of the system. Using surveys, Charles collected data that allowed him to look at differences by units within the hospital, by hospital within the network, and system-wide. His study also compiled data related to the frequency and type of medication delivery errors.

Further work leveraged the Bridge system to evaluate the impact of various nursing ratios on the clinical outcomes of patients. Using the rate of medications administered as a measure of workload along with unit layout, access to communication devices, and acuity of patient support need, he assessed the impact of the patient to nurse ratio for medical and surgical nursing units on complication rates, patient length of stay, and mortality rates. The study identified ratios that were strongly correlated with a lower rate of significant patient complications and was used by the Mercy hospitals to adjust staffing patterns to provide better quality care.

 

 

Minda Goodman
Evaluation of a Leadership Conference of an International Not for Profit Agency

Minda's evaluation of an Asian/Pacific Leadership Conference of an international not-for-profit service organization contributed to the organization’s history of evaluations in an effective, strategic way. The organization’s leadership was able to focus and navigate through the many issues that they wanted explored more logically, using a research-based framework that she could provide. The leadership was open to re-examining how they went about doing their evaluations, in order to gauge their effectiveness of their constituent’s grasp of the goals of a newly introduced national campaign.

 

 

 

Anne Heberger
Bibliometric Data and the Evaluation Theory Tree

Using bibliometric data from the Web of Science, Anne Heberger explored the works published by theorists on Alkin & Christie's (2003) Evaluation Theory Tree. This project provides a useful way to understand the benefits and limitations of studies using bibliometric data. References to articles published by the theorists are also examined. Anne's project shows that there are some consistencies in publishing patterns of the theorists across the three branches of the theory tree (Methods, Use, and Valuing), but each branch also exhibits a distinctive character of its own. Cited references to these articles reveal that the work of evaluation theorists extends beyond the areas in which the authors typically publish.

 

 

 

Shanna Livermore
Evaluation of the Sacramento Area Needle Exchange Program

Satellite Syringe Exchangers (SSEs) are individuals who access needle exchanges and obtain needles for others. They may or may not also be obtaining needles for themselves, as some SSEs are sober while others are active users. Satellite Syringe Exchange (SSE), otherwise known as secondary syringe exchange, is an innate activity in the Injection Drug Using (IDU) community. SSE has been going on all over the world for quite some time. The characteristics of SSEs vary by geographic location and community. The reasons SSEs obtain and distribute needles for others vary as well. Some exchange for broad altruistic reasons such as helping the IDU community. Some exchange to prevent their own circle of friends from getting Human Immunodeficiency Virus (HIV) or Hepatitis C because of their own HIV or Hepatitis C status. Other SSEs use syringes as a source of income by selling them and yet others are drug dealers who provide a free needle to those who purchase drugs from them. Though SSEs have existed for many years, their primary function has been to obtain and distribute needles, whatever their reason. Many public health programs now have a formal relationship SSEs.

One such program is the Sacramento Area Needle Exchange (SANE). The SSEs recruited into the SANE program have several functions. First, SSEs undergo a HIV risk assessment and then learn how to reduce their risk. Second, they learn information about HIV and Hepatitis C transmission and the respective harm reduction techniques associated with each of them. Third, they receive materials to distribute, which include: needles, cookers, and other injecting materials along with educational materials and a list of referrals for issues related to injection drug use such as wound care.

In the fall of 2007 brief survey was conducted to determine the type and degree of impact the work of SSEs working with the SANE program has had on the lives of the SSEs themselves and the people the SSEs exchange for. Eight SSEs working with the SANE program participated in a survey, conducted in person and via telephone, consisting of both qualitative and quantitative questions. Questions ranged from basic demographics to retrospective Likert Scales measuring the level of security felt when obtaining needles. Overall, the SSEs working with the SANE program reported highly positive changes after exposure to the SANE program and expressed great passion for and deep emotional connection to their work as SSEs.

 

 

 

James Luther
Mission Statements and Values of Evaluators

The association between social justice and evaluation has existed for over 25 years (Kushner, 2005). Newman and Brown (1996) point out how important it is for evaluators to have commitment to social justice because social justice can have a major impact on interpretation of findings. Donaldson and Christie (2006) argue that most evaluators have some commitment to social justice and point to professional association ethical standards and guidelines as supporting that argument on the professional level. Making a public commitment to incorporating social justice in evaluation would seem to be a first step toward the application of social justice commitment. Since Mission Statements help define the goals of the company, they are one means by which the organization can publicly state their values. In response to literature and research in this area, James Luther conducted a research project to explore mission statements of evaluators and to discover if there was evidence of a commitment to social justice on the part of these evaluators.

James’ project also sought to determine the “level” evaluators focused their evaluation efforts and potentially target their commitment to social justice – at the macro or micro level. Macro level social change models have the goal of shifting power imbalances at the societal level. Alternatively, micro level models are aimed at empowering individuals in organizations and at the program level. James obtained mission statements from the websites of 16 evaluation agencies and performed quantitative data analysis, looking at similarities across and differences by geographic region and for-profit status.

 

 

Murdoch Johnson
Imagination Library Case Study

The Imagination Library program is funded and run by the Dollywood Foundation in Tennessee. It mails a book of appropriate age level and content to each enrolled child (age birth through five) each month. The program is available nationally. In Central Minnesota, parents can enroll their children (free) through the local United Way (UW). UW's local Success by Six Initiative is the overall sponsor of the project. Local businesses provide financial support. Some also take an active part in encouraging their employees who are parents to participate by holding enrollment drives at the workplace and offering incentives for participation. Due to the overwhelming interest in the Imagination Library program in Central Minnesota, Murdoch Johnson designed a case study evaluation. The results will be used to inform funders and other stakeholders, as well as drive program improvements.

Murdoch’s design included surveys, focus groups, in-depth interviews and observations to collect both quantitative and qualitative data from parents, kindergarten teachers, early childhood specialists, and other stakeholders. The case study design is intended to provide a “rich description” regarding the effectiveness of the Imagination Library program in encouraging parents to read to their children, with intended outcomes of increasing early literacy and kindergarten readiness. Murdoch is currently completing the evaluation as a pro bono project.

 

 

Denis Marcheterre
The Inter-American Development Bank's Knowledge Management Capacity: IDB Employees' Perceptions on the Bank's Readiness to Become a Learning Organization, and Knowledge Management Experiences from Sister Organizations

A number of obstacles obscure knowledge management for institutional learning. Organizations are often faced with improper organizational culture to learning; staff turnover and loss of institutional memory; lack of knowledge management systems; risk-adverse and unproductive administrative requirements; little space for innovation and experimentation; and inappropriately designed monitoring and evaluation processes. Through its case studies, the American Productivity and Quality Center (APQC) has found that knowledge management within organizations generally occurs in five developmental stages: unique issues, requirements, strategies, tools, and actions.

With this in mind, Denis Marcheterre designed his evaluation project to provide information to his organization about employees’ perceptions of the organization’s readiness to become a learning organization. He also collected lessons learned and best practices from other organizations. Denis used interviews and survey data collected from employees, along with relevant information about two comparable organizations and case study information available from the APQC.

 

 

 

David Merves
Meta-Evaluation of a State Improvement Grant Evaluation

This project reviewed the Meta-Evaluation, as indicated by the Meta-Evaluation standard of the Program Evaluation Standards (PES), of an Evaluation of a State Improvement Grant. The Office of Special Education Programs in the US Department of Education funded this State Improvement Grant (SIG). Evaluators are required to be contracted for these grants, which typically last five years and have multiple initiatives. This grant had three initiatives: Comprehensive Literacy Development, Secondary Transition and Personnel Preparation. The findings from the analysis included a description of data sources, methods and conclusions about the evaluation utilizing the PES-based checklist developed by Stufflebeam, and four guiding questions. The paper concluded that this Evaluation of a State Improvement Grant was useful, feasible, proprietary and accurate.

 

 

 

Aimee Moreno
Building Evaluation Capacity in Small Nonprofit Organizations

Aimee Moreno's practicum project was developed to further what we know on how to build the internal capacity of small nonprofit and/or educational organizations. It was also intended to examine how to use evaluation both for external or reporting needs, and perhaps even more importantly, for ongoing program learning and improvement. Her project was conducted to deepen the understanding of the more collaborative and participatory evaluation theories and approaches, and to create and pilot a “first steps” evaluation training. The pilot would be geared towards begining the understanding of where a particular organization is starting from and what would be needed to move them forward. The knowledge from this study will be used to determine on a broader scale what may be needed for evaluation capacity-building in other small nonprofit organizations.

 

 

 

Kathleen Norris
Graduate Admissions at Plymouth State University

Plymouth State University (PSU), in New Hampshire, is faced with new challenges in its graduate admissions process. On the one hand, the shifting demographics at PSU are changing the admissions process. What was once a student body made up almost exclusively of part-time students who are also working professionals, now includes more full-time students and students coming directly from undergraduate programs. On the other, PSU has recently expanded the number and types of degree and certificate programs offered for graduate studies.

Dr. Kathleen Norris recognized the need to change the graduate admissions process to address these changes. As part of PSU’s strategic plan, Kathleen performed an evaluation of the PSU graduate admissions process, using surveys, focus groups, and admissions data. This will allow the university to move forward on sure footing, communicating in a way most appropriate for the new, less experienced applicants, while making sure nothing is lost in the shuffle with the rapid growth of the university.

 

Christiane Parry
Moving the Dial: A Strategy to Improve Whale Tail Grant Project Evaluations

Administered by the California Coastal Commission, the Whale Tail Grants Program funds non-profit organizations and government agencies to conduct marine education and restoration projects in California. The program awards about $700,000 in each annual grant cycle. The maximum award is $50,000; most grants are in the $10 - $30,000 range. The overarching goal of these projects is to engender a sense of stewardship for the state’s marine and coastal resources. Grantees are a heterogeneous group – from individual classroom teachers to large government agencies – with a wide range of evaluation expertise and experience.

Chris Parry’s practicum project aimed to improve program outcomes overall through better evaluation and by helping to build evaluation capacity where needed. It evaluated the current state of affairs in Whale Tail grantee evaluations, analyzed evaluation needs and success cases, researched tools and best practices, and based on this research, developed a strategy to assist grantees with program planning and evaluation. The strategy includes more detailed guidelines and requirements for grant proposals, and a menu of training and technical assistance options for grantees.

 

Paula Plonski
Seminar on Advanced Focus Group Skills

Paula Plonski created a training module for Praxis Research, Inc., a consulting firm that specializes in the evaluation of educational and human service programs. Praxis’ objective is to provide quality evaluation, planning, and organizational services to public and private entities. Praxis uses a variety of techniques to collect information for its clients. These include surveys, focus groups, interviews, structured observations, and case study analysis.

Paula’s training is designed to provide focus group moderators with advanced skills related to the physical setting details, moderator techniques and communication options, group processes, participant roles, and methodological issues. The training module also included diagnostic case studies covering familiar focus group issues.

 

 

 

Kathryn Race
Developing Program Model and Outcome Metrics in an Outcomes-based Evaluation

Kathryn Race’s practicum project demonstrated through an illustrative example how metrics can be developed to measure the core strategies or components associated with a specific program model and showed how this information might be used in subsequent outcomes-based analyses. The basic premise of this work asserted that it is not enough to describe an intervention through an articulated program model (although an important step); it is also necessary to empirically test it and that both program development and evaluation can benefit from this process (Donaldson, 2003; Chen, 2005).

The program Kathryn based her study on is a network of private high schools that provides high-quality, Catholic, college preparatory education to students from low-income, limited opportunity communities. These schools are primarily financed through an innovative Corporate Internship Program (CIP) where students work throughout high school, in entry-level business positions. As a result, the student earnings contribute toward their own tuition and the students obtain access to a corporate work experience and life outside their immediate neighborhood (Race & Brett, 2005).

First, colleague Belle Brett and Kathryn created a program model based on a review of internal documents, select published literature, and interviews with stakeholders and staff and used a “backwards” mapping process to link program strategies and student learning outcomes. Based on this model, a student survey was developed, which was designed to measure components of the model and designated learning outcomes as a first step in this metric development. Kathryn’s project then focused on how she tested the survey to see how well it measured contextual factors that could be used to assess the program model, its adequacy as a measure of student perceptions of select learning outcomes, and then described how these data could be used in subsequent outcomes-based analyses.

 

 

 

 photo of Veronica Smith

Veronica Smith
Design, Development and Implementation of a Strategic Planning Dashboard for a Public Radio Station Board of Trustees

KUOW is a public radio station that serves communities in the Puget Sound region, Western Washington and Southern British Columbia. The station is operated by KUOW Puget Sound Public Radio (PSPR) under an agreement with the University of Washington, KUOW's licensee. PSPR was established in 1999 and is a private, non–profit corporation governed by a community board consisting of 20 members. Veronica Smith led the effort to more systematically monitor and evaluate KUOW's progress against its strategic plan, resulting in the design and implementation of a strategic planning dashboard.The process began with a facilitated board discussion, which resulted in a list of committee action items, including creating and organizing metrics for more frequent board review. Veronica partnered with management and board to identify metrics for the organization’s strategic priorities. She also thoughtfully designed the dashboard to ensure effective and efficient communication. Finally, she created practices and protocols to sustain data quality, use, and accuracy. The dashboard roll out represented a successful strategic planning initiative that 1) resulted in improved collaboration between management and board and 2) increased quality of management and board self-evaluation, monitoring, and evidence-based decision making.

 

 

 

Warren Wylupski
Golden Spread Rural Frontier Coalition Project

Warren Wylupski used the evaluation tools he learned at CGU and lessons learned in the field to support the Golden Spread Rural Frontier Coalition with its application for grant funding from the State of New Mexico Office of Substance Abuse. As a newly funded entity, the Golden Spread Rural Frontier Coalition focuses on identifying substance abuse prevention needs and developing coalition-based activities to be implemented. A coalition program develops its community members and stakeholders so that they mobilize to affect substance use in the community.

Warren helped Golden Spread complete a needs assessment, community capacity assessment of gaps and resources, and a strategic plan. To address recruitment issues in community, Warren also completed a stakeholder analysis, a stakeholder reporting and communication plan, and a visual representation of the program theory as a logic model.

 

 

link to go to Evaluation Certificate program homepage link to go to Evaluation Certificate faculty page link to page on "Why earn a certificate in evaluation?" link to Evaluation Certificate components and costs webpage link to webpage of participants and graduates of Evaluation Certificate program link to Evaluation Certificate history webpage

link to page on "How to Join the Evaluation Certificate Program"