Certificate of Advanced Study in Evaluation

Examples of Our Graduates' Work

Below is a list of recent participants in our certificate program, and the titles of the work they completed as a portion of the program.  Project descriptions are coming for all participants, and will be posted when available.

Cindy Beckett Even Start Program Analysis The California Even Start Literacy Program in California currently consists of 154 programs that serve ESL learners and their families. The program consists of four components that are shown to enhance literacy and include Adult Education, Early Childhood Education, Parenting Education and Parent/Child Interaction (PCILA). The program is blanketed under the Federal “No Child Left Behind” legislation and serves to increase family literacy skills for both children and adults. The collection of data reflective of those indicators is required by programs on a state wide level in order to measure program outcomes. Indicators were developed at the national level and based on sound research in family literacy. Cindy Beckett’s study investigated areas of functioning at the program level by identifying the differences in program processes in a variety of program areas, the evaluation activities that support these processes, and the dynamics of evaluator roles. During the project year, programs were required to complete a “self-study” in order to measure progress and identify strengths and weaknesses. The results of the self study along with information gleaned from telephone interviews with project directors were utilized to explore the areas of interest. Cindy found that tools such as the self-study and the gathering of additional qualitative data help to identify areas of weakness that can help programs at the program level by creating efficient systems. As in Empowerment Evaluation, the self-study process helps programs to critically look at their processes and to strategize in order to change systems and create healthy functioning organizations.

Shannon Bettridge Development of a Logic Model: Lessons Learned The Law and Mental Health Program (LAMHP) is a distinct clinical program within a large, urban psychiatric hospital in Toronto, Ontario, Canada. The LAMHP is responsible for the assessment, treatment and supervision of a large number of individuals who have a mental illness and have come into contact with the criminal justice system. All patients are involuntary and the LAMHP itself has limited control over the referral or discharge of its own clients. At all times, staff and leadership in the LAMHP must balance the treatment needs of patients with the legal requirements and responsibilities bestowed upon forensic programs (including protection of the public) as outlined in the Criminal Code of Canada and other relevant legislation. As Manager of Program Development and Evaluation at LAMHP, Shannon worked with the directors, managers, and senior clinicians of the program to develop a system-level LAMHP logic model, from which other more detailed program-level logic models and a clearly prioritized and developed evaluation plan can be developed.

Kristine Chadwick Evaluation Purposes, Methods, & Relationship to Program Planning; Program Theory; Implementation Evaluation; and Outcome Evaluation (4 Module Training Series)

Jaime Chua Shifting Gear from Logical Framework to Theory Driven Evaluation: A Descriptive Study of Techno-Structural and Behavioral Change Management Jaime Chua's evaluation study describes an international nonprofit organization committed to the development of a peaceful, prosperous, just, and open Asia-Pacific region. To realize this mission, the organization supports high-impact programs that help improve governance and economic reform, women's empowerment, and regional relations in collaboration with civil society, private sector, and public partners. Specific activities include leadership training, organizational development, policy research, and institutional reform. In response to increasing expectations for performance accountability, Jaime evaluated existing evaluation policy at his workplace. Jaime designed his exploratory and descriptive evaluation study to: (1) assess the current program design and evaluation practice and (2) recommend and substantiate an alternative program design and evaluation methodology. Jaime used the CDC’s six-step evaluation framework as a guide and used in-depth interviews, document analysis, and participant observations for data collection. Based on his analysis, he also developed a set of technical, structural, and behavioral change management plan to convert the organization’s logframe-based evaluation model into a theory-driven evaluation model.

Craig Evans Challenges Faced by Emancipating Foster Youth

Sonja Evensen Basics of Evaluation (Modules 1 through 5) Sonja Evensen created a five-module evaluation training series for Pacific Resources for Education and Learning (PREL), an independent, nonprofit 501(c)(3) corporation that serves the educational community in the U.S.-affiliated Pacific islands, the continental United States, and countries throughout the world. PREL bridges the gap between research, theory, and practice in education and works collaboratively with schools and school systems to provide services that range from curriculum development to assessment and evaluation. Sonja created the training series in response to the January 2005 Federal Register Notice, which called on educational researchers to apply “scientifically-based research” to evaluate the effectiveness of their programs. The modules are: (1) Defining Evaluation: Meanings, Uses, Types; (2) Steps in Program Evaluation: Logic Model; (3) “Evaluation Consumer” Basics; (4) Data Basics; and (5) Experimental Design. The modules can be used as part of a series or as a stand alone and have been used in different settings both in Hawaii and among project partners in the Pacific Region.

Leslie Fierro What are we preparing for? A description of the status of evaluation coursework within master's level epidemiology programs Leslie Fierro is interested in enhancing existing capacity to conduct evaluations in the discipline of public health. During her career as an epidemiologist, she has noticed an increase in the need for evaluation and the number of requests for epidemiologists to assist or lead these evaluations. Epidemiologists are trained to study the distribution and determinants of diseases in populations. As part of this training, epidemiologists acquire a number of quantitative skills that may be valuable in an evaluation setting. Although epidemiologists have some tools that are useful for conducting evaluations, recent surveys assessing the training needs of practicing epidemiologists indicate that there is a need for basic instruction in evaluation. For example, in a recent 2006 National Assessment of Epidemiology Capacity Survey conducted by the Council of State and Territorial Epidemiologists, 45% of respondents agreed that additional training was needed for epidemiologists in developing “program logic models and theories of action” (CSTE, 2006) (p. 20). The goal of Leslie’s practicum project was to provide a description of the exposure master’s level epidemiologists get to introductory coursework in program and intervention evaluation as part of their academic training. Leslie systematically reviewed the websites for all schools of public health accredited by the Council of Education in Public Health between January 2007 and April 2007. Descriptions of required and elective coursework for students seeking a master’s level degree in epidemiology were extracted from 36 of the 38 schools of public health and classified based upon their emphasis on program or intervention evaluation. A total of 1,538 courses descriptions reviewed, few focused on program or intervention evaluation. Findings indicate that there is a need for additional coursework that emphasizes introductory concepts in program and intervention evaluation within master’s level epidemiology curricula.

Charles Gasper Evaluation of Nursing Perceptions Surrounding the Medication Delivery Process Affected by Bridge Bar Coding Implementation As noted in the evaluation study by Charles Gasper, a minimum of 44,000 deaths occur each year due to adverse drug events (Kohn, Corrigan & Donaldson, 1999). Nearly 78% of these errors, according to recent national data, are preventable using bar coding technology (Hicks, Santell, Cousins, & Williams, 2004). In 2004, the Sisters of Mercy Health System embarked upon the implementation of a bar coding and electric medication record to help alleviate this problem – Bridge. Charles Gasper conducted an evaluation of nurse perceptions related to the Bridge medication bar code scanning system at nine hospitals within the Mercy network.  His evaluation looked, in part, at the change in perceptions in patient safety pre- and post- implementation of the system. Using surveys, Charles collected data that allowed him to look at differences by units within the hospital, by hospital within the network, and system-wide. His study also compiled data related to the frequency and type of medication delivery errors. Further work leveraged the Bridge system to evaluate the impact of various nursing ratios on the clinical outcomes of patients. Using the rate of medications administered as a measure of workload along with unit layout, access to communication devices, and acuity of patient support need, he assessed the impact of the patient to nurse ratio for medical and surgical nursing units on complication rates, patient length of stay, and mortality rates. The study identified ratios that were strongly correlated with a lower rate of significant patient complications and was used by the Mercy hospitals to adjust staffing patterns to provide better quality care.

James Luther Mission Statements and Values of Evaluators The association between social justice and evaluation has existed for over 25 years (Kushner, 2005). Newman and Brown (1996) point out how important it is for evaluators to have commitment to social justice because social justice can have a major impact on interpretation of findings. Donaldson and Christie (2006) argue that most evaluators have some commitment to social justice and point to professional association ethical standards and guidelines as supporting that argument on the professional level. Making a public commitment to incorporating social justice in evaluation would seem to be a first step toward the application of social justice commitment. Since Mission Statements help define the goals of the company, they are one means by which the organization can publicly state their values. In response to literature and research in this area, James Luther conducted a research project to explore mission statements of evaluators and to discover if there was evidence of a commitment to social justice on the part of these evaluators. James’ project also sought to determine the “level” evaluators focused their evaluation efforts and potentially target their commitment to social justice – at the macro or micro level. Macro level social change models have the goal of shifting power imbalances at the societal level. Alternatively, micro level models are aimed at empowering individuals in organizations and at the program level. James obtained mission statements from the websites of 16 evaluation agencies and performed quantitative data analysis, looking at similarities across and differences by geographic region and for-profit status.

Murdoch Johnson Imagination Library Case Study The Imagination Library program is funded and run by the Dollywood Foundation in Tennessee. It mails a book of appropriate age level and content to each enrolled child (age birth through five) each month. The program is available nationally. In Central Minnesota, parents can enroll their children (free) through the local United Way (UW). UW's local Success by Six Initiative is the overall sponsor of the project. Local businesses provide financial support. Some also take an active part in encouraging their employees who are parents to participate by holding enrollment drives at the workplace and offering incentives for participation. Due to the overwhelming interest in the Imagination Library program in Central Minnesota, Murdoch Johnson designed a case study evaluation. The results will be used to inform funders and other stakeholders, as well as drive program improvements. Murdoch’s design included surveys, focus groups, in-depth interviews and observations to collect both quantitative and qualitative data from parents, kindergarten teachers, early childhood specialists, and other stakeholders. The case study design is intended to provide a “rich description” regarding the effectiveness of the Imagination Library program in encouraging parents to read to their children, with intended outcomes of increasing early literacy and kindergarten readiness. Murdoch is currently completing the evaluation as a pro bono project.

Denis Marcheterre The Inter-American Development Bank's Knowledge Management Capacity: IDB Employees' Perceptions on the Bank's Readiness to Become a Learning Organization, and Knowledge Management Experiences from Sister Organizations A number of obstacles obscure knowledge management for institutional learning. Organizations are often faced with improper organizational culture to learning; staff turnover and loss of institutional memory; lack of knowledge management systems; risk-adverse and unproductive administrative requirements; little space for innovation and experimentation; and inappropriately designed monitoring and evaluation processes. Through its case studies, the American Productivity and Quality Center (APQC) has found that knowledge management within organizations generally occurs in five developmental stages: unique issues, requirements, strategies, tools, and actions. With this in mind, Denis Marcheterre designed his evaluation project to provide information to his organization about employees’ perceptions of the organization’s readiness to become a learning organization. He also collected lessons learned and best practices from other organizations. Denis used interviews and survey data collected from employees, along with relevant information about two comparable organizations and case study information available from the APQC.

Kathleen Norris Graduate Admissions at Plymouth State University Plymouth State University (PSU), in New Hampshire, is faced with new challenges in its graduate admissions process. On the one hand, the shifting demographics at PSU are changing the admissions process. What was once a student body made up almost exclusively of part-time students who are also working professionals, now includes more full-time students and students coming directly from undergraduate programs. On the other, PSU has recently expanded the number and types of degree and certificate programs offered for graduate studies. Dr. Kathleen Norris recognized the need to change the graduate admissions process to address these changes. As part of PSU’s strategic plan, Kathleen performed an evaluation of the PSU graduate admissions process, using surveys, focus groups, and admissions data. This will allow the university to move forward on sure footing, communicating in a way most appropriate for the new, less experienced applicants, while making sure nothing is lost in the shuffle with the rapid growth of the university.

Paula Plonski Seminar on Advanced Focus Group Skills Paula Plonski created a training module for Praxis Research, Inc., a consulting firm that specializes in the evaluation of educational and human service programs. Praxis’ objective is to provide quality evaluation, planning, and organizational services to public and private entities. Praxis uses a variety of techniques to collect information for its clients. These include surveys, focus groups, interviews, structured observations, and case study analysis. Paula’s training is designed to provide focus group moderators with advanced skills related to the physical setting details, moderator techniques and communication options, group processes, participant roles, and methodological issues. The training module also included diagnostic case studies covering familiar focus group issues.

Kathryn Race Developing Program Model and Outcome Metrics in an Outcomes-based Evaluation Kathryn Race’s practicum project demonstrated through an illustrative example how metrics can be developed to measure the core strategies or components associated with a specific program model and showed how this information might be used in subsequent outcomes-based analyses. The basic premise of this work asserted that it is not enough to describe an intervention through an articulated program model (although an important step); it is also necessary to empirically test it and that both program development and evaluation can benefit from this process (Donaldson, 2003; Chen, 2005). The program Kathryn based her study on is a network of private high schools that provides high-quality, Catholic, college preparatory education to students from low-income, limited opportunity communities. These schools are primarily financed through an innovative Corporate Internship Program (CIP) where students work throughout high school, in entry-level business positions. As a result, the student earnings contribute toward their own tuition and the students obtain access to a corporate work experience and life outside their immediate neighborhood (Race & Brett, 2005). First, colleague Belle Brett and Kathryn created a program model based on a review of internal documents, select published literature, and interviews with stakeholders and staff and used a “backwards” mapping process to link program strategies and student learning outcomes. Based on this model, a student survey was developed, which was designed to measure components of the model and designated learning outcomes as a first step in this metric development. Kathryn’s project then focused on how she tested the survey to see how well it measured contextual factors that could be used to assess the program model, its adequacy as a measure of student perceptions of select learning outcomes, and then described how these data could be used in subsequent outcomes-based analyses.

Warren Wylupski Golden Spread Rural Frontier Coalition Project Warren Wylupski used the evaluation tools he learned at CGU and lessons learned in the field to support the Golden Spread Rural Frontier Coalition with its application for grant funding from the State of New Mexico Office of Substance Abuse. As a newly funded entity, the Golden Spread Rural Frontier Coalition focuses on identifying substance abuse prevention needs and developing coalition-based activities to be implemented. A coalition program develops its community members and stakeholders so that they mobilize to affect substance use in the community. Warren helped Golden Spread complete a needs assessment, community capacity assessment of gaps and resources, and a strategic plan. To address recruitment issues in community, Warren also completed a stakeholder analysis, a stakeholder reporting and communication plan, and a visual representation of the program theory as a logic model.