Professional Development Workshops
August 22-27, 2008
Presenter Bios
Click on the names and photos below to learn more about our presenters.
Workshop Descriptions
Friday, August 22, 2008
|
|
 
Stewart I. Donaldson & Christina A. Christie
Basics of Evaluation & Applied Research Methods
This workshop will provide participants with an overview of the core concepts in evaluation and applied research methods. Key topics will include the various uses, purposes, and benefits of conducting evaluations and applied research, basics of validity and design sensitivity, strengths and weaknesses of a variety of common applied research methods, and the basics of program, policy, and personnel evaluation. In addition, participants will be introduced to a range of popular evaluation approaches including the transdisciplinary approach, program theory-driven evaluation science, experimental and quasi-experimental evaluations, empowerment evaluation, fourth generation evaluation, inclusive evaluation, utilization-focused evaluation, and realist evaluation. A complimentary copy of Program Theory-Driven Evaluation Science: Strategies and Applications (2007: Erlbaum) will be provided to all registered participants. This workshop is intended to provide participants with a solid introduction, overview, or refresher on the latest developments in evaluation and applied research, and to prepare participants for intermediate and advanced level workshops in the series.
Questions regarding this workshop may be addressed to Stewart.Donaldson@cgu.edu.
|
|
Dale E. Berger
Introduction to Applied Quantitative Analysis
Descriptive and inferential statistics are essential for evaluating programs and for using data to guide policy decisions. This introductory/refresher course will focus on practical approaches to data analysis. Topics include first looks at data (preparing data for analysis, detecting problems, dealing with unusual data, visualizing patterns in data), selecting appropriate methods of analysis for the research questions of interest (treating your data properly, providing appropriate models, relating analyses to the motivating research questions, exploring data for unanticipated relationships, understanding limitations), and presenting your findings (interpreting results, describing results for a non-technical audience). Participants are invited to bring research questions to be used as case studies. We will discuss alternative approaches to analysis with a view toward providing a conceptual understanding of basic and advanced statistical techniques that may be used with consultants, in funding applications, and in reports. Prerequisites: None, although some familiarity with basic statistics will be helpful.
Questions regarding this workshop may be addressed to dale.berger@cgu.edu.
|
|
Michelle C. Bligh
Introduction to Qualitative Research Methods
This workshop is designed to introduce you to different types of qualitative research methods, with a particular emphasis on how they can be used in applied research and evaluation. Although you will be introduced to several of the theoretical paradigms that underlie the specific methods that we will cover, the primary emphasis will be on how you can utilize different methods in applied research and consulting settings. We will explore the appropriate application of various techniques, and review the strengths and limitations associated with each. In addition, you will be given the opportunity to gain experience in the use of several different methods. Overall, the workshop is intended to provide you with the basic skills needed to choose an appropriate method for a given project, as well as primary considerations in conducting qualitative research. Topics covered will include field observation, content analysis, interviewing, document analysis, and focus groups.
Questions regarding this workshop may be addressed to michelle.bligh@cgu.edu.
|
|
Allen M. Omoto
Writing Successful Grant Proposals
This workshop covers some of the essential skills and strategies needed to prepare successful grant applications for education, research, and/or program funding. It will provide participants with tools to help them conceptualize and plan research or program grants, offer ideas about where to seek funding, and provide suggestions for writing and submitting applications. Some of the topics covered in the workshop include strategies for identifying appropriate sources of funding, the components and preparation of grant proposals, and the peer review process. In addition, topics related to putting together a research or program team, constructing an appropriate budget, grants management, and the writing of an application will be discussed. The workshop is organized around key questions relating to grant support and how to become a successful grant-getter, including WHY seek grant funding or support? WHERE to look for support? WHO applies for funding and WHEN should one seek funding? WHAT is submitted in a grant application? And, HOW to structure an application and supporting materials? The workshop is intended primarily as an introduction to grant writing, and will be most useful for new or relatively inexperienced grant writers. Workshop participants are encouraged to bring their own "works in progress" for comment and sharing. At its conclusion, workshop participants should be well positioned not only to read and evaluate grant applications, but to assist with the preparation of applications and to prepare and submit their own applications to support education, research, or program planning and development activities.
Questions regarding this workshop may be addressed to allen.omoto@cgu.edu.
|
Saturday, August 23, 2008
|
|
 Huey T. Chen & Stewart I. Donaldson
Practical Program Evaluation: A Program Theory Approach
This workshop will focus on how to design practical and cost-effective program evaluations. A step-by-step approach will be used to teach participants how to conduct accurate and useful program evaluations. Key topics will include engaging stakeholders, formulating and prioritizing evaluation questions, needs assessment, expressing and assessing program theory, program implementation, managing resistance and excessive evaluation anxiety, formative evaluation and continuous improvement feedback, determining program outcomes/impact, and cost-benefit and cost-effectiveness analysis. A complimentary copy of Program Theory-Driven Evaluation Science: Strategies and Applications (2007: Erlbaum) will be provided to all registered participants.
Questions regarding this workshop may be addressed to stewart.donaldson@cgu.edu.
|
|
Hallie Preskill
Building Evaluation Capacity
Evaluation capacity building (ECB) has become a hot topic of conversation, activity, and study in recent years. Interest in ECB can be seen in the: 1) 242 AEA conference sessions between 2000 and 2007 that had a least one paper related to the subject , and the 2) birth of the American Evaluation Association’s Organizational Learning and Evaluation Capacity Building Topic Interest Group in 2006. Seeking to enhance stakeholders’ understanding of evaluation concepts and practices, and in an effort to create evaluation cultures, many organizations have been implementing a variety of strategies as a means of helping their members learn about and engage in evaluation practice. In this workshop, we will explore the purposes, motivations, objectives, and ten strategies for building organization members’ evaluation capacity. The workshop content will based on an extensive review of the literature as well as a newly developed multidisciplinary model of ECB. A combination of mini-lectures, individual work, small group activities, and large group discussions will be used to facilitate participants’ learning.
Questions regarding this workshop may be addressed to hallie.preskill@cgu.edu.
|
|
 Sandra Mathison
Advanced Qualitative Evaluation
This interactive workshop consists of two components: the first is the development of knowledge and skill in inductive approaches to focusing and conducting qualitative evaluation, and the second is the development of knowledge and skill in narrative and image based approaches to evaluation. The workshop will provide a framework for planning evaluations by and with stakeholders through the use of fourth generation evaluation techniques. Through the use of case examples and role playing participants will learn how to use the inductive approach of investigating stakeholder claims and concerns about an evaluand to develop evaluation questions and data collection strategies. The second part of the workshop will focus on cutting edge qualitative data collection, analysis, and representation strategies in evaluation. The workshop will focus specifically on narrative and image based approaches. By exploring examples of these approaches and through exercises in using these techniques participants will develop skills to use these approaches. Participants should be familiar with basic evaluation concepts and have an introductory knowledge of qualitative research methods.
Questions regarding this workshop may be addressed to sandra.mathison@ubc.ca.
|
|
 Michael Scriven
Logic of Evaluation and Causality: What Makes What Works Work?
The Logic of Evaluation and Causality: What Makes What Works Work is a combination of an overview of the discipline, with the reasons for thinking that it is a discipline, with an outline of the basic concepts that work across all subdivisions of the field, ie., personnel and product evaluation (and seven more) as well as program evaluation. An overview will also be provided of the two sides in the 'causal wars' and a summary of the case for doing something other than treating the randomly controlled trial (RCT) approach as 'the gold standard' by introducing a new model for all causal investigations.
Questions about this workshop may be e-mailed to scriven@aol.com.
|
Sunday, August 24, 2008
|
|
 Rodney K. Hopson
Considering Culture in Evaluation and Applied Research
How does culture matter in evaluation theory and practice? How does attention to cultural issues make for better evaluation practice? Does your work in an agency or organization require you to know what culturally responsive in evaluation looks like? What issues do you need to consider in building culturally competent and responsive evaluation approaches? How do agencies identify strategies for developing and disseminating culturally responsive evaluation information? This workshop aims to address these questions and others as we consider culture in evaluation and applied research for the student, professional, or academic in a plethora of non-profit, philanthropic, educational agencies and organizations.
This workshop will rely on various interdisciplinary social science theoretical frameworks to both situate and advance conversations about culture in evaluation and applied research. In particular, participants will receive information and materials that help them to develop expertise in the general topics of culture in evaluation, including understanding the value-addedness for the evaluation researcher or program specialist who needs to develop a general understanding of the topic itself. This workshop will also assist participants to assist their own agencies to become culturally responsive and competent. That is, participants will develop tips and tools for working with their own agencies who work with underserved, minoritized, or priority groups.
The instructor will encourage opportunity to raise questions for participants who address current issues in their agencies and organizations. Registered participants will be expected to communicate with the instructor prior to the workshop to best assess participant and/or agency needs for optimal learning and application upon completion of the workshop.
Level: Intermediate.
Questions regarding this workshop may be addressed to hopson@duq.edu.
|
|
 Thomas Chapel
Logic Models for Program Evaluation and Planning
Many programs fail to start with a clear description of the program and its intended outcomes, undermining both program planning and evaluation efforts. The logic model, as a map of what a program is and intends to do, is a useful tool for clarifying objectives, improving the relationship between activities and those objectives, and developing and integrating evaluation plans and strategic plans. First, we will recapture the utility of program logic modeling as a simple discipline, using cases in public health and human services to explore the steps for constructing, refining and validating models. Then, we’ll examine how to use logic models effectively in some key program evaluation steps, including gaining stakeholder consensus and determining an evaluation focus. Finally, we'll present, in an overview fashion, some ways in which logic models can help in selecting performance measures and in defining program mission/vision and priority areas for strategic planning. Course presentations are supplemented by extensive small group case study work.
NOTE: Those taking the workshop on Health Program Evaluation will find that some portions of this course's content overlaps.
Questions regarding this workshop may be addressed to tchapel@cdc.gov.
|
|
 Melvin M. Mark
Evaluation Use, Influence, Consequences: Research, Theory, and "Minding the Gap" to Practice
Evaluators (like other applied researchers) typically hope that their work will make a difference. But what kind of difference? And how? This workshop draws on the literature (conceptual, empirical, and practice-based) about the ways that evaluation can make a difference. Classic ideas about use are evolving to attend more broadly to evaluation influence and consequences. In addition, long-standing complaints about evaluation going unused are in some quarters giving way to concerns about automatic and thoughtless use of evaluation findings. The workshop draws upon the literature and on the practice-won wisdom of several seasoned evaluators, and sketches different models and evaluator roles related to evaluation influence. Exercises will include trying on alternative roles related to use/influence, and building an "influence plan," analogous to a program theory or logic model.
Questions about this workshop should be addressed to m5m@psu.edu.
|
|
 Ross Conner
Getting Outside the American Box: New Perspectives on Evaluation’s Function, Focus and Utility from Around the World
Evaluation is now occurring in many countries around the world, with over 100 formal and information national and regional associations and networks of evaluators in both developed and developing countries. With the recent spread of evaluation thinking and activity to locally-based evaluators, particularly in developing countries and regions, new perspectives have been added to the evaluation discussion. At this workshop, Ross Conner will share some of these new perspectives and ideas about evaluation’s function, focus, conduct and use. His goal is to have participants seeing, thinking and feeling about evaluation in new and different ways.
To do this, Conner will draw upon information from evaluators in a variety of countries, including Africa, Russia and the NIS (Newly Independent States), Malaysia, Romania and Israel. In 2006 and 2007, Conner served as the American Evaluation Association’s representative to and the first president of the International Organisation for Cooperation in Evaluation (IOCE), the ‘United Nations’-like organization of national and regional evaluation groups around the globe. In highly interactive, discussion-focused workshop sessions, he will share his IOCE experiences and learnings from different countries and regions, as ways to spark new perspectives on evaluation for workshop participants. As part of the workshop, he will share suggestions for evaluators who want to step outside their ‘usual cultural box,’ and begin to undertake work in other cultures. This discussion will include a focus on North American evaluators who want to work overseas, as well as on evaluators from outside the US who want to undertake work here.
Questions about this workshop should be addressed to rfconner@uci.edu.
|
Monday, August 25, 2008
|
|
  Tiffany Berry & Rebecca M. Eddy
Using RCTs (Randomized Control Trials) to Improve Educational Research and Evaluation
Current federal priorities for evaluation designs focus on increasing rigor in research and evaluation through the incorporation of Scientifically-based research (SBR). SBR generally involves conducting Randomized Control Trials (RCTs), which can be successfully carried out in a variety of applied settings. Despite a continuing controversy regarding the appropriateness of using these types of designs in educational settings specifically, local program and school district evaluators often find it necessary to conduct this type of research. The purpose of this workshop is to provide an overview of RCTs as well as practical strategies designed to improve educational research and evaluation. Based on our experience conducting RCTs on K-12 educational curricula, we will engage participants in class exercises and small-group discussions to examine issues related to the following: (1) recruiting districts, schools, and teachers; (2) collecting baseline and posttest data (involving multiple measures and methods) efficiently and accurately across schools; (3) monitoring and describing implementation fidelity; (4) analyzing data appropriate for designs dealing with nested data; and (5) reporting guidelines consistent with the What Works Clearinghouse (WWC). No evaluation experience required, but evaluation experience in educational settings is preferred.
Questions regarding this workshop may be addressed to rebecca.eddy@cgu.edu or tiffany.berry@cgu.edu.
|
|
Thomas Chapel
Evaluating Public Health Programs and Initiatives: Using CDC's Framework for Program Evaluation as a Guide
Public health presents some unique challenges for program evaluators. Long-term outcomes are often confounded by multiple factors, and the nature of community interventions makes the application of traditional research models either inappropriate or short-sighted. Using CDC's 6-step Framework for Program Evaluation in Public Health as an organizing structure, the class will work through a series of cases and find effective ways to address common evaluation challenges. In the process, we hope to demonstrate the added-value of a utilization-focused approach to identifying and surmounting these unique challenges. While the course uses public health programs as its focus, extensive knowledge of public health is not needed, and anyone with an interest in practical approaches to program evaluation will benefit.
Prerequisies: Some familiarity with general evaluation terminology.
NOTE: Those taking the workshop on Logic Models will find that some portions of this course's content overlaps.
Questions regarding this workshop may be addressed to tchapel@cdc.gov.
|
|
 Jeanne Nakamura
Introduction to the Measurement of Experience
What do people do with their time, and how do they feel about it? Within psychology, this question animates much basic, applied, and evaluation research. In recent decades psychologists have begun to study in earnest the quality of everyday life. Researchers at SBOS’s Quality of Life Research Center pioneered some of the methods that make it possible to get an accurate sense of how people actually live.
One-time, self-report measures are widely relied upon to assess people’s use of time, and their thoughts, feelings, motivations, and well-being. These measures have several advantages but depend on processes of estimation and retrospection that distort the reality they are intended to assess. In this workshop participants will learn about the strengths and limitations of various methods – surveys, diaries, and tools for real-time measurement of experience in natural settings. Primary attention will be given to the Experience Sampling Method (ESM), considered the "gold standard" of the field. Issues to be considered include how to select the appropriate tool for a researcher’s questions and resources, how to design an ESM study, and how to analyze ESM data to answer a variety of questions such as: Where during a school day are students most engaged? How do moods change in response to environmental factors? When do patients with chronic illness report the least pain?
Questions about this workshop should be addressed to loren.bryant@cgu.edu.
|
|
  Sharon F. Rallis & Matthew Militello
Beyond the Basics and the Traditional: Designing Qualitative Evaluation and Applied Research Studies
This interactive course creates an opportunity for participants to conceptualize and design evaluations or applied research studies drawing from the multiple traditional, alternative, and innovative strategies/techniques for data collection and analysis. Designing any study entails decisions; becoming a good qualitative evaluator or applied researcher means becoming a good decision maker. You use your knowledge and skills: to choose an approach; in developing sound rationales that support those choices; and in marshalling evidence. This course serves to hone this knowledge and skill set, as well as to explore among available tools and to match tools to purposes. For example, we will move beyond traditional interviews and observations to using methods such as photovoice, Q-sorts, and digital storytelling.
We introduce activities such as:
- Specifying the issue of interest; establishing purpose
- Defining program logic and building a conceptual framework
- Using the framework to center the inquiry and craft answerable and meaningful research questions
- Understanding the genres of qualitative research and the various tools each offers
- Using the framework to choose an appropriate and practical methodological approach
Through small-group work and critical feedback, participants will try out tools and build a preliminary conceptual framework and design for the conduct of a qualitative evaluation or applied research study of interest to you. We will begin with hands-on work using several innovative methods, allowing you to experience what purposes each might serve. We will link methods with research questions and program logic. Participants should come with a reasonably clear focus for a study to ensure optimal use of the course.
For more information about this workshop, contact sharonr@educ.umass.edu.
|
Tuesday, August 26, 2008
|
|
 Jason Siegel
Creating Effective Surveys for Applied Research
The focus of this hands-on workshop is to instruct attendees how to create reliable and valid surveys to be used in applied research. A bad survey is very easy to create. Creating an effective survey requires a complete understanding of the impact that item wording, question ordering, and survey design can have on a research effort. Only through adequate training can a good survey be discriminated from the bad. The day long workshop will focus specifically on these three aspects of survey creation. The day will being with a discussion of Dillman’s (2007) principles of question writing. After a brieef lecture, attendees will then be asked to use their newly gained knowledge to critique the item writing of selected national surveys. Next, attendees will work in groups to create survey items of their own. Using Sudman, Bradburn, and Schwatrz’s (1996) cognitive approach, attendees will then be informed of the various ways question ordered can bias results. As practice, attendees will work in groups to critique the item ordering from selected national surveys. Next, attendees will propose an ordering scheme for the questions created during the previous exercise. Lastly, using several sources, the keys to optimal survey design will be provided. As practice, the design of national surveys will be critiqued. Attendees will then work with the survey items created, and properly ordered, in class and propose a survey design. At the end of the day, attendees will never look at a survey in the same way again.
Questions about this workshop may be e-mailed to jason.siegel@cgu.edu.
|
|
 David Fetterman
Empowerment Evaluation
This session provides the basic steps of empowerment evaluation, including 1) mission; 2) taking stock; and 3) planning for the future. In addition, the workshop highlights guiding concepts including: critical friend, cycles of reflection and practice, community of learners, culture of evidence, and reflective practitioner. The principles of this approach are also explored, such as: improvement, community ownership, inclusion, democratic participation, social justice, community knowledge, evidence-based strategies, capacity building, organizational learning, and accountability. Real world examples are used to demonstrate the effectiveness of empowerment evaluation, ranging tobacco prevention and education programs and academically distressed school districts in Arkansas to a $15 million Hewlett-Packard Digital Village project (to bridge the digital divide in disenfranchised communities). Digital and online tools are also presented to facilitate the empowerment evaluation process and demonstrate how to transfer the technology to users.
Questions regarding this workshop should be addressed to fettermanassociates@gmail.com.
|
|
 Daryl G. Smith
Monitoring Progress on Campus Diversity
This workshop will focus on the question of how a campus can build its capacity to monitor progress on diversity and use the results to inform institutional decision making. At its core the workshop will ask –How Do we Know we are making progress. As diversity efforts increase, the growth of programs and initiatives leads campuses to believe that they are making progress. In this context, activity can be confused with institutional change. Designed for campus practitioners, evaluators, funders, and leaders, the workshop will use the results of work throughout the country to: share promising practices about what works in monitoring progress; introduce an institutional framework for campus diversity that is both inclusive and differentiated; and, suggest ways to incorporate these efforts into the center of an institution’s strategic and effectiveness efforts for excellence.
Questions about this workshop should be directed to Daryl.Smith@cgu.edu.
|
|
 William D. Crano
Quasi-Experimental Methods
Many introductory courses on research design start and end with the randomized experiment. This is a good beginning, but knowledge of "pure" experimental designs does not prepare researchers for the messier and more complicated (and often, more interesting) state of affairs commonly found in applied contexts. Just as important as knowledge of experimental designs is a thorough understanding of quasi-experimentation design and analysis. Though they often do not support causal explanations, "quasi" designs can provide evidence that may markedly reduce the range of possible explanations for an intervention or treatment. This workshop is designed to impart an understanding of quasi-experimental design and analysis.
The workshop begins with a consideration of randomized experiments, strengths and weaknesses, with particular emphasis on Campbell and Stanley’s threats to internal and external validity. Considerations of reliability validity will be an important feature of the beginning sections of our work. From here, we move to the land of the quasi-experiment. In this largest section of the workshop, we will consider designs that lack random assignment (a central feature of quasi-experiments), designs that lack a control group or pretest (or both), those that use one or another of these design features. Along the way, we will consider the dangers of matching groups to create initial equivalence, and the potential utility of the new propensity scoring approaches. After considering case control designs and analysis, we move to time series designs, with special emphasis on the use of interrupted time series studies. If time permits, we then consider regression/discontinuity designs. Analytic models useful for each of these design approaches will be discussed, but our emphasis is on design, not statistics.
Questions about this workshop should be addressed to william.crano@cgu.edu.
|
Wednesday, August 27, 2008
|
|
 Stewart I. Donaldson & Tarek Azzam
Using Technology to Enhance Applied Research and Evaluation
This workshop will focus on how to use a range of new technological tools to improve applied research and program evaluations. Specifically, we will explore the application of free or inexpensive software to engage clients and a range of stakeholders, collect research and evaluation data, formulate and prioritize research and evaluation questions, express and assess logic models and theories of change, track program implementation, provide continuous improvement feedback, determine program outcomes/impact, and to present data and findings. Participants will learn how to access tools such as Geographical Information Systems (GIS), data collection software, and interactive conceptual framing software to improve the quality of their applied research and evaluation projects. Participants will be provided with information sheets on each technological tool along with details about attaining free trials.
Questions about this workshop should be addressed to stewart.donaldson@cgu.edu.
|
|
 Annette Steinacker
Cost-Benefit Analysis
Cost-benefit analysis is a tool used to organize information in a way that can assist in decision-making regarding programs or policies: which program to choose to achieve policy goals, what level of the program to support, how to allocate a given budget to maximize efficient use of the funds. This workshop will cover the basics of the technique: who has standing, the different methods used to measure benefits and costs, discounting to achieve a net present value (NPV) for a project, and appropriate decision rules given the estimated NPV. We will also discuss several of the concerns that people raise with use of the analysis and ways to reduce those concerns. When dealing with some policy issues, ethical questions have been raised that certain items, such as health or freedom, should not be reduced to monetary values. Some argue that risks that are undertaken voluntarily and result in potential costs should be treated differently than risks that are not voluntary. There also can be distributional concerns given that those who benefit from a project are not always those who will pay for it. The workshop will included several in-class exercises to practice the cost-benefit techniques and presentation of the results to lead to informed decision-making.
Questions about this workshop should be addressed to Annette.Steinacker@cgu.edu.
|
|
 Katrina L. Bledsoe
How to Conduct Culturally Competent Community-based Program Evaluations
The dynamic cultural demographics of communities make it imperative to understand how to conduct evaluations that are culturally sensitive and competent in community-based settings. This understanding is especially important in working with communities with hard to reach populations (e.g., marginalized groups), and where evaluations, if not tailored to the community and cultural milieu, can easily overlook the mores of its members. The purpose of this workshop is multifaceted. First, it encourages participants to understand barriers that might arise in evaluative settings between evaluators, key stakeholders, and evaluation participants that can hamper the development and execution of culturally competent evaluations (e.g., power dynamics; and institutional structures that may intentionally or unintentionally promote the “isms”). Second, the workshop provides participants with basic strategies in enhancing cultural competent evaluations by addressing issues such as institutional review boards and research ethics; translation of evaluation terminology to communities; and strategies to garner community involvement and buy-in, and gain trust. Finally, participants will have the opportunity to develop a model of a culturally competent community-based program evaluation. Dr. Katrina Bledsoe will provide “real world” examples from her evaluation experiences in small urban communities, as well as a large national multi-site federally funded community-based initiative. This workshop assumes participants will engage in highly interactive discussions about program evaluations that they might have or may currently be conducting.
Questions about this workshop should be addressed to katrina.bledsoe@gmail.com.
|
Last updated 8/29/08.
|
