programeval

This course is designed to provide students with a broad overview of how program evaluation is carried out within a range of agency and organizational contexts, as well as to provide students with specific skills in developing and implementing a program evaluation plan with an agency or unit. "Program Evaluation" is designed to be an introductory course targeted to undergraduates seeking preparation for management and supervisory roles within a variety of human service settings.

Prerequisites:

An undergraduate course in introductory statistics is highly recommended (but not required) in preparation for completing this course.

Objectives:

Develop conceptual skills among students relative to what meaningful program evaluation entails.
Develop in students a chronological perspective of how one designs and evaluates new programs, as well as how one evaluates existing programs
Prepares students for accurately identifying and measure specific program outcomes (products) as well as the processes that are entailed in producing program outcomes.
Provides students skills in conceptualizing and applying program performance measurement techniques and statistics to agency/program outcomes.

Course Requirements:

Complete student portfolio of weekly homework assignments,
Present contents of portfolio to instructor and classmates at the end of the course
Actively participate in group discussions and exercises during class sessions
Complete the final exam in the class.

Grading:

Student Portfolio Completion 200 points

Student Participation 100 points

Student Presentation 100 points

Final Exam 200 points

A 600-525

B 524-450

C 449-325

D 324-275

Course Schedule

Week One:
Friday May 15, 1998, 6:00 - 9:00 P.M.
Introduction & Overview

Saturday May 16, 1998, 9:00 -4:30 P.M.
Statistical Review

Readings: Chapter 1 & 2, Key Concepts in Evaluation Research (Berk & Rossi): Dean, Keith (1997) Surfstat, University of Newcastle, Newcastle, Australia http://surfstat.newcastle.edu.au/surfstat/main/surfstat.html

Topics (Friday): Policy Space, Stakeholders, Meaning of Program Effectiveness, Validity, & Reliability

Time Use (Friday Evening):

6:00 - 7:00 Class Administrative Tasks and Course Overview

7:00 - 8:00 Program Effectiveness

8:15 - 9:15 Break

8:15 - 9:30 Reliability & Validity

Topics (Saturday): Qualitative & Quantitative Data, Discrete & Continuous Data, Modality & Symmetry, Central Tendency, Variability, Normal Distributions, Correlation & Mean Comparisons

Time Use (Saturday)

9:00 - 10:00 a.m. Qualitative & Quantitative Data

10:00 - 10:15 a.m. Break

10:15 - 11:15 a.m. Discrete & Continuous Data

11:15 - 12:00 noon Distribution (modality & symmetry)

12:00 - 1:00 p.m. Lunch

1:00 - 2:30 p.m. Measures of Central Tendency

2:30 - 2:45 p.m Break

2:45 - 4:30 p.m. Measures of Variability

Class Format: Lecture, Discussion, Presentation & Activities

Week Two:
May 18 - 28, 1998,
Homework & Self Study Assignment:

Readings: Berk & Rossi, Chapters 2-4; Martin & Kettner, Chapters 1-2

Portfolio Assignment: Identify four nonprofit agencies in the region, and develop a complete a list as possible of the service "products" each agency produces. This will require to visit or contact each agency to acquire this information. Thereafter, speculate as to how you imagine each product is statistically documented within each agency. Finally, describe how these statistics are used by the agency or unit, including the rationale for collection such information.

Internet Assignment: Go to the internet and use the search engine in Netscape to find the web pages for the following agencies:

March of Dimes

Alzheimer Association

American Red Cross

Planned Parenthood

Identify the mission, stated goals and objectives of these agencies as well as the services (products) they produce and explain (if possible) how the mission and goals of these agencies relate to the products they produce.

Stat Assignment: Go to SurfStat on the web and click "detailed contents." Then, complete all progress checks beginning with the Section entitled "Types of Variate" through "Measures of Variability" Copy work from the web page and include in portfolio.

Week Three:
Friday May 29, 1998, 6:00 - 9:30 P.M.
Design, Evaluation

Saturday May 30, 1998, 9:00 -4:30 P.M.
Performance Measurement

Readings: Berk & Rossi, Chapters 3-4; Martin & Kettner, Chapters 1- 6

Topics:

Designing New Programs, Basic Questions, Fitting Evaluation Strategy to the Problem, Policy Contexts of Evaluation, Policy Formulation Stages, Problem Definition, Determining Where the Problem Resides & it's Significance (Needs Assessment), Determining What Can Be Done About the Problem (Problem-Driven Research), Transforming Program Concepts into Programs, YOAA Problem, Determining Program Effectiveness, Practical Developmental Evaluation Approaches.

Evaluating Ongoing Programs: Determining Program Appropriateness, Program Integrity Research, Fiscal Accountability, Program Evaluability, Outcome Assessment, Research Design Models: True Experimental, Regression Discontinuity, Interrupted Time Series, Cross Sectional, Pooled Cross Sectional & Time Series; Cost Effectiveness

Performance & Accountability: Forces Promoting Performance Accountability, Identifying, Defining & Evaluating Social Problems, Measuring Output, Measuring Quality, Measuring Outcome.

Time Use (Friday):

6:00 - 7:00 Homework Assignment Review and Discussion

7:00 - 8:00 Presentation & Discussion on Program Design

8:00 - 8:15 Break

8:15 - 9:30 Presentation & Discussion of Program Evaluation

Time Use (Saturday)

9:00 - 10:30 Presentation and Discussion of Performance Accountability & Social Problems

10:30 - 11:00 Break

11:00 - 12:00 Presentation and Discussion on Measuring Output, Quality, and Outcome

12:00 - 1:00 Lunch (Stat Lab After Lunch)

1:00 - 2:00 Normal Distributions

2:00 - 3:00 Correlation

3:00 - 3:15 Break

3:15 - 4:30 Comparing Means

Class Format: Lecture, Discussion, Presentation & Activities

Week Four:
June 1, - June 11, 1998,
Homework & Self Study Assignment

Readings: Martin & Kettner, Chapters 1 - 11

Portfolio Assignment I:

Go to the internet and find the Web Page for the Texas Department of Aging. How many congregate meals were served last year, and how many people were served? How many one-way trips were made by transportation providers, and how many people were served? How many contacts were made by information and referral providers? How many homes were repaired, and for how many people? How many certified ombudsmen does the Department now have? How many of the following have these ombudsmen dealt with: complaints taken, complaints resolved, requests for information & assistance? How many case management hours has the Department funded, and how many people were served? How many hours of adult day care was provided and to how many individuals?
Go to the library and conduct a literature search to see what you can find about the Caregiver Strain Index. Develop an annotated bibliography of ten citations, of no more than 500 words per citation, that identifies the use of the Index.
Go to the library and conduct a literature search to see what you can find about the Barthel Index. Develop an annotated bibliography of ten citations, of no more than 500 words per citation, that identifies the use of the Barthel Index in inpatient aging settings.
Return to the home page of the Texas Department of Aging. Find the budget report for services rendered by the agency. Take this budget information and create a table and/or bar/line graph which integrates the service statistics which you have earlier identified with the corresponding budget information. For ease of comparison, you may want to just use total cost data from the agency.

Portfolio Assignment II:

From the library, find 6 articles having to do with total quality management (TQM) and summarize each article in approximately 500 words. Be prepared to present these annotated bibliographies in class.
Provide a three page (typed/double/12 font) report based upon at least three journal articles or government publication, explaining what service efforts and accomplishments reporting (SEA) is. Be prepared to present your paper in class.
Use the local newspapers (particularly the Fort Myers News Press) and find out what the "social problem" was that resulted in the organization "Lee County Pulling Together." Find out what event or events, resulted in this group finally being formed.
Go to the internet and find the Joint Commission on the Accreditation of Healthcare Organizations (JCAHO). Browse through this web site and see if you can find JCAHO's definition of quality. If you can, then copy it. If you can't find a definitive statement on quality, then determine from the content on the web site how JCAHO measures quality, and why healthcare organizations would want to be accredited by this organization.

Stat Assignment:

Go to INTRODUCTION TO QUANTITATIVE METHODS A Basic Statistics Course in the College of Education at Arizona State University, Instructor: Gene Glass. Read Chapter Six on Standard Scores and complete all excercizes with the exception of the assignement at the bottom of the chapter. Also read the Cliff Quick Review of Statistics (pp. 85-110) and familiarize yourself with "z" and "t" tests. Use your calculator and compute the examples provided in the text to better understand how to use these statistics. For class, write down questions that you had relative to these statistics, and identify concepts that you both understood and topics that you did not understand. Be prepared to report back to class regarding your experiences with this homework. Also, and MOST IMPORTANTLY utilize members of your study group and/or other members of the class to help you understand these concepts. Copy your work and include in portfolio.

Week Five:
Friday June 12, 1998, 6:00 - 9:30 P.M.
Four Types of Outcome Performance Measures & Issues in Collecting, Reporting & Using Outcome Performance Data

Saturday June 13, 1998, 9:00 - 4:30 P.M.
Student Portfolio Presentations

Readings: Martin & Kettner, Chapters 7-11

Topics: Numeric Counts, Standardized Measures, Level of Functioning Scales, Client Satisfaction Scales, Collecting, Reporting & Using Outcome Performance Data

Time Use (Friday)

6:00 - 7:00 Homework Assignment Review & Discussion

7:00 - 8:00 Presentation & Discussion on Types of Outcome Performance Measures

8:00 - 8:15 Break

8:15 - 9:30 Continued Discussion on Types of Outcome Performance Measures and Reporting & Using Outcome Performance Data

Time Use (Saturday)

9:00 - onward, Student Portfolio Presentations

Class Format: Discussion & Presentation

Week Six:
Friday June19, 1998, 6:00 - 9:00 p.m.
Final Exam

Statistic Problems

Standard Deviation Problems

Online Lecture Outlines, Stat Courses & Texts