Issue Number 4, March 2003
Monitoring and Evaluating Physical Activity Programs for
By: Anita M. Myers PhD, Professor, Department of Health
Studies and Gerontology, University of Waterloo; Adjunct,
Canadian Centre for Activity and Aging, University of Western
Program evaluation involves collecting information on service
delivery and clients to help you make informed decisions when
planning new initiatives or examining existing services. Evaluation
provides the information you need to support your case for
new, continued or increased program funding. It also demonstrates
your commitment to delivering the highest quality service
possible with available resources. While evaluation may be
required for funding or reporting, first and foremost it will
enable you to be accountable to your primary stakeholders—the
clients or participants in your program.
Older adults who take part in active living programs have
the right to expect a safe and beneficial program, tailored
as much as possible to their needs, abilities and expectations.
Conducting evaluation shows potential and current clients
that you value their input and feedback.
Benefits of Program Evaluation
Accountability to sponsors and participants
Informed planning and decision-making
Directions for improving service delivery:
• Recruitment and retention
• Meeting client needs and expectations
• Maximizing client benefits
• Minimizing adverse effects
• Maximizing efficiencies
Popular Misconceptions about Evaluation
While program evaluation sounds good in principle, you may
wonder if you have the time, money or expertise to conduct
it. First, it is important to understand that evaluation differs
from “traditional” research. Research examines
physical activity interventions under highly controlled conditions,
while evaluation examines programs under real-world conditions.
You do not have to be a researcher to evaluate your programs!
While evaluation does require time and effort, the alternative
(not doing evaluation) can be much more costly. The programs
at greatest risk of losing funding are those with no supporting
evidence. For example, back in the 1980’s when the Canadian
Red Cross Society underwent major restructuring, the Fun
& Fitness Program for Seniors survived because of
evaluation data. Not only did the findings convince the Board
to continue the program, the program coordinators also used
the evaluation to secure external funding to expand it across
In-house resources will affect the scope of evaluation projects
you can tackle. However, even grassroots programs with limited
resources can undertake simple evaluation activities. As you
will see, simple adjustments to record-keeping practices can
produce valuable information for decision-making.
Another popular myth is that evaluation may produce negative
findings, leading to funding cuts and staff layoffs. This
apprehension stems from a misunderstanding that evaluation
only looks at whether the program can demonstrate “success”
in terms of client outcomes or benefits. Outcome evaluation
is only one type of evaluation. And most show that programs
tend to be relatively beneficial (for some clients
more than for others). It is not productive to think in terms
of success or failure. By setting realistic performance objectives
and examining various aspects of service promotion and delivery,
evaluation will identify which aspects are working well and
which may need improvement.
Let’s look at the different types or approaches to
program evaluation, as well as the distinction between periodic
evaluation studies and ongoing data collection or monitoring.
Then we will look at ways to improve client information gathering
Types of Program Evaluation
Program Evaluation Approaches
Needs assessment (planning; market analysis)
Formative evaluation (piloting; market testing)
Process evaluation (routine operation)
Outcome evaluation (effectiveness)
Cost evaluation (efficiency)
The particular stage of program development determines which
approach to evaluation is most appropriate. For instance,
when planning new programs, it is important to conduct a needs
assessment (or market analysis) to examine what services
already exist in your area to avoid duplication and to see
whether your target audience (potential users) is interested
in such a program, given possible fees, location, scheduling,
and so on. Once you have made the decision to proceed and
begin developing materials (such as brochures, videos, posters),
it is important to test the reactions of potential clients
through formative evaluation (market or pilot
testing). Are materials understandable? Appealing? Motivating?
Evaluation at the draft stage can save time and money.
Imagine producing 1,000 copies of an exercise video only to
discover that older adults did not identify with the actors
or found it boring or difficult to follow.
Once your program is up and running, it is important to conduct
process evaluation. Are you reaching your
intended target audience (recruitment)? Who is using your
program or services, how often? Are your classes or groups
filled to capacity? Can you meet the demand (is there a wait
list, how long)? To what extent are you meeting client expectations?
Do participants tend to continue participating or complete
the program (retention)? Do some areas of program delivery
After you have modified your program, to optimize recruitment
and/or delivery, you may want to consider outcome
and cost evaluations. Outcome evaluation examines
the extent of client improvement, maintenance or prevention
(depending on the program’s objectives). It assesses
clients before they begin the program and after some period
of participation using measures or indicators such as level
of physical activity, physical or psychological functioning.
Cost analysis, on the other hand, can be fairly simple (cost
per case) or highly complex (relating program inputs to program
Evaluation and Monitoring
The good news is that most evaluation studies need to be
conducted only periodically. Once you have supporting
information from your needs assessment, you can proceed with
developing your new program. You need to conduct further periodic
needs assessment only when if you want to expand your services
to new areas or target groups. For example, if you wanted
to attract more ethnically diverse older adults in your region
you would conduct another needs assessment. You would conduct
another process evaluation if you made substantial changes
to your promotional or delivery strategies. You may not have
to do an outcome evaluation (pre- and post- client comparisons)
if studies documenting client benefits of similar programs
Conversely, certain information should be collected on an
ongoing basis. Every program should collect background
information from each and every new participant and record
individual attendance (for class or group-based programs.)
Such information is essential for developing a profile
of your clientele, including dropout rates. Comparing
the same information over time (monitoring or tracking)
allows you to examine patterns or trends. For instance, you
may see that enrolment patterns are higher at certain times
of the year. Or, you may find that your client profile has
changed in the last few years (such as more early retirees
or different cultural groups or persons with more severe arthritis).
Usage patterns may also change (such as greater demand for
morning classes, more women using the weight rooms, more men
in dance classes). Today’s clients may have different
needs and expectations, requiring you to modify promotional
and delivery strategies. For instance, if your clientele now
includes individuals with more severe arthritis or osteoporosis,
you may need to help them to safely enter and exit the pool.
Likewise, it may take more time to explain home exercises
to clients with cognitive impairments. Either of these scenarios
would justify additional program resources.
As well as monitoring possible changes in your client profile
and usage patterns, you need to determine how well you are
meeting client needs and expectations. This can be done either
periodically or on an on-going basis.
Obtaining Participant Feedback
Many programs use client satisfaction questionnaires (also
referred to as “evaluation” or “feedback”
forms) because they are easy to develop and score. However,
you should also understand their limitations.
Easy to develop
Easy to adapt
Hard to interpret
- Did you know that 80 to 85% of consumers
say they are moderately to highly satisfied?
This response bias has been found across all types of services
and programs. Why? Most people don’t like to complain
(at least formally) for fear of losing the service or not
wanting to ruin it for others. Others simply can’t
be bothered filling out the form. Since such surveys tend
to be distributed at the end of the session or year-end,
less frequent attendees may not be there the day of the
survey. And dropouts, those for whom the program has not
met their needs and expectations, are already long gone.
- Did you know that two-thirds of consumers
switch services due to staff indifference?
This holds true whether examining health care services,
fast foods or active living programs. It is debatable whether
participants can judge the technical skills or competency
of service providers, but they do know whether they were
treated in a friendly, courteous manner. If people like
the service provider, they tend to give high overall ratings
to the program itself.
- What can you learn from client satisfaction
Such surveys typically entail checking or circling a response
ranging from highly satisfied to highly dissatisfied (or
excellent to poor) for various program features (such as
instructor, location, scheduling, music). Even if some people
check “dissatisfied,” for such things as music
or scheduling, what do these findings tell you? Since such
surveys are anonymous, there is no way of determining which
clients are dissatisfied, and why. Although surveys may
include space for open-ended comments, most people do not
bother. Even if you receive some specific comments, you
would not want to act without first knowing the extent to
which such concerns represent the views of your clientele
as a whole.
How to improve client satisfaction surveys:
- Choose more than one class or time period for distribution
(not close to holidays).
- Have someone other than the group instructor or leader
distribute these surveys.
- Include a confidential envelope and a drop-off box for
- Mail the survey (with an addressed, postage paid envelope)
to all registrants. This will help reach less frequent users
and those who have left the program.
- Always report the response rate (percent completing the
survey divided by the total number of registrants) when
presenting the findings.
You should also consider other feedback strategies. Suggestion
boxes and comment cards should be readily available and promoted.
A phone call to absent members (shown to reduce attrition
in commercial fitness clubs) can be used to both offer support
and gather information. You should also periodically conduct
Focus groups are an excellent way to obtain in-depth client
feedback. Focus or discussion groups allow several participants
(usually 8 to 12 per group) to exchange ideas, elaborate on
impressions and explore issues in more detail. This method
is highly recommended for needs assessments and formative
evaluation (to solicit the input of potential clients) and
for process evaluation (to solicit feedback from current clients).
Where to start?
Start by examining your current record-keeping practices
concerning client information. Is the information easy to
access? Is it in usable form? When working with community
recreation and seniors’ centres, we found that most
collected enrolment or registration data. Many instructors
also recorded class attendance using registration lists, although
some did total head counts only. While instructors knew a
lot about individual participants, typically only client age
and gender were available from registration forms (usually
stored in file folders).
A simple background questionnaire given to all new registrants,
together with individual attendance/usage data, is all that
you need to develop a more detailed profile of your clientele.
As well as age and gender, you may want some other demographic
information (such as marital status, education), some basic
health information (such as mobility, vision or hearing problems
that may affect participation), as well as reasons for joining,
how they heard about the program, how they normally get to
the program, other physical activities, possible reservations,
and so on. Simple databases can help you manage and link different
sources of information (to compare characteristics of frequent
versus infrequent users, adherers versus dropouts), and monitor
patterns over time.
(Simple Ways to Get Started)
- Record individual attendance or usage
- Gather background on new clients
- Set up a simple database to manage, link and track client
- Set up a procedure for contacting absent clients
- Conduct periodic focus groups to solicit in-depth client
There are simple ways to improve your current record-keeping
systems to begin profiling your clients and monitoring your
programs. Depending on your program’s stage of development,
you should also consider periodic evaluation studies. Most
participants will be happy to provide information for evaluation
purposes once you explain why it is important and how it will
be used (refer to Tips for Older Adults). Invite some of your
program’s participants to sit on your evaluation planning
committee to ensure appropriateness of tools and relevance
of activities. Involving them in the evaluation process shows
you value their ideas. Undertaking evaluation shows that you
are committed to improving service delivery and supplying
evidence for continued or increased program support and funding.
The Canadian Centre for Activity and Aging (CCAA) systematically
evaluates all its programs, covering both programs that are
conducted at the Centre and the community outreach programs,
and also spanning the continuum from healthy to frail older
adults. The book Program Evaluation for Exercise Leaders
uses examples from the CCAA, as well as examples from other
sectors (such as public health, community recreation, rehabilitation)
to help program managers and deliverers, particularly those
working with older adults, evaluate their own programs. The
book contains prototypes for developing performance objectives,
client consent forms and background questionnaires and protocols
for contacting absent clients. It also provides a 12-step
guide for conducting focus groups, shows you how to protect
client information (by assigning unique codes based on the
date and order people register), how to select tools for different
types of evaluation, and how to manage, analyze and interpret
If you are trying to promote active living in your community,
you can learn a great deal from the evaluations of the Community
Health Activities Model Program for Seniors (CHAMPS)
in California, and the Aging Well and Healthy initiative
in the Netherlands. If you work with frail older adults, look
at the evaluation articles on the CCAA Home Support Exercise
and Functional Fitness Long-term Care Programs. Articles
tracking the participation and adherence patterns of older
adults in various classes at the CCAA, as well as the one
by T.S.Gavin (profiling older adults who join Tai Chi and
line dancing programs offered at recreation and seniors’
centres), provide important benchmarks or comparisons
for other program deliverers. But remember, you still need
to evaluate your own programs.
References and Resources: For the list
of references in this article, please contact the ALCOA office
or visit our web site. For more information on the book Program
Evaluation for Exercise Leaders (ISBN 0-88011-709-5), go to
www.HumanKinetics.com or call toll free in Canada 1-800-465-7301.
We would like to thank Liz Cyarto, instructors and participants
at the CCAA, for the valuable input on this article. The views
expressed are those of the author and not necessarily those
of Health Canada.
to Research Update