Magistrates Early Referral Into

Treatment

MERIT

Program Evaluation & Monitoring

Framework

………………………………

1
Revised August 2005

Published by the Crime Prevention Division

NSW Attorney General’s Department
Level 19
Goodsell Building
8-12 Chifley Square
Sydney NSW 2000
Telephone 9228 7206
Fax 9228 8559
Website: www.lawlink.nsw.gov.au/cpd

ISBN: 0 7347 2823 9

2
TABLE OF CONTENTS

1.0 Introduction
1.1 Purpose of program evaluation and monitoring 4
1.2 Program description 4
1.3 MERIT program evaluation and monitoring group 6
1.4 MERIT database 6
1.4.1 Data quality and availability 6
1.4.2 Program resources 7
1.5 Evaluation map 7

2.0 Monitoring
2.1 Purpose of monitoring program process and outcomes 9
2.2 Summary of monitoring reports 9
2.3 Monitoring process 10
2.3.1 Operations and activities 11
2.3.2 Performance 11
2.3.3 Target participation 11
2.3.4 Service utilisation 12
2.3.5 Court coverage 12
2.4 Monitoring outcomes 12
2.4.1 Program, post program and unintended outcomes 12

3.0 Evaluation
3.1 The purpose of evaluating program process and outcome 12
3.2 Summary of evaluation reports 12
3.3 Evaluating process 13
3.3.1 Program fidelity 13
3.3.2 Program equity 13
3.3.3 Stakeholder attitudes 13
3.3.4 Resources 14
3.3.5 Follow up care 14
3.4 Evaluating outcomes 14
3.4.1 Program outcomes 14

References 15

List of Tables
Table 1.0 Summary of monitoring strategies and reports 9
Table 2.0 Summary of evaluation strategies and reports 12

List of Figures
Figure 1 Evaluation map 8

3
1.0 Introduction

1.1 Purpose of program evaluation and monitoring

The measure of performance is a key element in the NSW Government’s
commitment to managing results in the public sector. A program’s performance can
be effectively measured and assessed through ongoing monitoring of the program
and by formal evaluation. Social programs such as MERIT aim to:

ameliorate a social problem or improve social conditions. It follows that is

appropriate for the parties who invest in social programs to hold them
accountable for their contribution to the social good. Correspondingly, any
evaluation of such programs that is worthy of the name must evaluate – that is,
judge – the quality of a program’s performance as it relates to some aspects of
its effectiveness in producing social benefits (Rossi, Lipsey and Freeman,
2004:17-18).

Evaluation and monitoring are crucial in distinguishing effective social programs from
ineffective ones. The commitment to engage in evaluation of the MERIT program is
reflected in the existence of a formal evaluation strategy and a working party (the
MERIT Program Evaluation Group) that exists to specifically develop and implement
the evaluation strategy. This document provides an overview of the MERIT program
evaluation strategy.

1.2 MERIT Program description

As part of the Government Plan of Action to implement the resolutions of the NSW
Drug Summit, a commitment was made for a range of innovative schemes to be
introduced and trialled. These schemes allow police and courts to exercise discretion
and divert offenders from prosecution and custodial sentences. The schemes aim to
break the drugs-crime cycle by trying to engage offenders at various stages in
treatment and rehabilitation programs as a way of addressing their drug dependency.
The MERIT program is one initiative that responds to the recommendations of the
NSW Drug Summit.

MERIT is an inter agency initiative between the NSW Attorney General’s Department
(lead agency), the NSW Department of Health (Centre for Drug and Alcohol) and
NSW Police.

MERIT scheme has two levels of management: central and regional. At both levels
of management the membership involves a range of key stakeholders that may vary
between sites and over time. The NSW Attorney General’s Department convenes
the MERIT Statewide Steering Group which is responsible for the overall policy and
design of MERIT as well as supervising the implementation of MERIT across the
state.

MERIT targets defendants with a demonstrable drug problem who are eligible and
suitable for release on bail and who are motivated to engage in treatment and
rehabilitation for their illicit drug problems. An important feature of the program is that
participation is voluntary and that the defendant is not required to enter a plea to the

4
charges that bring them before the court in order to participate in the program. A
person’s agreement to become involved in MERIT is not an admission of guilt to the
offence(s) for which s/he has been charged. Once charged, the defendant is eligible
for the MERIT program if they:

• are eligible for release on bail

• have a demonstrable illicit drug problem
• are an adult
• consent to voluntarily participate in the MERIT program
• are not involved in offences related to violence, sexual offences or wholly
indictable drug offences
• have no matters pending of a violent or sexual assault nature
• have a treatable problem
• have been deemed suitable for the program
• usually reside in the defined treatment catchment area, and
• are approved to participate in the program by the Magistrate

MERIT provides for referrals from police at the time of arrest, from solicitors following
arrest, or at court by the Magistrate. Referrals from other sources are considered.

The outcomes of the MERIT program may be divided into specific program
outcomes, post program outcomes and unintended outcomes.

Program Outcomes
The intended outcomes for the MERIT program are to:
• Decrease drug related crime by participating defendants for the duration of the
program
• Decrease drug related crime by participating defendants following program
completion
• Increase community protection
• Improve health and social functioning for the duration of the program and in the
post program period
• Reduce sentences due to better rehabilitation prospects

Post-Program Outcomes
Once a defendant has completed the MERIT program, it is hoped they will:
• Cease their drug related crime
• Cease their illicit drug use
• Make lifestyle changes that support a drug-free existence, and
• Improve their health and social functioning

Unintended Outcomes
Possible unintended program outcomes are:
• Increased remand numbers if offenders fail to comply with bail conditions
• Increased number of court appearances by MERIT clients due the provision of
judicial supervision
• Added strain on some of the Area Health Services to provide the full range of
drug treatment services

5
1.3 The MERIT Program Evaluation Group
The MERIT Program Evaluation Group was established to oversee MERIT evaluation
and monitoring. The Evaluation Group meets on a quarterly basis with
representatives from the NSW Attorney General’s Department, NSW Health and
NSW Police. This groups reports to MERIT Statewide Steering Group.

The role of the Group is to develop and implement the evaluation strategy. The
specific activities are:

1. Strategic issues
To share recent research literature in the field of drug and alcohol diversion
To identify strategic challenges in MERIT evaluation strategy and develop
strategies to oversee these challenges
To identify challenges in current evaluation projects and propose ways of
overcoming them

2. Monitoring
To monitor the MERIT evaluation strategy
To monitor existing evaluation projects
To monitor the health and criminal justice outcomes of MERIT across all MERIT
Teams
To monitor data collection across NSW Area Health Service MERIT sites
To review and revise the MERIT data dictionary

1.4 The MERIT database

A specific database, the ‘MERIT Information Management System’ (MIMS) provides
data for the measurement of process and outcome evaluation indicators. A data
dictionary provides a description of each item recorded. The database provides the
basis for the monitoring reports as well as fulfilling reporting requirements to the
Commonwealth. Each year a complete set of program data from the time of program
inception is downloaded by the MERIT database manager in order that a whole of
program analysis might be made. This data is analysed and included in the MERIT
Annual Report completed by the MERIT project officer.

The MERIT Evaluation Group monitors the extent to which the data measures may
effectively be used to address evaluation questions. The continual monitoring of data
in this way means that ideas concerning better measurement are identified and
addressed.

At times supplementary information from external sources must be used in

conjunction with MIMS data to adequately answer evaluation questions. For
example, the Bureau of Crime Statistics and Research annually match their data on
criminal records with MERIT client data in order to answer evaluation questions
concerning recidivism patterns.

6
1.4.1 Data quality and availability
The quality of data is affected by a number of factors including the consistency of
recording of data between MERIT program sites. Training sessions have been held
to promote consistency. Ongoing training and updates are provided by the fulltime
MERIT database manager.

All stakeholders in the MERIT program have a responsibility to ensure that there is
provision for program evaluation that includes measurement of the program’s
intended health and criminal justice outcomes. The data collected for this purpose
are largely undertaken by health workers and may generally be made available to
other stakeholders with adequate individual privacy safeguards.

1.4.2 Program resources

Program resources include a budget allocation, special purpose funding and
additional monies approved on an ad hoc basis following from budgetary roll
overs. There are three identified positions across the NSW Attorney General’s
Department and NSW Health working on MERIT evaluation.

1.5 Evaluation map

An activities map of the evaluation strategy is presented in Figure 1. This map is
used to report on progress for current studies and is updated on a regular basis to
reflect new evaluation strategies.

7
 MERIT EVALUATION ACTIVITIES

Monitoring Evaluation

Process Impact Process Impact

Operations and activities Program Outcomes Program fidelity Program outcomes

Activity & Quarterly reports* Annual Reports*** Process study*** Lismore evaluation**
Health outcomes study***
Criminal justice outcomes study****

Performance Post Program Outcomes Equity Efficiency

Key performance indciators *** Annual Reports*** Aboriginal Participation Study** Cost-Benefit Study (Lismore Evalution)**
Women**** Cost-effectiveness Study***
Cultural background & Mental Health**** Costing Study***

Target participation Unintended Outcomes Stakeholder attitudes

Activity reports* **** Magistrates Survey**
Police Survey***
Legal Practitioner Survey****

Service utilisation Follow up care

Residential rehabilitation* Aftercare study***

Court coverage* Participant satisfaction

Lismore Evaluation (participant survey)
Health Outcomes Study (questions to participants)

* Ongoing projects
** Completed projects
*** Projects in progress
**** To be developed

8
2.0 Monitoring

2.1 The purpose of monitoring

Monitoring reports seek to convey timely and relevant data regarding particular
indicators of the program’s outputs. These serve to highlight areas of variation in the
program implementation between different health areas as well as monitor critical
indicators of the program’s operation. Ongoing monitoring will lead to the
establishment of key performance indicators and benchmarks.

2.2 Summary of monitoring reports

Table 1.0 presents a summary of the monitoring components, indicators and
reporting formats for the MERIT program.

Table 1.0: Summary of monitoring strategy and reports

Monitoring Indicators Indicators Format
component
Monitoring Operations and Referral numbers and rate Monthly activity &
of process activities Quarterly reports
Source of referrals Monthly activity &
Quarterly reports
Acceptances into MERIT Monthly activity &
Quarterly reports
Program completion Monthly activity &
Quarterly reports
Breaches Monthly activity &
Quarterly reports
Data issues (data dictionary, NGO Ongoing reports
involvement)
Target participation Monthly activity &
Quarterly reports
Program availability Court coverage On request
Monitoring Program, post Decrease drug related crime by Annual reports
of outcomes program and unintend participating defendants for the duration
outcomes of the program prospects
Decrease drug related crime by Annual reports
participating defendants following
program completion
Increase community protection Annual reports
Improve health and social functioning for Annual reports
the duration of the program and in the
post program period
Reduce sentences due to better Annual reports
rehabilitation
Cease their drug related crime; Annual reports
Cease their illicit drug use; Annual reports
Improve their health and social Annual reports
functioning. Health outcomes study
Make lifestyle changes that support a drug-free Annual reports
existence
Increased remand numbers if offenders Annual reports
fail to comply with bail conditions
Increased number of court appearances Annual reports
by MERIT clients due the provision of
judicial supervision;
Added strain on some of the Area Health Servic Annual reports
to provide the full range of drug treatment servic

9
Monthly Reports
The Centre for Drug and Alcohol, NSW Department of Health produce monthly and
quarterly reports after individual sites have sent their data at the end of each
month. The aim of these reports is to measure the level of program activities across
health areas in the period. Measures include:

• Number referred for assessment

• Number partially assessed
• Number accepted
• Number ineligible, declined or did not appear
• Number breached, removed or withdrew voluntarily
• Number remaining on program
• Number completed program/court requirements

The monthly report provides data on the cumulative numbers who have been
involved in the program at the State and individual Area Health Service level. It is
possible to derive rates of referral acceptance and program completion rates. These
comparisons need to be interpreted with care as the MERIT program in each area
varies in terms of development, resources, demographics, and offender profiles.

Quarterly Reports
Quarterly monitoring reports include a more extensive and detailed coverage of
program measures. Indicators include the source of referral, gender, Aboriginality,
principle drug problem, offence type, number with previous gaol sentences, numbers
receiving particular types of health services and court findings. As well each health
area reports on its MERIT program implementation.

Annual Reports
The NSW Attorney General’s Project Officer (Monitoring and Evaluation) produces
an Annual Report on MERIT. As well as summarising and discussing monthly and
quarterly reports, the Annual Report includes data collected for outcome monitoring.
Differences between area programs are analysed as are MERIT outcomes for
particular groups such as women and Aboriginal and Torres Strait Islanders. The
Annual Report presents an overview of trends in the MERIT program and includes
some statistical analyses associating indicators with outcomes.

The NSW Health Department produces an Annual Report for the Commonwealth
focusing on health outcomes.

2.3 Monitoring process

In order to ascertain whether a program is operating as it intended it is necessary to
continually and systematically monitoring how the program is actually working in
practice, that is, the program process. The monitoring of these indicators will, over
time, lead to the development of key performance indicators that may then become a
standard for which to compare performances. Thus process monitoring involves the
repeated measurement of performance indicators over time.

10
2.3.1 Monitoring operations and activities

Referral Numbers and Rate

Referral rates are key indicators of demand for MERIT. Areas where there are
particularly low numbers of referrals may indicate problems with the process. This
could reflect on the level training or non-acceptance of the scheme by referral
agencies within an area. Alternatively it may be a true reflection of low demand due
to differing patterns of drug use and offending in different areas. Where significant
differences appear in the monitoring reports, further investigation may require a more
qualitative approach involving in-depth interviews with the parties concerned, such as
magistrates or police. This may highlight attitudes, practices and resource issues that
are producing a pattern of low referral rates.

Source of Referrals
The source of referrals is a potentially important process indicator. The source of
referral to the program can provide an indicator of the way the program is working
within an area. Low rates of referral from a particular source may indicate that further
information or training is required.

Acceptances into MERIT

As well as the acceptance rate it is important to monitor reasons for non-acceptance
onto the program. This may highlight areas where resources or training are required
to make the program more efficient. The proportion of acceptances and non-
acceptances is monitored by health area and recorded in monthly and quarterly
reports. Further analysis of demographic characteristics, drug use and criminal
history provides a description of MERIT participants. This can be used to identify
particular groups who are not being accepted into the program as well as those that
the program is reaching.

Breaches
Breaches are monitored as part of the standard reporting system. Significant
variability in the relative proportion of breaches in each area could indicate
operational program differences requiring further investigation. Higher rates of
breaches may be the result of the nature of participants referred to the program in a
particular area, for example differing criminal histories and/or levels of drug offending.
They may also indicate significant variation in practice.

2.3.2 Performance
As well as indicating area differences in the program, the monitoring process can
provide information on program quality and standards. Key performance indicators
are currently being devised for the MERIT program.

2.3.3 Target participation

The characteristics of clients referred, participating and completing MERIT is
monitored by the collection of data on demographic characteristics. These are
reported in the monthly and quarterly data reports.

11
2.3.4 Service utilisation and MERIT expansion
The number and nature of residential rehabilitation services is monitored. The
number of courts operating the MERIT program is monitored and reported on at the
MERIT Statewide Steering Group.

2.4 Monitoring outcomes

While it is important to monitor process related indicators, it is also essential to
systematically monitor program outcomes. Monitoring of program outcomes allows
the documentation of progress in meeting goals. Monitoring outcomes helps to
generate information for program managers in order to efficiently administer and
improve programs. The monitoring of outcomes is not a substitute for formal impact
evaluation but rather a gauge of progress to date.

2.4.1 Monitoring program, post program and unintended outcomes

Progress on meeting selected program outcomes are reported periodically in the
MERIT Annual Report. The data collected in the Annual Report may then be
compared to data collected in prior years in order to gain a sense of change over
time. Trends in the meeting of program outcomes may be identified. The trends in
data may highlight specific issues that can be acted upon, or may highlight areas of
future evaluation research.

3.0 Evaluation

3.1 The purpose of evaluation

The systematic and continual monitoring of program process and outcomes is
essential in providing a gauge of the working nature of a program and procedural
issues arising on an everyday basis. However to assess whether a program is
achieving its intended outcomes it is essential to conduct more formal impact
evaluations. It is only then that a reliable and valid assessment of the worth of the
program may be made. Ensuing decisions, with significant social implications, may
then be justifiable as they are based on a legitimate foundation of research.

3.2 Summary of impact evaluation reports

Table 2.0 provides a summary of the components of both process and outcome
based MERIT evaluation. While impact studies will reflect evaluation of each of the
program’s outlined objectives, formal evaluation of a program’s process often cover
issues such as program fidelity, equity, attitudes, financial considerations and follow
up care. Other process evaluations may occur premised on the identification of
specific issues in the monitoring reports, or premised on specific requests from
stakeholders.

12
Table 2.0: Summary of impact evaluation strategy and reports
Evaluation Indicators Key Question Format
component
Evaluating Program fidelity What does the program look like in practice? Process Study
process Does it match the intended process?
Program equity Are clients accessing the program in a fair and Aboriginal
equitable manner? In particular, given participation study
research on low referral and completion rates
of Aboriginal clients in diversion program, what
is Aboriginal participation and completion like
in the MERIT program?
Stakeholder attitudes What are the attitudes of key stakeholder Magistrates, Police
groups (in particular Magistrates and Police) and Legal Practitioner
to the MERIT program? survey
Resources What are the financial issues in the practice of Cost effectiveness
MERIT? Do they match the intended costs? study
Follow up care What are the follow up needs for MERIT Aftercare study
clients? Are these being met in the current
process?
Evaluating Program, post program To what extent does the MERIT program lead Lismore evaluation
outcomes unintended outcomes to the intended criminal justice outcomes? report
To what extent does the MERIT program lead Health outcomes
to the intended health outcomes? study

3.3 Evaluating process

Evaluation of process related variables is important in helping determine whether a
program matches the program model and therefore will likely achieve its program
goals and objectives.

3.3.1 Evaluating program fidelity

To complement existing process evaluation indicators an evaluation of MERIT at a
local level is being undertaken. The MERIT Process/Standards Study will assist in
identifying current process and practice challenges being experienced by MERIT
across the State, and will assist in developing program standards and performance
indicators. This information will also assist in the review of the MERIT Operational
Manual to best meet the current operational needs of MERIT.

3.3.2 Evaluating program equity- Aboriginal participation

Part of any process evaluation strategy should be the specific study of participation.
This data will be used in particular to answer questions about the equitable
participation of groups traditionally under represented in diversion programs. The
Aboriginal Participation Study investigates the participation and completion of
Aboriginal clients in the MERIT program. The findings from this study will highlight
areas of the program that might be improved in order to increase access and
completion rates.

3.3.3 Evaluating stakeholder attitudes – Police and Magistrates

It is important in process evaluation to study the views of those stakeholders who
have a specified role in the program’s implementation. In the MERIT program both
Magistrates and Police play important roles in referring, and for Magistrates
overseeing, the program. A study into Magistrates attitudes to MERIT has been
completed, while the Police attitudes survey is in progress.

13
3.3.4 Evaluating costs and benefits
It is important to examine whether each of the program’s benefits exceed their costs,
and whether diversion into treatment offers economic benefits to the community in
terms of reduced crime and improved health and social functioning as compared with
the costs of the status quo where there was no diversion option for eligible offenders.
A cost benefit analysis was undertaken as part of the Lismore evaluation and
provided an assessment of the cost savings of diversion into treatment. NSW Health
is currently developing a Health Costing study.

3.3.5 Evaluating follow up care

Process evaluation of programs with significant social implications should also
include study of the follow up care available to clients. This will help determine the
likely long term effects of program outcomes. The MERIT Aftercare Study addresses
this part of the process evaluation. This study will examine evidence on best practice
in aftercare for offenders and non offenders following drug treatment. Following this
review a trial model of aftercare may be developed, implemented in one or two
regions and evaluated.

3.4 Evaluating outcomes

Formal impact evaluation of all program outcomes is important in gauging whether
the program meets its stated objectives. Evaluation of outcomes attempts to discern
the extent to which the change in outcomes is attributable to the program itself.

3.4.1 Evaluating program outcomes

A full impact evaluation was carried out on the MERIT pilot program in Lismore
(Passey, 2003). The evaluation examined a number of program outcomes and
included study of: program and participant profiles, court outcomes and recidivism,
health and social functioning, participant perspectives, stakeholder views, economic
assessment and a review of legal issues.

A separate Health Outcomes Study is in the process of being conducted by NSW

Health and will focus on improved health and social functioning for the duration of the
program and in the post program period.

14
References

Passey, M (2003). Evaluation of the Lismore MERIT Pilot Program. NSW Attorney
General’s Department.

Rossi, P.H., Lipsey, M.W., & Freeman, H.E. (2004). Evaluation: A Systematic
Approach. 7th Edition, Sage Publications, Thousand Oaks.

15