The purpose of this paper is to describe a pilot to test an approach to measuring inter-rater reliability of the Outcomes Star suite of tools. The intention, in publishing this account, is to show transparency in on-going development of the tool, and to invite further co-operative development.
In total, 24 workers, trained to use the first edition Family Star, scored a tested case study. Scoring was analysed using two metrics on the ten-point scale and the underlying five-point Journey of Change. The case study approach and metrics were evaluated for validity and accessibility.
This initial evaluation suggests this edition of the Family Star has good inter-rater reliability for the five-point Journey of Change, reaching the accepted threshold of 0.8 for the inter-rater reliability coefficient when three outlying workers are excluded. The reliability for the full ten point scale was moderate.
The sample size of 24 raters is small, though sufficient for an initial test of the approach, which will now be applied to larger samples, using other versions of the Outcomes Star.
The findings indicate that it is important that service providers test worker understanding of the scales to ensure consistency of use. The second edition of the Family Star incorporates more precise definitions of the ten-point scales to help improve the reliability.
The case study method and metrics provide an accessible measure of reliability, both for Star development and to enable managers to assess the reliability of an organisation's client data for internal and external purposes.
Emerald Group Publishing Limited
Copyright © 2014, Emerald Group Publishing Limited