Rating need for care in the Ontario Common Assessment of Need: A reliability assessment
In 2008, pilot testing of the Ontario Common Assessment of Need (OCAN) decision-making tool began in the community mental health sector in Ontario with the aim of standardizing, enhancing, and improving the efficacy of the client assessment process. The pilot implementation has since expanded across the province, led by the Community Care Information Management project (CCIM).
A cornerstone of the OCAN is the Camberwell Assessment of Need, a standardized, comprehensive set of items that measure need for care in persons with severe mental illness (categorized by Mental Health and Risk Needs; Basic Needs and Functioning Needs; Social Needs; and, Service Needs), as well as formal and informal help received across these domains. Studies support the reliability of the Camberwell when administered during research. Less studied, however, has been its reliability when the measure is completed in the field as part of routine care delivery by many staff working in diverse organizational settings (i.e. varying in size, approach and programs offered).
The study, Rating Need for Care in the Ontario Common Assessment of Need: A Reliability Assessment, led by Drs. Janet Durbin and Elizabeth Lin, Centre for Addiction and Mental Health, assessed the reliability of the Camberwell need for care items when collected through the OCAN implementation in Ontario. Specifically, the study tested the consistency of ratings made by 123 trained program staff from different organizations across Ontario based on the same consumer information, and the accuracy of staff ratings compared to a ‘gold’ standard (established by a consensus of four experts).
Performance was good in most of the mental health and risk domains, and almost all of the basic needs and functioning areas; however, few of the social and service domains achieved consistency in ratings. Social and service needs may be areas of assessment that are newer for some staff or where a standardized approach to assessment is not common. Reliability may improve over time as staff develop more knowledge, experience and comfort with rating these areas.
The study team suggests that periodic re-assessment of reliability performance, which is done for other established health-related databases in Canada, is important to ensure that this aspect of data quality is supported as effectively as possible in the implementation of the OCAN assessment tool and process in Ontario.
See “Rating Need for Care in the Ontario Common Assessment of Need: A Reliability Assessment,” December 2010, available on the OMHAKEN website at www.ehealthontario.ca.
For more information about OMHAKEN, please contact Keri West at email@example.com.