The challenge of evaluating care integration programmes

Hubertus Johannes Maria Vrijhoef (Department Patient & Care, Maastricht University Medical Center, Maastricht, The Netherlands) (Family Medicine and Chronic Care, Vrije Universiteit Brussel Faculteit Geneeskunde en Farmacie, Brussel, Belgium)
Axel Kaehne (PANAXEA B.V., Amsterdam, The Netherlands and EPRC, Edge Hill University, Ormskirk, UK)

Journal of Integrated Care

ISSN: 1476-9018

Article publication date: 22 July 2019

Issue publication date: 22 July 2019



Vrijhoef, H.J.M. and Kaehne, A. (2019), "The challenge of evaluating care integration programmes", Journal of Integrated Care, Vol. 27 No. 3, pp. 190-192.



Emerald Publishing Limited

Copyright © 2019, Emerald Publishing Limited

The challenge of evaluating care integration programmes

Care integration evaluation is characterised by methodological pluralism and diversity. Moreover, researchers of care integration have chosen different research philosophical paradigms, i.e. basic belief systems or world views to guide their investigations. In order to measure integration from the perspective of patients or healthcare providers, interpretivism has played an important role (e.g. Bautista et al., 2016). However, positivism has dominated the field since the early days as policy makers were mostly interested whether integrated care is (cost-)effective. With most reviews and meta-analyses summarising the insights derived from quantitative data being mostly inconsistent (e.g. Damery et al., 2016; Valentijn et al., 2018), a shift towards realism is taken place in the last few years. In doing so, researchers argue that integration needs to be regarded as a tailor-made approach with evaluation paying attention to what works for whom in what circumstances, and how (e.g. Busetto et al., 2016; Turner et al., 2018).

Notwithstanding the value of individual research philosophical paradigms, their independent application limits the generalisability of evaluation findings and contributes to the rather low level of shared learning across the sector. Moreover, the methodological quality of evaluation in the field is concerning. This applies to studies with empirical data (e.g. Looman et al., 2018), reviews (e.g. Briggs et al., 2018) and studies on the development and validation of instruments to measure integrated care (e.g. Bautista et al., 2016).

Against this background, the Journal of Integrated Care invited authors to submit their work paying attention to the (many) challenges in evaluating care integration programmes. We were also very much interested in learning about novel methodologies to overcome such challenges.

In their case study evaluation, Hinde report on an extensive analysis of the Vanguard programme, exploring whether the implemented integrated response services resulted in any meaningful change in secondary healthcare activity. Data access, poorly specified aggregate secondary care data, and a poorly specified intervention were put forward as the main challenges confronted with. The methodological challenges of one of the Vanguard sites is also discussed by Grimwood (2019), who points our attention to what (modified) realist evaluation has to offer when dealing with complexity, strategy and rhetoric. By applying a multi-method evaluation protocol, Grooten et al. (2019) are looking into the implementation of a scaling-up strategy for integrated care initiatives. Pragmatism may perhaps be the best way to describe their approach as they plan to use realist evaluation and implementation science.

With the increasing interest in implementation science in this field, the viewpoint paper by Sadler et al. (2019) discusses three potential contributions from implementation science that can help clinicians and researchers to design and evaluate more effective integrated care programmes for older people with frailty. Another relative novel method is discussed by Gray and Shaw (2019). They explore summative, process and developmental approaches to evaluating complex interventions and suggest three guiding principles in doing so. A final contribution to this special issue comes from Kaehne (2019) who explores how Big Data may help us in the evaluation of care integration programmes and whether the evaluation and research community in this field is ready to take on what Big Data has to offer.

The challenge of shared learning in the field of care integration and its evaluation goes beyond a special issue or a section on evaluation in the Handbook Integrated Care (Amelung et al., 2017). It requires a culture where people share what they know, where people receive constant and constructive feedback, and where the expertise of team members is actively promoted. When care integration programmes and their evaluations take place in separate worlds with different dynamics and languages, it is no surprise that shared learning is not happening. In that sense, the Vanguard Programme has much to offer, including briefing series, blogs, podcasts, vanguard videos, resources, reports and site visits (NHS Confederation).

Researchers need to step out of their comfort zone as shared learning is not what most of them do on a regular base. Moreover, keeping dogmatically to existing study designs and outcome measures for publication purposes is not what evaluation of care integration programmes needs. If integration is indeed about the transformation of service delivery into new models of care, why then apply methods that are most suitable for the evaluation of existing services?

The same holds for editors of scientific journals. We need to embrace the challenges that come with change too, for example, by sharing what we know, by providing authors with constructive feedback, and by referring authors to each other, in short, by actively creating a community of practice. This special issue is an example of such an effort as both the Journal of Integrate Care and the International Journal of Care Coordination want to foster shared learning about how the evaluation of care integration programmes can further be improved to support informed decision making by all those involved.


Amelung, V., Stein, V., Goodwin, N., Balicer, R., Nolte, E. and Suter, E. (Eds) (2017), Handbook Integrated Care, Springer International Publishing AG.

Bautista, M.A.C., Nurjono, M., Lim, Y.W., Dessers, E. and Vrijhoef, H.J.M. (2016), “Instruments measuring integrated care: a systematic review of measurement properties”, The Milbank Quarterly, Vol. 94 No. 4, pp. 862-917.

Briggs, A.M., Valentijn, P.P., Thiyagarajan, J.A. and Araujo de Carvalho (2018), “Elements of integrated care approaches for older people: a review of reviews”, BMJ Open, Vol. 8 No. 4.

Busetto, L., Luijkx, K.G., Elissen, A.M.J. and Vrijhoef, H.J.M. (2016), “Context, mechanisms and outcomes of integrated care for diabetes mellitus type 2: a systematic review”, BMC Health Services Research, Vol. 16, p. 18.

Damery, S., Flanagan, S. and Combes, G. (2016), “Does integrated care reduce hospital activity for patients with chronic diseases? An umbrella review if systematic reviews”, BMJ Open, Vol. 6 No. 11.

Gray, C.S. and Shaw, J. (2019), “From summative to developmental: incorporating design-thinking into evaluations of complex interventions”, Journal of Integrated Care, Vol. 27 No. 3, pp. 241-248.

Grimwood, T. (2019), “Realism and rhetoric in the evaluation of a new care model”, Journal of Integrated Care, Vol. 27 No. 3, pp. 204-214.

Grooten, L., Alexandru, C.A., Alhambra-Borrás, T., Anderson, S., Avolio, F., Cotanda, E.V., Gütter, Z., Henderson, D., Kassberg, A.C., de Manuel Keenoy, E., Lange, M., Lundgren, L., Pavlickova, A., Txarramendieta Suarez, J., Whitehouse, D., Zabala, A.F., Rementeria, J.I.Z. and Vrijhoef, H.J.M. (2019), “A scaling-up strategy supporting the expansion of integrated care: a study protocol”, Journal of Integrated Care, Vol. 27 No. 3, pp. 215-231.

Kaehne, A. (2019), “Big data and what it means for evaluating integrated care programmes”, Journal of Integrated Care, Vol. 27 No. 3, pp. 249-258.

Looman, W.M., Huijsman, R. and Fabricotti, I.N. (2018), “The (cost-)effectiveness of preventive, integrated care for community-dwelling frail older people: a systematic review”, Health & Social Care, Vol. 27 No. 1, pp. 1-30.

NHS Confederation. available at: (accessed 10 June 2019).

Sadler, E., Sandall, J., Sevdalis, N. and Wilson, D. (2019), “The contribution of implementation science to improving the design and evaluation of integrated acre programmes for older people with frailty”, Journal of Integrated Care, Vol. 27 No. 3, pp. 232-240.

Turner, A., Mulla, A., Booth, A., Aldridge, S., Stevens, S., Begum, M. and Malik, A. (2018), The International Knowledge base for New Care Models Relevant to Primary Care-Led Integrated Models: A Realist Synthesis, NIHR Journals Libraryx, Southampton.

Valentijn, P.P., Pereira, F.A., Ruospo, M., Palmer, S.C., Hegbrant, J., Stemer, C.W., Vrijhoef, H.J.M., Ruwaard, D. and Strippoli, G.F.M. (2018), “Person-centered integrated care for chronic kidney disease: a systematic review and meta-analysis of randomized controlled trials”, Clinical Journal of the American Society of Nephrology, Vol. 13 No. 3, pp. 375-386.

Further reading

Hinde, S., Setters, J., Bojke, L., Hex, N. and Richardson, G. (2019), “Does the integration of response services lead to meaningful change in healthcare activity? A case study evaluation”, Journal of Integrated Care, Vol. 27 No. 3, pp. 193-203.

Related articles