This paper aims to examine the relationship between advocacy on behalf of libraries and impact evaluation in a national public library development context in which the boundaries between these two roles are likely to be blurred, creating ethical issues for all concerned.
The authors draw upon their broad experience of helping various international organisations to develop strategic approaches to impact planning and evaluation for public libraries, as well as their work in helping to develop library practice at national level, in order to focus on and examine the creative tensions between impact evaluation and advocacy.
There are particular issues for all key participants (international programme funders, policy shapers, service managers, evaluators and advocates) in planning, supporting and delivering impact evaluation programmes. Most of these can be addressed directly but some (such as balancing programme requirements against local priorities, or achieving a balance between collecting evidence based on predetermined impact indicators and progressive focusing) entail management of the tensions between conflicting pressures.
Specific ways forward are offered to encourage ethical advocacy and impact evaluation at national library development level. These principles are equally applicable to education and health library development and to public library development at regional or local levels.
The particular issues of advocacy and impact evaluation in the national public library development context have not previously been recognized in the international development literature addressing advocacy and evaluation or in the library and information services research literature.
Streatfield, D. and Markless, S. (2011), "Impact evaluation, advocacy and ethical research: some issues for national strategy development?", Library Review, Vol. 60 No. 4, pp. 312-327. https://doi.org/10.1108/00242531111127866Download as .RIS
Emerald Group Publishing Limited
Copyright © 2011, Emerald Group Publishing Limited
This paper charts some consequences of the growing pressure for libraries to prove their impact in terms that resonate at national policy level, examines various pressures that may influence methodological choices and their possible consequences and discusses how to create conditions for ethical exploitation of research evidence.
From a starting point that, especially in emerging countries, advocacy is critical to secure sustainable services by attracting ongoing funding from government in the face of competing priorities, the authors show how demands of funding bodies can put pressure on researchers to focus on evidence of success rather than evidence of fundamental change. They show the nature of the pressure on practitioner‐researchers who need to keep their own services going and who may be “researching for survival” and point out some of the pressures on new evaluators enter the library and information services (LIS) field as they engage with others in the complexities of shifting the focus from statistically based service monitoring to evaluation based on qualitative research methodologies and methods.
The authors then outline an approach to deeper level engagement with the issues surrounding evaluator and advocate integrity when undertaking national impact evaluation work.
2 Impact evaluation and advocacy
What do we mean by impact evaluation and by advocacy? Following Fitz‐Gibbon (1996), we define impact as any effect of the library service (or any of its activities or initiatives) on an individual, group or community. This effect:
may be positive or negative;
may be intended or accidental; and
may affect library staff, senior managers, users/customers, pupils, teachers, parents, local politicians, etc.
quality of life, e.g. self‐esteem, confidence, feeling included, work or social prospects; and
educational and other outcomes, e.g. skills acquired, educational attainment, levels of knowledge.
Advocacy is defined in the Concise Oxford Dictionary (Allen, 1990) as “verbal support or argument for a cause, policy, etc.” In the LIS context, advocacy involves articulating the case for LIS services, creating and sustaining networks of people who will help make the case, and persuading policy shapers that LIS can help meet their strategic objectives. Although the messages that make up the case are not confined to evidence from evaluation, evaluation evidence should be the strongest and most convincing element of the case. Evaluation evidence should make the difference between evidence‐based decision making and persuasion through propaganda. However, as we shall argue below, advocacy may not always be important or even appropriate in attempting to secure library development.
3 Context: developing library services – the Golden Age (if any) and after
When we think about strategic development of libraries, it is important to take account of their history and, in the context of this paper, the history of library service evaluation, of advocacy, and of service evaluation more generally. How libraries have traditionally been managed and developed in different countries is likely to significantly influence the scope and prospects for evaluation‐based development and for effective advocacy. For the purposes of this paper, we will confine ourselves to three versions of library development history, whilst recognising that some countries will have different experience of whether and how evaluation and advocacy fit into their library development experience. The issues and ways forward may resonate differently in these countries but we hope that there will be enough recognition of issues to make the ways forward that are proposed at least tenable.
Turning to our three variations on a theme, as in most areas of human endeavour, the recent history of library service development is frequently presented as a story of ad hoc and opportunist development (often from a low base); of steady progress (in formerly or currently centralist or totalitarian states) or in countries with well‐developed library services, as a descent from a more or less mythical Golden Age (usually positioned one or two generations prior to the one in question).
In countries where there is no strong tradition of library development, evaluation of library services, and often of public services more generally, was likely to be confined to token gathering of performance statistics (sometimes called “busyness statistics”) except in the case of externally funded projects. The priority for evaluation of public sector projects (including library development projects) was usually to provide the funder with assurance that the project had been accomplished, so that any evaluation normally focused primarily on change to the services on offer rather than to users of services or to the communities affected. The exception was where funders insisted that grantees collect evidence of the broader impact of services. Such insistence could create problems in countries where there was limited expertise in such evaluation. Advocacy in this context was usually confined to making the case to meet the next external project funding opportunity and would be largely shaped by funder priorities and requirements.
In the former Communist states of Eastern Europe and in other countries where the government tended towards totalitarian authority, monitoring and evaluation requirements were usually prescribed centrally in some detail and, in the library development context, were heavily reliant upon service performance data as a (more or less token) confirmation that prescribed targets were being met. Public advocacy had little significance in this context although aligning library services with potentially influential political patrons could be important.
We will give a little more attention here to the countries with well‐developed library services, because attempts have recently been made to export the New Managerialist evaluation and advocacy “solutions” that we will describe below into the former Communist states and to other countries in Asia and South America (with widely varying levels of apparent success).
In the library development version of the Golden Age myth (as in other areas of public service provision), the politicians and other policy shapers engaged in supervising national, regional or local library services relied heavily on historically established budgets when allocating resources and put great reliance upon the professional competence of professional staff (who were only incidentally seen as service managers). These professional staff expected to advise policy decision makers about the development of services and to take prime responsibility for organising appropriate professional delivery of the agreed services. They used a range of service performance data to monitor their services and expected to take professional decisions if problems were revealed or suspected. Where significant changes in the direction of service delivery were contemplated, the librarians might combine performance data with appropriate demographic evidence to illustrate the extent of reach and penetration of these services. Advocacy in this environment was relatively straightforward, since libraries were universally accepted as “a good thing” and professional advice was taken seriously, even if tempered by recognition of the element of vested interest involved when librarians made proposals (You would say that as a librarian, wouldn't you?).
In this version of the Golden Age myth everything began to change with the advent of New Managerialism in the public services in some countries during the 1980s and 1990s. Critics of New Managerialism (a term which is usually applied pejoratively) claim that it denigrates professionalism in favour of generic service management, emphasises public scrutiny of services, opts for complexity in articulating service objectives and focuses on evaluating apparent effects on “customers” (who replace “service users” in the New Managerialist vocabulary) by means of arbitrary service delivery targets (for a more positive view, see Boston et al., 1996). Deem et al. (2007) characterised the consequent effect on public services as a descent “from regulated autonomy to institutionalized distrust.” Pursuing this theme, Macfarlane (2009, p. 159) claims that:
The performative culture is symptomatic of a society in which there has been an erosion of trust in the professions and those working in the public sector […] The effect has been to create a contemporary research environment that places a strain on many of the virtues essential to the ethical conduct of research […].
We will return to the theme of ethical research later.
Clearly, New Managerialism is not a universal phenomenon and there has been some reaction against its extreme forms in countries which at first embraced its tenets with the greatest enthusiasm. However, where this approach to public sector management has been even partially adopted, it has fuelled demand for introduction of service impact assessment to gauge performance levels, often as part of the espousal of “evidence‐based decision making” (which, by narrowly prescribing the nature of acceptable evidence may become a convenient rationing device to limit service innovation to proposals which can meet these prescriptions – House (2006) offered a powerful critique of this approach as applied by the US Government in the 1990s, labelling it as “methodological fundamentalism”). One effect of this demand for evidence has been to trigger a reaction which involves turning to advocacy to ensure the survival of valued services.
Taking an optimistic view, as service evaluation and advocacy become increasingly part of the discourse of public sector development, more benign and democratic forms of evaluation are likely to be considered in many countries, echoing the academic debate of recent years (within the LIS field, international organizations such as the Global Libraries Initiative and IFLA have led the way in encouraging localized approaches to evaluation and encompassing disadvantaged groups within the evidence gathering, if not yet full democratization of control of the evaluation programmes).
Meanwhile, library service managers in North America and Western Europe have responded to the evaluation and advocacy demands of their new political environments in various ways:
Calling on or collaborating with academic researchers to conduct large‐scale evaluations of the economic and social effects of public libraries (Bryson et al., 2002; University of North Carolina, School of Information and Library Science, 2006; Griffiths et al., 2006), or of public access to computers and the internet through public libraries (Griffiths and King, 2008; Bertot et al., 2009; Becker et al., 2010); or of school libraries (Todd and Kuhlthau, 2004), the results of which can be used for advocacy (Scholastic Research Foundation Paper, 2008).
Hiring consultants to evaluate the economic impact of their services by, for example, the British Library (Spectrum Strategy Consultants and Indepen Consulting, 2004) or Manjarrez et al. (2007).
Engaging in professional development to equip themselves and their staff to undertake qualitative impact evaluation of services, through action research, such as the Impact Implementation Programme involving 22 university library teams (Payne and Conyers, 2005; Markless and Streatfield, 2005) or through professional association workshops (Markless and Streatfield, 2006).
Joining international library development projects with strong components of impact evaluation and advocacy at national level, for example, the Global Libraries Initiative (Fried et al., 2010).
Participating in an international library development programme focused on strengthening library associations' capacity, notably in relation to advocacy – see the IFLA Building Strong Library Associations Programme to be announced today at this Congress.
4 The changing world of evaluation
Of course, impact evaluation was not being conducted within an LIS cocoon (nor even within a public services bubble). Fundamental changes in how evaluation and particularly qualitative evaluation is conceived and interpreted were taking place over a similar time period (Guba and Lincoln, 1987; Shaw et al., 2006). These changes in thinking about the political nature of evaluation (Taylor and Balloch, 2005; Vestman and Conner, 2005; Abma, 2006), and about the relationship between qualitative evaluation and government policy (House, 2005) need to be taken into account in any strategic approach to evaluating library services at national or international levels. Various ideas about the notion of a more inclusive and democratic evaluation embracing a wider range of stakeholders including marginalised groups and involving them in designing the evaluations and in interpreting the findings are particularly important (Mertens, 2003; Greene, 2006). Central here is the idea of preventing qualitative evaluation from becoming just another way of enforcing the existing power relationships between governments and their people.
Reflecting on these trends from an LIS perspective, it should be noted that real inclusiveness is scarcely achievable within national evaluations of library services. The best that can be hoped for is to engage with a range of informants drawn from all the main groups of people likely to be affected by changes in services, including marginalized groups and to ensure that minority views are accorded weight. This is likely to be more than enough of a challenge for evaluators in many countries!
5 Ethical evaluation
Much of this reinterpretation of evaluation touched on or was fuelled by consideration of ethical evaluation or more general consideration of values in evaluation (House and Howe, 1999). This evaluation debate has led various evaluation professional bodies to seek to establish a shared professional stance on ethical issues (e.g. The American Evaluation Association, 1995). The concept of ethical evaluation has been further pursued by individuals, such as Datta (1999) who examined whether published guiding principles are helpful in aligning a range of different interpretations of ethical positions, and Simons (2006, p. 243) who pointed to the four main levels of socio‐political interactions – with government and other commissioners of evaluations, with programme participants, with the evaluation profession and with the wider audiences, before emphasising that:
Evaluation has to operate in this multilayered context of different interests, providing information to inform decisions while remaining independent of the policies and programmes themselves. In such a context it is not surprising that ethical dilemmas arise as to which is the best course of action to take.
Nor is it surprising that the presentation of evaluation findings creates a further series of ethical concerns for advocates. There is relatively little material published specifically on the role of evaluator as advocate, although in reviewing ethical evaluator positions, Datta cited various authors who argued that evaluators should not be advocates but should strive to produce non‐partisan evaluations (Chelimsky, 1997; Scriven, 1997). She then contrasted their views with those of a range of prominent evaluators (Lincoln, 1990; Greene, 1995; Mertens, 1995) who asserted that evaluators should be advocates whose role is to align with “the underdog, the oppressed, the marginalized in their fight for social justice.” Finally, she charted a middle path, drawing on Lincoln et al. (1998, p. 108), who wrote:
When I talk about advocacy I don't mean taking sides in that more specific sense. What I mean rather refers to becoming an advocate for pluralism, for many voices to feed into the evaluation.
Interestingly, because evaluators should be advocates for democracy and the public interest, they should not be advocates for particular stakeholder groups in which perceived interests are viewed as impervious to evidence and are promoted come what may (Greene, 1995, uses the term advocacy in one sense and Chelimsky, 1998, uses it in the other, so unfortunately they are talking at cross‐purposes, pp. 95‐6).
Datta concluded that:
Diverse evaluators “agree” that the evaluator should not be an advocate (or, presumably, an adversary) of a specific programme in the sense of taking sides, of a preconceived position of support (or destruction).
There is “agreement” that the evaluator should be an advocate for making sure that as many relevant cards as possible get laid on the table, face up, with the values (worth, merit) showing.
There is “agreement” that the evaluator must be aware of how less powerful voices or unpopular views, positions, information can get silenced and make special efforts to ensure that these voices (data, view, information) get heard.
6 Evaluating national library development programmes
We consider that it is important for people who are involved, or who are considering whether to become involved, in evaluating national library development programmes to take account of the history of public service evaluation in the relevant countries as well as the more general debate about ethical evaluation and advocacy. Some of the implications of both are summarised below along with comments drawn from our experience of helping library development programme staff to engage with evaluation and advocacy issues in what we hope is an ethical manner.
6.1 Some issues for international programme funders
Programme funders have the unenviable task of trying to steer a path through the minefield of ethical evaluation and at the same time to achieve an appropriate balance between central programme evaluation requirements and national or local autonomy in choosing where and how to focus local evaluations. This tension may give rise to a number of issues:
Broadening the evaluation agenda. To what extent, if at all, should funders seek to extend the evaluation repertoire of potential grantees? If grantees are comfortable with the simple logic model of evaluation even when undertaking complex and innovative development programmes with uncertain outcomes, should they be expected to adopt more sophisticated and nuanced versions of evaluation? If so, should these expectations apply equally in all countries, whatever their history and experience of evaluation? What are the likely implications for grantees of any such shift in terms of expertise and resources? Who should pay for more demanding evaluation?
Towards democratic evaluation. Should funders encourage potential grantees to adopt more inclusive models of evaluation and advocacy? If they do so there is likely to be a knock‐on effect on the way in which programmes are planned and managed, which may not be acceptable in some countries. Again, should any such encouragement be offered in all countries involved with a programme or only those with appropriate prior experience?
Proportionate evaluation. How much evaluation should national participants be expected to undertake and how much will be done centrally? For smallish scale programmes is it necessary or feasible to seek to systematically gather impact evidence from service users? If so, on what scale should this be attempted and over what period of time? And should funders insist that socially excluded groups are included in any evaluation?
Programme requirements versus local priorities. What is the appropriate balance between requiring countries involved in a programme to standard types of evidence in prescribed ways and encouraging countries to focus on areas that they regard as important? Should freedom to decide on local priorities be extended to a free choice of how to evaluate whether these priorities are being met? If a minimum of standard evidence is required from all programme participants, is an opportunity being lost for learning about the wider effects of programmes across a range of settings?
The doubtful path from impact evidence to policy change. The usual approach to influencing policy through advocacy appears to be based on a simple transmission model of change which involves collecting and assembling impact evidence, interpreting and presenting this through advocacy and expecting the results to be adopted as policy changes. Is there any evidence that this model works except in some areas of the health policy field? (Gray, 1997; there is, however, substantial literature to suggest that the relationship between evaluation evidence and policy change is complex and often frustrating – Weiss, 1999; Davies et al., 2006; Carden, 2009).
6.2 Some issues for policy shapers
The people who influence and formulate library development policy at national level are also faced with balancing a number of evaluation and advocacy issues:
Target setting. There may be growing governmental pressure to set national development targets for libraries but the efficacy of such targets is questionable, especially if the current state of libraries is widely disparate. Is it possible to find a balance between targets that are so general as to be useless, so complex that they require disproportionate administrative efforts to reach them and begin to corrupt the service in the process, or so precise that they can readily be corrupted by imaginative service managers?
Evaluation evidence gathering. Collection of qualitative evaluation evidence is time consuming and has to take account of respondents' availability, service delivery requirements, time for information analysis and synthesis and other factors, all of which reduce the likelihood of the evaluation process fitting comfortably into the annual planning cycle.
Separation of evaluation and advocacy. At the height of the New Managerialist fashion, there was a tendency for research commissioners to state their requirements in terms that elided evaluation and advocacy. Even if this trap is avoided, there is some potential danger of designing evaluations to demonstrate the effectiveness of the library service rather than to establish whether the library service is effective, especially where service managers play a key role in securing the evaluation.
Calling for appropriate evaluation. The traditional and still dominant model of impact evaluation (sometimes called the simple logic model) is based on identifying impact indicators and collecting evidence against them. However, as libraries increasingly seek to take advantage of ICT developments by introducing new services, they are moving into the area of “emergent evaluation” in which it is not straightforward to predict where change will appear and therefore, where to focus the evaluation. Rogers (2010) distinguished between simple logic models and complicated logic models/programme theory, which she characterizes as interventions with a multiplicity of components, agencies and causal strands. She then articulated a further category of complex interventions and logic models/programme theory where the paths from action to impact are complex (in her terms, recursive causality with reinforcing loops), with disproportionate relationships (where at critical levels, a small change can make a big difference) and emergent outcomes (which cannot readily be specified at the outset). The more advanced library services appear to have all the characteristics of the complex logic model. To what extent should evaluators be encouraged to embrace emergent evaluation when the services being evaluated are innovating?
6.3 Some issues for service managers
The main issues relating to evaluation and advocacy for library service managers are about what is practical given the constraints of time and resources. To some extent this depends upon why the managers are embracing evaluation: the main reasons that we have encountered in running training workshops on evaluation for library managers are:
to secure additional resources;
to ensure the survival of the service;
for external accountability at national or regional level;
for internal accountability;
as a focus for development of the library service;
to enhance the status/profile of library staff who are seen to be taking service improvement seriously; and
for professional pride/job satisfaction of library staff, knowing that they are doing a good job.
Some of the practical decisions required are whether to:
Employ or train evaluators rather than rely on external experts. Securing in‐house evaluation capacity offers flexibility in undertaking short‐term and small‐scale evaluations of specific initiatives and should help to secure staff support for evaluation work. Many managers would also argue that it is better to do evaluation yourself rather than having arbitrary impact indicators and targets imposed from outside. On the other hand, internally conducted evaluations may give rise to suspicion that the results have been manipulated and only the most dedicated internal evaluator will be able to conduct the more sophisticated forms of evidence gathering.
Align impact areas appropriately. It may be possible to choose impact areas with those of central government; or with those of the organization of which the library service is part; or to focus on local development priorities. In some countries, of course, decisions about impact areas and indicators may be reached without reference to library service managers. However, if there is a choice to be made or recommended, alignment with national or regional government priorities should make it easier to construct an advocacy case that resonates at policy levels. On the other hand, since each library service will have a different history and current circumstances (including the state of the library) evaluation focused on local priorities may be more important. Whatever the choice, it is important to ensure that the impact areas and indicators are chosen to show what difference the library service can make; it will not be possible to show what difference the library makes if a national or regional priority is adopted without indicating how the potential library contribution can be evaluated.
Find enough time to assemble evidence. In many countries, there may be pressure on library services to conduct evaluations to fit within annual funding or planning cycles or, if international or national funding is involved, to complete evaluation programmes within two or at most three years. This is problematic at the one‐year level since it will be difficult to find any evidence of significant change within such a short‐time period, but it may still be too early to evaluate change at the three‐year level, since it is well established that major change (such as introducing public access ICT through public libraries) usually takes from three to five years (Fullan, 2007). Hurried evaluations may lead to misdirection in focus onto what can be measured in time rather than what should be evaluated; they may result in insufficient evidence of sufficient quality to make an effective advocacy case for service sustainability (in this context, the Global Libraries Initiative offers a good practice example by committing to a five‐year programme). It is interesting to note that there have been relatively few longitudinal studies of library service impact.
Address social inclusion. The literature on democratic evaluation emphasises the importance of including marginalised people within evaluation programmes. This raises the issue of what social exclusion means in the context of library services. If people are excluded from equitable access to libraries through access problems, bureaucratic rules or other means, will this make it difficult to include them in service evaluation? How can such difficulties be overcome?
Take a main role in advocacy. Should library service managers take a leading advocacy role or should they rely upon respected “champions” who are not seen to have a vested interest in libraries? Is it possible to take a leading advocacy role and to play a significant part in service evaluation or will this inevitably lead to focusing on only collecting positive evidence?
6.4 Some issues for evaluators
Although evaluators should be independent of the library services that they are evaluating, in many smaller countries, this may not be feasible in practice, since evaluators, policy shapers and library service managers are all likely to operate within the same circles. Retaining an independent stance becomes more problematic if the evaluator is employed by a library service or is part of the LIS community. Whatever their position, evaluators have a number of issues to manage:
Pressure from key participants. Demands of funders, policy shapers and library service managers can put all pressure on evaluators to focus on evidence of success rather than evidence of fundamental change and to minimize the amount of evidence collection required, since this may put additional burdens on the library services. Giving way to this pressure may lead to inappropriate methodological choices for evidence collection (usually resulting in over‐reliance on routine performance data collection which tells the evaluator little about changes being produced by new or modified library services). In turn, this may lead to going through the motions of evaluation without focusing on significant change, or focusing on library activities instead of on their effects on users.
Insufficient time for effective evaluations. Unrealistic expectations about what level of change can be seen in one, two or three years may lead to hurried evaluations, which may in turn lead to distortion in interpretation of results, especially by practitioner‐evaluators who need to keep their own services going and who may be “researching for survival.”
Retaining research integrity. As new evaluators enter the LIS field and engage with others in the complexities of shifting the focus from statistically based service monitoring to evaluation based on qualitative research methodologies and methods, this may lead to neglect of core evaluator values.
Opting for the simple logic model of evaluation. Although many organisational interventions are likely to be complicated or complex, it is perhaps not surprising that organisational managers or programme evaluators usually choose the politically expedient route of regarding impact evaluation as a simple logic mechanistic process (Ramirez, 2007). It is particularly difficult for the internal evaluator to propose and conduct more complex evaluation models, especially if these do not provide the option of comparing progress over time against baseline evidence.
Evaluator as advocate. Is it possible for LIS evaluators to sustain an ethical advocacy position when they are broadly (or in the case of evaluators employed by library services, specifically) identified with the existing services? Returning to Datta's principles of ethical advocacy for evaluators listed above:
Is an LIS evaluator inevitably an advocate of a specific programme in the sense of taking sides, of a preconceived position of support?
Can the LIS evaluator ensure that as many relevant cards as possible get laid on the table, face up, with the values (worth, merit) showing?
To what extent can the LIS evaluator ensure that less powerful voices or unpopular views get heard?
6.5 Some evaluation issues for advocates
Advocacy is critical, especially in emerging countries, to secure sustainable services (that can continue after initial project funding ends) by attracting ongoing funding from government or other sources in the face of competing priorities. But where does this role fit with the evaluation role?
Building a base for evidence‐based advocacy. Should the advocate encourage the evaluator to build and sustain an appropriate evidence base to create conditions for ethical exploitation of research evidence?
Identification with the library service. To what extent should advocates identify with the values, priorities and reputation of the library services for which they are assembling an advocacy case?
Beyond evidence‐based advocacy. Is it legitimate for the advocate to:
try to present a complete picture of the service under consideration, including its weaknesses and areas requiring attention as revealed by evaluation?
present a positive picture but firmly grounded in the evaluation evidence?
select only success stories from the evaluation and re‐present these to make the advocacy case more strongly?
rework the evaluation evidence to present as positive a picture as possible?
7 Some ways forward
What can the LIS evaluator and advocate do to strengthen their endeavour to conduct their work both effectively and ethically and how can policy shapers and library service managers help them to do this? We suggest that they discuss any of the issues raised in the previous section that they regard as relevant. Here are some closing thoughts about how this work can be supported, wherever it is undertaken.
7.1 Evaluator and advocate integrity
The authors of this presentation are currently completing another paper on the ethics of impact evaluation focused on libraries. They have adapted a framework for ethical research developed by Macfarlane (2009) in order to examine ethical research positions and the difficult decisions to be taken at each stage in the research process. Since the current paper is focused on the role of the advocate and on the evaluator as advocate the most relevant equivalent stage in the research process identified by Macfarlane is the dissemination phase. We noted that this is the stage of impact evaluation at which political agendas and differences become most apparent. Since there is often a close potential link between positive evaluation and future organisational funding or even survival it is hardly surprising if people are more concerned about disseminating impact evaluation findings than about most other forms of research reporting. Questions about who should be told what and how and about how evaluation findings should be used in advocacy (aimed at securing greater resources and more support for the service being evaluated) are all inherently political.
We decided that the most appropriate virtue associated with the ethical evaluator at this stage in the process is resoluteness in the face of likely manipulativeness and partiality on the part of other key participants. We concluded that the key to survival as an ethical evaluator (and, we would now argue, the ethical advocate) at this stage is to keep listening to the impact evidence and resist any attempt by others to distort the findings. This is doubly difficult because dissemination of impact evidence is usually conducted in a politically charged environment in which there will be attempts to manipulate findings to secure vested interests. Is this an unrealistic stance for the advocate to adopt?
7.2 Broadening the debate
In a recently published examination of the consequences of 23 research projects funded by Canada's International Development Research Centre (IDRC), Carden (2009) identified three ways in which research can contribute to better governance, as:
encouraging open inquiry and debate;
empowering people with the knowledge to hold governments accountable; and
enlarging the array of policy options.
7.3 Evidence‐based advocacy
Effective advocacy at national level requires that the advocates have a strong sense of what changes are occurring in library service provision and in user responses to these services so that they can convey this convincingly. Such an overall picture requires evidence of service impact, especially where services are being enhanced – ideally this should entail baseline evidence and further evidence showing change over time. Qualitative research methods will be needed to build up an in‐depth picture: this is time consuming but worthwhile to secure a reliable evidence‐base showing both “good and bad news.”
7.4 Professional judgements
One unfortunate consequence of the adoption of evaluation approaches as part of New Managerialism was the insistence upon the ascendency of customer views at the expense of professional expertise, to the extent that if the majority of customers polled wanted a service to do “A”, but if the professional staff involved in running the service thought that they should do “B”, then the service did “A”. We believe that impact evidence is important but, unlike the New Managerialists, we also value professional judgement. In particular, we think that there is a vital role for professional judgements by practitioners in helping to provide nuanced interpretations of impact evidence – and of translating these into advocacy messages.
About the authors
David Streatfield is Principal of Information Management Associates, a research and consultancy team established in 1991, working in the public sector and focused on education, health and libraries (www.informat.org). He was previously Head of Information Research and Development at the National Foundation for Educational Research.
Sharon Markless and David Streatfield have worked together on national and more recently international level library service impact evaluation for a number of years and distilled their early experience in their joint book Evaluating the Impact of Your Library published by Facet Publishing in 2006. They are both currently Independent Impact Consultants to the Bill and Melinda Gates Foundation's Global Libraries Initiative and are also helping with impact assessment/evaluation work for the Global Impact Study (jointly funded by the Gates Foundation and by the International Development Research Centre) and for the International Research and Exchanges Board, as well as for the International Federation of Library Associations. Sharon Markless is the corresponding author and can be contacted at: firstname.lastname@example.org
Abma, T.A. (2006), “The practice and politics of responsive evaluation”, American Journal of Evaluation, Vol. 27 No. 1, pp. 31‐43.
Allen, R.E. (Ed.) (1990), The Concise Oxford Dictionary of Current English, Oxford University Press, Oxford.
American Evaluation Association (1995), “Task force on guiding principles for evaluators”, in Shadish, W.R., Newman, D.L., Scheirer, M.A. and Wye, C. (Eds), Guiding Principles for Evaluators, New Directions for Program Evaluation No. 66, Jossey‐Bass, San Francisco, CA.
Becker, S., Crandall, M.D., Fisher, K.E., Kinney, B., Landry, C. and Rocha, A. (2010), Opportunity for All: How the American Public Benefits from Internet Access at US Libraries, Institute of Museum and Library Services, Washington DC, available at: http://tascha.washington.edu/usimpact (accessed 20 October 2010).
Bertot, J.C., McClure, C.R., Wright, C.B., Jensen, E. and Thomas, S. (2009), Public Libraries and the Internet 2009: Study Results and Findings, available at: www.liicenter.org/Center_for_Library_and_Information_Innovation/Public_Libraries_and_the_Internet_Reports.html (accessed 20 October 2010). [Latest in a series of reports to the Bill and Melinda Gates Foundation and The American Library Association.].
Boston, J., Martin, J., Pallot, J. and Walsh, P. (1996), Public Management in the New Zealand Model, Oxford University Press, Auckland.
Bryson, J., Usherwood, R.C. and Streatfield, D.R. (2002), Social Impact Audit for the South West Museums, Libraries and Archives Council, Centre for Public Library and Information in Society, University of Sheffield, Sheffield.
Carden, F. (2009), Knowledge to Policy: Making the Most of Development Research, Sage/IDRC, Thousand Oaks, CA, available at: www.idrc.ca/openebooks/417‐8/ (accessed 20 October 2010).
Chelimsky, E. (1997), “The political environment of evaluation and what it means for the development of the field”, in Chelimsky, E. and Shadish, W. (Eds), Evaluation for the 21st Century: A Handbook, Sage, Thousand Oaks, CA.
Chelimsky, E. (1998), “The role of experience in formulating theories of evaluation practice”, American Journal of Evaluation, Vol. 19, pp. 35‐55.
Datta, L.‐E. (1999), “The ethics of evaluation: neutrality and advocacy”, New Directions for Evaluation, Vol. 82, Summer, pp. 77‐88.
Davies, P., Newcomer, K. and Soydan, H. (2006), “Government as structural context for evaluation”, in Shaw, I.F., Greene, J.C. and Mark, M.M. (Eds), Handbook of Evaluation: Policies, Programs and Practices, Sage, London.
Deem, R., Hillyard, S. and Reed, M. (2007), Knowledge, Higher Education, and the New Managerialism: The Changing Management of UK Universities, Oxford Scholastic Online Monographs, Oxford University Press, Oxford.
Fitz‐Gibbon, C.T. (1996), Monitoring Education: Indicators, Quality and Effectiveness, Cassell, London.
Fried, S., Kochanowicz, M. and Chiranov, M. (2010), “Planning for impact, assessing for sustainability”, Performance Measurement & Metrics, Vol. 11 No. 1, pp. 56‐74.
Fullan, M. (2007), The New Meaning of Educational Change, 4th ed., Teachers College Press, New York, NY.
Gray, J.A.M. (1997), Evidence‐based Health Care: How to Make Health Policy and Management Decisions, Churchill Livingstone, London.
Greene, J.C. (1995), “Evaluation as advocacy”, Evaluation Practice, Vol. 18, pp. 25‐36.
Greene, J.C. (2006), “Evaluation, democracy and social change”, in Shaw, I.F., Greene, J.C. and Mark, M.M. (Eds), Handbook of Evaluation: Policies, Programs and Practices, Sage, London.
Griffiths, J.M. and King, D.W. (2008), InterConnections: The IMLS National Study on the Use of Libraries, Museums and the Internet, Institute of Museum and Library Services, Washington, DC.
Griffiths, J.M., King, D.W. and Aerni, S. (2006), Taxpayer Return of Investment in Pennsylvania Public Libraries, available at: www.statelibrary.state.pa.us/libraries/lib/libraries/PAROIreportFINAL7.pdf (accessed 20 October 2010).
Guba, E.G. and Lincoln, Y. (1987), Fourth Generation Evaluation, Sage, London.
House, E.R. (2005), “Qualitative evaluation and changing social policy”, in Denzin, N.K. and Lincoln, Y.S. (Eds), The Sage Handbook of Qualitative Research, Sage, London.
House, E.R. (2006), “Democracy and evaluation”, Evaluation, Vol. 12, pp. 119‐27.
House, E.R. and Howe, K.R. (1999), Values in Evaluation and Social Research, Sage, Thousand Oaks, CA.
Lincoln, Y.S. (1990), “The making of a constructivist: a remembrance of transformations past”, in Guba, E. (Ed.), The Paradigm Dialog, Sage, Thousand Oaks, CA.
Lincoln, Y.S., Ryan, K., Greene, J., Lincoln, Y.S., Mathison, S. and Mertens, D.M. (1998), “Advantages and disadvantages of using inclusive evaluation approaches in evaluation practice”, American Journal of Evaluation, Vol. 19, pp. 101‐22.
Macfarlane, B. (2009), Researching with Integrity: The Ethics of Academic Enquiry, Routledge, London.
Manjarrez, C.A., Cigna, J. and Bajaj, B. (2007), Making Cities Stronger: Public Library Contributions to Local Economic Development, Urban Libraries Council, Evanston, IL, available at: www.urban.org/publications/1001075.html (accessed 22 October 2010).
Markless, S. and Streatfield, D.R. (2005), “Facilitating the impact implementation programme”, Library and Information Research, Vol. 29 No. 91, pp. 10‐19.
Markless, S. and Streatfield, D.R. (2006), Evaluating the Impact of Your Library, Facet, London.
Mertens, D.M. (1995), “Identifying and respecting differences among participants in evaluation studies”, in Shadish, W.R., Newman, D.L., Scheirer, M.A. and Wye, C. (Eds), Guiding Principles for Evaluators, New Directions for Program Evaluation No. 66, Jossey‐Bass, San Francisco, CA.
Mertens, D.M. (2003), “The inclusive view of evaluation: implications of transformative theory for evaluation”, in Donaldson, S.I. and Scrivens, M. (Eds), Evaluating Social Programs and Problems: Visions for the New Millennium, Lawrence Erlbaum, Mahwah, NJ.
Payne, P. and Conyers, A. (2005), “Measuring the impact of higher education libraries: the LIRG/SCONUL impact implementation initiative”, Library and Information Research, Vol. 29 No. 91, pp. 3‐9.
Poll, R. (2009), “Bibliography: ‘impact and outcome of libraries’”, available at: www.ifla.org/files/statistics‐and‐evaluation/publications/bibliography‐impact (accessed 20 October 2010).
Ramirez, R. (2007), “Appreciating the contribution of broadband ICT within rural and remote communities: stepping stones towards an alternative paradigm”, The Information Society, Vol. 23, pp. 85‐94.
Rogers, P.J. (2010), “Using programme theory to evaluate complicated and complex aspects of interventions”, Evaluation, Vol. 14 No. 1, pp. 29‐48, available at: http://evi.sagepub.com/cgi/content/abstract/14/1/29 (accessed 20 October 2010).
Scholastic Research Foundation Paper (2008), School Libraries Work!, 3rd ed., Scholastic Library, Danbury, CT.
Scriven, M. (1997), “Truth and objectivity in evaluation”, in Chelimsky, E. and Shadish, W. (Eds), Evaluation for the 21st Century: A Handbook, Sage, Thousand Oaks, CA.
Sey, A. and Fellows, M. (2009), Literature Review on the Impact of Public Access to Information and Communication Technologies, CIS Working Paper No. 6, Center for Information and Society, University of Washington, Seattle, WA, available at: www.cis.washington.edu2009/05/06/cis‐working‐paper‐no‐6 (accessed 22 October 2010).
Shaw, I.F., Greene, J.C. and Mark, M.M. (2006), Handbook of Evaluation: Policies, Programs and Practices, Sage, London.
Simons, H. (2006), “Ethics in evaluation”, in Shaw, I.F., Greene, J.C. and Mark, M.M. (Eds), Handbook of Evaluation: Policies, Programs and Practices, Sage, London.
Spectrum Strategy Consultants and Indepen Consulting (2004), Measuring Our Value: Results of an Independent Economic Impact Study, British Library, London, available at: www.bl.uk/aboutus/stratpolprog/increasingvalue/measuring.pdf (accessed 22 October 2010).
Taylor, D. and Balloch, S. (Eds) (2005), The Politics of Evaluation: Participation and Policy Implementation, The Policy Press, Bristol.
Todd, R.J. and Kuhlthau, C.C. (2004), Student Learning through Ohio School Libraries, Ohio Educational Library Media Association, Columbus, OH, available at: www.oelma.org/studentlearning/default.asp (accessed 20 October 2010).
University of North Carolina, School of Information and Library Science (2006), Taxpayer Return‐on‐Investment (ROI) in Pennsylvania Public Libraries, Report for the Commonwealth of Pennsylvania, University of North Carolina, School of Information and Library Science, Chapel Hill, NC, available at: www.palibraries.org/associations/9291/files/FullReport.pdf (accessed 22 October 2010).
Vestman, O.K. and Conner, R.F. (2005), “The relationship between evaluation and politics”, in Shaw, I.F., Greene, J.C. and Mark, M.M. (Eds), Handbook of Evaluation: Policies, Programs and Practices, Sage, London.
Weiss, C.H. (1999), “The interface between evaluation and public policy”, Evaluation, Vol. 5 No. 4, pp. 468‐86.
Ryan, K., Greene, J., Lincoln, Y.S., Mathison, S. and Mertens, D.M. (1998), “Advantages and disadvantages of using inclusive evaluation approaches in evaluation practice”, American Journal of Evaluation, Vol. 19, pp. 101‐22.