The purpose of this paper is to report on performance measurement and impact assessment progress made in 14 countries as part of the Global Libraries initiative, starting with the early country grants in Mexico and Chile. For the mature grants in Bulgaria, Botswana, Poland, Romania, Ukraine and Viet Nam which were recently completed or are approaching completion, the nature of the country program is outlined, before the impact assessment work is described and some recent results and conclusions are reported. A similar approach is adopted with pilot and new grants in Colombia, Indonesia, South Africa, Turkey and Lithuania.
The country reports are presented as a series of case studies, in some cases supplementing those in an earlier special issue of this journal.
Where appropriate, recent country-specific survey findings are reported.
This paper shares Global Libraries IPA learning at country level with people in other countries who may be contemplating public library evaluation at regional, national or local level or who are interested in performance measurement and impact evaluation.
These cases studies reflect concentrated impact assessment and performance measurement work at country level across a range of countries over more than 12 years.
Al, U., Andrade Blanco, P., Chiranov, M., Cruz Silva, L.M., Devetakova, L.N., Dewata, Y., Dryžaite, I., Farquharson, F., Kochanowicz, M., Liubyva, T., López Naranjo, A., Phan, Q.T., Ralebipi-Simela, R., Soydal, I., Streatfield, D., Taolo, R., Trần, T.T.T. and Tkachuk, Y. (2015), "Global Libraries impact planning and assessment progress", Performance Measurement and Metrics, Vol. 16 No. 2, pp. 109-131. https://doi.org/10.1108/PMM-05-2015-0015
Emerald Group Publishing Limited
Copyright © 2015, Authors. Published by Emerald Group Publishing Limited. This work is published under the Creative Commons Attribution (CC BY 3.0) Licence. Anyone may reproduce, distribute, translate and create derivative works of the article (for both commercial and non-commercial purposes), subject to full attribution to the original publication and authors. The full terms of this licence may be seen at http://creativecommons.org/licenses/by/3.0/legalcode .
This paper traces the long Global Libraries journey into the world of impact planning and assessment (to use the GL preferred term for what is sometimes called impact evaluation). The international public library development work of the Bill & Melinda Gates Foundation (referred to from here on as the Foundation) began with two grants to Chile and Mexico commencing in 2002. The first section of this paper looks briefly at these grants and at some of the subsequent impact assessment and performance measurement work conducted in Chile. The next group of Foundation country grants was funded under the umbrella of the Global Libraries initiative and began to address impact assessment in a more concerted way, as described below. These reports concentrate on updating the country-level impact assessment work undertaken since the previous reports published in an earlier issue of this journal (Vol. 13 No. 1) in 2012. The third wave of pilot projects and country grants is then introduced: these have had the benefit of being able to tap all the earlier Global Libraries learning about performance measurement and impact assessment and have coincided with a reinforcement of the outcomes-focussed approach adopted by GL.
2. Early grants
The first two country grants were awarded to Mexico and Chile in 2002. When the e-Mexico program began, only 2 percent of their public libraries had computers. By March 2008, 2,728 of Mexico’s public libraries (38 percent) were connected to the internet through 15,000 computers (fewer than 4 percent of the population had internet access in 2002, with little or no access in the rural south and south east). Although the evaluation concentrated on performance measurement the program was able to show substantial change, with libraries in all 31 states and the Federal District participating in the program, which involved providing more than 300,000 hours of computer training for library staff and volunteers and 450,000 hours of computer training of library users.
The BiblioRedes Program in Chile began in 2002 and since then three grants have been received from the Foundation, leading to three impact assessment studies being conducted.
Table I shows the features and methods used in each of these studies.
As we can see from the different impact studies applied, mixed models have been developed, seeking to account for the effects of the project and ICT on public library users and the communities where they are served.
The team began all these studies with a diagnostic process which identified a social problem which they set out to mitigate with the program: in the first instance the digital gap was determined as the main problem; and then from 2006 the problem became more complex and was defined as a low digital inclusion of citizens.
For the study developed from 2002 to 2005 the team decided to undertake a panel longitudinal study to establish a baseline and then to allow monitoring of users and non-users of public libraries throughout this period on the effect produced by the BiblioRedes Program (see Arnau, 1995). To achieve this, they monitored the same public library users, or library users with similar characteristics, to determine the main changes or effects in their lives, and the same procedure was conducted for non-users. (A similar approach has been taken more recently for the proposed country grant in Jamaica).
For that purpose, the team designed a study that involved qualitative and subjective aspects of using ICTs, such as social values, fears about use, characteristics of users and library staff and existing cultural relationships. At the same time, and from the quantitative perspective, the team sought to distinguish the population of users and non-users of public libraries by targeting variables associated with internet access at home, skills and use of ICTs (differentiating by gender, age, ethnicity and rural or urban location).
The team derived the assessment variables for this study from a theoretical base that addresses concepts such as cultural assets (Bourdieu, 1997), social assets (Putnam, 1993; Bourdieu, 2000), access levels, levels of use, and ICT Skills. This study design enabled the team to define areas where the BiblioRedes Program impact was more significant, resulting in reduction of the digital gap through providing internet access and training in the use of ICTs (see also, Bardin, 2002).
From this perspective, the first study (2002-2005) showed the importance of internet access in the public library. Some of the indicators with greatest impact are shown in Table II.
These results showed the cultural asset increase of public library users, through the skills index, rising 2.3 points in three years; while non-users of public libraries with similar socio-economic and cultural characteristics grew only 0.6. The baseline indicated that the biggest gaps were in the female population and in rural areas; three years later the results for these groups showed the highest increase in the whole country. A similar result was obtained from the data associated with internet access, where the use of ICTs in public libraries was most intense in communities where there had been low penetration.
However, by 2009 the internet access situation had changed as a result of a mass increase in computers at home, use of smartphones and greater connectivity in the country. The problem became more complex and digital inclusion increased in importance compared with the digital divide, and with this the relationship between internet applications and simple access to the internet.
In response to this added complexity, the team introduced evaluation measures associated with the suitability and use of ICTs, by adding new skills in the index, such as the use of social networks and the use of e-government. The new variables that resulted were: use of social networks was added to the social assets; and web promotion of economic activities was added to the creation of economic assets. These demonstrated that the program was very significant in rural and, especially, isolated areas, and also that public libraries were the only place with free internet access.
To conclude, creating a good baseline allowed the team to have comparable data from 2002 to date, obtained through user studies and statistics. The effects of the BiblioRedes Program have been observed over time by monitoring the processes and characteristics of digital inclusion, because the theoretical construction and part of the application of variables remains consistent.
The use of data of national interest, such as accessing and using ICTs, were shared with decision makers through press coverage of key data, publishing reports and targeting services to places where the baseline showed the biggest gaps. Thanks to this work, the program has been fully funded by the State of Chile since 2005.
3. Mature and completed grants – the programs
The focus and approach to program delivery, as well as the location of the project teams of the next wave of grants varied between countries. Most projects formed partnerships with the relevant government ministries (responsible for public library and ICT development, or more broadly for culture or community enhancement) and the national library or public library system (some teams were embedded in one of these).
The Glob@l Libraries-Bulgaria Program (GLB) was implemented between May 2009 and February 2014. The main results of the implementation were:
An IT infrastructure covering 960 public libraries in 911 settlements, in 71 percent of which the population is below or up to 2,500 inhabitants.
Enhancement of human capital, both of libraries and local communities. GLB provided training for 3,789 librarians who in turn trained 43,989 users in groups and delivered more than 217,000 individual consultations.
A major issue from the beginning of the program was how to maintain and broaden the program impact after the grant. The answer was to establish the Global Libraries Bulgaria Foundation in August 2013. The Managing Board includes representatives of relevant ministries which had been involved in the GL program, as well as the Library and Information Association, the National Assembly of Municipalities, the “13 Centuries Bulgaria” Fund, and the association of national chitalishte (cultural communes). The new entity is responsible for continuing the development initiated through the GL program. One objective is to maintain the existing situation with the program target libraries as well as to include more libraries and different activities. A strategic goal is to make public libraries focal points of e-government in accordance with the development of this strategy at a political level.
In Botswana, the Sesigo Project was a collaboration between the relevant ministry and the Foundation, and was project managed by the African Comprehensive HIV/AIDS Partnerships. The project’s implementation partner was the National Library Service, which manages the public library system. The project’s vision of success was to see Botswana transformed into an information society with effective modern public libraries that offer free access to computers and internet for all citizens. This required a strong public library system and improved access to information that would contribute toward attainment of the country’s socio-economic goals. The project was due to run for four years from July 2009 but was extended until March 2014. The beneficiaries for the project were all members of the community, including children and youth, rural communities, the unemployed, the elderly, small population groups and any other disadvantaged members of the community.
By contrast, the Polish Library Development Program was oriented toward libraries in rural areas and small towns. The program started in 2008 and eventually worked with 3,808 libraries, aiming at making them more dynamic and vivid, helping them introduce new services and acquire new users and partners and, in doing so, becoming a hub for social activities in their communities. Among other things they delivered training and equipment, organized conferences and workshops, ran a library portal and set up several formal and informal library networks. The LDP completed its work in March 2015, but the Information Society Development Foundation (an institution implementing the Program) will continue to provide support.
Biblionet Romania started in March 2009 as a partnership between IREX (an NGO based in the USA which also managed the Ukraine program), the relevant ministries, Educating for an Open Society Foundation, the National Association of Public Libraries and Librarians, Intuitext and Romanian public libraries. The aim was to build a modern public library system by: equipping 2,280 public libraries with technology, so that citizens have better access to information; 3,250 training librarians in ITC and library management, to provide better services to library visitors; establishing 41 new local training centers and five regional training centers for librarians across the country; and helping to develop the capacity of the National Association of Public Libraries and Librarians to advocate on behalf of libraries and librarians. The program ended late in 2014.
Since 2008 the Bibliomist Ukraine program has worked to meet the informational needs of citizens, helping libraries better serve their communities through a countrywide system of public internet access in libraries. In total, 1,930 of the 18,001 public libraries were equipped with technology, or 11 percent of the total and 24 regional training centers were equipped with 240 computers. As of March 2014, 18 of the training centers and the National Training Center had also received smartboards and training in their use. IREX installed creation spaces (or “makerspaces”) at five regional-level libraries to give patrons the opportunity to create videos, recordings, music, art, and other materials. In addition to providing equipment, the Bibliomist program cultivated hundreds of new partnerships between libraries, NGOs and other private actors to develop new library services to meet the information needs of citizens. Examples of these services include e-government services, financial literacy, immigrant assistance, and serving internally displaced persons (IDPs).
The Viet Nam Project has provided computers, access and training in 40 provinces (out of 63 provinces and cities) over the past four years. The project targets communities which are in difficult conditions, in remote and rural areas, via 40 provincial libraries, 378 district libraries and 1,572 cultural post offices (CPOs), which are local official services offering basic post office and information services at commune level. The project has been delivered in three phases, involving 12 provinces in step one (from 2011), 16 provinces in step two since 2012, and the other 12 in step three from 2013. The project has reached millions of users through 13,375 internet-connected workstations in 1,900 access points in CPOs and libraries, enhanced and broadened public library services provided by 1,887 trained staff and through more than 4,000 advocacy campaigns and events. The Project is implemented by the Ministry of Information and Communications, in collaboration with the Ministry of Culture, Sports and Tourism and the People’s Committees of 40 provinces. The project aims at innovation through a new vision that focusses on impacting the poor, disadvantaged groups of people and those living in extremely difficulty, by providing opportunities to have equal, effective and sustainable access to information technology and socio-economic benefits from such access.
4. Mature and completed grants – impact assessment
Since impact assessment was not a core feature of the second cluster of country grants at the outset, it is not surprising that there is variation in how the country teams (if not the impact specialists) saw the overall significance of this work. Other factors also came into play: for example, where country teams were embedded within the national public library structure (e.g. Latvia and Lithuania) there were usually fewer problems in sustaining the work (and its assessment) after the expiry of the country grant. In other countries (such as Bulgaria, Romania and Ukraine) sustainability of the performance measurement and assessment work became a significant preoccupation. Some of this variation is reflected in these reports.
Library statistics in Bulgaria are collected annually by the Ministry of Culture; the National Statistical Institute also collects information but not annually. The debate on library statistics standards was renewed in 2009 after the adoption of the most recent law on public libraries. Since these standards do not include impact indicators required by the GL program, quarterly reports were introduced for the target libraries. The new Global Libraries Bulgaria Foundation will continue to collect information from the target libraries via online reports. The aim is to adopt the IPA-Lite modification of the GL evaluation approach in order to ease librarians into impact data collection and data usage for advocacy purposes. Data will be very important as evidence when the new entity applies for funds. At the same time, the Bulgarian Library and Information Association would like to use the data collected by the libraries in improving library standards, thus contributing to impact assessment at a national level.
To continue the data collection from target libraries, for the period February 2014-February 2015, target libraries were asked to report data three times – a quarterly report in spring 2014 just after the end of the grant; a survey among target libraries under the “Sustainable Services Delivered Through the Library Network” project, conducted in September-October 2014 among the target libraries; and an annual report. The information gathered through the survey was used to inform the strategic plan of the new entity. Data gathered via the annual reports will be used in discussions on library standards updating. Finally, but not less important, all data gathered on the impact of the GLB program (see Devetakova, 2012) will be used for advocacy purposes by the new team.
Like the other grantees, the Sesigo Project in Botswana conducted impact assessment to track progress and gauge the benefits that the project brought to the country and its citizens. The impact assessment was intended to show whether the project was being conducted effectively, as well as whether and how it was making a positive impact on people’s lives, including groups, organizations, and communities. The assessment was based on a framework developed in January 2009 as part of the proposal for the overall project. It comprised a set of outcomes and impacts to be measured, with corresponding indicators. Based on this framework, four annual studies were conducted during the implementation of the project: the baseline (2009), annual impact studies in 2010 and 2011 and the final impact study in 2012. The final impact assessment built on and improved the work of the previous studies and incorporated salient aspects of the studies to provide information expected to be used by stakeholders for learning, improvement, advocacy, awareness-raising, and sustainability.
Since the report in the earlier special issue of this journal (Resego, 2012), the final impact assessment study was completed in the first quarter of 2013 culminating in a stakeholder workshop to interrogate the findings and results and map the way forward for the project beyond June 2013 when the project was due to end. The results were widely disseminated in different ways such as newspaper inserts for the summarized results, and radio drama and video on the findings to reach a larger audience.
In the Polish case, planning for impact was based on extensive research: an in-depth needs assessment that was carried out during the planning phase of the program in 2008. Based on their findings, the team was able to set out aims, design activities, prepare an impact assessment framework and select reference points for some key indicators. During the program’s implementation they carried out ongoing evaluation of the activities and then an ex post evaluation after each of the three rounds of the program (a new batch of libraries was added each time). This allowed not only for constant refining and improvement of team activities, but also gave initial input of data for assessing impact (for an earlier report on IPA work in this program, see Kochanowicz, 2012).
However, the key activity was the comprehensive impact assessment, carried out first in 2012 (Why Poles need libraries?), and then again in 2014, within the framework of the Common Impact Measurement System (CIMS) developed by GL for country-level impact assessment, as described in Paley et al., 2015.
Apart from the studies already mentioned, the team have also carried out other types of research, the more interesting of them recently being:
A comparison of libraries participating in the program with those not participating using external data about library activities and performance gathered by the Central Statistical Office. The Polish CSO has been collecting basic data on library performance for many decades. As participation in the program was competition-based (some eligible libraries were included; others not) the team was able to compare those two groups (non-participating ones serving as a kind of a control group) over a longer period of time. They have seen, for example, that libraries participating in the program attract non book-lending users much more readily than non-participant libraries, and that participating libraries are able to secure a larger (and growing) share of municipal budgets.
A special online tool to allow libraries to combine country impact data with presentation of their own achievements. Almost 1,000 libraries decided to do this and their input provided valuable study material, which was analyzed. As expected, librarians were most proud of offering access to culture (books, but also exhibitions or meetings – altogether 62 percent) and saw this as their biggest contribution to the well-being of their communities, but this was followed by giving access to educational opportunities (42 percent; where, for example, teaching foreign languages figured prominently) and teaching specifically ICT (mostly to senior citizens – 24 percent). After that came the library as a meeting place (reflecting one of the program’s aims – 23 percent), developing hobbies and private interests (21 percent) and offering a sense of belonging – being a part of the community (16 percent).
From the beginning of the Biblionet Romania program, the team aimed to create an easily maintained and cost-efficient impact assessment system which would be useful to librarians after the program end. They tried to incorporate librarians’ opinions into the tools and processes that were developed: the team hoped to contribute to program sustainability by producing something technically relevant to librarians. The team preferred practicality over a sophisticated academic approach, the whole purpose being to make it easy for all the stakeholders involved to make connections between real life evidence, program achievements and the data presented and reported. To do this the team tried to make data as friendly as possible and used tailored data in communication with stakeholders.
One of the critical points was to improve the image of libraries held by public administrators. Until Biblionet, most of the communication with public administrators was done using traditional library statistics. Very few librarians succeeded in breaking out of this vicious circle: “public administration shows little interest in libraries – librarians are mostly presenting statistics which might not be relevant to public administration.” The team planned to do something about this.
They collected data through various channels (for an interim report on IPA work in Romania see Chiranov, 2012a):
librarians’ reports (monthly and quarterly);
quantitative studies and one qualitative study using a professional research company;
an online pop-up survey system to “interview” library users (Chiranov, 2011); and
activity reports from the program team.
The tools were discussed with the librarians in the design phase, tested together, adjusted based on experience and then implemented. Occasionally the team initiated ad hoc surveys on SurveyMonkey.com with the librarians to collect their opinions on various topics.
In May 2010 the team initiated a voluntary group of 13 librarians, which they called the Impact Group, to work closer with the Biblionet Impact Assessment Manager. They met four to five times a year to work on various topics. In September 2011 the librarians decided to have these workshops only at weekends so that they would not be away from their daily tasks too long. This continued until December 2014 and the team consider this to be great evidence of their commitment. The Impact Group plans to continue its work independently.
As another part of the effort to ensure sustainability of the evaluation work, ten librarians (half of them members of the Impact Group) volunteered to created an evaluation and statistics in public libraries group, which took over the most important impact assessment data collection tools at the end of the program.
Impact data usage is as important as impact data collection. For the past four years the team has sent out annual tailored letters to about 1,400 local councilors in all counties to inform them about the positive impact that libraries are having within their counties, in the hope that they will develop their understanding of what libraries are doing. The team also prepared one page infographics presenting the most relevant data obtained. Periodically, they sent reports and information to library managers and county coordinators, containing processed program data and also the raw data for their reference (see Chiranov, 2012b).
The Bibliomist Ukraine impact planning and assessment activities were fairly exhaustive:
monthly online reports from librarians;
training center reports;
the National Citizen Surveys conducted in 2009, 2012, 2014;
community analyses in 2010, 2012, 2014;
ongoing pop-up surveys;
CIMS surveys in the Autumn of 2013 and 2014;
qualitative librarian surveys in 2009, 2012, and 2014;
postal surveys of librarians in 2010, 2012, 2014; and
Ukrainian Library Association organizational capacity assessments in 2008, 2010, 2012, and 2014.
In 2014, Bibliomist focussed on building the capacity of librarians to calculate return on investment (ROI) and to use this information for advocacy purposes. 20 librarians from across Ukraine convened to learn how to estimate the social and economic impact of their work and how to show the value in monetary terms. Trainers from a partner NGO specializing in political and social analytics explained the basics of the costs-and-benefits approach to library services and presented the ROI calculation methodology. The participants discussed ways to find alternate value for common library services and practised presenting the numbers to different audiences. As a follow-up to the training, the participants are calculating the monetary value and ROI for the five most popular services their libraries provide (for more information about the Ukraine approach to ROI, see Streatfield et al., 2015).
Bibliomist’s recent program to train librarians to provide services to IDPs has had significant results already. Since October 2014, Bibliomist has trained 8,095 librarians in 1,624 libraries to provide services to IDPs in the community. This has attracted 5,816 new IDP patrons who have had 30,453 consultations with librarians to get help in registering for elections, looking for local housing, recovering important documents, and searching for jobs.
For each step of the Viet Nam program, a baseline survey was conducted in various localities before project intervention to inform the design of the program. Staff of public libraries and CPOs were trained in monitoring and evaluation and impact assessment, to try to ensure that they understand its significance, as well as how and when to implement monitoring and evaluation tools.
After implementation of each step, mid-term and periodic assessments were carried out by external evaluation firms, to collect performance and impact metrics and to analyses the project implementation progress and impact against the original work plan. At the end of the project, a comprehensive impact assessment for all 40 provinces will be conducted to measure the impacts of public access computing at different stages for lesson sharing and to help in seeking sustainability.
The team has developed and applied an observatory system, involving a server linked with software installed in project computers, to monitor, evaluate and summarize use at project sites (including computer/internet use time, monitoring the applications and software used, and the history of internet access such as web sites and entertainment programs). The interface is designed for project staff to collect and enter data via a web interface gateway.
The team regularly updates, adapts and improves the monitoring tools, oversees the technical planning, implementation and delivery of periodic assessments conducted by subcontracted external evaluators, and shares findings, lessons learnt and recommendations among stakeholders and partners for project enhancement.
5. Mature and completed grants – recent results and conclusions
Public access to computers and internet was provided in Botswana through 398 computers (the target was 412) installed at 72 public libraries (target 78). Daily and weekly usage of computers increased from 19-16 percent in 2009 to 28-56 percent in 2012, respectively.
Library users continued to place a high value on the information and services they received from libraries with the most appreciated information being on health (84 percent), business (83 percent), science and technology (83 percent), education (80 percent), agriculture (73 percent), languages (71 percent), culture (67 percent), e-governance (62 percent) and politics (56 percent). There was a significant rise in library user satisfaction levels, from 62 percent in 2009 to 81 percent in 2012.
Library staff ICT skills were enhanced, with 54 percent of them reporting they were proficient in using a computer in 2012 compared to 30 percent in 2009. More specifically, 79 percent of librarians reported that they were generally computer literate and 79 felt that they were proficient in helping the public to use the internet. Other areas of proficiency reported included social networks (e.g. Facebook), e-mail, software use and sharing programs.
Some progress was made in building partnerships to strengthen the library system. At local level, libraries reported having engaged and partnered with the Kgotla (community-based organizations), parent teachers associations and village development committees in reaching out to communities.
In total, 96 percent of library users viewed libraries as contributing to the development of communities and improvement of lives of users.
On a larger scale, the Polish team sees the change exercised by the program at two levels:
Change within the libraries as a direct effect of the program’s activities. Libraries have tremendously extended the scope of activities, especially non-traditional activities (that are not book/reading oriented). The number of participants in various courses, workshops and events has increased many times (the subjects covered include ICT, language courses, art classes, activities for pre-school youth, sport and healthy lifestyle or exhibitions). At the same time libraries are operating in a more dynamic and professional way – they cooperate more with partners (both other libraries and from different sectors); offer more training opportunities to their employees; extend opening hours (especially during weekends) and acquire more financing (both from municipalities and from other sources, such as various grants or donations).
Users benefiting from changes within libraries as an impact of the program. This can be classed into several areas, for example ICT (more than 400,000 people made their first click in the libraries; more than 200,000 have access to the internet only in the library; 1.15 million have acquired new skills); communicating with others (using online communication such as Skype – 800,000; more frequent contact with friends through the library services – 950,000); the job market (looking for jobs – 450,000; getting one – 90,000); education (doing homework – 750,000; improving educational achievements – 1.5 million); or health (looking for health information – 650,000; having their health actually improved – 500,000).
Biblionet Romania focussed its recent evaluation research on the letters sent to county councilors. Ten counties (25 percent of the total) reported a higher budget in 2014 compared with 2013. Asked what the most important activities were in helping to improve the library relationship with the county council, library managers identified meetings with the county council on a regular basis as the third most important activity and county coordinators cited the letters sent by Biblionet to county council members as the fourth factor:
Of the total 84 percent (managers and coordinators) reported that their county council now involved them in various activities, or partnerships, or are supporting them in various ways.
In total, 81 percent had noticed some, or a significant, improvement in county council attitudes toward public libraries in 2014, compared with 2009, when Biblionet was launched.
In all, 93 percent were satisfied with the impact data they received over time. A county coordinator added, “along with our efforts to promote the public library and its results in the community at local level, the information you provided helped maintain and strengthen the trust of public authorities in our institution” and a library manager said “We’ve used the results of the questionnaire to inform the local authorities and the media about the state of the public library system in the county.”
Totally, 51 percent felt that the county library should collect, process and report impact data in the future (39 percent of the managers and 65 percent of the coordinators); one library manager added “We will continue to apply what we’ve learned during Biblionet program” and another said “Our library has had good practice in collecting and processing data, and in developing statistical bulletins and materials.”
In total, 42 percent considered that the data collection, processing and reporting would be better done by an independent, external body (48 percent of the managers and 35 percent of the coordinators), but “it’s hard to believe that someone else would be able to do it at same professional level as Biblionet did.”
By the end of the program (in December 2014) more than 600,000 people had used computers and the internet for the first time in public libraries.
The Romania team conclude that change can be reached within a reasonable time horizon by using properly planned and implemented activities. Involving librarians in program design and implementation is a key factor in reaching sustainability and in solving library problems.
Communication is kept alive and effective by using the right data, which is well understood by each important stakeholder. In their letters to county councilors the team did not speak about traditional library statistics, they only mentioned library achievements that were contributing to local development goals (e.g. people finding jobs using technology in the library; farmers applying online for agricultural subsidies; students looking for education-related information) (see Cottrill et al., 2015, for more information on the Romanian approach to using impact evidence for advocacy).
For Ukraine, their 2014 CIMS survey (using the GL impact assessment tool) revealed much about the role of libraries and technology use in Bibliomist program libraries:
In total, 86 percent of public library visitors use modern technologies. If people use ICT in public libraries, they visit them more often.
In all, 68 percent of library visitors learned at least one IT skill as a result of using library services. This is most visible in rural areas: 78 percent of village library visitors and 53 percent of elderly people (aged 60 years+) learned at least one skill at the library.
Totally, 27 percent of visitors used the internet for the first time in a public library. For older people, the share of first-time users is notable: 39 percent for 45-59 year-olds; and 36 percent for those aged 60+.
In total, 61 percent of users only have free access to the internet through a public library.
In all, 7 percent of visitors used e-government services; nearly all of these said that it allowed them to save time.
Most users of ICT say that technology helps save money (83 percent), increases their awareness of various events and activities (71 percent), and encourages people to read more (76 percent).
The third wave of the national survey demonstrates shifts in the library sector resulting from Bibliomist’s library modernization efforts:
One out of nine public library visitors (11 percent) use the internet/work with library computers, compared to 5 percent in 2009.
In total, 25 percent of public library visitors think of their public library as a venue to use the internet, a significant increase from 11 percent in 2009. While in 2009 only 18 percent agreed that their public library was a place to use computers, in 2014 that proportion more than doubled to 41 percent.
Since 2009, those who consider the library to be a community center increased from 59-72 percent. Those who think of the library as a place to get help in finding a job increased from 16-27 percent.
Totally, 37 percent of visitors think that librarians could teach them how to use computers and internet, an increase from the 17 percent who thought so in 2009.
Bibliomist’s focus on locally responsive library services resulted in special outreach to marginalized groups to ensure that they benefit from equal access to ICTs. For example, a library in Rivne offered computer literacy training for over 1,000 women with disabilities, nearly 40 percent of whom said that they had improved their health thanks to information obtained online. Libraries all over Ukraine offered special assistance to elderly patrons.
In Viet Nam, the team completed a mid-term assessment of the 12 step-one provinces and both the needs assessment and baseline survey for the ten step-three provinces in 2014.
The mid-term assessment reviewed the project implementation, user behavior and information use, as well as the benefits achieved during the 20 months since ICT installation and ten months since library and CPO staff training. The assessment used longitudinal study, user and non-user analysis, focus groups and ethnographic tracing. The consultants interviewed 2,063 project site visitors, including 1,626 computer and internet users, as well as 1,357 non-users. Public library and CPO technical and managerial staff were also interviewed, as well as people responsible for management, policies and funding of libraries and CPOs at various levels.
Headline findings included: more than 80 percent of library and CPO users made use of computers and the internet; and about 13 percent first used computers and the internet at these locations. The most frequent computer and internet users were students and government staff (47 and 26 percent, respectively). Use by other target user groups such as retired people, farmers, hired laborers and unemployed people remains low, accounting for the remaining 27 percent. The main areas of use are culture and leisure (91 percent), community connection and information sharing (63 percent) and education (56 percent). Only 30 percent of users reported interest in health, e-government and governance, and economic development. The habit of using public libraries and CPOs for information access and learning is only beginning to emerge for a limited number of people in remote areas in Viet Nam (observatory statistics show that computers are being used for an average of 99,015 hours per week, equivalent to only just over 20 percent of total possible use time).
The report findings and proposed solutions have helped in team discussion and reflections on shortcomings, notably that the number of people who have used computers and the internet remain low. Team members have been asked to identify priorities as well as to work together to improve the situation. These findings have also helped in charting the sustainability plan, including specific programmatic interventions to increase the number of users and learn more about the needs of various communities.
The step three needs assessment and baseline survey again focussed on delivery requirements and the information needs at the localities. A survey to collect performance indicators will be conducted in mid-2015 to ensure that the implementation is on track. Also, between April and December 2015, a mid-term assessment of step two provinces and another assessment of step one provinces will be conducted simultaneously. The results of these upcoming assessments will feed into the Global Libraries Data Atlas (see Paper 2) and will help in building toward the development of public libraries as dynamic community learning hubs and helping to realize the government’s new rural development program until 2020.
6. Pilot and new grants – the programs
The next group of reports covers countries where pilot projects are being conducted in preparation for a country grant proposal to the Foundation (or in the case of Colombia, the country grant has recently been secured). The exception is Lithuania, which was one of the second group of countries and has completed its country grant. However, the program team has since secured a new grant which operates in a different way and presents a series of fresh impact evaluation challenges.
The Colombia pilot project was funded by the Ministry of Culture and the Foundation, and was implemented from 2012 to 2014. This involved 26 libraries in various regions of the country and began with an assessment of the infrastructure of the libraries and of the information needs of their users and non-users.
Key actions during the pilot were to:
Conduct library advocacy to improve the librarians’ perception of their role, and the public library’s role, creation of alliances and securing of resources.
Train library staff in public library advocacy in the municipality, based on creative and innovative use and application of ICT in the delivery of services to the community.
Provide complementary technologies (e.g. reporter screen, projector, tablets, multifunction printer, sound system and headphones) to encourage the use of computers and existing connectivity, and expand the provision of library services.
Monitor and measure changes in user profiles using a web application which tracks uses and users of public access computers in libraries.
Implement innovative and creative services with ICT (e.g. autonomous learning of English, use of digital and physical resources for health and autonomous digital literacy) which show how libraries contribute to the development of the municipality. After the training in each library, ICT services were developed, promoted from the library or implemented within the framework of national projects.
The country grant for project implementation runs from May 2014 through March 2018. The implementation will enhance the technology of 1,200 public libraries and train 1,800 librarians, to ensure that the public libraries are recognized as spaces that offer services, using technology creatively to meet the needs of people, contributing to their human, social and economic development.
Since the main objective of the project is to achieve recognition and support to public libraries by the state, the private sector and civil society, the impact expected to be generated through the project is:
increase in the number of library users and the number of users of technology in public libraries;
train more people in the use of ICT and information; and
identify the benefits people obtain through the use of ICT resources in public libraries.
The PerpuSeru pilot program in Indonesia started in 2011. Their current grant with the Foundation will end in July 2015 and they are hoping to implement the scale-up phase through to 2025 (although GL funding will cease by December 2018). During the “supplementary phase” where the program is preparing the “bridge” between current activities and the scale-up phase, they are trying to replicate current good practices in district level libraries at village level libraries. In total, 19 district libraries were selected out of the existing 34: in these, three to four villages were selected from each district. Most of the training and mentoring for village library staff was conducted by district library staff and local facilitators (known as “core facilitators”). Each library was provided with three computers by the program.
In South Africa, Mzansi Libraries OnLine is a two-year pilot project, ending in December 2015. It is hosted by the National Library under the auspices of the relevant government department. The over-arching purpose of the project is to strengthen and enhance public libraries in South Africa, empowering them to fulfill their role as vibrant community hubs contributing proactively toward improving the lives of their library users.
In total, 27 public libraries are involved in the pilot. A purposive stratified sampling technique was used to select three from each of the nine provinces. The libraries were chosen by the partners with the Library and Information Services Association and represents a wide range of library provision types across urban, semi-urban and rural settings.
The Librar-e Turkey Planning and Pilot Project (LT-PP aims to contribute to the improvement of the life quality of Turkish society in general, and the disadvantaged and disempowered in particular. They will do this by transforming all the public libraries in the country into centers of public attraction, where people can have free access to the hardware, software, and customized training required to optimally benefit from ICT. LT-PP is a collaborative effort by Hacettepe Technopolis-Technology Transfer Center, the relevant directorate of the ministry and the Foundation. Within the scope of the 28-month project, the aim is to achieve significant improvements in ICT infrastructure and computer-internet literacy of both library staff and users of 78 public libraries (which vary in size and are in 26 provinces that statistically represent Turkey). The Project consists of different work groups that have particular responsibilities – advocacy, finance, ICT, training and impact.
Since the earlier report on the Lithuania project in this journal (Lipeikaite and Sadunisvili, 2012) the National Library has launched their new project Libraries for Innovation 2 (December 2013-December 2016), again funded by the Foundation with the relevant Ministry. Successful implementation of the previous project Libraries for Innovation (2008-2011) helped to overcome disparities in the urban and rural informational communication infrastructure. The earlier project strengthened the position of public libraries as modern, contemporary, attractive public places. The goal of the new project is to enhance the capacities of public libraries to meet the developing needs of communities and consolidate the libraries as sustainable community institutions able to improve people’s life quality. The project covers 60 central libraries and their branches; the Library for the Blind and five its branches; and five county libraries. In all there are 1,292 libraries in Lithuania. As a key part of this project, calls for proposals were sent to all public libraries early in 2015, for projects that are important for communities and that will further involve local residents in library activities. The selected projects will receive some financial aid for project implementation.
7. Pilot and new grants – impact assessment
Apart from a common concern with needs assessment, the new grantees signal different impact assessment and performance measurement preoccupations, which reflect both their hugely varied work contexts and the stage in the process that they have reached.
The Colombia team has developed a plan for monitoring, evaluation and impact measurement that will allow them to identify, not only the results obtained in the beneficiary communities, but also the impact in improvement of the quality of life of users of public libraries through the implementation of ICT. This plan was created through a Logical Framework exercise in which the objectives and activities to be undertaken during the planning phase were identified. The expected changes in strategic stakeholders were identified through outcome mapping, which identified direct project partners, as well as the changes needed for each. Various stakeholders were involved in this planning, including the National Library and other entities related to the library sector.
The resulting plan identified four main stakeholders (librarian, community, library and local authorities) and for each of them the expected changes, the aspects to measure and their impact indicators were identified.
To implement the plan, various actions were carried out:
initial needs assessment of the pilot program libraries;
a baseline survey to collect data before and during the initial implementation of the project, using existing performance measurement tools, and surveys of librarians and users; and
continuous monitoring and constant measurement during the implementation of the project using the tracking system, surveys and interviews with those involved in the process.
In parallel, a national needs assessment of public libraries, and information needs and perceptions assessment of users and non-users of public libraries were performed. This information served as a basis for defining key aspects of the proposed national implementation.
For this phase, a logic model of the project was made to align project actions with the expected impacts and relate these to the GL performance indicators and common impact measurement system. As well as this results and impact indicators framework, the team plans to develop this evaluation model using the contingent valuation and double difference methods.
The PerpuSeru team in Indonesia is working with the district library staff to count the number of library visits (Since the National Library does not require libraries to use their software, each library partner has its own software, although some use that of the National Library).
The team established an online reporting system to ensure that target groups are being reached through activities in the library. This system tracks the community engagement activities conducted by the libraries, as well as their advocacy activities to see if these have results. Library staff report their activities into the system. Each library has a dedicated team member and each report is reviewed by relevant library managers and project team members. Where needed, reports are returned to library staff for clarification.
CIMS surveys were conducted in 2014 and in 2015; the latest survey data are now being analyzed.
For the South Africa project, two factors significantly affected the design of the impact planning and assessment framework. First, the project started several months later than planned, because of challenges in the project inception phase, including staff recruitment and selection. The Impact Specialist was finally appointed in October 2014. As a result, several project activities – while contributing significantly to stakeholder engagement and advocacy – were conducted without the benefit of an over-arching and co-ordinated IPA framework. Innovative ways had to be found, therefore, to fast-track the IPA process, while adhering to core GL guidelines and requirements. Second, the implementation of the pilot project is running in parallel with the preparation of the country grant proposal, as a consequence of the winding down of GL. Despite these challenges, the condensed timeframe prompted a great deal of creative thinking – and ultimately an opportunity to trial an adapted IPA model, appropriate to South Africa’s unique circumstances.
The IPA planning process confirmed the need for a national needs assessment survey, as well as a CIMS survey of library visitors, in order to provide robust data for the needs assessment component of the IPA; and a baseline for a planned mid-term and final impact survey during the country grant phase. Additional funding was secured from GL, who appointed the TNS survey company to undertake the work. Five South Africa-specific, closed questions were developed for a general population “omnibus” survey. In total, 15 questions were added to the CIMS survey, including questions in the core focus areas of education; economic development and health. In addition, some of the wording in the demographics section of the surveys was adapted to suit the country context.
The fieldwork for the CIMS survey is scheduled for completion by the end of April 2015. Library visitors are being interviewed from the 27 pilot sample libraries as well as an additional sample of 27 comparison libraries. The latter provides a potential control group for a planned mid-term and final impact CIMS survey during the country grant. The final reports are expected in June 2015 and will include information on the target groups that have been identified – the youth; women; the unemployed; small medium and micro enterprises/ entrepreneurs/ co-operatives; the elderly and people living with disabilities (particularly the visually impaired).
Between February and March 2015, nine provincial focus groups were facilitated by project team members. Each pilot library nominated three candidates; in some cases, a representative from provincial library services also participated. During these sessions, the team collected rich qualitative data for the baseline and needs assessment about public libraries (areas of strength; challenges; how library staff envisage online ICT access benefiting their users; changes that would need to occur in libraries to make them true “community hubs”; changes librarians might need to make; possible barriers to change; how these could be overcome and finally, strategies for ensuring sustainability into the future). The focus groups provided the team with an opportunity to verify data on the training and development needs of library staff, gathered through online training needs assessment surveys.
In Turkey, before the implementation part of the LT-PP, some needs assessment studies were conducted as Stage 1, starting with two online surveys: a library data collection survey and a library staff data collection survey, to investigate the current situation of the pilot libraries. The aim was to identify whether the general ICT infrastructure, personnel and space conditions of these libraries were convenient for the targeted training and advocacy activities. In this needs assessment part of the project, two CIMS-based paper-pen questionnaires were also applied in the pilot libraries and locations (a public library usage survey and a public library potential usage survey). Although these questionnaires were based on CIMS, some questions about users’ ICT levels and needs were added and the logical flow also reorganized according for the local audiences. The results of these questionnaires were reported to the Turkish community in two academic articles (Al and Soydal, 2014a, b).
For the needs assessment stage, the team created an “Online Data Visualization Platform” using the StatPlanet open source tool to merge the distributed data about public libraries in Turkey. This platform helped them to synthesize different types of data which were obtained from various sources such as the government and the project’s first stage needs assessment surveys. Through this platform, it became possible to present different aspects of the current situation of public libraries (such as the number of users, number of collections and the distribution of these parameters according to districts). Some infographics were also created by using the data from the CIMS questionnaires to present in the meetings with decision makers and potential partners. All the needs assessment studies conducted by the project’s impact assessment work group were combined and presented in an online timeline which was created by using the Knightlab free online tool. This first stage of the project helped the project team to see the needs of libraries and users and helped individual work groups to create and implement their roadmaps.
In Lithuania, there are three major challenges already in preparing the project IPA framework:
how to secure an appropriate balance between quantitative and qualitative evaluation methods;
achieving a balance between evaluation evidence being collected by library staff, the project team and local consultants; and
not being able to complete the impact planning and assessment implementation plan until the grantees have been selected, because it is likely that the various libraries involved will focus on different target groups and have different intervention strategies.
The three main project objectives are to:
empower public libraries to create new services based on their community needs;
strengthen librarians’ capacities through trainings and partnership between libraries and other institutions; and
secure libraries as sustainable institutions equipped to meet their community needs, react properly to them and find good partners.
The main evaluation focus is on changes in librarians’ perceptions, behavior and motivation. However, the team also plans to extend impact evaluation to other target groups (library partners, local policy makers and library visitors), and to involve librarians as much as possible in this process to help them learn.
After the grantees are selected, the team plans to deliver impact assessment training for librarians to help them to implement their own impact assessment. They will be offered help in devising appropriate assessment tools, as well as guidelines for focus groups, interviews and story collection. They will be given sets of performance metrics to collect (modified according the individual projects), and help will be given to librarians in planning when and what kind of data to collect based on the target groups for their projects. The team expects that, with their help, librarians will collect data from their partners, local policy makers and their visitors.
The team has surveyed public librarians to explore their response to the project web site and social network. They intend to follow this up with another survey covering partnership, advocacy, alternative funding, and the effectiveness of the training provided by the project. Another survey will be administered near the end of the project to see how the situation changes over time.
Two other challenges in the IPA work are to:
teach librarians to do the IPA of their projects, because librarians do not have very strong methodological skills; and
logistical challenges: the team works with librarians from all Lithuania, so it is difficult to get them into one space at the same time and to organize focus groups or other discussions.
8. Pilot and new grants – recent results and conclusions
In Colombia, once the pilot program of the 26 libraries was finished and the results and what was learnt from each of the project components were consolidated, various relevant actions for the development of the national initiative were identified.
Two infographics were created from the results of the national assessments, to share with different strategic stakeholders. These help to visualize the physical and technological infrastructure conditions, the library services and library staff of the national public library network; and the information needs, perception and use of technology in libraries by users and non-users. A highly qualified organization was hired to develop a detailed analysis of the data obtained, allowing comparisons with other Latin American countries.
The activities undertaken in the impact assessment part of the project helped to identify the physical and technological infrastructure conditions of public libraries. Identifying information needs of library users and non-users provided motivation for other stakeholders to be involved in ongoing development work in public libraries, to build new services in libraries that meet the main needs identified by users and non-users.
The evaluation conducted so far has demonstrated that providing ICT in public libraries increases the amount of visits, and that users get to know and use more technology. This has allowed them to identify some benefits to improve their quality of life, such as reading more by using available technology in the library, and, for the students, improving their academic performance using these devices.
In Indonesia, some key results from the 2014 CIMS survey are:
In total, 70 percent of respondents said that they visited public library at least once a week.
On perception of library services, 79 percent identifed free internet services provided by the library as extremely important and 71 percent mentioned having free computers that they could use. In total, 55 percent saw the library as a place where they could learn about (or receiving training on) computers and the internet.
Totally, 80 percent reported that they use their own laptop/smartphone when accessing the internet at the library – 52 percent said this was because they did not want to wait for their turn to use library computers, 38 percent mentioned that there is a time limit for using library computer and 68 percent had to work on their assignment at the library and preferred to use their laptop.
The team is currently analyzing the 2015 CIMS survey results: data snapshots show that library users tend to mention that their quality of life has improved through internet access.
Changes that happen in the community also give useful information for library staff advocacy to gain more budget or support from government (e.g. one library partner in Jepara, Central Java used the evidence of recorded library visitors to advocate for a bigger allocated annual budget for the library – adding more books, changing the layout to make it more comfortable for library visitors, as well as strengthening the internet bandwidth).
The team are currently doing endline evalation to mark the end of current program (before moving into the scale-up phase). Some changes have been happening across the program:
some IT trainees mentioned they are now receiving higher salaries as a result of attending training provided by library;
a library security officer was inspired to improve his chicken business as a result of receiving training by the library and spent time reading books in library; and
training for local small entrepreneurs gave them new access/network to market their products (whether their earnings improved has yet to be seen).
One important lesson for the potential scale-up is about data and its consistency. Since the National Library does not firmly stipulate data collection requirements any library can have its own method for collecting data (or none). There is a rising need for data to support advocacy, but most library staff do not know how to do this. This is something that the program will be focussing on to build staff capacity during the scale-up phase.
In South Africa, at the time of writing (March 2015), it would be premature to comment on impact, as the two core components of the intervention are not yet complete. The delivery of ICT equipment to all pilot libraries is scheduled for March-April 2015 and three training and development academies for library staff, support staff and “training the trainers” will be completed during June-August 2015. Nonetheless, “green shoots” are visible. The work accomplished so far has contributed toward a heightened awareness and support of the project among all stakeholders, as well as a growing excitement about how library services will be strengthened and enhanced. A librarian during one of the focus groups spoke of how the project will finally bring to life his dream of empowering library users in his community. He plans to ask them to: “help me […] to help you […] to help yourself.”
In Turkey, the needs assessment results show that public library users in rural areas are in a more disadvantaged position in terms of technology ownership, compared to urban areas. Moreover, there is an inverse relationship between the ages and library visit frequencies of users. The most common aim of public library users is to borrow or read books, but many express willingness to attend the free computer literacy training programs which will be given in public libraries. The results from the potential public library users showed that people living in rural areas tend to benefit less from public libraries than those in urban areas. Library use patterns differ depending on such variables as education, employment status, age, income level, and place of residence by geographic region. Potential users also demanded some free training on ICT or online courses of interest at public libraries.
In the second stage of the project (the implementation of training and advocacy activities), an additional log analysis program created by the ICT work group has been used for computer and internet monitoring using Data Giraffe software. Data Giraffe also provided a way to ask some questions to the users of the computers in the pilot libraries. The second stage of the CIMS survey results are still being analyzed, which will help the team to see if there are any changes in the perception of users toward libraries and library services. The results will enable the team to prepare an impact assessment report for the pilot implementation which will be used for the country program proposal. By the end of the project it is expected that registered users of libraries will have increased, especially in libraries in which some serious training and advocacy activities have been conducted.
The Lithuania team has already conducted two structured focus groups (with library managers and project coordinators) and have used the results to create a modified Delphi forecasting instrument. This approach combines elements of qualitative and quantitative evaluation but is essentially intended to provide an in-depth qualitative picture of attitude change over time. This instrument asks librarians to respond to a series of statements about public libraries in the future. Responses have been collected from 56 (out of 65) library managers. The team has now prepared a report on this research (More information about modified Delphi Forecasting can be found in Streatfield et al., 2015).
The team is also evaluating other parts of the project, such as:
The effects of initiatives taken by libraries and sponsors as part of a program that involves getting famous Lithuanian people to return to their home libraries and become more involved with them.
A battle game called Biblio involving 136 libraries and 1,200 participants.
The 2014 librarian training program covering topics such as outreach, advocacy, project management, user need analysis and innovation, impact assessment, project initiation and public procurement. These training sessions involved 837 participants (a total of 534 librarians).
The final round of Global Libraries country grants will have commenced by the time this special issue is published. We can reasonably expect that the country teams and especially their impact specialists will continue to learn from the deepening GL experience of performance assessment and that there will be at least one more chapter to write in the GL impact assessment story.
“Mzansi” is derived from the Xhosa word “uMzantsi”, meaning “south.” The name of the project is deeply meaningful for the People of Mzansi – each and every South African – and reflects important nation-building imperatives: the redressing of inequities caused by apartheid; the promotion of social inclusion and citizenship and a healing of the divisions of the past.
David Streatfield can be contacted at: firstname.lastname@example.org
Al, U. and Soydal, İ. (2014a), “Kütüphan-e Türkiye Projesi: halk kütüphanesi kullanım araştırması (Librar-e Turkey Project: public library usage survey)”, Türk Kütüphaneciliği , Vol. 28 No. 3, pp. 288-307.
Al, U. and Soydal, İ. (2014b), “Kütüphan-e Türkiye Projesi: halk kütüphanesi potansiyel kullanım araştırması (Librar-e Turkey Project: public library potential usage survey)”, Türk Kütüphaneciliği , Vol. 28 No. 4, pp. 526-546.
Arnau, J. (1995), Diseños longitudinales aplicados a las ciencias sociales y del comportamiento , Noriega Ediciones.
Bardin, L. (2002), Análisis de contenido , Editorial Akal, España.
Bourdieu, P. (1997), Capital Cultural, Escuela y Espacio Social , Editorial S.XXI, Madrid.
Bourdieu, P. (2000), Poder, Derecho y Clases Sociales , Ed. Desclée, España.
Chiranov, M. (2011), “Applying pop-up survey software to incorporate users’ feedback into public library computing service management”, Performance Measurement and Metrics , Vol. 12 No. 1, pp. 50-65.
Chiranov, M. (2012a), “Impact planning and assessment: making it happen – Romania”, Performance Measurement and Metrics , Vol. 13 No. 1, pp. 24-31.
Chiranov, M. (2012b), “Combining qualitative and quantitative evidence: forming strategic collaborations – and telling stories”, in Streatfield, D.R. , Paberza, K. , Lipeikaite, U. , Chiranov, M. , Devetakova, L. and Sadunisvili, R. (Eds), Developing Impact Planning and Assessment at National Level: Addressing Some Issues Performance Measurement and Metrics , Vol. 13 No. 1, pp. 63-65.
Cottrill, J. , Fernando Letelier, F. , Andrade Blanco, P. , García, H. , Chiranov, M. , Tkachuk, Y. , Liubyva, T. , Crocker, R. , Vanderwerff, M. , Čistovienė, G. , Krauls-Ward, I. , Stratilatovas, E. , Mount, D. , Kurutyte, A. and Triyono , (2015), “From impact to advocacy: working together toward public library sustainability”, Performance Measurement and Metrics , Vol. 16 No. 2.
Devetakova, L. (2012), “Impact planning and assessment: making it happen – Bulgaria”, Performance Measurement and Metrics , Vol. 13 No. 1, pp. 19-23.
Kochanowicz, M. (2012), “Impact planning and assessment: making it happen – Poland”, Performance Measurement and Metrics , Vol. 13 No. 1, pp. 32-37.
Lipeikaite, U. and Sadunisvili, R. (2012), “Impact planning and assessment: making it happen – Lithuania”, Performance Measurement and Metrics , Vol. 13 No. 1, pp. 53-57.
Paley, J. , Cottrill, J. , Errecart, K. , White, A. , Schaden, C. , Schrag, T. , Douglas, R. , Tahmassebi, B. , Crocker, R. and Streatfield, D.R. (2015), “The evolution of global libraries’ performance measurement and impact assessment systems”, Performance Measurement and Metrics , Vol. 16 No. 2.
Putnam, R.D. (1993), “The prosperous community: social capital and public life”, The American Prospect , Vol. 4 No. 3, pp. 35-42.
Resego, T. (2012), “Impact planning and assessment: making it happen – Botswana”, Performance Measurement and Metrics , Vol. 13 No. 1, pp. 38-43.
Streatfield, D.R. , Andrade Blanco, P. , Chiranov, M. , Dryžaite, I. , Kochanowicz, M. , Liubyva, T. and Tkachuk, Y. (2015), “Innovative impact planning and assessment through global libraries”, Performance Measurement and Metrics , Vol. 16 No. 2.
Quick, S. , Prior, G. , Toombs, B. , Taylor, L. and Currenti, R. (2013), “Cross-European survey to measure users’ perceptions of the benefits of ICT in public libraries”, (funded by the Bill & Melinda Gates Foundation), available at: https://digital.lib.washington.edu/researchworks/handle/1773/22718 (accessed 22 September 2015).
© Authors. Published by Emerald Group Publishing Limited. This article is published under the Creative Commons Attribution (CC BY 3.0) licence. Anyone may reproduce, distribute, translate and create derivative works of this article (for both commercial & non-commercial purposes), subject to full attribution to the original publication and authors. The full terms of this licence may be seen at http://creativecommons.org/licences/by/3.0/legalcode
Supported by the Bill & Melinda Gates Foundation.