This paper aims to describe the main impact and performance measures employed by the Global Libraries – Biblionet Romania programme (GL‐Biblionet) and reports on the pilot application of a pop‐up survey designed to provide focused user‐feedback to inform the development of the programme.
GL Biblionet aims to ensure that community informational needs are increasingly met at public libraries, and that these in turn provide better services to users and the community. An efficient performance management system of public library computing service is required for these purposes, including an adequate data/feedback collection system, and a flexible mechanism to analyse and report the findings. A pop‐up survey mechanism was developed and tested as a key part of the user feedback element of impact and performance measurement.
The pilot implementation of the pop‐up survey was broadly successful in providing useful data but the survey results need to be contextualised and triangulated with other impact and performance measurement data to assess their level of reliability. More nuanced questions about users' self‐assessments of competence in using ICT need to be developed.
Although only at the pilot stage, the GL‐Biblionet approach to pop‐up survey development offers a potentially useful tool for long‐term use by public libraries in Romania and for other countries faced with similar public library development and user feedback issues.
This paper reports on a novel and potentially valuable application of pop‐up survey technology for national systems of public libraries and for local or regional application.
Chiranov, M. (2011), "Applying pop‐up survey software to incorporate users' feedback into public library computing service management", Performance Measurement and Metrics, Vol. 12 No. 1, pp. 50-65. https://doi.org/10.1108/14678041111124298Download as .RIS
Emerald Group Publishing Limited
Copyright © 2011, Emerald Group Publishing Limited
The Biblionet Program – Romania is part of the Global Libraries initiative funded by the Bill and Melinda Gates Foundation. The program's overall objective is to improve peoples' lives by facilitating access to information through the internet.
GL‐Biblionet has four major objectives:
Objective 1 – Community members' informational needs are increasingly met at public libraries.
Objective 2 – Librarians are increasingly able to provide better services to the community.
Objective 3 – The Library Association's capacity to provide services to their members is increased.
Objective 4 – Government increasingly supports library development.
The prospect of managing high quantities of data brings exciting challenges and new problems to resolve. It is important to have a reliable system for data collection and processing – one, which is error free, user friendly and cost efficient to maintain.
GL Biblionet program is implemented by IREX, a US‐based non‐profit organization. One of the main goals of the program is sustainability, and this determined us to be strongly focused on cost effective solutions, which can easily can be taken over and used by Romanian librarians after the programme end in 2014.
In an ongoing effort to learn the effects of implementation, GL‐Biblionet collects data through a variety of methods (Figure 1):
Librarian reports – this action has two different purposes:
To understand the librarians' views and help them to have a proactive approach to data collection, in order to be sure that the data is relevant and helpful to understand library patrons' needs.
To strengthen the connection between the county library and local libraries. Library activity is decentralized in Romania and funding comes from county (local) level authorities. In some counties, it is a good relationship between the county library and local libraries; but in other circumstances the communication and cooperation is dangerously close to nothing. When GL‐Biblionet was planned, IREX created an informal position of “County Coordinator” to take care of Biblionet activities at county level. The County Coordinators are expected to collect data from local librarians, as well as to coordinate training, hardware delivery and other activities involving GL‐Biblionet resources. They are the main contact points in their county.
Pop up surveys – IREX Romania designed and tested automated software to collect users' feedback on their experience of using GL‐Biblionet computer in public libraries, to learn about successes and find out improvement opportunities.
Both quantitative, nationally relevant, surveys and qualitative longitudinal studies have been planned over the course of the next four years. The purpose of conducting both types of study is to thoroughly identify the impact of such a wide‐spanning programme and its direct effects on public institutions, public libraries, and library users. IREX will use the information collected to present overarching trends and information that will benefit the entire GL‐Biblionet community as well as to guide IREX management in developing coherent programmatic changes, and furthermore to provide local stakeholders with valuable information that will be used in local and national advocacy efforts, especially in promoting services that have demonstrated a high level of local appreciation.
Main characteristics of the GL‐Biblionet data collection processes
These report formats have been agreed with the County Coordinators and are constantly reviewed for consistency. There are different types of reports:
Monthly reports: containing information on the number of visits, including the number of visits to use Biblionet computers.
Quarterly reports: including data on additional services delivered (e.g. workshops on various topics: labor, health, banking) and success stories collected by librarians.
Yearly reports: data on library materials circulation.
A Pop‐up Survey is a “questionnaire that appears on the screen of a website visitor. The term can also be used to refer to an invitation to participate in a survey. Pop‐up surveys are usually triggered by some mechanism, such as the nth visitor, or a visitor who shows evidence of interest in a topic, or a visitor who goes to a particular section of the web site, etc.” (www.decisionanalyst.com/glossary/pglossary.dai).
The surveys are flexible tools, which allow IREX to ask any questions to Biblionet library users. IREX is interested in learning about library users' demographics, user experience of using computers/internet, what type of information they look for and whether they succeeded in finding it. They also collect information on library patrons who have solved a personal or a professional problem using library computers and ask them to share their experiences with us.
IREX is conducting national surveys in each year of programme implementation.
The general scope of the study is to measure levels of access to information, perceptions of public libraries, and the information and computer literacy of the Romanian population. Study results will demonstrate whether the popularity of libraries, informational search patterns, perception of libraries by respondents, and levels of public participation in the library activities, events and projects change over time. Samples will be chosen to statistically represent the Romanian population. The whole approach (methodology, sampling and questions) will be designed and implemented in such a way as to conserve the comparability of some of data, which is considered relevant to demonstrate programme impact and progress.
IREX is contracting qualitative studies that capture changes related to three major areas of interest: the changing role of libraries and librarians within their communities; the evolution of information needs amongst community members and whether and how these can be satisfied at public libraries; and how the relationship between the librarian and the local council can be improved. These studies should be useful to create tailored case studies, and to find areas for deeper research into the scope for new services or activities, which will result in better community services delivered by public libraries.
This paper presents an overview of the pilot phase of the public libraries users' feedback collection system used by GL‐Biblionet, which was designed to build up an understanding of user satisfaction with public libraries computing service. This system was intended to help the programme team to design and administer an ongoing survey of PAC users in order to understand:
what types of information the library patrons look for;
whether they succeed in solving a personal or professional problem using the information obtained;
and how friendly they find the library environment, as well as to;
collect suggestions for future activities and improvements.
Collecting data from such a large number of libraries has not been a simple task and doing this whilst meeting the need for cost efficiency, data quality and reliability does not make this job any easier. IREX quickly decided that it was probably best to put an automatic system in place to collect and process data, which would pop‐up short questionnaires on GL‐Biblionet computer in the hope that library patrons would provide the requested feedback. We anticipated that appropriate database configuration to store, select and process data should not be problematic.
Benefits of the system
The benefits that this system offers can be summarised as:
Once‐only investment in the solution; with minimal operating costs.
The possibility to ask any question at any time to patrons in any public library that receives GL‐Biblionet computers and training.
Scope to collect and process data nationwide, or at regional, county or local level.
A real time data collection system in place, that is flexible and easy to tailor, based on various needs.
Very good snapshots on qualitative aspects of the service.
A direct link with library patrons – giving them a chance to communicate directly what they like or dislike about the PAC service.
A chance to collect information about best practices direct from source.
A way to speed up the whole cycle of data collection – data processing – management decision‐making – and action.
A strong and useful tool to assess programme impact.
A practical and flexible management tool.
Weaknesses of the system
The main weak points in using such a data collection system are:
Uncertainties about the reliability of the data: although we have a good image of the socio‐demographic profile of the people answering the questions there are several factors that are difficult to quantify and gauge:
whether and how many people offered multiple answers;
whether and how many people just play when answering the on‐line survey;
whether the librarians want to be polite to GL Biblionet and answer themselves in order to not disappoint us, or to make their libraries look good;
whether some of the library patrons just want to be nice to the librarians, and always respond positively about the library and PAC.
IREX obtained answers only from libraries, which are open and where computers are available – these answers do not necessarily represent the views of all library users across the country. This could mean that the countrywide situation might be different, so it will be necessary to triangulate the data in order to understand the potential sources of error, and the margin of error within these surveys.
3. Approach to implementation
When IREX decided to initiate the implementation of the system we were concerned about several factors:
The need to guarantee privacy – we held extensive discussions with various librarians, in Romania and abroad, about how such a tool should be designed and used in order to ensure that privacy rights are strictly respected. The system does not collect any personal information (library patron's name, ID, or other data) which can be used to identify users or their computer activity.
Securing stakeholder involvement – GL Biblionet presented the concept to Romanian librarians involved in the programme and to the Romanian Association of Librarians and Libraries (ANBPR); they have had the chance to offer feedback to influence process design, and also to contribute to question design. They are constantly informed about the results, and in this way all libraries participating in GL Biblionet have the chance to better understand what their patrons are saying and what they like when using PAC within the library.
Keeping interference with library patrons at minimum, in order to respect their time, so that they are not prevented from using the PAC services for their own purposes. This resulted in our providing a short explanation at the beginning of the questionnaire detailing the purpose of the survey and how the results would be used. We placed an easy to see button “I do not want to take the survey” at the beginning of the survey, to allow them just do their job without answering the survey if they preferred.
Alerting library patrons to the possibility that they might be asked to answer a short questionnaire – all pop‐up windows are limited to the number of questions, which can be captured in one screen. In this way all potential respondents will see from the very beginning that there is no, or minimal, scroll down necessary to answer the questions. Hopefully everyone then understands that the questionnaire will be over after five or six clicks. This is expected to make them comfortable in deciding to answer the questions. On the other hand, this requirement somewhat complicated our job since we had to divide our range of questions into several small mini‐surveys, aligned with our overall aims.
Sharing the results with library patrons – after a questionnaire is filled and the library patrons press the “submit” button, a new windows opens, thanking the submitter for the input and offering a link to the GL Biblionet website on a specific page where some charts and success stories based on the questionnaires answers are presented. In this way we expect to help library patrons understand the value of their opinion and stimulate them to revisit the page to see what has changed based on their input.
4. System description
To be able to implement the pop‐up surveys we brought together several major components (see Figure 2):
The Server – used to manage the pop‐up survey content, to trigger the frequency and to store the answers.
Public library's computers – at the time that this paper was prepared we were linked to 994 computers located in 235 public libraries across Romania.
Pop‐up software – the core is installed on the server as a small component in each of the 994 computers in public libraries.
Create, change and adjust any survey: basically this is a tools which allows us to deliver any kind of message to computers used within public library which have the software installed.
Decide the triggering method; this can based on pre‐set times, or when a specified application is closed by the library patron (see Figure 3).
Select where the pop‐up survey will be delivered – it is possible to review the locality level and to select localities based on various criteria. We can select a whole county, the whole of Romania or any other combinations we want (see Figure 4).
Data visualisation and processing: we can easily view any survey results based on similar criteria for: the whole country, specific counties, a group of localities from various counties, or one locality. We can see whole survey results or replies to a specific question. It is also possible to export the results in XML and convert it into an Excel file, which can be easily sorted and filtered (see Figure 5).
5. Survey test
The testing of the pop‐up tool was carried out in 235 Romanian public libraries and involved 994 computers. We started testing it in all Romanian public libraries included in Round 1 of the Global Libraries programme. Over a period of four months we received 11,377 inputs from library patrons across the country. This represented about 4.2 per cent of the public library visits during that period. We consider this to be a high response rate, since we had anticipated a response rate of around 1 percent, with a likely maximum of 1.5 per cent. This high response rate was probably caused by great interest in computer services offered by public libraries in rural areas, where it was a real novelty for a lot of people. The most important conclusions are: over one third of the respondents had first begun to use the internet through GL Biblionet computers. Also about one third of library patrons reported that they have less than two years experience in internet usage, this is an extra reason to consider that it might be useful to offer assistance in improving internet use skills. About 27 per cent declared they have al least three years experience in using internet.
Over 55 per cent of patrons claim to have used computers for less than two years, 18 per cent have used computers for between three and five years, 25 per cent for more than five years.
Of the respondents 71 per cent said that they were frequent library users before the GL‐Biblionet donated computers. However, more than 25 per cent of library users declared they came only (6 per cent) or mainly for internet use. This seems an accurate reflection since we know (Biblionet Quantitative study, 2010) that many people do not come to the library for a variety of reasons: they may want other types of resources or books than the public library provides, or they may not read much, and some prefer their own books, whilst others would like to come if the public library had better premises, or new books. But they do come for the GL‐Biblionet computers! This is a good sign and is also a great opportunity for libraries, which can gain more users and eventually reinvent themselves if they learn what these users want and set out to meet their needs (Figure 6).
These figures seem impressive – more than 50 per cent of respondents declared that they come to use GL‐Biblionet computers at least two or three times a week. This offers librarians the chance to see what other services can be delivered to these patrons, as well as a good opportunity for public administrators: if they want to meet citizens, they are going in increasing numbers to the public library! This is also a challenge for the library and librarian – these people will understand fast what can be improved and if they do not see any change in reasonable time they will certainly notice (Figure 7).
About 58 per cent of the total respondents are under 20 years old. This leaves 42 per cent of adult patrons, which begins to contradict opinions heard on various occasions (Biblionet stakeholders at the programme outset) that public libraries are filled mostly with “kids playing games”.
We asked respondents, “What type of information do you look for (generally, not necessarily today)?” The answers are shown in Table I.
Clearly, education is in the first place (53 per cent), which is unsurprising because about 58 per cent of patrons are under 20 and many of them are likely to come to the library for home work. Equally understandable is easy communication with relatives abroad, in second place (27 per cent) – since probably about 10 per cent of Romanian citizens are working abroad (although there are no reliable official statistics on this). Third place for “Finding the solution to a medical problem” is again not surprising since we know that the Romanian public health system needs a lot of restructuring. These areas of interest identified by library patrons are in line with the results of the Citizen's Needs Assessment performed in 2008 by an independent survey company, so we can consider this information as reliable and relevant.
Of the respondents 78 per cent said that they had been successful in solving a personal problem relating to one of a list of subject areas by using the GL‐Biblionet computers. This seems impressive. At the beginning of our work we were not very happy about this question since we were envisaging people using the GL‐Biblionet computers to find a better school or university, obtain a new job, or other tangible results. After a while we came to understand that people may also use the GL‐Biblionet computers to find information about something, and not necessarily take action based on that information. Delivering the right information needed by library patrons can also be considered a success. This drew us to the conclusion that people who reported success here include both those who found the information they needed, and also the ones who took action based on that information. Later we will look more closely at the people who were proactive and used the information to get tangible results. The areas in which they were successful are shown in Table II.
It was notable that this list closely matches the general list offered earlier, suggesting that people are relatively consistent in the subject areas in which they search for information via the GL‐Biblionet computers.
We asked respondents whether they are experienced users of Microsoft Office (33 per cent of respondents think that this is completely true) and whether they can easily find the information they want on the internet (34 per cent completely agreed). Given that 38 per cent of the respondents started using the internet after the GL‐Biblionet computers became available and 57 per cent have been using the computer for less than two years, there appears to be scope for training and support. However, the present form of the questions used to probe this area do not sufficiently distinguish levels of competence as ICT users. This issue is discussed further, in the Conclusions section.
6. Challenges in data interpretation – data quality checking
We have already pointed out the need to triangulate pop‐up survey results in order to understand the sources of potential distortion, and the magnitude of any such distortion. The next steps planned to understand these problems are to:
Disaggregate data by locality, in order to be able to select localities which provided “good” data and “bad” data and to select some communities from each category for further investigation. To choose samples we will explore various criteria to select “good” or “bad” data. The following criteria will be used as a start:
What communities showed the largest number of pop up survey answers?
What communities showed the smallest number of pop up survey answers?
What communities showed the highest interest in education?
What communities showed the highest interest in online communication?
What communities showed the highest interest in health‐related information?
What communities showed the highest interest in labour market information?
Visit these sites and ask public library users similar questions to the ones in the pop‐up surveys and additional ones to clarify issues.
Compare pop‐up survey scores with site responses; and explore the similarities and differences.
Decide what margin of error can eventually be applied to all or most of the pop‐up survey findings.
7. Other benefits of the system
We happily shared the information on the system and its findings with our stakeholders, the Romanian librarians. Analysis of their feedback helped us to understand how each individual community answered particular questions. This feedback enabled us to understand where some subject areas are most in demand. We were able to identify communities where there is a high interest in labour market or health information. The local librarian is able to use this information to decide what new activities can be developed for library patrons. Some of them have already invited people from the local labour office to present at a workshop and to explain to library patrons how to find a job, or how to register for a course leading to qualification. Other librarians have invited bank experts to present bank services, such as how to do on line payments and what security measures to take.
We produced statistics on the number of completed pop‐up surveys for each community. We observed a group of communities with no returns. Sharing this information with our County Coordinators (librarians who coordinate GL‐Biblionet in their county) helped them to understand more about the situation within their county. Since they have a full time job with the County Library they are not always able to catch up with the latest local news within their county. We learnt that the mayor in one community was asking the librarian to stay in city hall to do other work so that the library was mostly closed. The County Coordinator called the mayor and asked for the librarian to stay mainly in the library to deliver services. They agreed on a well defined scheme to deliver GL‐Biblionet computer services and the problem was solved. Another County Coordinator recently discovered that one of her local libraries had contracted a computer virus. Somebody offered to help clean the computers but in doing so, deleted all the software, including the pop‐up survey software. The librarian realised that we needed to reinstall the pop‐up survey software. These small things helped us to raise the interest of our stakeholders in this management tool, and offered them unexpected support in doing their jobs better.
8. Users perceived benefits
One of the surveys includes an open box where library patrons are able to share their personal experiences in using GL‐Biblionet computers within public libraries. Here are a few of their contributions:
I found information that the library didn't have.
Issues of health or law.
Information, bibliographies, stories, drawings …
I think this is a very good service for us.
I recently gave birth to my second child and I didn't know that I could take advantage of social service payments. I went online and found that indeed I am entitled to this premium which nobody told me about. I found the article online, the legislation, and even the application form.
I helped the school to make a cleaner world by finding online where garbage should be placed. We would also like to separate the garbage bins: with a basket for paper, one for plastic, and another for toxic materials.
I take part in the eco‐geography club where we urge people to throw garbage away in the right places.
I am in the school eco geography club and I urged people not to throw trash where they aren't supposed to, but in places equipped with bins and trash cans.
We did practice tests for a driver's licenses and a practice examination, [I achieved a] B Category.
Education and training
I downloaded tests for the scholastic Olympics.
I don't have computer at home and here at the library I'm able to do my high school homework.
I've searched and found a theatre play that we can bring live to our school.
I'm unemployed with two children. I searched online and found courses in Galati. I am very happy. Maybe soon I will become a hairdresser.
I am going to get a certificate for completing a course at Eurocor.
I have searched for some sociology and psychology papers for my mother.
I am a 5th grade student. I often come to the library to do my homework and search for useful information on the internet.
I have managed to write out two projects and apply for Rompetrol financing of [a major] Contest for my 8th grade students.
I could check online the result on my exam.
I made an appointment for the Fundeni Hospital, Urology Department.
I made an online appointment for my mother at a clinic in Bucharest.
I have found a doctor I needed and scheduled an appointment.
Talking to relatives abroad
I talked to my mother abroad, and we saw each other with our webcams.
I communicate with relatives abroad.
I can communicate with my daughter and son‐in‐law who live in France.
I have found a teaching job in Neamt County on the internet.
My name is Tedora and I am in kindergarten. I am going to school in the fall and I come to the library to play Barbie games.
I have become a better pastry chef after gathering recipes and tips on the internet.
I have found verification tests that helped me win a Romanian language and literature contest.
The internet helps me to solve problems regarding my printing house business; it make the communication with my business partners easier.
I have gathered information regarding the financing of a green project.
9. Conclusions and next steps
These findings suggest possible future actions.
It appears that this GL‐Biblionet computer service is highly needed in Romania, partly as an aid to designing and implementing activities to assist library patrons to improve their internet use skills in order to maximize the impact of the service. Various types of IT and Internet training have already been delivered by public libraries in Romania, but still there are more unexplored opportunities.
We will study more options to find out how skilled library patrons are in using computers, finding the expected information on internet, or using the Microsoft Office package. The first set of questions on this subject was extremely general, and we obtained information, which is not consistent with other findings. For example, people who are new Internet users might be very satisfied about their IT skills if they found the expected information after a 30 minutes search. On the other hand, someone with more experience of using the internet might consider themselves as having average skills in internet searching when they spent only five minutes in looking for what they wanted. We need more sophisticated ways of capturing this type of information so that we can better understand library patrons' skills and their need for future assistance.
We found that many children come to use the GL‐Biblionet computer because they are interested in finding information and to do their homework, which provides a good motivation for Biblionet stakeholders and the IREX team. This can be the basis for developing better tailored services for this target group.
A lot of library patrons have shown interest in health information, the labour market, legislation and other specific types of information. GL‐Biblionet, together with the managers of the libraries should be able to decide how this interest can be better focused through other activities. GL‐Biblionet is already delivering Modern Library Services training to help librarians define and plan alternative services to the traditional ones related to book and multimedia lending.
We expect to deliver this measurement process and tool to the librarians to help them collect library patrons' feedback in an efficient way, to understand local needs and to deliver tailored service in order to better serve their communities.
As noted earlier, the data reported above will be triangulated with other evidence we use, such as full quantitative data surveys, a qualitative data study, librarians' reports and programme team data and reports. Together, they help us to better understand the programme progress, challenges and opportunities in order to design and implement better services for Romanian citizens.
We are committed to building a sustainable action plan to continue for long after GL‐Biblionet ends and we have a lot of this work in place. Through this approach we are working to understand our stakeholders' needs and to respond to these needs so that they in turn can deliver better services to Romanian citizens. We have already created a group of volunteers (Coordinators in charge of managing Biblionet operations at county level) who are interested in working with us on data collection and processing. We expect to pass them over the pop‐up survey tool, which is one of the reasons why we wanted something simple and cost effective to run. In the next few years we plan to work with this group to refine this process and eventually to further refine it to meet both local and regional needs. In this way the pop‐up survey tool should become a central piece in our sustainability plan; and the first results and degree of interest already raised seem to confirm this.
Marcel Chiranov can be contacted at: firstname.lastname@example.org
Biblionet (2010), “Biblionet – Quantitative study report”, February 2010.
Chiranov, M. (2010), “Real life impact of the Global Libraries: Biblionet Romania Program”, Performance Measurement and Metrics, Vol. 11 No. 1, pp. 93‐106.
Eckerson, W.W. (2005), Performance Dashboards: Measuring, Monitoring and Managing Your Business, Wiley, New York, NY.
Fried, S., Kochanowicz, M. and Chiranov, M. (2010), “Planning for impact, assessing for sustainability”, Performance Measurement and Metrics, Vol. 11 No. 11, pp. 56‐74.
Project Management Institute (2008), A Guide to the Project Management Body of Knowledge, Project Management Institute, Newtown Square, PA.
World Bank (2004), “World Bank Romania health sector program reform documents”, available at: www‐wds.worldbank.org/external/default/main?pagePK=64187835&piPK=64187936&theSitePK=523679&siteName=WDS&menuPK=64187283&callBack=&projectId=P078971&siteName=WDS&menuPK=64258544&callBack (accessed 12 December 2010).