Issues of e-government services quality in the digital-by-default era – the case of the national e-procurement platform in Czechia

Abstract

Purpose

Scholarly research on e-procurement has been limited and, like e-government, e-procurement has been researched primarily from the perspective of adoption/non-adoption. This paper aims to focus on public administration employees’ perceptions of the quality of národní elektronický nástroj (NEN) – the Czech national e-procurement tool they are required to use.

Design/methodology/approach

The paper is based primarily on statistical analysis of data obtained through two questionnaire surveys addressed to contacts from of all Czech central state administration bodies using NEN; 175 completed questionnaires were gathered in 2020 and 128 in 2022 and subjected to statistical analysis in SPSS.

Findings

NEN was launched as fully operational in August 2015. The research indicates that in 2022 there were still important gaps in the quality of NEN as perceived by public employees.

Social implications

The paper has important practical implications for e-procurement policymakers. It shows that making the e-procurement system compulsory is not sufficient. The government needs to guarantee that it would be competitive with tools that would otherwise be preferred. Otherwise, the application of the digital-by-default principle may lead to institutionalisation of services that are not user-friendly. This has important implications for e-government/e-procurement management and change management.

Originality/value

Little is known about public employees’ perceptions of the quality of e-government and e-procurement. Although e-procurement is an area where the digital-by-default principle was implemented rather early, the quality of e-procurement has still received limited attention in research.

Citation

Špaček, D. and Špačková, Z. (2023), "Issues of e-government services quality in the digital-by-default era – the case of the national e-procurement platform in Czechia", Journal of Public Procurement, Vol. 23 No. 1, pp. 1-34. https://doi.org/10.1108/JOPP-02-2022-0004

1. Introduction

E-government has been one of the most important and popular elements of public sector reform in developed and developing countries for the past two decades (Zhang et al., 2014). It has received a lot of attention in theory, research and practice (Špaček et al., 2020). Researching e-government is even more important in current situations where central governments are increasingly adopting the “digital-only” or “digital-by-default” principle, and e-government services are becoming compulsory. In these situations, available e-government adoption and diffusion theories may not be fully valid and may become obsolete (Märien et al., 2016), and it seems more appropriate to deal instead with the quality of services, which determines the user satisfaction, that is important for the success of the project (Alawneh et al., 2013).

Public authorities were obliged to use digital technologies much earlier than citizens. Although recent literature-reviews show that there has been an increase in the amount of e-government research and articles, e-government research has been following a tradition of diffusion-/adoption-focused research and deals, especially with the citizens’ point of view (Wirtz and Daiser, 2018). The government-to-government (G2G) dimension, where duties to use information and communication technologies (ICTS) are often specified, has received limited scrutiny, and little is known about public administration employees’ satisfaction and perceptions of the quality of e-government.

Public e-procurement is an important area where ICTS have been used intensively. E-procurement systems proliferated in the late 1990s due to the growth of information technology and the internet, the tremendous potential savings achievable with this tool and growing attention from the public sector (Bulut and Yen, 2013). Still, scholarly research on the implementation of e-procurement is limited (Bromberg and Manoharan, 2015) and, similarly to e-government, e-procurement is researched above all using the adoption/non-adoption perspective and theories. This is relevant in contexts where public procurers may choose which e-procurement system they will use but is not appropriate in contexts where legislation requires them to use a national one. In the EU, directives drive mandatory practices. The first EU directives aimed at making public e-procurement progressively mandatory came into force in 2014 (Buyse et al., 2015; OECD, 2019).

With this paper, we want to contribute to the scholarly research on e-participation quality, which is not a frequent topic in the available literature. In the paper, we do not apply the adoption perspective because we concentrate on the e-procurement tool that some public authorities are required to use. This may contribute to the e-procurement literature, which has, to some extent traditionally, applied the adoption/non-adoption perspective that is not fully appropriate for researching compulsory e-tools. We focus on the quality of the Czech National e-Procurement Tool (“Národní elektronický nástroj”, NEN) as perceived by public employees who are legally required to use it in public procurement and are not allowed to decide whether or not to use it. If they are not satisfied with the system, they cannot escape and use another e-procurement tool. Their perceptions may indicate to what extent NEN actually fits their needs and determine their satisfaction with it. Our research was also motivated by the fact that costs for the development and maintenance of NEN exceeded CZK 737m (i.e. almost EUR 30m) since 2014 when its development began (MMR, 2021). So we wanted to know whether, in the perception of the public employees, the quality of the system has been improving too. For this, in this paper, we linked findings of previous research and our own findings based on two questionnaire surveys (conducted in 2020 and 2022). We also want to contribute to another research gap – we concentrate on an e-procurement case from a transition country, context that is usually omitted in available e-government literature (Ahmad et al., 2019). Our finding may be relevant for policymakers and practitioners from these countries who intend to launch a national e-procurement tool that is expected to be compulsory.

The paper is structured as follows: In Section 2, we outline the theoretical support for our paper and summarize main points from the literature on e-government service quality and e-procurement. In Section 3, we briefly introduce the study context – the Czech e-procurement strategies and the development of NEN. In Section 4, we introduce our research design and methodology and continue with a summary of our findings in Section 5, discussion (Section 6) and conclusion (Section 7).

2. Theoretical support

2.1 E-government service quality literature

Service quality has been recognized as having the potential to deliver strategic benefits (Rowley, 2006). Because it is widely believed that user satisfaction is a crucial factor for the success or failure of e-government projects (Kumar et al., 2007; Alawneh et al., 2013; Rowley, 2011) and that satisfaction of e-government users is determined by their perceptions of the quality of e-government services (Gupta et al., 2016; Jacob et al., 2019), implementation of the digital-by-default principle clearly requires e-government services to be of high quality.

Based on the research into user satisfaction with e-services, various e-government quality models have been proposed in the literature adapting models from e-service quality literature (Sá et al., 2016). But there is no consensus on the component dimensions (Papadomichelaki and Mentzas, 2012) and researchers usually approach e-government service quality from the perspective of citizens. Due to this approach, they have been working intensively with various e-government adoption theories (Gupta et al., 2017; Gil-Garcia et al., 2018; Wirtz and Daiser, 2018; Malodia et al., 2021). But approaches used in looking at the quality of e-government services may not be fully suitable for evaluating the G2G services due to different perspectives of government employees (Martin et al., 2020). Also, e-government research has been taking a global and integrated approach, and quality dimensions are usually assumed to be generic (Sá et al., 2016). But e-government is a broad, heterogeneous field, containing various areas of public e-services, including specific areas like e-procurement. Therefore, it has been clearly emphasized in the literature that quality may have different faces and meanings in different contexts of e-government services – different contexts may yield very different meanings of quality, and although definitions of quality often encourage the use of one perspective, there is a risk of not fully understanding what it means to work with quality in practice (Martin et al., 2020).

Not much is known about the satisfaction with and perceptions of quality among employees of public authorities. According to Rana et al. (2013), constructs such as job relevance, privacy, security, perceived benefits, perceived knowledge, assurance, anxiety, perceived quality, income and output quality – even though they possess great significance in contributing to the analysis of employee adoption behaviour for e-government services, were largely under-represented. Gupta et al. (2017) deal with factors influencing employee adoption of e-government, but again they follow the trend to focus on adoption rather than on the quality of e-government systems that are compulsory in public administration. There are also limited studies assessing the success of e-government systems – e.g. Stefanovic et al. (2016) suggested using approaches based on the success of information systems (IS) in e-government evaluation from an employee perspective; Janita and Miranda (2018) suggested quality dimensions from the perspective of public sector employees comprising a security dimension, an efficiency dimension and a communication dimension, but they did not use it to evaluate the quality of a specific e-government service.

2.2 E-procurement and its quality

E-procurement is viewed as a disruptive innovation (Mohungoo et al., 2020). It is defined as “any technology designated to facilitate the acquisition of goods by a commercial or government organization over the Internet” (Reddick, 2004). Inspired by e-business solutions (Walker and Brammer, 2012), public e-procurement is an inter-organizational system that is intended to facilitate Government-to-Business (G2B) and G2G electronic communication, information exchange and transaction support (Mohungoo et al., 2020). E-procurement is a collective term for a range of different technologies that can be used to automate the internal and external processes associated with the sourcing and ordering process of goods and services (Bof and Previtali, 2010). A number of benefits of e-procurement have been reported (e.g. operational and cost efficiency, enhanced transparency and accountability and increased internal customer satisfaction). Various challenges and problems have been discussed in the literature too (e.g. costs, requirements on various skills of staff involved in the management and monitoring of e-procurement systems, security and authentication, compatibility and interoperability of support structures and systems) (Vaidya et al., 2006; McCue and Roman, 2012).

The literature also emphasizes that moving public procurement to the internet raises high expectations. National e-procurement tools may be developed to deal with the situation when sophisticated public procurement services are scarce at the local level, but a full and legally compliant e-procurement process between contracting authorities and economic providers requires a considerable number of functionalities (Huntgeburth et al., 2012), covering the public procurement cycle (OECD, 2019). Therefore, public e-procurement systems may be rather complex (Chen et al., 2021). In addition, e-procurement systems are increasingly connected to other management ISs (e.g. budgeting and financial systems, e-invoicing systems) and external databases (OECD, 2019). These factors may determine the perceived quality of e-procurement systems.

E-procurement is studied above all using the adoption/non-adoption perspective and theories (Wirtz et al., 2010; Huntgeburth et al., 2012; Bromberg and Manoharan, 2015; Chen et al., 2021). This is also apparent in the approach of Brandon-Jones and Carey (2010), who worked with the construct of user-perceived e-procurement quality and defined it as (p. 76): “a multi-dimensional construct incorporating user perceptions of an e-procurement system and the support provided to use it”. They found strong evidence of a positive relationship between user-perceived e-procurement quality and system and contract compliance and suggest that achieving the full potential benefits of e-procurement ultimately depends on delivering a system and support in a way that meets users’ expectations. We have not identified any literature that would deal with the quality of e-procurement systems as perceived by employees of public authorities who are required to use the system by law and/or internal rules and who, when performing their jobs, deal directly with the e-procurement system. Views of employees regarding the e-procurement quality are rather important, and the literature suggests that they may be different (less positive) than views of policymakers or regulators (Ahmad et al., 2019).

3. Study context

E-procurement has been included in Czech national strategies since 2006. The first e-procurement strategy was prepared for the 2006–2010 period. It viewed decentralization as a cornerstone of future development and anticipated a co-existence of the national e-procurement infrastructure (called “NIPEZ”, consisting of systems for publication, e-marketplaces and NEN) and the so-called “individual electronic tools” (IENs). In contrast to NEN, e-marketplaces were expected to focus on different segments of public tenders for fast operational purchases according to the needs of contracting authorities. NEN was to provide contracting authorities with support for complex purchases in all categories of public tenders. IENs were to coexist with the national tools, and it was anticipated that they would be developed by the contracting authorities themselves or contracted out.

The next strategies put greater emphasis on the centralization of e-procurement through the use of NEN. They anticipated the finalization of NEN development by August 2012, the launch of pilot operations by June 2013 and full operation by 2014. But deadlines for the NEN development and launch changed several times. The Ministry of Regional Development (MMR) attributed the delays and changes to amendments to public procurement legislation, as a consequence of which NEN had to be adjusted. The MMR accepted NEN and assumed ownership on 31 March 2014. NEN was launched as fully operational in August 2015. Since the beginning, operation of NEN has been contracted to the private companies that developed it.

The 2016–2020 e-procurement strategy required the MMR to propose and implement measures to establish requirements for central state authorities to use NEN by 30 June 2017. After some changes in deadlines, the government decided (in its resolution no. 467/2017) that central authorities and their organizations would be required to use NEN as a central e-marketplace from 21 June 2017 and for other contracting procedures from 2018 (with some possible exemptions). For this reason, in most central authorities, NEN has replaced the IENs and the e-marketplaces. The requirements were slightly modified by government resolution no. 408/2018, which remains in force and requires central authorities and their organisations (again with some possible exemptions determined by the government) to use NEN for public contracts above CZK 500 thousand (exclusive of VAT) (Ječný, 2020).

The MMR has been presenting NEN as a crucial module of the national e-procurement infrastructure (called “NIPEZ”) and a fully autonomous system that offers complex functionalities for public procurement that can be used by all types of contractors. NEN has been undergoing various improvements since 2017, including the conversion of NEN’s Silverlight foundation into HTML 5, which was expected in 2017, but was delayed (Šálková, 2020). During 2018, a tool called “Simplified Walkway” was incorporated into NEN to guide users and simplify the system for public contractors. The HTML 5 version for public contractors was launched at the end of November 2019, but not fully. A full transformation was expected by the end of 2020, but the section “News for users” on NEN Web pages still informed about incremental conversion of functionalities for public contractors in February and April 2021. This was accompanied by training and updates of NEN user manuals. Some of the functionalities were to be prepared in HTML 5 even later, but the last information on updates is from April 2021 (as of 2 June 2022).

4. Research methodology

4.1 Research steps

Our research had the following steps: in the first step, we reviewed previous approaches to e-procurement and e-procurement quality with an aim to identify e-procurement quality dimensions. Since the literature on this topic is relatively scarce, we also had to review the literature on e-government quality. During the literature review, we also looked for literature evaluating NEN to identify dimensions that had been used for its evaluation, following the idea from the literature that quality may have different faces and meanings in different contexts of e-government services (see the sub-section 2.1).

The first step helped us identify the quality dimensions we used in the second step when we dealt with the preparation of a research design and methods used for data collection. In sub-section 4.2 below, we outline the quality dimensions we used, and the methods we used for data collection in sub-section 4.3. As noted, we conducted two questionnaire surveys. The original paper we submitted to the journal was based on the questionnaire survey we carried out from February to April 2020. After closing the survey, we analysed the data obtained, interpreted them and provided our findings. The methods we used in the data analysis are presented in sub-section 4.4 below. Since our analyses were exploratory, not confirmatory, we did not formulate any hypotheses for testing. Based on the feedback received in the reviews on the original paper, we re-ran the survey to update the data. We used the same questionnaire as in 2020, only adding questions on whether respondents had been working with the HTML 5 version of NEN and (in a separate question) the “Simplified Walkway” tool in order for the survey to reflect more extensively the developments of NEN. In the data analysis, we used the same methods as in the analysis of data from the first survey. Again, we did not formulate any hypotheses, and used new data, especially for (partial) comparisons of findings between the two survey years. On the other hand, the data analysis allowed us to indicate associations between some variables (as outlined in sub-section 4.4 and presented in the findings).

4.2 Quality dimensions used

Based on the literature review, our research worked with the quality dimensions outlined in Table 1. Due to the research gaps and the scarcity of relevant e-procurement literature, we considered the following when preparing our research:

• quality dimensions that are presented and discussed in recent meta-analyses of e-government literature (we consider especially the paper of Sá et al., 2016);

• the points regarding gaps in employee-oriented e-government research made by Rana et al. (2013);

• the role of customer-centric design and user-friendly technologies, the availability of guidelines, interoperability, awareness of expectations, continual evaluation and the importance of user feedback, as emphasized in the literature on critical success and sustainability factors of e-government implementation (Rose and Grant, 2010) and the e-government paradox (Savoldelli et al., 2014);

• the suggestions to work with IS success models in e-government evaluation (Stefanovic et al., 2016). We particularly considered the following constructs: information quality (sufficiency and usefulness of information), system quality (user-friendliness, ease of use, usability), service quality (quality of service that users of an e-government system receive from IS personnel) and user satisfaction (users’ attitude towards the system, users’ general satisfaction with the e-government applications, perceived utility, fulfilled expectations and whether it is worthwhile to use the e-government system); and

• findings of the previous research on NEN (Špaček et al., 2017). The previous study was exploratory and was based on 16 interviews. It was conducted in February and March 2017, so in our research, we also wanted to determine if the previous findings remain valid.

Some constructs traditionally used in e-government adoption literature were not used in our methodology due to the implications of the digital-by-default principle. In particular, we did not use those that relate to intention to use e-government services and systems. Our method had to be adapted to (public) e-procurement too.

4.3 Data collection and respondents

The empirical material for this paper was collected based on a mixed-method approach. The research combined the following:

• We mapped improvements of NEN based on an analysis of secondary sources – we worked primarily with information about NEN and its improvements as published on NEN web pages (https://nen.nipez.cz/) and with information available on the Public Procurement Portal (https://portal-vz.cz), including annual reports published by the MMR on NEN since 2015 (as of the beginning of June 2022, the last available was on the situation in 2020).

• We carried out four semi-structured interviews – two with employees of ministries that were using NEN (the Ministry of Culture and the Ministry of the Interior; we focused on ministries that had already participated in the pilot testing of NEN or decided to use it before the use of NEN became compulsory) and two with employees of ministries that had obtained an exemption from using NEN at that time (the Ministry of Health care and the Ministry of Agriculture). The interviews were carried out in March 2020, prior to the first questionnaire survey (see below) with the goal of obtaining input for the preparation of the survey (questionnaire items). The interviews took the form of conversations, and the questions followed a prepared interview protocol. They lasted 41 min on average and were transcribed verbatim and in their entirety and saved in Microsoft Word documents.

• We conducted two questionnaire surveys. Each was addressed to all central state administration bodies that were using NEN according to information available in NEN (i.e. 12 out of 14 ministries and 14 out of 16 other central state administration bodies; two ministries and two other central bodies obtained an exemption from using NEN). For the first survey, we prepared a database of contacts on all the central authorities, based on information on public procurers in NEN. The first survey was conducted between 23 March and 18 April 2020 and was distributed to 232 contacts from the central authorities. The second questionnaire survey was conducted between 25 April and 18 May 2022 and addressed to 226 contacts (we revised the database of contacts using the information in NEN as of 23 April 2022). In both surveys, we asked the respondents to also distribute the information about the survey to employees of their organizations (i.e. organizations under a ministry) if relevant (i.e. such organizations exist and were using NEN). We were aware that this would make it impossible for us to control for the return rate, but we wanted to increase the number of completed questionnaires we could work with. For the same reason, in the case of both the surveys, we extended the deadline for completing the questionnaire and sent three calls to the contacts to participate in the surveys.

In the case of the first survey, we gathered 175 completed questionnaires; in the second, it was 128. In both years, the survey was completed by respondents from heterogeneous organizations. Although the samples were not identical, which limits the possibilities to compare perceptions between the two years, the majority of respondents who participated in both surveys had long (more than five years) work experience in public procurement and had worked with NEN for more than three years (i.e. before and after its functionalities started to be converted into HTML 5). Also, in both surveys, public employees who participated had experience with simplified public procurement procedures for small-scale contracts in NEN (over 84% from respondents of both surveys), but a majority also with more complex procedures (over 54% of respondents indicated experience with open procedures in NEN in the first survey and over 65% of respondents indicated the same in the second). A majority (88%) of the respondents of the 2022 survey were actively working with the HTML 5 version of NEN; 57% had used the “Simplified Walkway” tool (this tool is available in NEN for several public procurement procedures – for small-scale public tenders, open procedures, simplified below-the-threshold procedures and negotiated procedures with prior publication). Many of the public employees who participated in both surveys also had experience with other e-procurement tools (IENs), which enabled them to compare NEN with other e-tools they had used for their work (most of them were experienced with major NEN competitors – E-marketplace Tendermarket, Tender Arena or E-marketplace Gemin). Characteristics of respondents are specified in Table A1 in Appendix 1.

4.4 Data analysis

The empirical data obtained through the interviews were analysed by closely reading through the transcriptions. Since the interviews were used especially to obtain input for the preparation of the first questionnaire survey and, therefore, their number was intentionally small, they were not subjected to a robust qualitative analysis. Still, they followed the same interview protocol and helped us with the preparation of questionnaire items.

In the case of the data obtained through the questionnaire surveys, all the data analyses were run under the SPSS statistical software. The analytical methods used were selected according to the character of the data – the majority of data analysed were categorical – ordinal or nominal, only a few variables were continuous. When we looked into potential associations between selected variables that were both ordinal, coefficients of association (Somers’s D for asymmetric associations and Kendall’s Tau-C for symmetric associations) were calculated, together with their approximate statistical significance. For detecting potential effects of a dichotomous variable on an ordinal one, the non-parametric Mann–Whitney U test was used.

As we had no prior hypotheses related to the associations (our analyses were exploratory, not confirmatory), we did not strive to design any overall regression model. As presented in Section 5 dedicated to findings, in the case of both surveys, our analyses revealed only some and partial associations between the evaluation of particular aspects by public employees and particular variables investigated. The fact that we observe these partial or particular effects and not any global tendencies in our data (typically, the evaluation of one particular aspect is associated with one particular variable) is, together with the size of our sample, another reason why we do not aim to present any overall (regression) model in this paper.

5. Findings

In this section, we present the main findings of our research. This section contains tables with results of the 2022 survey. In the text, we link them with findings of the 2020 survey. The results of this older survey are included in Appendix 2.

5.1 Evaluation of quality dimensions as perceived by public employees

At the beginning of the survey, public employees were asked to evaluate the overall quality dimensions. The grading system used was similar to grading in Czech primary and secondary education, going from 1 (excellent) to 5 (poor/unsatisfactory). Findings are summarized in Table 2.

As Table 2 indicates, public employees working with NEN were rather critical, especially of its performance (speed for work they need to do in the system) and, to a lesser extent, with the ease of use. They evaluated the other dimensions positively. In 2020, public employees gave worse marks not only to the performance and the ease of use but also to the design and complexity. In both surveys, public employees gave the best marks to the user support.

The individual quality dimensions were operationalized with follow-up close-ended and open-ended questions in the questionnaire survey.

Usability was specified by several statements, and the respondents were asked to indicate to what degree they agreed with them. Findings of the 2022 survey are outlined in Table 3.

In the 2022 survey, public employees were more positive regarding the surveyed aspects of usability of NEN than in 2020. They were most critical of its intuitiveness, ease of understanding for a new user, and they also frequently disagreed with the statement that the system provided them with accurate information and instructions necessary to complete tasks. This confirms the limits of the HTML 5 conversion and corresponds to the negative marks public employees gave to ease of use in their overall evaluation of NEN. Thanks to the HTML 5 conversion, the compatibility of NEN with some internet browsers has also increased, which improves its usability.

In the survey, public employees were also asked if, in their view, NEN requires users to have any special technical knowledge to operate it efficiently. According to most respondents (77%) to the 2022 survey, this is not the case (and the opinion of most public employees was similar in 2020). Only 9% of respondents stated the opposite; they often used comments like “it is rather hard to use NEN without any support”.

Table 4 summarizes results for views on the adequacy of NEN functionalities. It indicates that most public employees considered NEN functionalities to be sufficient for their contracting authorities. On the other hand, they were not so positive about functionalities meant to make their work in the system easier – with the adequacy of templates to simplify e-procurement for public employees, possibilities to generate information and documents for control purposes, and search possibilities offered by NEN. The 2020 data indicated the same.

Findings on the views of public employees on NEN’s performance are presented in Table 5. Public employees were more positive about the statements used in the survey compared to the 2020 findings and positive perceptions about the statements prevailed, except with the last statement, which they mostly did not know how to answer. NEN was not perceived as so slow (including when compared to other e-procurement systems), which seems to be a result of the conversion to HTML 5. Still, views of public employees were rather heterogeneous and were not so positive about the frequency of errors and number of unnecessary steps they need to take in the system, which is in line with their views on the usability. In their follow-up comments, they often compared NEN with other e-procurement tools they used. In both surveys, they stressed that compared to the other tools, NEN was “not well-arranged”, “not intuitive”, “not user friendly”, “hard to operate”, “operable without assistance on the phone”, “too complex even for small scale public contracts”/“requiring needless steps”. In both surveys, public employees also stated that “every step is lengthy”, “NEN gets stuck often”, “NEN is unreliable” or that “NEN is too slow”. Some added “even after the conversion”.

The need to improve the performance of NEN was also included in public employees’ replies to an open-ended question regarding five key changes that needed to be made in order for them to be able to work with NEN more efficiently. More than half of the 72 public employees who responded to the question required increasing speed or response, decreasing complexity (including comments like “decrease the number of steps needed to finish a job”, because “even simple tasks like sending take a lot of time”) and improvement of intuitiveness, including NEN’s design. About one third of them required new functionalities that would make the work in NEN faster – e.g. automated calculations of VAT, automated prefilling of information on public contractors, templates, a preview of public tenders currently administered on the introductory screen, better user control (to avoid situations like “I never know what will get published”), better navigation for users, an option to pre-set some functionalities and customize NEN or better interlinking of NEN with Tenders Electronic Daily (i.e. the national system for publication of information on public tenders required by law). A few public employees pointed out that compared to the older (non-HTML 5) version of NEN, some functionalities were missing in the new version (e.g. they pointed to automatically generated calls for small public tenders).

Security appeared to be rather difficult criterion for public employees in both survey years. It was difficult for them to assess whether NEN was secured against data losses or unauthorized use. On average, almost 60% of public employees did not know how to answer related questions in both the surveys. About 24% agreed that NEN was secure, but most probably, this is a consequence of their trust rather than their technical knowledge on security measures applied in the system. Compared to 2020, when 82 respondents (i.e. 47%) stated that they had faced unexpected situations related to security, 30 public employees (i.e. 23%) stated this in the 2022 survey. In the 2020 survey, public employees mostly pointed to two operational events – one occurred in February 2019 and the second in February 2020. Due to the first, some public contracts were lost from NEN and had to be submitted repeatedly, sometimes in cooperation with other contractors. This delayed some public tenders for five weeks or longer and was frequently perceived as “unbelievable” or “alarming” by public employees. During the second event, the help desk mistakenly assigned certain rights to some users, enabling them to access information and documents that should not have been accessible to them. In the 2022 survey, in their comments responding to the follow-up question, they again pointed especially to these events. Only one public employee referred explicitly to a more recent event when they had faced some security-related issue – a failure of the system in April 2022.

Interoperability/compatibility could not be assessed from the survey data because most respondents (64% in 2022 and 80% in 2020) stated that their organization had not connected another system to NEN (e.g. record management system) or they simply did not know (33% in 2022, 18% in 2020). Therefore, they did not know how to respond to the survey question asking whether NEN could be easily interlinked with another system (82% chose this answer to the question).

Findings on the user support dimension are summarized in Table 6. In both survey years, they indicated that NEN relied primarily on assistance provided to users outside the system. Public employees were relatively more negative about the availability of information within the system that would help them with problems they were unable to solve on their own. Still, positive perceptions prevailed and, similarly to the 2020 findings, user support was viewed most positively among the quality dimensions surveyed. This is consistent with the findings on the overall evaluation of NEN.

Public employees sometimes commented that “user guides are long”, “not always updated promptly” and “too general in the case of some functionalities” or “too specific in the case of the new version of NEN”. In the 2020 survey, public employees were rather positive about the staff of the Helpdesk and expressed this in a large number of comments they made for the follow-up open-ended survey question. Such statements were rather scarce in the 2022 survey. This may be attributed to growing experience of public employees with NEN – because they are more skilled in using NEN, they might contact the help desk less often than in the past.

Concerning the system management and development, most respondents (61% in 2022 and 50% in 2020) agreed that NEN had been developing continuously to meet the needs of users, but the speed of changes to address important needs of users was not perceived as positively (in 2022 40% of public employees agreed and 34% disagreed that the speed of these changes was sufficient; in 2020 27% agreed). In the 2020 survey, in their additional comments, they often pointed to difficulty in changing the system quickly because “changes are subjected to public procurement, and it is very difficult to deal with all requirements because they occur over time”. In the 2022 survey, comments on this were rather scarce, and most of the respondents made statements like “I have no objections” in their replies (we obtained only 12 comments, most stating that the responsible ministry – MMR – was not sufficiently active in changing the system and did not sufficiently communicate with public contractors).

5.2 Impact of selected factors on the evaluation of Národní elektronický nástroj

As noted in the methodological part of this paper, in our data analysis, we also looked into potential associations between respondents’ overall evaluation of NEN and selected (control) variables. For instance, we were interested in whether the length of public employees’ experience with public procurement affected marks (1–5) given to the quality dimensions. With regard to the amount of experience with public procurement, the values of the coefficients of association (Somer’s D and Kendall’s Tau-C) for the majority of aspects evaluated were less than 0.10, which indicates trivial or zero association. In the analysis of the 2020 data, we observed a low but statistically significant positive association in the case of “availability of functionalities” and “complexity” (i.e. the longer the experience with public procurement, the worse the evaluation). Analysis of the 2022 data did not reveal any of these associations (for the values of coefficients and their approximate significances, see Table 7), but we observed low, but statistically significant positive association in the case of performance (speed for the work public employees need to perform); results indicate that the longer a respondent’s experience with public procurement, the worse their evaluation of this dimension.

Regarding the amount of experience using NEN, the 2020 data analysis suggested a low and statistically significant positive association in the case of the aspect “Availability of functionalities” (i.e. the greater the amount of experience with NEN, the worse the evaluation of the aforementioned aspect). But the 2022 data analysis did not indicate any statistically significant associations, i.e. the length of experience with NEN did not significantly influence the marks given to individual dimensions.

We were further interested in seeing whether the marks might be influenced by experience with specific selected e-procurement tools. Because e-procurement tools that were the main competitors of NEN were developed by two companies, we focused on public employees’ experience with these two groups of tools: those developed by Tendersystems (that is, Tendermarket and Tender Arena) and those designed by QCM (E-ZAK and Gemin). To detect potential effects of experience with these two groups of tools on public employees’ evaluation of NEN, we examined our data using the non-parametric Mann–Whitney U Test (M–W Test). (For both groups of tools, the sample of respondents was divided into two groups: experienced and inexperienced respondents.) Results of the 2022 data analysis are presented in Tables 8 and 9 (note: NS = not significant).

The 2022 data indicate that for most dimensions, average marks assigned by public employees experienced with Tendersystems tools were worse than those given by inexperienced respondents (see Table 8). The only exception was the dimension “System stability”, for which the relationship was reversed. However, we do not observe any statistically significant differences in distribution of marks between the two groups. The data thus did not confirm our 2020 finding revealing that public employees experienced with the Tendersystems tools evaluated the aspect “Quality of user support (help desk)” significantly more positively (at p = 0.031) than the inexperienced respondents.

For most quality dimensions, average marks given by public employees experienced with QCM tools were worse than those assigned by inexperienced respondents. The only exception was the dimension “Performance”, for which the relationship was reversed. However, only in the case of the dimension “System stability” was the difference statistically significant (at p = 0.016). We also observed a borderline statistical significance in the case of the ease of use (at p = 0.057).

Open procedure is among the most complex public procurement procedures. Therefore, we were interested in whether experience with this procedure in NEN would affect its evaluation (marks assigned). Findings are presented in Table 10.

As we can see in Table 10, for all dimensions, the average marks assigned by respondents experienced with the open procedure were worse than those given by the inexperienced respondents. The 2022 data thus did not confirm the finding of 2020, which had indicated that public employees experienced with the open procedure gave considerably better marks to the aspect “Ease of use” than did the inexperienced (The differences in the distribution of marks between the two groups were statistically significant at p = 0.045). On the other hand, in the 2022 data, we observed statistically significant differences in the distribution of marks between the groups in the case of the dimensions “Performance” and “Stability”; for both of these dimensions, public employees experienced with the open procedure in NEN gave considerably worse marks than did the inexperienced public employees. This is probably determined by the complexity of the tool (its functionalities for the open procedure). This is partly confirmed by findings regarding differences in marks given to “Design” and “Stability” by respondents experienced with the “Simplified Walkway” tool (see Table 11) (The influence of experience with the HTML 5 version of NEN was not possible to validate because only 15 public employees indicated they did not have such experience).

As Table 11 indicates, for all the dimensions, average marks assigned by respondents with experience with the “Simplified Walkway” were worse than those assigned by respondents without this experience. These differences in the distribution of marks between the groups were statistically significant in the case of the dimensions “Design” and “Stability”.

One question the respondents were asked was, “If you could decide for your public authority about using NEN for e-procurement, would you decide to use it?” (They could choose from among four possible answers “Definitely YES – Probably YES – Probably NO – Definitely NO”. In 2022, 78% of respondents answered “Definitely YES” or “Probably YES”; in 2020, the figure was almost 58%. We inquired into potential associations between public employees’ evaluations of the quality dimensions and their answers to the aforementioned question. For both years, respondents’ evaluation of NEN and their (personal hypothetical) decision about the use of NEN by their public authority were strongly positively associated (positive association meaning that the better NEN is evaluated by public employees, the higher the willingness to use it). For all dimensions evaluated, the associations were statistically significant, and except for the “Quality of user support” dimension, the levels of association were from medium to substantial to very strong (for 2022 findings, see Table 12).

In the case of the “Quality of user support” dimension, the (positive) association was low (but still statistically significant). This may be explained by the fact that this dimension was evaluated very positively by most public employees (the most frequent mark assigned was 1, see Table 2).

6. Discussion

Service quality must be assessed based on the consumer’s experience of the service (Lindgren and Jansson, 2013). Governments forcing their citizens to use e-government services that, for example, do not meet their needs, may lead to frustration of citizens and dissatisfaction with the government’s performance (Ghareeb et al., 2019). Our paper indicates that the same applies to perceptions of public employees as compulsory users of e-procurement.

E-procurement is in the G2G area of e-government. The G2G area has more to do with the institutional environment than with technological criteria (Zheng et al., 2013). Our paper suggests that in the G2G area as well, quality perceptions determine the perceived failure of e-government (e-procurement) services and low perceived quality of the system that employees of public authorities are required to use may raise questions about severe direct and indirect financial costs. This has already been asked in available literature which deals with challenges and risks of e-government management, development and implementation (Kumar et al., 2018; Heeks, 2006; Garson, 2006).

Our research indicates that most of the issues that were presented in the 2017 study (Špaček et al., 2017) remained valid five years later. Although NEN was launched as fully operational in August 2015 and more than EUR 30m have been invested in its development and maintenance, public employees were still critical about most of the quality dimensions surveyed. Although evaluations and perceptions of public employees were more positive in all the dimensions surveyed in 2022 compared to our 2020 findings, public employees were still relatively critical of the speed of NEN for work they need to do (i.e. NEN’s performance) and, to a lesser extent, with the ease of its use and stability.

The paper, therefore, suggests that the application of the digital-by-default principle may be risky in e-procurement because it may lead to the institutionalization of services that are not designed in a user-friendly way. Better perceptions of public employees regarding the quality dimensions indicated in 2022 probably resulted from a combination of various factors like the conversion of NEN into HTML 5, its simplification, and, also, the growing length of use of NEN. Still, our research indicates limits of the conversion, which, on the one hand, might increase the speed of the system, but, on the other hand, did not lead to its major transformation, as is suggested by the levels of disagreement with statements we used in the survey. For instance, in both survey years (2020 and 2022), public employees were critical of NEN’s intuitiveness and its ease of use. Findings also suggest that the transformation did not increase the speed of the system sufficiently because some public employees were still critical of speed which is, as the findings suggest, a result of the complexity of the system (even in the case of functionalities available to public employees for the administration of small-scale public tenders).

Similar to findings from 2017, in both our surveys, public employees were still of the opinion that NEN had not been updated quickly enough to respond to their important needs. The speed of changes has been determined by the fact that they had to be procured, and this takes time. But the perceived poor usability of NEN and the slow speed of its changes were also determined by the fact that NEN was developed by a company that lacked sufficient experience with public procurement processes and other e-procurement tools (Špaček et al., 2017). This confirms that service quality is vital for the development stage of an e-government (e-procurement) system (Stefanovic et al., 2016), and this has to be reflected sufficiently in the preparation as well as procurement of these systems.

NEN was not developed in the situation when sophisticated public procurement services were scarce in Czechia. Various e-procurement tools had been developed and used already before NEN was officially launched. This was a result of the first national e-procurement strategies that did not require a compulsory use of some e-procurement tool, but, on the other hand, they emphasized decentralization and public contractors could decide which e-procurement tool they would use (although only those certified by national bodies were allowed). Our paper clearly indicates that developers of NEN did not consider the quality of competing systems developed, especially by QCM or Tendersystems. But more comparative research on perceptions of these instruments and their functionalities would be needed to identify how NEN could be improved by learning from the competing tools.

The paper clearly suggests that especially experienced users may help significantly during the development of the system and that their opinions may be vital for the development of a system that is to become compulsory. Central bodies considering the creation of a compulsory national e-procurement tool should bear in mind that the inclusion of experienced users in the system development may be beneficial and could save money in the future.

If we apply the first definition of e-services quality (suggested by Zeithaml et al., 2000) to e-procurement and a user perspective, e-procurement quality may be generally defined as the extent to which an e-procurement system facilitates efficient and effective work and the accomplishment of tasks. By these standards, the quality of NEN would be assessed as low considered the findings presented in the paper. Performance and ease of use still received the lowest overall score in 2022, and in the paper, this is linked more to public employees’ level of agreement with specific statements regarding NEN. NEN was not perceived by public employees as an intuitive and easy system. Rather, it was seen as unreasonably complex.

On the other hand, although public employees using NEN gave it a lower average score for usability, at the same time, a significant number of them were positive about surveyed aspects of usability, so the survey indicates mixed results. This may indicate that some public employees had grown accustomed to the complexity of the system; they still perceived the system negatively, as rather unreasonably complex and difficult to use, but were able to operate it and since they simply have to use it to accomplish their tasks, they found ways to do so. The survey also indicates that compared to the situation in 2017, public employees in 2020 were not as negative about functionalities available in NEN, and the 2022 survey indicated that most public employees considered NEN functionalities to be sufficient for their contracting authorities. This may also be reflected in their opinions on other items we used in the surveys. It also highlights the importance of continuous regular evaluation of the quality of the e-procurement systems.

As in the 2017 study, the user support (help desk), in particular, was perceived positively by public employees participating in the 2020 and in the 2022 surveys. Findings from both surveys indicate that if public employees needed it, they were more satisfied with the support they received externally (from the help desk) than with support available to them within NEN. This clearly suggests that the role of the user support (help desk) should not be underestimated and requires some integration of online and offline support services available to users. This supports the findings of Fan and Yang (2015) that users’ perception of offline service quality has an effect on improving their perceptions of online service quality.

Our research also raises a number of points related to methodologies used or designed to be used for the quality evaluation of e-government services. The paper suggests that existing e-government quality models may not be fully appropriate for the evaluation of e-procurement quality and seem to need revisions because e-procurement represents a specific field and, also, e-government models have often been built on the user acceptance constructs that are generally defined as an initial decision made by the individual to interact with and use the technology. The digital-by-default principle makes the use of e-government services compulsory and standard e-government adoption and diffusion theories may not be fully valid because surveying intentions to use e-government services and explaining a lack of interest in using them no longer seems to be so relevant.

On the other hand, existing e-government quality models may still be used to evaluate the quality of e-government services that are legally required to be used. Some factors or criteria they work with may be extremely important, especially in the phase of designing compulsory e-government services, e.g. the relative advantage or observability, perceived functional benefit or user acceptance (Weerakkody et al., 2013). Even in the digital-by-default era, poor quality of e-government services is risky – it may still lead to resistance and may negatively affect the spread of e-government, as suggested by Ghareeb et al. (2019). This is also relevant in the case of e-procurement, where some public authorities may be required to use a national e-procurement system. Due to this, for instance, a construct of social influence, used in the unified theory of acceptance and use of technology model (Venkatesh et al., 2003) or user acceptance (Weerakkody et al., 2013) may be relevant, especially when the degree to which a user perceives that others believe he or she should use a particular system is low due to the system’s low quality. Even in the digital-by-default era, criteria like adequate marketing campaigns (suggested by Ghareeb et al., 2019) may be rather important, for instance, in the case of highly fragmented administrative systems with a high number of small municipalities (which is true of Czechia, which has more than 6,200 municipalities, most of which have fewer than 1,000 inhabitants).

Our research also suggests that it may be hard for regular users to assess system security, which is a usual part of e-government quality models. With regard to security, the role of trust has been discussed in literature. Alawneh et al. (2013) suggest that the role of trust is usually linked to trust in an e-government service rather than to its provider. Our paper indicates the opposite. Although about a quarter of public employees explicitly agreed that NEN was secure in the 2020 and 2022 surveys, this was probably not determined by their technical knowledge of security measures implemented in NEN but rather by their trust in the central body responsible for the system (they probably trusted that the MMR would not require them to use an insecure system). This confirms the positive role of the institution-based trust for satisfaction with e-government services as suggested by Alzahrani et al. (2017).

7. Conclusion

In the paper, we outlined findings of our research, that was based on an assumption that approaching e-procurement from the adoption/non-adoption perspective is not fully appropriate if its use is compulsory. We intended to contribute to the scholarly research on e-procurement quality, which is not a frequent topic in the available literature, although it is a crucial factor for discussing the success of e-procurement projects. Our approach was built on the service quality literature that assumes that the satisfaction of users of e-services is determined by their perceptions of the quality. In our research, we worked with quality dimensions used in the e-government literature, adapted them to e-procurement and used them for the evaluation of the quality of a selected e-procurement system. We also focused on perceptions of public employees, a topic that is not common in e-government/e-participation literature, although public employees were required to switch to digital means earlier than citizens.

We focused on NEN – the Czech national e-procurement tool that is considered by national bodies to be a crucial system of e-procurement infrastructure in the country. NEN was not developed in a situation when sophisticated e-procurement services were scarce, and the research suggests it cannot be considered as a disruptive innovation that would successfully challenge competitive e-procurement tools. The paper clearly shows why it is important to evaluate the quality of e-government systems in the digital-by-default era. Still, almost seven years after NEN was launched as fully operational and the investment of a rather large amount of money in it, NEN was not perceived as satisfactory by public employees who were required to use it – although their perceptions were more positive than our findings from 2020, users in 2022 were still critical of its speed (performance), ease of use (due to steps and information used within the system) and stability, i.e. factors that may significantly determine the efficiency of work within the system. Thanks to the conversion of NEN into an HTML 5 version, the speed and compatibility of NEN has increased, which, as the findings suggest, increased its usability. But the research also clearly shows that the conversion did not bring a transformation of the system that would decrease its complexity in the case of functionalities for small-scale public contracts. In their follow-up comments, public employees often compared NEN to other available e-procurement tools.

The paper raises especially the following points for related e-government/e-procurement theories, methodological approaches and research:

• Quality dimensions need to be discussed and revised to be usable in specific fields of e-government, e.g. e-procurement, and to be appropriate for the digital-by-default era.

• Our research indicates that the digital-by-default principle may lead to institutionalization of e-procurement services that are not fully user-friendly. Future research may answer to what extent this can be seen in other specific e-government fields.

• Continuous research is needed to validate findings, including those on statistical significance.

Our findings suggest the following main lessons for policymakers and practitioners from countries where it is intended to launch a compulsory national e-procurement tool:

• National policymakers need to challenge, discuss and evaluate the quality of e-procurement systems because the better the quality of an e-procurement systems is evaluated by public employees, the greater the willingness to use it.

• Policymakers should be aware that the strict application of the digital-by-default principle may be risky because it may lead to the institutionalization of services that are not user-friendly. Quality perceptions determine the perceived failure of e-government services, and low perceived quality of systems that employees of public authorities are required to use may raise questions about severe direct and indirect financial costs.

• In a situation where national bodies intend to make an e-procurement system compulsory for public contractors, it is necessary that the system be competitive and compare favourably in terms of quality to competing and comparable systems already in use.

• Quality is vital for the development stage of an e-government system, and experienced users especially may help significantly during the development of the system. This may increase the probability that the system is perceived as user-friendly and may reduce costs of future revisions and transformations of the system, and should be reflected in the project management and related change management activities.

• Developers should pay sufficient attention to assistance that is to be provided to users within the system (guiding information as well as their control). Also, the role of the help desk should not be underestimated and requires some integration of online and offline services.

It is necessary to point out that the research on which this paper is built had several limitations that should be considered. We consider the following to be important:

• Although we made every effort in both surveys to increase the number of questionnaires completed, the number of questionnaires we were able to work with was not high and the research must be considered as rather exploratory. Still, we attracted a relatively high number of public employees who are required to use NEN for public contracting and especially public employees with long experience with public procurement and e-procurement tools participated in the surveys.

• Other stakeholders, especially economic operators, should also be included in the evaluation of the quality of the system. In this paper, we especially wanted to discuss G2G aspects of e-procurement and point to issues of digital-by-default approaches in e-government/e-procurement development using perceptions of public employees who are required to use the e-government/e-procurement systems (also because the focus on public employees’ views is infrequent in available literature). Further research on the perceptions of other stakeholders is definitely needed to have more complex data and findings on the quality of NEN.

• The research is based on perceptions. Some real-time testing of operations in NEN, including comparisons with other available e-procurement tools (IENs), would be useful for confirmation of our findings. On the other hand, quality of e-services is always, to some extent, subjective, since it is determined by perceptions and levels of satisfaction of users.

• Also, future research findings could be more integrated with literature dealing with the failures of e-government projects.

Table 1.

Quality dimensions used in our research

Quality dimension	Components researched
Usability	The system is intuitive (it is easy for users to find their way)
	The system is easy for new users to understand
	Users do not need special technical knowledge to operate the system
	The system provides users with accurate information and instructions necessary to complete tasks
	The system is reliable (i.e. functional as described in available guides)
	The system is transparent (it informs users about the current step and about upcoming steps that must be completed)
	The system monitors users (to prevent them from making mistakes)
	The system is compatible with a variety of internet browsers
Functionality	The system offers functions that a contracting authority needs
	The system offers a sufficient number of templates that simplify e-procurement for users
	The system enables the generation of information and documents that a contracting authority needs for the purposes of future control
	The system enables a contracting authority to easily search for information and documents
Performance	A tender can be announced in several easy steps
	A tender can be announced quickly
	The system does not require a user to repeatedly perform steps unnecessarily
	The system works better than other available systems
	Users do not face frequent errors when they work with the system
System stability	The system has no problem handling a large number of users/operations
Security	The system is secured against data losses
	The system is secured against unauthorized use
Interoperability/ compatibility	It is easy to connect the system to other systems (e.g. with records management systems)
User support	The system contains information that users need when they face a problem
	User guides are available to users
	E-learning on how to use the system is available
	Offline training is available
	Users can contact a user support desk
	Users are informed about planned temporary shutdowns in a timely manner
System management and development	The system is sufficiently evaluated by a responsible central authority
	The central authority responsible for the system monitors user experiences and needs sufficiently
	The central authority responsible for the system is an experienced e-procurer
	The system is updated quickly to respond to the important needs of users

Table 2.

Overall evaluation of NEN by public employees

Dimension	Frequency of individual marks
	1	2	3	4	5	Mean	Median	Mode
Ease of use	17	50	40	17	4	2.54	2.00	2
Design (graphical design and outline of individual functionalities in NEN)	20	53	42	12	1	2.38	2.00	2
Availability of functionalities (the system offers functionalities I need)	22	61	36	7	2	2.27	2.00	2
Performance (speed for work I need to do)	16	31	35	30	16	2.99	3.00	3
Complexity (in terms of steps required to complete a task)	22	60	29	14	3	2.34	2.00	2
System stability	22	49	42	12	3	2.41	2.00	2
User support in case of problems (help desk)	73	36	14	4	1	1.63	1.00	1

Table 3.

Perceived usability of NEN

Statements	Frequency of answers
	Strongly agree	Agree	Disagree	Strongly disagree	Do not know	Mean	Median	Mode
NEN is intuitive (it is easy for users to find their way)	6	66	37	18	1	2.53	2.00	2
The system is easy for a new user to understand	10	58	41	15	4	2.49	2.00	2
The system provides users with accurate information and instructions necessary to complete tasks	7	62	47	11	1	2.49	2.00	2
The system is reliable (i.e. functions as described in available guides)	18	79	20	5	6	2.10	2.00	2
The system is transparent (it informs users about an actual step and about further steps ahead that must be completed)	27	72	26	1	2	2.01	2.00	2
The system monitors users (to avoid mistakes)	5	73	32	8	10	2.36	2.00	2
The system cannot be used with some Internet browsers (is incompatible with them)	15	40	13	7	53	2.16	2.00	2

Table 4.

Perceived adequacy of NEN functionalities

Statements	Frequency of answers
	Strongly agree	Agree	Disagree	Strongly disagree	Do not know	Mean	Median	Mode
The system offers functionalities that a contracting authority needs	16	79	15	7	11	2.11	2.00	2
The system offers enough templates that simplify e-procurement for users	10	43	32	9	34	2.43	2.00	2
The system enables the generation of information and documents that a contracting authority needs for the purposes of future control	17	55	28	8	20	2.25	2.00	2
The system enables a contracting authority to easily search for information and documents	11	65	34	6	12	2.30	2.00	2
NEN makes it possible for a supplier to easily access information and documents on public contracts	15	58	15	3	37	2.07	2.00	2

Table 5.

Perceived performance of NEN

Statements	Frequency of answers
	Strongly agree	Agree	Disagree	Strongly disagree	Do not know	Mean	Median	Mode
It is possible to announce a tender in several easy steps	24	77	19	8	0	2.09	2.00	2
It is possible to announce a tender quickly	21	62	35	9	1	2.25	2.00	2
The system does not require a user to repeatedly perform unnecessary steps	13	64	33	10	8	2.33	2.00	2
Compared to other systems with which I have experience, NEN is slow	28	28	31	11	30	2.26	2.00	3
When users work with the system, they do not face frequent errors	14	55	42	10	7	2.40	2.00	2
The system handles a large number of users/operations well	4	13	24	15	72	2.89	3.00	3

Table 6.

Perceptions on user support

Statements	Frequency of answers
	Strongly agree	Agree	Disagree	Strongly disagree	Do not know	Mean	Median	Mode
The system contains information that users need when they face a problem	11	75	35	5	2	2.27	2.00	2
Clear user guides are available	18	75	22	4	9	2.10	2.00	2
Sufficient e-learning on the use of the system by public contractors is available	12	49	14	4	49	2.13	2.00	2
When users face a problem, they can quickly solve it with user support	60	57	7	0	4	1.57	2.00	1
Users are informed about changes to the system and related operational measures	22	65	18	6	17	2.07	2.00	2

Table 7.

Effects of the amount of experience with public procurement on the evaluation of NEN

Control variable: length of experience with public procurement (from less than one year to more than 25 years)
Dimension assessed (mark 1–5)	Somer’s D	Kendall’s Tau-C	Approximate significance	Level of association
Ease of use	0.03	0.03	0.717	None
Design	0.01	0.01	0.892	None
Availability of functionalities	0.11	0.10	0.151	Low
Performance	0.17	0.16	0.024	Low, but statistically significant
Complexity	0.06	0.06	0.449	Trivial
System stability	0.08	0.07	0.313	Trivial
User support (help desk)	−0.00	−0.00	0.959	None

Table 8.

Effects of experience with Tendersystems tools on evaluation of NEN

	Public employees experienced with Tendersystems tools (n = 70)			Public employees not experienced with Tendersystems tools (n = 58)			Significance
Aspect evaluated	Mean	Median	Mode	Mean	Median	Mode	(M–W test)
Ease of use	2.60	2.00	2	2.47	2.00	2	NS
Design	2.46	2.00	2	2.29	2.00	2	NS
Availability of functionalities	2.31	2.00	2	2,21	2,00	2	NS
Performance	3.06	3.00	3; 4	2.91	3.00	2	NS
Complexity	2.41	2.00	2	2.26	2.00	2	NS
System stability	2.34	2.00	2	2.50	2.00	2	NS
User support (help desk)	1.63	1.00	1	1.62	1.00	1	NS

Table 9.

Effects of experience with QCM tools on evaluation of NEN

	Public employees experienced with QCM tools (n = 50)			Public employees not experienced with QCM tools (n = 78)			Significance
Aspect evaluated	Mean	Median	Mode	Mean	Median	Mode	(M–W test)
Ease of use	2.72	3.00	3	2.42	2.00	2	NS (p = 0.057)
Design	2.40	2.00	2	2.37	2.00	2	NS
Availability of functionalities	2.36	2.00	2	2.21	2.00	2	NS
Performance	3.24	3.00	3	2.83	3.00	3	NS
Complexity	2.42	2.00	2	2.29	2.00	2	NS
System stability	2.66	3.00	2	2.26	2.00	2	0.016
User support (help desk)	1.64	1.00	1	1.62	1.00	1	NS

Table 10.

Effects of experience with open procedure in NEN on its evaluation

	Respondents experienced with the open procedure in NEN (n = 95)			Respondents inexperienced with the open procedure in NEN (n = 80)			Significance
Aspect evaluated	Mean	Median	Mode	Mean	Median	Mode	(M–W test)
Ease of use	2.56	2.00	2	2.50	2.00	2	NS
Design	2.46	2.00	2	2.23	2.00	2	NS
Availability of functionalities	2.26	2.00	2	2.27	2.00	2	NS
Performance	3.15	3.00	3; 4	2.68	2.50	2	0,031
Complexity	2.35	2.00	2	2,34	2.00	2	NS
System stability	2.60	3.00	3	2,07	2.00	2	0.001
User support (help desk)	1.67	1.00	1	1.55	1.00	1	NS

Table 11.

Effects of experience with the “simplified walkway” tool

Aspect evaluated	Answer = YES (n = 73)			Answer = NO (n = 55)			Significance
	Mean	Median	Mode	Mean	Median	Mode	(M–W test)
Ease of use	2.62	3.00	2	2.44	2.00	2	NS
Design (graphical design and outline of individual functionalities in NEN)	2.53	3.00	3	2.18	2.00	2	0.019
Availability of functionalities (the system offers functionalities I need)	2.37	2.00	2	2.13	2.00	2	NS
Performance (speed for work I need to do)	3.07	3.00	3	2.89	3.00	2	NS
Complexity (with regard to steps needed to be done to finish a task)	2.44	2.00	2	2.22	2.00	2	NS
Stability	2.56	2.00	2	2.22	2.00	2	0.049
Quality of user support in case of problems (HelpDesk, etc.)	1.66	1.00	1	1.58	1.00	1	NS

Table 12.

Associations between evaluation of NEN and the decision about its use by the public authority

Evaluation of (selected aspects of) NEN vs (personal hypothetical) decision about using NEN by the administration (Definitely YES – Probably YES – Probably NO – Definitely NO)
Dimension assessed (mark 1−5)	Somer’s D (symmetric)	Kendall’s Tau-C	Approximate significance	Level of association
Ease of use	0.58	0.51	<0.001	Substantial to very strong
Design	0.46	0.40	<0.001	Medium to substantial
Availability of functionalities	0.36	0.31	<0.001	Medium to substantial
Performance	0.36	0.33	<0.001	Medium to substantial
Complexity	0.45	0.40	<0.001	Medium to substantial
Stability	0.35	0.30	<0.001	Medium to substantial
Quality of user support (help desk)	0.18	0.14	0.011	Low

Table A1.

Respondents of our questionnaire surveys

Organization	2020		2022
	n	% (out of 175)	n	% (out of 128)
Women	95	54	75	59
Men	76	43	53	41
18–24 years old	3	2	1	1
25–34 years old	33	19	17	13
35–44 years old	47	27	43	34
45–64 years old	86	49	30	23
65+ years old	3	2	2	2
With secondary education degree	64	37	27	21
With higher education degree	110	63	101	79
Work experience in public procurement
• less than one year	6	3	3	2
• 1–2 years	17	10	17	13
• 2–5 years	59	34	25	20
• 5–10 years	49	28	45	35
• more than ten years	44	25	38	30
Work experience with NEN
• less than one year	10	6	6	5
• 1–2 years	30	17	19	15
• 2–3 years	58	33	17	13
• 3–4 years	44	25	26	20
• 4–5 years	20	11	31	24
• more than five years	13	7	29	23
Experience with other e-procurement tools		0		0
• E-tržiště Tendermarket	78	45	58	45
• E-ZAK	69	39	49	38
• Tender arena	62	35	49	38
• E-tržiště Gemin	54	31	29	23
• Contractor’s profile	51	29	28	22
• Other	34	19	24	19
Value of public contracts of their organization
• not exceeding CZK 5m	41	23	30	23
• between CZK 5m and 10m	19	11	10	8
• between CZK 10m and 20m	21	12	8	6
• between CZK 20m and 50m	16	9	9	7
• between CZK 50m and 100m	12	7	14	11
• exceeding CZK 100m	66	38	57	45
From the sector under/organization
• Ministry of Agriculture	3	2	3	2
• Ministry of Culture	13	7	2	2
• Ministry of Defence	7	4	31	24
• Ministry of Education, Youth and Sports	12	7	4	3
• Ministry of Finance	9	5	8	6
• Ministry of Foreign Affairs	3	2	0	0
• Ministry of Health care	7	4	3	2
• Ministry of Industry and Trade	8	5	3	2
• Ministry of Labour and Social Affairs	13	7	15	12
• Ministry of Regional Development	10	6	0	0
• Ministry of the Environment	5	3	2	2
• Ministry of the Interior	22	13	6	5
• Ministry of Transport	4	2	3	2
• Ministry of Justice	26	15	0	0
• Office of the Government	5	3	1	1
• Czech Mining Office	0	0	1	1
• Czech Statistical Office	2	1	4	3
• Czech Telecommunication Office	0	0	1	1
• Industrial Property Office	1	1	1	1
• Land Surveying and Cadastre	16	9	26	20
• National Cyber and Information Security Agency	0	0	2	2
• National Security Authority	1	1	1	1
• Office for Government Representation in Property Affairs	4	2	2	2
• Office for Personal Data Protection	0	0	1	1
• Office for the Protection of Competition	2	1	1	1
• State Material Reserves Administration	1	1	1	1
• State Office for Nuclear Safety	1	1	2	2

Table A2.

Overall evaluation of NEN by public employees

Dimension	Frequency of individual marks
	1	2	3	4	5	Mean	Median	Mode
Ease of use	12	46	62	39	16	3.01	3	3
Design (graphical design and outline of individual functionalities in NEN)	21	49	59	29	17	2.84	3	3
Availability of functionalities (the system offers functionalities I need)	24	64	62	19	6	2.54	2	2
Performance (speed for work I need to do)	9	36	42	33	55	3.51	4	5
Complexity (in terms of steps required to complete a task)	26	52	45	38	14	2.78	3	2
System stability	20	70	37	34	13	2.71	2	2
User support in case of problems (help desk)	90	51	21	8	5	1.78	1	1

Table A3.

Perceived usability of NEN

Statements	Frequency of answers
	Strongly agree	Agree	Disagree	Strongly disagree	Do not know	No answer	Mean	Median	Mode
NEN is intuitive (it is easy for users to find their way)	8	58	49	58	2	0	2.91	3.00	2;4
The system is easy for a new user to understand	12	52	58	50	3	0	2.85	3.00	3
The system provides users with accurate information and instructions necessary to complete tasks	13	63	68	30	1	0	2.66	3.00	3
The system is reliable (i.e. functions as described in available guides)	23	90	31	24	7	0	2.33	2.00	2
The system is transparent (it informs users about an actual step and about further steps ahead that must be completed)	47	76	37	11	3	1	2.07	2.00	2
The system monitors users (to avoid mistakes)	26	89	39	6	14	1	2.16	2.00	2
The system cannot be used with some internet browsers (is incompatible with them)	51	40	25	22	34	3	2.13	2.00	1

Table A4.

Perceived adequacy of NEN functionalities

Statements	Frequency of answers
	Strongly agree	Agree	Disagree	Strongly disagree	Do not know	No answer	Mean	Median	Mode
The system offers functionalities that a contracting authority needs	28	79	41	10	17	0	2.21	2.00	2
The system offers enough templates that simplify e-procurement for users	6	63	43	22	40	1	2.60	2.00	2
The system enables the generation of information and documents that a contracting authority needs for the purpose of future control	13	71	43	14	34	0	2.41	2.00	2
The system enables a contracting authority to easily search for information and documents	17	75	50	15	14	4	2.40	2.00	2
NEN makes it possible for a supplier to easily access information and documents on public contracts	27	59	29	15	42	3	2.25	2.00	2

Table A5.

Perceived performance of NEN

Statements	Frequency of answers
	Strongly agree	Agree	Disagree	Strongly disagree	Do not know	No answer	Mean	Median	Mode
It is possible to announce a tender in several easy steps	22	74	53	25	1	0	2.47	2.00	2
It is possible to announce a tender quickly	20	52	61	39	2	1	2.69	3.00	3
The system does not require a user to repeatedly perform unnecessary steps	23	65	53	19	13	2	2.43	2.00	2
Compared to other systems with which I have experience, NEN is slow	65	42	28	13	24	3	1.93	2.00	1
When users work with the system, they do not face frequent errors	15	79	53	21	7	0	2.48	2.00	2
The system handles a large number of users/operations well	8	10	22	36	96	3	3.13	3.00	4

Table A6.

Perceptions of user support

Statements	Frequency of answers
	Strongly agree	Agree	Disagree	Strongly disagree	Do not know	No answer	Mean	Median	Mode
The system contains information that users need when they face a problem	13	87	49	12	14	0	2.37	2.00	2
Clear user guides are available	31	82	39	10	13	0	2.17	2.00	2
Sufficient e-learning on the use of the system is available	15	55	27	7	69	2	2.25	2.00	2
Because the training on NEN is insufficient, we obtain information from other authorities	10	28	53	52	29	3	3.03	3.00	3
When users face a problem, they can quickly solve it with user support	75	75	15	5	5	0	1.71	2.00	1;2
Users are informed about changes to the system and related operational measures	23	79	27	10	34	2	2.17	2.00	2

Table A7.

Effects of the amount of experience with public procurement on the evaluation of NEN

Control variable: length of experience with public procurement (from less than one year to more than 25 years)
Dimension assessed (mark 1–5)	Somer’s D	Kendall’s Tau-C	Approximate significance	Level of association
Ease of use	0.10	0.10	0.117	Low
Availability of functionalities (the system offers functionalities I need)	0.17	0.16	0.008	Low
Complexity (with regard to steps necessary to finish a task)	0.13	0.13	0.029	Low

Table A8.

Effect of amount of experience using NEN on its evaluation

Control variable: length of experience using NEN (from less than one year to five years)
Dimension assessed (mark 1–5)	Somer’s D	Kendall’s Tau-C	Approximate significance	Level of association
Availability of functionalities (the system offers functionalities I need)	0.13	0.12	0.038	low

Table A9.

Effects of experience with Tendersystems tools on evaluation of NEN

	Respondents experienced with Tendersystems tools (n = 94)			Respondents not experienced with Tendersystems tools (n = 74)			Significance
Aspect evaluated	Mean	Median	Mode	Mean	Median	Mode	(M–W test)
Quality of user support (help desk)	1.65	1.00	1	1.99	2.00	1	0.031

Table A10.

Effects of experience with QCM tools on evaluation of NEN

	Respondents experienced with QCM tools (n = 68)			Respondents not experienced with QCM tools (n = 99)			Significance
Aspect evaluated	Mean	Median	Mode	Mean	Median	Mode	(M–W test)
Stability	3.00	3.00	1	2.52	2.00	2	0.005

Table A11.

Effects of experience with open procedure in NEN on its evaluation

	Respondents experienced with the open procedure in NEN (n = 95)			Respondents inexperienced with the open procedure in NEN (n = 80)			Significance
Aspect evaluated	Mean	Median	Mode	Mean	Median	Mode	(M–W test)
Ease of use	2.84	3.00	3	3.20	3.00	3	0.045

Table A12.

Associations between the evaluation of NEN and the decision about its use by the public authority

Evaluation of (selected aspects of) NEN vs (personal hypothetical) decision about using NEN by the administration (Definitely YES – Probably YES – Probably NO – Definitely NO)
Dimension assessed (mark 1–5)	Somer’s D (symmetric)	Kendall’s Tau-C	Approximate significance	Level of association
Ease of use	0.60	0.58	<0.001	Substantial to very strong
Design (graphical design and outline of individual functionalities in NEN)	0.44	0.43	<0.001	Medium to substantial
Availability of functionalities (the system offers functionalities I need)	0.41	0.39	<0.001	Medium to substantial
Performance (working speed)	0.51	0.50	<0.001	Substantial to very strong
Complexity (with regard to steps necessary to finish a task)	0.54	0.53	<0.001	Substantial to very strong
Stability	0.45	0.44	<0.001	Medium to substantial
Quality of user support (help desk)	0.25	0.22	<0.001	Low to medium