Twenty years of S&T priority setting in Russia: lessons learned

Anna Sokolova (Institute for Statistical Studies and Economics of Knowledge, National Research University Higher School of Economics, Moscow, Russian Federation)
Anna Grebenyuk (Institute for Statistical Studies and Economics of Knowledge, National Research University Higher School of Economics, Moscow, Russian Federation)
Alexander Sokolov (Institute for Statistical Studies and Economics of Knowledge, National Research University Higher School of Economics, Moscow, Russian Federation)


ISSN: 1463-6689

Publication date: 10 September 2018



This paper aims to present a retrospective analysis of the experience gained in the course of 20 years’ history of S&T priority setting and critical technologies’ identification, in terms of expected and actually achieved effects and lessons learned.


The methodology is based on analysing project documentation and reports, as well as on interviewing project team members. Each project’s effects are evaluated in terms of the six key foresight functions.


The key factors affecting success of priority S&T areas and critical technologies’ selection and implementation have been identified. They include focusing on practical implementation, linking S&T with socio-economic goals, combining thematic priorities with infrastructural and functional ones, as well as integrating priority selection in the S&T policy process.

Research limitations implications

The task of evaluating priority setting exercises over a long period requires a substantial information base to provide a comprehensive comparative analysis. The projects considered in the paper also need to be analysed in a context of socio-economic development.

Practical implications

The lessons learned presented in the paper could contribute to further development of approaches to selecting science and technology priorities and critical technologies, and their more efficient implementation.

Originality value

Priority setting has significant influence on policymaking and decision-making at the national and industry level. The evaluation of a unique 20-year experience provides substantial information and practical hints for further increasing efficacy of this instrument.



Sokolova, A., Grebenyuk, A. and Sokolov, A. (2018), "Twenty years of S&T priority setting in Russia: lessons learned", Foresight, Vol. 20 No. 5, pp. 449-466.

Download as .RIS



Emerald Publishing Limited

Copyright © 2018, Emerald Publishing Limited

1. Introduction

Most of developed countries set national science and technology (S&T) priorities that serve as a basis for shaping their science, technology and innovation (STI) policies. Approved lists of S&T priorities and critical technologies significantly influence countries’ development and allocation of public resources, which stresses the need to evaluate priority setting efforts in terms of their practical results and effects. In Russia, national-level S&T priority setting practices go back to 1995, with five project rounds completed during the 20-year period. The goal of this paper is to present a retrospective analysis of the experience gained in the course of 20 years’ worth of priority setting and critical technologies’ identification, in terms of expected and actually achieved effects and lessons learned, which would contribute to further development of approaches to selecting priority science and technology areas and critical technologies (PAs and CTs), and their more efficient implementation.

The scale of research has been steadily growing during the last few decades, along with the level of its multidisciplinarity and the role it plays in global innovation-based development. In certain countries, expenditures on research and development (R&D) have reached 3-4 per cent of the GDP ( – the level, which is difficult to further increase. However, even countries such as the US and China (the leaders in terms of R&D expenditures) cannot afford full-scale research covering the complete range of subject areas. Therefore, setting system of adequate STI priorities becomes particularly important as they affect the prospects not just for scientific, but to a large extent socio-economic development as well (OECD, 2012; Gassler et al., 2004). This is especially relevant for countries where public resources available for R&D funding are limited. In most countries, priority setting is seen as a key S&T policy tool, designed to support strategic planning, management and coordination while trying to accomplish major socio-economic objectives (Popper et al., 1998; Technologies clés, 2010, 2006; Klusacek, 2004; Salo and Liesio, 2006; Sokolov, 2007).

The critical technologies approach was first applied in the USA in the 1970-1980s and then diffused further to France, Germany and other countries. It is usually considered a tool to concentrate limited resources on the most important innovative manufacturing technologies (Keenan and Cervantes, 2010). At the same time, it can be used as an instrument to support design of STI policy and creation a favourable environment for its implementation, e.g. in public – private cooperation, research infrastructures and international cooperation (BILAT-USA, 2010). Priorities are selected on different levels: national (UK – Department for Business, Innovation and Skills, 2011; Technologies clés, 2010, 2006), international (EU – European Commission, 2011; BRICS – Sokolov et al., 2017), sectoral (Department for Business, Innovation and Skills, 2013). In most cases, they are identified on the basis of extensive broad consultations with the expert community, and application of foresight techniques (OECD, 2010; BILAT-USA, 2010). Recently, these methods have been increasingly combined with quantitative tools, such as bibliometric and patent analysis, big data analysis et al. (Sokolov et al., 2017). The system of criteria for priority setting depends on their type and level, but usually, they are quite broad and reflect technologies’ contribution to socio-economic development and capacities available for their successful practical implementation.

High importance and widespread application of foresight practices suggest the need to evaluate them in terms of practical results and achieved effects. A lot of theoretical and practical experience has been accumulated by now regarding evaluating national level foresight exercises’ results. Authors of theoretical studies suggest models and criteria for evaluating national foresight projects (Georghiou et al., 2004; Georghiou and Keenan, 2006; Kalle et al., 2010; Hassanzadeh et al., 2015), identify various success factors (Calof and Smith, 2008; Meissner, 2012; Habegger, 2010) and impact areas (Popper et al., 2010; Havas, Schartinger and Weber, 2010; Harper, 2013). Results of national foresight studies have been evaluated since 1990; better-known efforts of this kind include the evaluation of FUTUR programme (Germany; Cuhls, 2003; Giesecke, 2008), the Hungarian Technology Foresight Programme (Kováts et al., 2000; Rader, 2003), the third round of the United Kingdom Foresight Programme (Miles, 2002; Miles and Keenan, 2003; Georghiou and Keenan, 2006), the Vision 2023 Technology Foresight (Turkey) (Saritas et al., 2007), the Colombian Technology Foresight Programme (Popper et al., 2010) and the Russian National S&T Foresight 2030 (Sokolova, 2015). However, practical experience of evaluating projects devoted specifically to identifying S&T priority areas and critical technologies has not been described in literature yet.

In Russia, work on identifying national thematic[1] S&T priority development areas and drafting lists of critical technologies has been going on since 1995; five rounds of relevant studies were completed in 1995-2015, with the ultimate goal of setting basic reference points for shaping and implementing national STI policy. The priority setting methodology improved from round to round, with an increased number and a more balanced mix of experts involved in the exercise and better-quality resulting priority areas (PA) and critical technologies (CT) lists, all of which except the latest one were approved on a high level of government. However, the PA and CT lists were criticised too, primarily for a limited scope for their practical application. For example, the 2015 list agreed with all key stakeholders and with the Government Commission has not been formally approved in the end, and further work on PA and CT selection was stopped. An attempt to set new S&T development priorities was made in the course of designing the S&T Development Strategy for the Russian Federation approved in 2016. Therefore, evaluating the previously implemented projects seems to be relevant and important – to compare the evolving approaches to priority setting using an adequate empirical basis.

The objective of the study is to retrospectively analyse the experience accumulated after 20 years of selecting S&T priority areas and critical technologies in terms of expected and actually achieved results, to contribute to further development of approaches to PA and CT selection, and more efficient practical application of the latter. The study’s methodology bases on a combination of analysing project documentation (such as terms of references and contracts), reports and publications as well as interviewing project team members. The documents’ analysis allowed making a detailed overview of all cycles of the project and preparing materials for interviews. The interviews, which were conducted with five project team members (including two authors of this paper who directly participated in the last three of the five rounds of the project) were aimed at understanding and analysis of planned and achieved effects of the project and lessons for the future. The positions of interviewed team members varied from project manager to project leader.

The important feature of the methodology is consideration of the achieved effects of the project through the main foresight functions. It allows to structure analysis of the effects to connect them to broader categories of potential areas of foresight influence and ensure that all kind of effects are considered. These main foresight functions are (according to Da Costa et al., 2008):

  • Informing policy: generating insights regarding the dynamics of change, future challenges and options, along with new ideas and transmitting them to policymakers as an input to policy conceptualisation and design.

  • Facilitating policy implementation: enhancing the capacity for change within a given policy field by building a common awareness of the current situation and future challenges, as well as new networks and visions among stakeholders.

  • Embedding participation in policymaking: facilitating the participation of civil society in the policymaking process, thereby improving its transparency and legitimacy.

  • Supporting policy definition: jointly translating outcomes from the collective process into specific options for policy definition and implementation.

  • Reconfiguring the policy system: in a way that makes it more apt to address long-term challenges.

  • Symbolic function: indicating to the public that policy is based on rational information.

The paper is structured as follows: Section 2 presents project descriptions including the external context, goals and objectives, methodology and results. In Section 3, the main expected and actually produced effects of each project are analysed, along with the lessons learned after their implementation. Finally, Section 4 presents an analysis of the projects’ external and internal evolution in terms of applied methodologies, achieved results, produced effects and conclusions made based on the analysis.

2. Presenting the projects: selecting priority S&T areas and critical technologies in 1995-2015

2.1 The 1995-1996 project

The first project on identifying PAs and CTs was initiated in 1995. After the collapse of the USSR, the Russian economy was in a critical state: in 1990-1995, the GDP dropped by almost 40 per cent, the previously established economic ties were disrupted, numerous high-tech enterprises and most of applied R&D organisations connected with the heavy industry and the defence sector went bankrupt. The economy has shifted towards mining industries; the need for technological modernisation was dire (Gokhberg and Sokolov, 2017). However, owing to the crisis public R&D expenditures were being cut faster, and more extensively, than other budget appropriations. Meanwhile, the S&T sector inherited from the USSR needed to adjust to the new economic and political system. Preserving the core S&T potential was one of the most crucial objectives of the period. Creating an economy based on advanced technologies was supposed to become a decisive factor of achieving growth and technological independence, making Russian S&T products competitive and increasing the quality of life. S&T priority setting was expected to provide a basis for emergence of advanced high-tech industry in Russia.

In a situation of severe budgetary constraints, the objective was to identify a limited number of CTs to receive public financial support, whose development would sufficiently quickly generate tangible socio-economic benefits. Accordingly, only technologies matching national development goals, and with a potential to be developed within the next ten years were considered for inclusion in the list.

While identifying S&T priority areas, the experts considered relevant priorities set by technologically developed countries (e.g. ICT, medicine and health, nanotechnology, etc.), and areas whose advancement would match specific features of the Russian economy (such as transport, fuel and energy and efficient environmental management). The first draft of the CT list was composed on the basis of a multistage Delphi survey involving hundreds of experts from academic institutes, universities and industrial companies.

To select critical technologies, experts were asked to consider how they affect quality of life, competitiveness of Russian products and services, how economically efficient they were, whether they could provide a basis for developing a large number of other technologies, and significantly affect various sectors of the Russian economy.

At the next stage, the experts’ suggestions were reviewed on the basis of consultations with scientists, and then finalised by a working group established by the Russian Government. The final result was the list comprising seven priority areas and 70 critical technologies, approved by the RF Government’s Expert Council.

2.2 The 2001 project

In 1998-1999, a major expert evaluation study (with participation of more than 1,000 experts) was conducted to assess the current state and development prospects of PAs CTs. The experts assessed each technology in terms of its contribution to economic and social development, national security, improving the environment and various other criteria. On the basis of the evaluation results, it was proposed to significantly reduce the number of critical technologies, by eliminating the ones the experts did not believe had sufficiently good prospects.

The new project on updating PA and CT lists was implemented in the aftermath of the 1998 crisis, which has led to a more than 5 per cent GDP reduction. The S&T sphere was affected more severely than other sectors of the economy: gross expenditures on RD had dropped by 9 per cent. On the other hand, the weak rouble created favourable conditions for Russian producers, and, thanks to growing energy prices, government revenues started to grow in the early 2000s. This subsequently allowed increasing investments in the R&D sector. Accordingly, the second round of priority setting was concentrated on identifying technologies with a potential to make the biggest contribution to fully implementing the country’s S&T potential, and making best use of national competitive advantages (Gokhberg and Sokolov, 2017).

During the first stage of the project, based on the previously approved list comprising 7 PAs and 70 CTs, a more detailed list of 258 technologies was drafted (the third level of the hierarchy). Then, a Delphi survey was conducted using this detailed list. Scientists, science managers and leading S&T authorities took part in the poll. Altogether, more than 800 experts have filled in the questionnaires. Based on the results, rankings were calculated for each second and third level technology using various specific criteria; following this analysis, the expert groups established in specific priority S&T areas prepared proposals to reduce the number of critical technologies, to concentrate resources in the more important fields. Experts from 39 government ministries and departments, science foundations, the Russian Academy of Science (RAS) and industrial academies were involved in adjusting the list of critical technologies. The involvement of such a broad circle of experts promoted lively professional debates, but at the final drafting stage the quality of priority lists has significantly deteriorated owing to lobbying by certain influential scientists. The main result was an essentially more extensive list of critical technologies, though their nominal number was reduced[2].

At the final stage of the project, the lists were agreed with government ministries and departments at the level of deputy ministers responsible for S&T development, who suggested other areas for inclusion in the lists. The ultimate result was a list comprising 9 priority development areas and 52 critical technologies.

2.3 The 2004-2005 project

In 2004-2005, PA and CT selection exercise took place against the background of rapid economic growth, and favourable economic conditions created by growing oil prices. Accordingly, the objective was to identify S&T development areas for investing the growing research and development appropriations in the most efficient way, and to promote innovation activity by encouraging research. Moving on towards research-intensive economy, increasing efficiency of innovation and supporting leaders became the keynote of the Russian S&T policy during that period. Along with priority setting, the first national-level S&T development strategies were designed. Particular attention was paid to commercialisation of R&D results and production of competitive products.

To correct the faults of the previous priority lists, it was decided to radically alter selection methodology and organisational procedures. The focus was placed on the following aspects: reducing the number of priorities; highlighting a limited number of particularly important technological areas; orienting towards achieving sustainable economic growth; adopting the “market pull” principle; minimising lobbying by government agencies and groups of scientists; linking priority setting with procedures for S&T policy shaping and implementation.

The first methodological innovation was reducing the number of priority setting criteria. Previously, owing to a large number of such criteria and their insufficiently clear interpretation, practically any technology could be included in the “priority” category. To avoid this, it was decided to apply just two criteria: contribution to accelerating GDP growth and increasing competitiveness of the Russian economy, and to strengthening Russia’s national security (including its technological aspects).

Another important innovation of the project, arising from the newly adopted “market pull” basic principle, was active involvement of businesses – to identify more promising products and services. Through this approach, and taking into account suggestions by federal executive agencies and results of polling external experts, the initial lists of products and technologies were drafted and subsequently discussed in the scope of moderated debates on each priority area. The main questions for expert events were as follows: production of which competitive products can be launched in Russia in the next ten years, and which new technologies would be required for this?

On the basis of the results of expert debates, draft lists of critical technologies and major innovative products were composed. Compared with the previous round, the number of critical technologies was not only visibly reduced (from 52 to 34), but significantly modified. At the same time, on the whole the structure of priority S&T areas remained practically unchanged, with the following categories: information and telecommunication systems; nanosystems industry and materials; live systems; efficient environmental management; energy and energy saving; transport, aviation and space systems; security and counterterrorism; prospective weapons, military and special-purpose equipment.

2.4 The 2009 project

The global 2008 crisis led to reduced industrial output and GDP and, as a consequence, to severe budgetary constraints. Implementation of anti-crisis policies required reallocating resources; accordingly, R&D expenditures were cut again. The STI policy was focused both on accelerating GDP growth and diversifying its structure, leading to further reduction of the number of CTs. They were identified paying particular attention to expected results of their application; technologies with a high potential for quick commercialisation were favoured and those with significant socio-economic effects.

The results of the Russian S&T Foresight 2025 study conducted in 2007-2008 served as the main source of information for this round (Sokolov, 2009). As in the previous project, the “market pull” approach dominated. Prospective markets were identified, and then innovative products and services with a potential to affect the growth of these markets, followed by identifying technologies best suited for launching competitive production of such products and services.

Overall, this project’s methodology was similar to that of the previous round. CT selection criteria remained unchanged, with a new one added: potential for quick commercialisation of R&D results. PA and CT lists again were drafted in the scope of moderated expert discussions; participants were asked to take into account the results of initial expert polling, an analysis of the Russian S&T Foresight 2025 results and proposals of government ministries and departments regarding prospective products and technologies. An important methodological innovation suggested during this round was developing CT passports containing more detailed information about the technologies, such as their description, structure, application areas, markets, effects and implementation mechanisms. In the end, a list of 9 S&T priority areas and 27 critical technologies was agreed upon.

2.5 The 2014-2015 project

Following Crimea’s joining Russia in 2014, the country’s increased isolation and introduction of sanctions, a dire need emerged to develop own technologies for various sectors of the economy. Accordingly, during this round of CT selection particular attention was paid to security-related aspects including technological security, and substituting imports of critically important products including mass market ones. Also, owing to reduced oil prices and economic stagnation, severe budgetary constraints were introduced, dictating the need to strictly focus public support on a narrow range of particularly important technology areas.

The project’s main methodological principle was orientation towards prospective products and technologies with a potential to make the highest contribution to accomplishing socio-economic objectives. Technologies’ potential for development and practical application in Russia was assessed, along with related risks and limitations.

A multilevel expert evaluation system was designed for in-depth analysis of the draft product and technology lists: 1) the Inter-Departmental Working Group (IDWG) comprising representatives of federal agencies, development institutes, leading R&D centres and the business community; 2) working groups for ten specific subject areas – members were selected on the basis of recommendations by federal agencies, publication and patenting activity and suggestions by leading experts in relevant fields (“snowball selection”); and 3) the overall expert community.

PA and CT lists were adjusted with participation of the IDWG, federal executive agencies and experts. In the end, a list comprising 9 priority areas and 27 critical technologies was agreed.

All five rounds of PA and CT selection are briefly described in Table I.

3. Analysis of effects, reasons and lessons learned

In line with the adopted methodology, effects were analysed in terms of the six main functions of Foresight studies conducted to support STI policy shaping: informing policy, facilitating policy implementation, embedding participation in policymaking, supporting policy definition, reconfiguring the policy system and the symbolic function. Key effects were identified for each round and each function (Table II). Next, these effects, external and internal factors affecting emergence of planned and actually produced effects, and lessons learned at the time are described for each project.

3.1 The 1995-1996 project

During the implementation of this project, a broad circle of scientists for the first time ever got involved in decision-making process. In the course of finalising the PA and CT lists at the working group meetings, a common vision of the S&T sphere was forged, shared by key stakeholders including representatives of 19 Russian Government ministries and departments.

The resulting PA and CT lists were approved by the RF Prime Minister; it was also decided to review it every 2-3 years, which implied that importance of the PA- and CT-based tool was recognised at the highest level, and that the government was committed to make S&T policy more efficient and increase returns on public R&D investments. A federal R&D programme in the S&T sphere was developed and approved on the basis of the identified PAs and CTs. The Government Commission instructed federal agencies and regional authorities to use the approved PA and CT lists as a basis for setting industry-specific priorities, adjusting concepts, forecasts and socio-economic development programmes. However, no work on identifying industry-specific PAs and CTs was actually done owing to the national economic crisis. Government agencies supervising specific industries had to fight for their survival, so they provided practically no support for development of industry-level R&D; no PA and CT lists were drafted.

This project was the first attempt to conduct a major foresight study in modern Russia. It laid a methodological foundation for identifying PAs and CTs, and because the project results were approved at the highest level, it gave an impulse to further development of the foresight culture. However, the main expected effect – concentrating public financial support on projects and programmes in priority areas, and development of critical technologies – was not achieved, though nominally, all applications for project funding in the scope of the federal S&T development programme were supposed to match one of the approved CTs.

The main reason for this was excessively broad coverage of S&T subject areas included in the priority lists (development of even a small proportion of the 70 critical technologies couldn’t possibly be fully supported with public resources). This was due to both the faulty methodology (excessively broad scope, lack of criteria for matching PAs and CTs under consideration with the actual demand in the real sector of the economy, availability of ground work results, competitiveness on international markets, a wide range of application areas) and to the direct lobbying of their vested interests by influential groups of scientists.

The main lessons learned for the next round included the need to reduce the number of CTs and fine-tune priority selection criteria. The need to involve representatives of the business community (as key consumers of R&D results), and minimise lobbying by influential scientists (frequently unable to produce any R&D results actually useful for the economy) have not been fully realised by that point in time.

3.2 The 2001 project

Many more experts were involved in PA and CT selection during this round – around 800, mostly representing the research sector with a small share of industry representatives. The emergence and high-level approval of PA and CT lists signalled to many researchers the need to demonstrate the “high priority” of their respective areas of study; against the background of the serious economic crisis of 1998-1999, influential groups of scientists fought with everything they had to get their subject areas included in the list of priority ones. At the adjustment and completion stage, a common vision of the PA and CT list was forged by representatives of 39 government ministries, departments, science foundations, the RAS and industrial academies.

The resulting list was approved at an even higher level than before, by the RF President. The selected PAs and CTs were also included in the document entitled “Basic RF Policy on S&T Development Until 2010 and the Subsequent Period”. Awarding official status to the selected priorities, and their approval by the country’s highest authorities provided an important incentive for further development of relevant approaches and organisational procedures.

However, in the end, the resulting CT list did not become an efficient S&T policy tool owing to several reasons. Firstly, despite the fact that the number of CTs was formally reduced from 70 to 52, they were formulated excessively broadly (in many cases the cutting was achieved by merging CTs together), which did not allow to concentrate public resources in really essential areas. Secondly, the PA selection methodology was based on the “technology push” principle – which has led to inclusion of numerous items with no serious market potential. Thirdly, the declared priorities not infrequently remained just that – i.e. declarations and played no role in shaping S&T policies. Finally, no efficient implementation mechanisms were put in place.

The need to apply a radically new approach became obvious, oriented towards practical application of technologies, real reduction of their number (together with the scope of priority areas) and elimination of lobbying.

3.3 The 2004-2005 project

Significant methodological progress was made during this round. For the first time, the accent was placed on demand by the real sector of the economy. Many scientists were not ready for this, and simply could not name specific products or services, which could be made by applying their R&D results. This allowed to significantly cut the lobbying and subsequently resulted in a number of researchers moving on to other, more applied subject areas. Moreover, in the scope of the new approach, representatives of the business community for the first time got involved in the dialogue about the research agenda, which provided yet another incentive for the R&D sector to produce more relevant practical results. Finally, work on developing critical technology passports began, which included the technologies’ detailed description, application areas, markets, effects and implementation mechanisms.

The final list comprising 8 PAs and 34 CTs was approved by the RF President, with the PAs and CTs providing a basis for the new Federal Targeted S&T Development Programme (FTSTP). After the PA and CT lists were approved, all applications for funding in the scope of the FTSTP had to include a reference to specific CTs whose development would be significantly supported by the proposed research. This time, largely thanks to the improved methodology and reduced lobbying, it indeed became possible to concentrate public funding in a limited number of high-priority areas.

One of the expected effects of this round of PA and CT selection was strengthening connections between business and science. To achieve that, applications for FTSTP funding had to be submitted jointly with industrial companies. However, this requirement remained a formality and did not contribute to promoting businesses’ demand for Russian technologies. Given the high growth rate of the economy, companies preferred to buy ready-made turnkey solutions abroad (such as equipment and integrated services) instead of having to struggle with often incomplete, unfinished domestically developed technologies. On the other hand, the Russian R&D sector was unable to offer complete turnkey solutions.

An important lesson of the project was realising the need to more thoroughly analyse critical technologies’ practical application prospects. A detailed, comprehensive assessment of groundwork technological results, human and production resources, opportunities and risks, market entry barriers and other factors was expected to provide a basis for making better-substantiated decisions to grant integrated support along the whole value chain, from R&D to manufacturing competitive products and selling them on the Russian and international markets. Another conclusion made after this PAs and CTs selection round was the need to continue this work on the level of specific industries and regions, coordinate S&T priorities with socio-economic strategies, and integrate the priority setting system into S&T policy shaping processes.

3.4 The 2009 project

The methodology applied during this round was largely similar to the last one. By this time, methodological and organisational procedures had settled, and were formalised by a relevant RF Government Regulation. For the first time, the results of the national foresight study completed a year earlier served as the information and analysis basis for priority-setting exercise – which can be seen as the beginning of emergence of an integrated technology foresight system in Russia.

The final PA and CT lists were approved by the RF President. Building on this work, several ministries have drafted detailed industry-level CT lists (in sectors such as space, health and energy). Also, the list now served as a reference point for designing other strategic documents (development programmes, technology platforms, clusters, etc.). Thus political influence of PAs and CTs was increased.

Following the approval of the PA and CT lists, a system of monitoring their implementation was put in place; it was applied to estimate the amount of resources spent for implementing the PAs and CTs, and assess the obtained results. The objective of strengthening the priority setting system with this new mechanism was to improve the quality of S&T policy.

The project also contributed to advancement of foresight culture in Russia. The system was based on the results of the Russian National S&T Foresight 2025 study; subsequently, this practice was adopted and extended by various government ministries and departments who conducted their own industry-specific foresight projects.

Despite the Ministry of Education and Science’s efforts to establish the PA and CT lists as a reference point for the whole national innovation system, private businesses did not show much enthusiasm about them. Largely, it was because of the lack of efficient tools supporting implementation of PAs and CTs. A regulation on providing tax breaks for organisations that developed solutions in line with the approved CTs was issued, but its effect was much weaker than expected.

The main lesson learned after this round was the need to clearly reflect on the most acute national S&T requirements; therefore, it was decided to group priorities around major socio-economic objectives. Another lesson was realising the need to design priority implementation tools, not only for the national level but also for specific businesses. The task of integrating the PA and CT system with other S&T policy tools also has not been fully accomplished yet.

3.5 The 2014-2015 project

The improved priority setting methodology applied during this round, which implied application of a multilevel expert evaluation system, actually allowed involving all key stakeholders in the work. Even representatives of the Presidential Administration took part, for the first time ever – which indicated that priority setting’s importance by now was realised at the highest level. However, in a way their participation not so much accelerated the work as slowed it down. Among other things, it was owing to their insistence on actually measuring the expected effect of CT implementation – which was quite a challenge even for individual large projects, and practically impossible to do for broad technology areas.

The final list was included in the draft RF Presidential Decree, approved by all federal agencies and by the IDWG, but at the final stage its official approval was not realised. One of the reasons was disappointment with the results of thematic priorities’ implementation, lack of S&T breakthroughs and achievements, which would significantly contribute to economic development. As one of the top officials put it, “the priorities have not been implemented, therefore they had to be the wrong ones”.

A whole host of barriers hindered implementation of thematic priorities in the existing national innovation system. No working mechanisms for involving private businesses in development of major technology areas were put in place. Accordingly, the real sector’s companies, even when offered certain tax breaks, believed the risks associated with investing serious resources in R&D were too high. This provided an incentive to look for new approaches to priority setting, in particular identifying, in line with the need to accomplish major socio-economic objectives, S&T areas with a potential to make a significant contribution to meeting grand challenges. This approach was reflected in the new S&T Development Strategy for the Russian Federation approved in 2016.

4. Discussion and conclusions

Analysis of the five rounds of selecting priority S&T development areas and critical technologies implemented during the last 20 years allowed to identify major expected, actually produced and unrealised effects, study the reasons and draw a number of lessons to further advance the PA and CT selection methodology.

Starting in 1995, PA and CT selection projects were planned as a tool for direct implementation of STI policy, whose main objective at the time was preserving the core Russian research potential in a situation of severe funding cuts, and concentrating the available resources in more important areas. Subsequently, PA and CT lists were also used as a tool for dealing with other current STI policy objectives, such as involving businesses in S&T development, developing domestic technologies to ensure technological security, making use of national competitive advantages and strengthening competitiveness of Russian producers.

The project’s underlying concept, methodology and implementation arrangements have also radically changed during the 20 years. The original approach applied in 1995-1996, directly connected with the then current STI policy objectives, was aimed at identifying prospective S&T areas to support with public resources, where Russia already had important groundwork results. Subsequently, the methodology developed towards reducing the number of critical technologies, applying the “market pull” principle (since 2004), integrating priority setting with the Russian National S&T Foresight 2025 study (since 2009) and identifying CTs on the basis of their contribution to accomplishing major socio-economic objectives (2014). The set of CT selection criteria was adjusted, and significantly reduced (from 6-7 in 1994 and 1999 to just two key ones from 2006 onwards). The mix of experts involved in the exercise has also significantly changed. If in 1995-1996 the majority of them were scientists, since 2006 representatives of the business community were actively drafted, and in 2014-2015 a multilevel expert evaluation system was put in place, comprising all key stakeholders (representatives of the R&D sector, public authorities, businesses and industry).

The bulk of work on drafting and adjusting PA and CT lists was done by working groups and (since 2004) thematic expert panels, who were given initial draft lists put together on the basis of Delphi surveys’ results (in 1994 and 1999), desk research (since 1999) and polls and interviews. Since 2009, the experts’ work was based on the results of national S&T Foresight studies. The number of CTs on the list has been steadily reducing, from 70 in 1996 to 27 in 2015.

Analysis of actual effects achieved in terms of the key foresight functions reveals that the most significant effects were accomplished in the scope of three functions: facilitating policy implementation, embedding participation in policymaking and supporting policy definition. Policy implementation was facilitated by forging a common vision of S&T development shared by the key stakeholders in the scope of working groups and expert panels. Note that in each new round of priority setting, the mix of key stakeholders was getting wider and better balanced. The five project rounds also promoted rapid development of the foresight culture in Russia. The 1995-1996 project was the first major foresight project implemented in the country, and the high-level recognition of its results provided a strong impulse for further development of foresight studies. During the subsequent rounds, methodological and organisational procedures for PA and CT selection were significantly improved, and the experience was also applied in other national foresight projects. After 2011, numerous ministries started conducting their own foresight exercises. It has resulted in quite widely recognised need to build long-term visions through Foresight projects by public authorities, businesses and R&D organisations.

The project also significantly contributed to involving new participants in the decision-making process. Hundreds of professionals were involved in the first project (1995-1996), and though most of them were scientists, it was the first time so many people have actually took part in setting national S&T development priorities. Subsequently, the number of participants increased (up to 800 experts in 2001) and their mix steadily became more representative (quite strict qualification requirements were applied during expert selection) and better balanced (in terms of representation of the R&D sector, business community, industry and public authorities).

Possibly the biggest effect was achieved in terms of supporting policy definition as from the very start the project was considered as a policy tool. The final PA and CT lists drafted in the scope of the first four project rounds were approved at a high government level, and served as a basis for designing other policy documents (R&D programmes, basis RF S&T development policy). Since 2001, the PA and CT lists served as reference materials for development of new strategic documents (e.g. technology platforms and clusters); more detailed CT lists were drafted for specific industries in the scope of the first industry-level priority-setting exercises.

Policy informing took place directly in the course of project implementation, during working group sessions with participation of government representatives discussing draft materials on more promising S&T development areas, possible application areas, risks and threats associated with their implementation and other aspects. Also, PA and CT lists, and since 2006 CT passports as well, were forwarded to regional authorities to be taken into account in the course of developing their own strategies, concepts and forecasts – thus strengthening their information and analysis basis.

The symbolic foresight function was implemented by adoption of PA and CT lists as important S&T policy tools (through approval of these lists at a high government level). Subsequently, their status increased from round to round: starting from the second project, the lists were approved by the RF President, the lists drafted during the fourth round served as a basis for numerous other policy documents, and the fifth round was conducted with direct participation of the RF Presidential Administration.

The most modest effects were achieved in terms of reconfiguring the policy system. These include application of PAs and CTs as a mechanism for allocating public resources (starting from the third round), and putting in place a system for monitoring PA and CT implementation (from the fourth round).

Though evaluating the actual effects is certainly important, probably an even more important task would be analysing the ones, which have not been achieved (along with possible reasons why), and learning lessons for the future (Table III).

Table III shows that after the first two rounds, the most important expected but unachieved priority setting effect was concentrating public financial support in a limited number of promising S&T development areas. In both cases, the main reason of this failure was related to project implementation: faulty methodologies and scientists being able to successfully lobby their vested interests. Which is hardly surprising, keeping in mind it was the first foresight project ever implemented in Russia.

After the first round was completed and the resulting lists evaluated by experts, the main lesson learned from the experience was the need to reduce the number of CTs and improve the methodology of their selection. The importance of limiting lobbying has not yet been realised by then. After the second round, which did not lead to significantly improved final PA and CT lists, it became obvious that the priority setting methodology must be radically improved. It was done, and did bring results: after the third round the project team succeeded in selecting a limited number of priority areas to provide public financial support in. However, by that time it was realised that simply providing funding for R&D in priority S&T areas would not be enough: companies’ demand for results of such R&D should be promoted, but this has not been accomplished at the time.

By the fourth round, expectations from the PA and CT selection project kept growing; it was believed the identified priories would serve as defining vectors of S&T development throughout the national innovation system, on the federal, regional and industry levels. However, private businesses did not take them into account while planning their strategies – owing to lack of efficient tools for supporting PA and CT implementation, because no targeted systemic effort was made to design them. The fifth round (which involved all key stakeholders) has led to a possibly quite natural result, i.e. realisation of the fact that priority setting in its current form did not work, so it was decided to look for new STI policy tools.

Looking back, it can be argued that one of the main lessons – the need to design practical mechanisms for PA and CT implementation – up to a point was learned after the first round of the project already. During the subsequent rounds certain steps were taken along this direction (adoption of the R&D programme in 1996, issuing the RF Government Regulation on tax breaks in 2001, etc.), though only a small proportion of relevant studies have actually yielded practical results (e.g. atomic force microscopes, technologies for forecasting of the environment in offshore and Arctic conditions and nuclear power plants). Throughout the 20 years of the project’s implementation, the effort was mostly focused on priority setting – how to select them, with less attention paid to ensuring their practical implementation. There is a strong need for the effective implementation mechanisms to make critical technologies a successful instrument of STI policy that will allow concentrating resources on selected priorities, distributing the responsibility among main participants, monitoring their implementation and assessing their socio-economic effects (OECD, 2012; Georghiou and Harper, 2011). Otherwise, PA and CT remain a declarative list with minimal positive effects on economy and society. Nevertheless, the issue of practical implementation of priorities has not been given much attention in scientific literature and requires additional in-depth analysis.

Summarising the lessons learned during the years, it can be concluded that key factors affecting success of priority S&T development areas’ and critical technologies’ selection and implementation include the following:

  • analysing the NIS’s ability to orient towards such priorities and their practical implementation;

  • linking S&T priority selection to these areas’ potential contribution to accomplishing major socio-economic objectives;

  • adopting an integrated approach, i.e. supplementing thematic S&T priorities with infrastructural, functional, etc. ones;

  • designing mechanisms for practical implementation of the priorities, and importantly, doing so before or along with drafting priority lists;

  • integrating the PA and CT selection project with other STI policy tools; and

  • planning the desirable effects of priority implementation in advance, and designing detailed roadmaps for obtaining practical results in selected priority areas.

Main characteristics of PA and CT selection projects, 1995-2015

Characteristics 1995-1996 2001 2004-2005 2009 2014-2015
Goal Identify S&T areas supporting which could produce the strongest socio-economic effect in the medium term
Responsible agency RF State Committee on Science and Technology RF Ministry of Science and Technology RF Ministry of Education and Science RF Ministry of Education and Science RF Ministry of Education and Science
Methodology Principles:
– PAs and CTs must match national development goals
– the initial list was drafted taking into account international PA lists
1. The initial CT list drafted in the framework of a two-stage Delphi survey
2. Submitted proposals adjusted on the basis of consultations with experts and scientists
3. Descriptions of critical technologies prepared
4. PA and CT lists finalised by the working group comprising representatives of 19 government ministries and departments
– Reduction and elaboration of the 1996 lists, based on their 1999 re-assessment
1. Cabinet analysis. Preparing a more detailed list of areas on the basis of the 1996 list (258 technologies)
2. Agreeing the list with key experts
3. Conducting a two-stage Delphi survey to evaluate the detailed list
4. Expert groups reduce the number of CTs on the basis of their rankings calculated in the scope of the Delphi survey
5. PA and CT lists agreed with government ministries, departments and the Government S&T Policy Commission
– the “market pull” approach
– reduced number of CT selection criteria
– reduced number of PAs and CTs
1. Obtaining stakeholder ministries’ and departments’ input on products and technologies
2. Conducting interviews with management of largest Russian companies; expert polling
3. Expert panels draft lists of competitive products manufactured with new technologies in each PA
4. Expert discussions to review and adjust the current CT list
– PAs and CTs identified on the basis of the results of the Russian S&T Foresight 2025 study
– the “market pull” approach
1. Expert poll to draft a list of innovative products (and relevant required technologies) likely to emerge in Russia during the next 10 years
2. Review of the Russian National S&T Foresight 2025 study results, ranking subject areas on the basis of their importance indices and development prospects
3. Collecting input about products and technologies from stakeholder ministries
4. Expert discussions and polls to draft PA and CT lists on the basis of Stage 1 results
5. Drafting CT passports
– focusing on CTs with a potential to contribute to accomplishing major socio-economic objectives
– setting up a multilevel expert evaluation system
1. Drafting a list of socio-economic objectives based on analysis of official forecasts and strategic documents
2. Identifying (jointly with experts) prospective products and technologies, which would contribute to accomplishing socio-economic objectives on the basis of federal agencies’ inputs
3. Drafting initial PA and CT lists based on stage 2 results
4. Finalising and agreeing PA and CT lists with federal agencies, the IDWG and experts
Results Lists comprising 7 PAs and 70 CTs Lists comprising 9PAs and 52 CTs Lists comprising 8 PAs and 34 CTs Lists comprising 8 PAs and 27 CTs Lists comprising 10 PAs and 27 CTs

Achieved effects by main foresight functions

Foresight function 1995-1996 2001 2004-2005 2009 2014-2015
Informing policy PA and CT lists distributed among federal agencies and regional authorities with an instruction to use them as guidance for developing/adjusting their forecasts/programmes Contributing to creating an information and analysis basis for development of industry-specific programmes (CT passports were discussed with industry experts) Creating an information and analysis basis in key S&T development areas
Facilitating policy implementation Forging a common vision of S&T development shared by key stakeholders (representatives of 19 Government ministries and departments)
Developing a methodological basis for selection of PAs and CTs
Forging a common vision of S&T development shared by key stakeholders (representatives of 39 government ministries and departments, science foundations, the RAS and industry academies) Forging a common vision of S&T development shared by representatives of government authorities, science and business
Significantly extended range of methods and techniques
Forging a common vision of S&T development shared by representatives of government authorities, science and business
Various government ministries started to implement industry-specific Foresight projects
Setting up a multilevel expert evaluation system involving all key stakeholders
Encouraging the search for other, more efficient approaches (strategy development instead of priority setting)
Embedding participation in policymaking Involvement of new stakeholders in decision-making (hundreds of experts, mainly researchers) Involvement of new stakeholders in decision-making (about 800 experts) Involvement of business community representatives in decision-making process Consolidating all key stakeholders in the scope of PA and CT selection (including the Presidential Administration)
Supporting policy definition PA and CT lists approved by the RF Prime Minister
Adoption of a federal R&D programme in the S&T sphere based on the identified PAs and CTs
PA and CT lists approved by the RF President
Adoption of new policy document “Basic RF Policy on Development of Science and Technology” based on PAs and CTs
PA and CT lists approved by the RF President
Adoption of a federal R&D programme in the S&T sphere based on the identified PAs and CTs
PA and CT lists approved by the RF President
Setting first industry-specific priorities based on the PA and CT lists (State Space Corporation Roscosmos, Ministry of Health, Ministry of Energy)
Numerous strategic documents became oriented towards the PA and CT lists (such as technology platforms, clusters, etc.)
Reconfiguring the policy system Practical application of PA and CT lists as a mechanism for resource allocation/ Setting up a system for monitoring implementation of the PAs and CTs
Symbolic function PA and CT lists recognised as an important S&T policy tool Further recognition of PA and CT selection’s importance Strengthening the status of PAs and CTs

PA and CT selection projects: unachieved effects, reasons and lessons

Project Expected but not achieved effect Reason Lessons learned at that time
1995-1996 Concentrating public financial support on projects and programmes related to PAs and CTs Excessively broad scope of S&T areas
 Methodological faults (excessively broad coverage, insufficiently matching the real sector’s needs, insufficiently matching existing groundwork results and global trends, occasionally too narrow scope for application, etc.)
A scope for scientists to directly lobby their vested interests
The number of CTs should be reduced, and selection criteria adjusted
Contributing to shaping industry-level S&T policies (using the approved PA and CT lists as the basis for drafting relevant industry-specific lists) Industry-level PA and CT lists were never drafted despite an instruction to do so by the Government Commission
 Almost no support of industry-level R&D in the situation of economic crisis
2001 PA and CT lists becoming efficient STI policy tools for concentrating public financial support Excessively broad scope of S&T areas, with some of them having no market potential at all
 Methodological faults (the “technology push” approach)
A scope for scientists to directly lobby their vested interests
Declarative nature of the PA list
No integration with other policy tools
No practical arrangements for implementing PAs and CTs
A radically different approach should be applied, oriented towards practical application of CTs, real reduction of their number and reduced scope of PAs
Lobbying must be eliminated
Practical arrangements for implementing PAs should be developed
2004-2005 Emergence of stronger links between businesses and science (increased demand by companies for Russian technologies) Companies’ unwillingness to buy incomplete, not perfected Russian technologies, opting for turnkey imported solutions (equipment, integrated services, etc.)
 R&D sector’s inability to offer ready-made turnkey solutions
Prospects for practical application of CTs need to be analysed more thoroughly
PA and CT selection should be continued on industry and regional levels
S&T priorities should be coordinated with socio-economic strategies
The priority-setting system should be integrated into S&T policy shaping process
2009 Orientation towards PAs and CTs by non-government participants of the NIS Lack of efficient tools to support implementation of PAs and CTs PAs and CTs should be more clearly linked to the country’s socio-economic objectives
Tools for implementing selected priorities should be designed not just for the public, but also for the private sector
The PA and CT system should be more closely integrated with other STI policy tools
2014-2015 PA and CT lists becoming a core element of the national Foresight system PA and CT lists have not been officially approved
 The authorities’ disappointment in thematic priorities because they did not produce expected results in the framework of the existing NIS
Other tools should be designed to replace PA and CT lists



Three kinds of priority areas are usually distinguished in literature, depending on the objectives of priority setting exercises (Gassler et al., 2004; Glod, Duprel and Keenan, 2009; OECD, 1991, Georghiou, Harper, 2011): thematic, mission-oriented and functional priority areas. This paper focuses on the practice of setting thematic priorities, which are frequently presented in the form of critical technology lists of the national level, though similar lists may also be drafted for specific industries, subject areas or regions.


Among other things, this resulted in areas of incommensurable size and importance being included in the final lists, e.g. “Exploration, production, refinery, and pipeline transportation of oil and gas” and “Mechatronic technologies”.


BILAT-USA (2010), “Analysis of S&T priorities in public research in Europe and the USA//BILAT-USA G.A. n°244434”.

Calof, J. and Smith, J. (2008), “Critical success factors for government led foresight”, Paper Presented at the Third International Seville Seminar on Future Oriented Technology Analysis: Impacts and implications for policy and decision-making, 16-17 October, Seville, Spain.

Cuhls, K. (2003), “From forecasting to foresight processes–new participative foresight activities in Germany”, Journal of Forecasting, Vol. 22 Nos 2/3, pp. 93-111.

Da Costa, O., Warnke, P., Cagnin, C. and Scapolo, F. (2008), “The impact of foresight on policy-making: insights from the FORLEARN mutual learning process”, Technology Analysis & Strategic Management, Vol. 20 No. 3, pp. 369-387.

Department for Business, Innovation and Skills (2011), “UK innovation and research strategy for growth”.

Department for Business, Innovation and Skills (2013), “Nuclear industrial strategy: the UK’s nuclear future”.

European Commission (2011), “Horizon 2020 – The framework programme for research and innovation”,

Gassler, H., Polt, W., Schindler, J., Weber, M., Mahroum, S., Kubeczko, K. and Keenan, M. (2004), “Priorities in science & technology policy – An international comparison”, Project report commissioned by the Austrian Council for Research and Technology Development.

Georghiou, L. and Harper, J.C. (2011), “From priority-setting to articulation of demand: foresight for research and innovation policy and strategy”, Futures, Vol. 43 No. 3, pp. 243-251.

Georghiou, L. and Keenan, M. (2006), “Evaluation of national foresight activities: assessing rationale, process and impact”, Technological Forecasting & Social Change, Vol. 73 No. 7, pp. 761-777.

Georghiou, L., Acheson, H., Cassingena Harper, J., Clar, G. and Klusacek, K. (2004), “Evaluation of the hungarian technology foresight programme”, Report of an International Panel.

Giesecke, S. (2008), “Futur – the german research dialogue”, in Giesecke, S., Crehan, P., Elkins, S. (Eds), The European Foresight Monitoring Network. Collection of EFMN Briefs – Part 1, Office for Official Publications of the European Communities, Luxembourg.

Glod, F., Duprel, K. and Keenan, M. (2009), “Foresight for science and technology priority setting in a small country: the case of Luxembourg”, Technology Analysis and Strategic Management, Vol. 21 No. 8, pp. 933-951.

Gokhberg, L. and Sokolov, A. (2017), “Technology foresight in russia in historical evolutionary perspective”, Technological Forecasting and Social Change, Vol. 119 No. 119, pp. 256-267.

Habegger, B. (2010), “Strategic foresight in public policy: reviewing the experiences of the UK, Singapore, and The Netherlands”, Futures, Vol. 42 No. 1, pp. 49-58.

Harper, J. (2013), Impact of Technology Foresight: compendium of Evidence on the Effectiveness of Innovation Policy Intervention, University of Manchester, Manchester.

Hassanzadeh, A., Namdarian, L., Majidpour, M. and Elahi, S. (2015), “Developing a model to evaluate the impacts of science, technology and innovation foresight on policy-making”, Technology Analysis and Strategic Management, Vol. 27 No. 4, pp. 437-460.

Havas, A., Schartinger, D. and Weber, A. (2010), “The impact of foresight on innovation policy-making: recent experiences and future perspectives”, Research Evaluation, Vol. 19 No. 2, pp. 91-104.

Kalle, A., Piirainen, A. and Gonzalez, J. (2010), “A systematic valuation framework for future research”, Futures, Vol. 44 No. 5, pp. 464-474.

Keenan, M. and Cervantes, M. (2010), Priority Setting for Public Research: Challenges and Opportunities, Presentation at Fagerberg Committee, Norway, 5 November 2010.

Klusacek, K. (2004), “Technology foresight in the Czech Republic”, International Journal of Foresight and Innovation Policy, Vol. 1 Nos 1/2, pp. 89-105.

Kováts, F., et al. (2000), “The hungarian technology foresight programme”, Report by the Steering Group, Budapest.

Meissner, D. (2012), “Results and impact of national foresight studies”, Futures, Vol. 44 No. 10, pp. 905-913.

Miles, I. (2002), “Appraisal of alternative methods and procedures for producing regional foresight”, Report prepared by CRIC for the European Commission’s DG Research Funded STRATA – ETAN Expert Group Action, Manchester.

Miles, I. and Keenan, M. (2003), “Ten years of foresight in the UK”, Paper presented at NISTEP’s second international conference on technology Foresight, Tokyo.

OECD (1991), Choosing Priorities in Science and Technologies, OECD Publishing, Paris.

OECD (2010), Priority Setting for Public Research: Challenges and Opportunities, OECD Publishing, Paris.

OECD (2012), Meeting Global Challenges through Better Governance. International Co-Operation in Science, Technology and Innovation, OECD Publishing, Paris.

Popper, R., Georghiou, L., Keenan, M. and Miles, I. (2010), Evaluating Foresight: fully-Fledged Evaluation of Colombian Technology Foresight Programme, Universidad del Valle, Santiago de Cali, Colombia.

Popper, S., Wagner, C. and Larson, E. (1998), New Forces at Work. Industry Views Critical Technologies, RAND, Washington, DC.

Rader, M. (2003), “Hungary – hungarian foresight programme (TEP) 1997/99”, FISTERA Report WP 1 – Review and Analysis of National Foresight.

Salo, A. and Liesio, J. (2006), “A case study in participatory priority setting for a scandinavian research program”, International Journal of Information Technology & Decision Making, Vol. 05 No. 1, pp. 65-68.

Saritas, O., Taymaz, E. and Tumer, T. (2007), “Vision 2023: turkey’s national technology foresight program: a contextualist analysis and discussion”, Technological Forecasting & Social Change, Vol. 74 No. 8, pp. 1374-1393.

Sokolov, A. (2007), “Method of critical technologies”, Foresight-Russia, Vol. 1 No. 4, pp. 64-74.

Sokolov, A. (2009), “Future of S&T: delphi survey results”, Foresight-Russia, Vol. 3 No. 3, pp. 40-58.

Sokolov, A., Shashnov, S., Kotsemir, M. and Grebenyuk, A. (2017), “Identification of priorities for S&T cooperation of BRICS countries”, International Organisations Research Journal, Vol. 12 No. 4, pp. 32-67.

Sokolova, A. (2015), “An integrated approach for the evaluation of national foresight: the russian case”, Technological Forecasting & Social Change, Vol. 101, pp. 216-225.

France – Technologies clés (2010, 2006), Les Editions de L’Industrie, Ministère de l’Economie, des Finances et de l’Industrie, Direction Générale des Entreprises, Paris.

Supplementary materials

FS_20_5.pdf (14.4 MB)


The paper was prepared within the framework of the Basic Research Program at the National Research University Higher School of Economics (HSE) and supported within the framework of a subsidy by the Russian Academic Excellence Project “5-100”.

Corresponding author

Alexander Sokolov can be contacted at:

About the authors

Anna Sokolova is based at Institute for Statistical Studies and Economics of Knowledge, National Research University Higher School of Economics, Moscow, Russian Federation.

Anna Grebenyuk is based at Institute for Statistical Studies and Economics of Knowledge, National Research University Higher School of Economics, Moscow, Russian Federation.

Alexander Sokolov is based at Institute for Statistical Studies and Economics of Knowledge, National Research University Higher School of Economics, Moscow, Russian Federation.