Towards a robust approach for evidencing quality in higher education learning: The new CALOHEE model

Abstract

Purpose

Key to transnational higher education (HE) cooperation is building trust to allow for seamless recognition of studies. Building on the Tuning Educational Structures initiative (2001) and lessons learnt from the Organisation for Economic Cooperation and Development (OECD)-Assessment of Learning Outcomes in Higher Education (AHELO) feasibility study, this paper offers a sophisticated approach developed by the European Union (EU)-co-financed project Measuring and Comparing Achievements of Learning Outcomes in Europe (CALOHEE). These evidence the quality and relevance of learning by applying transparent and reliable indicators at the overarching and disciplinary levels. The model results allow for transnational diagnostic assessments to identify the strength and weaknesses of degree programmes.

Design/methodology/approach

The materials presented have been developed from 2016 to 2023, applying a bottom-up approach involving approximately 150 academics from 20+ European countries, reflecting the full spectrum of academic fields. Based on intensive face-to-face debate and consultation of stakeholders and anchored in academic literature and wide experience.

Findings

As a result, general (overarching) state-of-the-art reference frameworks have been prepared for the associated degree, bachelor, master and doctorate, as well as aligned qualifications reference frameworks and more detailed learning outcomes/assessment frameworks for 11 subject areas, offering a sound basis for quality assurance. As a follow-up, actual assessment formats for five academic fields have been developed to allow for measuring the actual level of learning at the institutional level from a comparative perspective.

Originality/value

Frameworks as well as assessment models and items are highly innovative, content-wise as in the strategy of development, involving renown academics finding common ground. Its value is not limited to Europe but has global significance. The model developed, is also relevant for micro-credentials in defining levels of mastery.

Citation

Wagenaar, R. (2024), "Towards a robust approach for evidencing quality in higher education learning: The new CALOHEE model", Journal of International Cooperation in Education, Vol. ahead-of-print No. ahead-of-print. https://doi.org/10.1108/JICE-11-2023-0038

Introduction

Trust building and recognition have been core pillars in European higher education (HE) since the 1990s, initially evidenced in the development of the European Credit Transfer System (ECTS) and the Lisbon Recognition Convention. The Bologna Declaration, signed by 27 European countries in 1999, starting the Bologna Process, resulted in additional initiatives. Two stand out: the development of qualifications frameworks at European, national and degree programme level and the Standards and Guidelines for Quality Assurance in the European Higher Education Area (EHEA), in short ESG. All of these European ‘tools’ served as inspiration for other world regions. In 2005 the Ministers of Education endorsed the Framework of Qualifications for the EHEA (FQ for the EHEA) and the ESG at their Bologna Follow-up Conference held in Bergen, Norway. This framework is meant to serve as a general reference for meeting European quality standards of all cycles of HE learning.

In 2004 the European Commission (EC) published a new ECTS Users’ Guide, reflecting the extension of a transfer system towards a transfer and accumulation system (EC, 2004). This new Guide had been prepared by the project Tuning Educational Structures in Europe, initiated in 2000, in close cooperation with the European University Association (EUA). The Tuning initiative enjoyed strong political and financial support (2001–2008 2.5M euro) from the EC. Besides revising ECTS, the project also developed a model to reform HE degree programmes and it defined reference points for disciplinary fields to meet Europe wide agreed standards (González & Wagenaar, 2008). These reference point documents equalled the benchmark papers prepared by the Quality Assurance Agency (QAA) UK. This was confirmed in the Transnational European Evaluation Project (TEEP project) co-financed by the EC (ENQA, 2004). Tuning introduced Europe wide, the ‘paradigm change’ of outcome based/student-centred learning, based on the concepts of generic and subject-specific competences and learning outcomes. Learning outcomes being defined as level of competence (to be) achieved as a result of the learning process. Applying the term ‘competences’ (being competent) as used in the labour market.

In 2008 the European Union (EU) established its European Qualifications Framework for Lifelong Learning (EQF) encompassing all learning. This EQF has since competed with the FQ for the EHEA, endorsed by the signatory countries of the Bologna Declaration. Although, it has been claimed that both overarching European Frameworks are compatible, in reality they differ in design and purpose: the first focussing foremost on the educational process and the second on the outcomes of the process relating in particular to the needs of the labour market. As a next step for both frameworks national ones were developed based on country-specific features. In the context of the EQF also sectoral frameworks were defined. Tuning developed those for the Humanities and the Social Sciences (Tuning Educational Structures in Europe, 2010, 2012). All these initiatives, being intended to assure comparable and compatible quality of learning in the countries comprising the EHEA, to allow for automatic recognition of learning in (inter)national context.

Although the European tools indicated above, offered a good reference what to expect from an associated degree, a bachelor, a master and a doctorate, they did not evidence - in comparative perspective - the level of learning achieved. In addition to the frameworks, the European Standards and Guidelines for Quality Assurance were introduced as an indicator for meeting quality standards. Although highly useful, in practice the ESG is very much process steered and fully depending on peer interpretation of what quality implies. It was the OECD, having the experience of the Programme for International Student Assessment (PISA) tests for secondary education, that took the bold step to launch the Assessment of Higher Education Learning Outcome (AHELO) feasibility study. Its leading idea: quality can best be evidenced by transnational comparative assessments at system level (comparing countries performance). Although, not perceived as a success, it inspired further activities, which are outlined in this contribution.

This paper raises the question whether existing instruments are (still) sufficiently adequate to evidence not only the quality of learning but also its relevance given present-day societal challenges and the revolutionary effects of information technology affecting all parts of everyday life. In response, it also offers approaches and models forward, which are meant to be state-of-the-art and forward-looking reflecting new opportunities and challenges for alignment and trust building in the EHEA and beyond. Can quality of learning be assured and/or measured in transnational perspective and is this useful? If so, what is conditioned to make this possible?

Assessment of Higher Education Learning Outcomes (AHELO)

In 2012 the OECD published two huge volumes on the outcomes of the AHELO feasibility study (Tremblay, Lalancette, & Roseveare, 2012). Before the third volume was made public, the OECD organised a meeting of experts at its headquarters in Paris (OECD, 2013). The reception of the study proved to be rather critical. It did not offer the outcomes the OECD had hoped for and seem to confirm the opinions of its critics (Shahjahan & Torres, 2013; Ashwin, 2015).

The study was implemented in the years 2008–2012 and was based on three strands: (1) an international comparative test of generic competences – in practice focussing on critical thinking – and (2 and 3) the disciplinary fields of civil engineering and economics. Seventeen countries and three USA states signed up to participate in one or more strands. The study was financed by country contributions and Foundations. The EU did not contribute to the study. The study suffered from its expensive design and lack of funding. It was content wise based on successive steps. The first one was the development of qualifications frameworks of the two disciplinary fields, for which the Tuning initiative was made responsible. It involved academic experts from different world regions. A consortium of consultancy firms was selected to make the next steps: (1) the application of these frameworks as a foundation for making assessment frameworks, (2) the development of actual sets of test items and (3) finally the implementation of the tests in practice. For the generic assessment an existing USA standardised test, the Collegiate Learning Assessment (CLA), was selected, which was not globalised for purpose in terms of its topics and design for the AHELO feasibility study. As a result of educational and cultural differences the test did not allow for reliable and fair comparison. The disciplinary tests offered more useful outcomes, although also had fundamental flaws to be useful for identifying strength and weaknesses at system level. First of all, the summative assessments required individual teachers to assess the examination papers. Although, the pool of assessors was trained, this could not fully overcome cultural differences in judgements. Fully computerised tests would have avoided these challenges, but the fear was that such formats involved too much standardisation of testing, which was perceived as an abomination amongst in particular many USA academics.

A more fundamental problem was that the assessments were based on measuring disciplinary knowledge, not taking high-level subject specific and generic competences into account. It implied that in a world of internationalisation and a rapid development of information technology it could not do justice (anymore) to current developments and societal needs.

In retrospective, it seems fair to conclude that the idea and implementation of global comparative testing was insufficiently thought out at the time, relating to scope, methodology but most of all type of competences measured. AHELO was implemented in the years when the countries composing the EHEA embraced the change from expert-driven education to output based or student-centred education and active learning; promoted in the UK and by the Tuning Projects from 2001 and launched in 1995 in the USA (Barr & Tagg, 1995) as the new way forward and included in the Louvain-la-Neuve/Leuven Communique of the ministers of education of 2009.

Although, AHELO was clearly not a success, the OECD did not give up the idea of comparative international testing at system level. In 2015 it announced planning to roll out a full fledge scheme (OECD, 2015). The initiative obtained fear criticisms from experts (American Council on Education ACE and Universities Canada, 2015), summarised in a short article by Philip G. Altbach and proved not able to find the donors for financing such an initiative. His main arguments against a full-scale implementation of the AHELO model related to major differences between national secondary educational models, the high variety in interpreting competences such as communication and critical thinking and the significant differences at tertiary level in implementing curricula and courses. In his wording, it would be difficult to identify the ‘gold standard’ in different disciplines between countries. AHELO would be testing apples and oranges (Altbach, 2015).

As a result, the OECD was forced to limit its ambitions to building a test focussing on critical thinking only, called the “CLA + International Initiative”, involving 6 countries. The outcomes of the study were published in 2022 (Van Damme & Zahner, 2022). In addition, it launched the project Fostering and assessing students' creative and critical thinking skills in HE aiming to ‘identify the key contextual factors and effective approaches to foster these skills in HE settings, develop and implement exemplary instructional practices and assess the effects of innovative pedagogies on students and faculty members’. The project involves 25 universities from 13 countries, results planned to be published in 2023 (OECD, 2019).

The only initiative at disciplinary level building on the AHELO findings and known to the author is the one of the National Institute of Education Research (NIER) of the Japan Ministry of Education in the ‘Development of a Mechanical Engineering Test Item Bank to promote learning outcomes-based education in Japanese and Indonesian higher education institutions’. Its outcomes have been published in the Tuning Journal for Higher Education (Cross et al., 2017). Another - national - example, is the Netherlands ‘interuniversitaire Voortgangstoets Geneeskunde (iVTG), (n.d.) (Interuniversity Progress Test for Medical Studies) which was introduced at national level in the 1990s. At present, each test is taken by approximately 10.000 students. It obtained international attention and imitation (iVTG; Tio et al., 2016; Garcia Reberti et al., 2019). However, both seem to be rather exceptional examples of interuniversity, e.g. transnational testing, but at the same time they indicate possibilities and usefulness.

Why taking new initiatives?

As part of the Bologna Process and the development of EHEA, three key commitments were defined: comparable degree structure, an agreed system for quality assurance and recognition of studies. The EC Bologna Implementation reports of 2012 and 2015 (European Commission/EACEA/Eurydice, 2012, 2015) as well as reports of the EUA (Sursock, 2015) and the European Student Union showed that disappointing progress was made regarding the aims pursued (ESU, 2012, 2015). Making the paradigm change from expert-driven education to student-centred and active learning, introduced in Europe in 2001 by the Tuning initiative and embraced by the European ministers in their Bologna Louvain-la-Neuve/Leuven Communiqué of 2009, which was thought key for meeting the key commitments, proved to be a serious challenge for many national systems and HE institutions. In theory accepted, but not very well implemented.

The main reason identified was the absence of staff training and development of HE academic staff, already stipulated in 1998 by United Nations Educational, Scientific and Cultural Organisation (UNESCO) as a crucial factor (Fielden, 1998). This was confirmed by a study implemented by the International Tuning Academy, established in 2011 at Deusto University and one year later at the University of Groningen (Birtwistle, Courtney, & Robert, 2016). The Academy is the natural result of the many Tuning projects initiated and implemented around the world by the two universities mentioned. How could modernisation of strategies and approaches of teaching, learning and assessment take place when the vast majority of teachers were operating in a vacuum, lacking theoretical and practical knowledge and skills about what the profession of an academic teacher encompasses? It might be the one of few high-level professions in the world that does not require an evidenced set of competencies to operate successfully: ‘flying a plane with the experience of a passenger’.

In the Paris Communiqué of 2018 for the very first time an explicit reference was included concerning the need for teacher training of academic staff: “We will promote and support institutional, national and European initiatives for pedagogical training, continuous professional development of HE teachers and explore ways for better recognition of high quality and innovative teaching in their career”. It builds on a phrase included in the Yerevan Communiqué of 2015: ‘We will encourage and support HE institutions and staff in promoting pedagogical innovation in student-centred learning environments’. Too late, too little, because in practice, these rather vague promises were not turned into serious action in the vast majority of European countries in the years to come.

At European level the situation outlined, was taken more seriously, when preparing the next Bologna Ministerial Conference to take place in Rome in 2020. Its coordination group, the Bologna Follow-up Group of national representatives, established an Advisory Group on Learning and Teaching which prepared the report ‘Recommendations to National Authorities for the Enhancement of Higher Education Learning and Teaching in the EHEA’, outlining in detail steps to be made. The report was adopted by the ministers as an integral part of the Rome Communiqué (Rome Ministerial Communiqué, 2020).

It was one initiative of several to boost the Bologna aims and objectives, in which the EC set the tune. In the context of the European Community Action Scheme for the Mobility of University Students (ERASMUS) + Programme it introduced the Erasmus Charter for Higher Education (ECHE) in 2014 and renewed in 2021. It provides the general quality framework (a set of rules) for European and international cooperation activities HE institution may carry out within Erasmus+. Its award made a pre-requisite to obtain EU funding. In July 2023, 5646 HE institutions in Europe had obtained this quality label (ERASMUS Charter, 2021).

Triggered and inspired by the French president Emmanuel Macron’s speech held on 26 September 2017 entitled ‘Towards a European University’, the EC developed its flagship initiative European Universities to further greater collaboration between HE institutions within the EU, meant to improve the competitive position of the European HE sector in global context and to contribute to innovation. In the wording of the EC: “European Universities are transnational alliances that will become the universities of the future, promoting European values and identity, and revolutionising the quality and competitiveness of European higher education.” The initiative has been structured in such a way that universities are expected to fully align with the Bologna Process aims and objectives and assist each other to get the job done by peer learning and by conditioning that alliances are composed of universities well spread over the EU.

The initiative is one out of four flagships to boost the European dimension in HE (European Universities, legal statute for European alliances, joint European Degree and the European Student Card Initiative) that serves as the basis for the Communication from the Commission on a European Strategy for Universities published in January 2022. This policy paper from the Commission sets out a vital role for HE, formulated in four key objectives: (1) to strengthen the European aspect of education and research; (2) to position HE institutions as beacons of the European way of life; (3) to strengthen their role as actors in efforts towards the twin green and digital transition; and (4) reinforcing their leading role within the EU as a world player (EC, 2022).

All these initiatives can be perceived as being based on a top-down approach, although bringing in particularly HE management into play. Addressing directly universities’ management have been a target of the EUA, EURASHE (Universities of Applied Sciences) and the European Student Union. All these organisation prepared studies based on stakeholder involvement. Having done terrific and necessary work, however, they have not really been able to reach the individual academic teacher and learner. National Rectors’ Conferences did not do much better. The point has been made that ‘change’ in HE will not take place when there is not full involvement of all stakeholders (Wagenaar, 2018-1; Wagenaar, 2022).

From its start the Tuning Initiative had applied a bottom-up approach, based on the philosophy of a multi-level governance approach; each decision level having its own responsibilities, but with a key role for practitioners, that is teams of academics plus students being the key components of learning. So far Tuning had limited its role to developing a methodology for reforming HE programmes and developing reference points or indicators to define the quality of degree programmes. Because, the change of paradigm of learning was taken place so slowly, it might be time to confront HE stakeholders with the limitations of present-day degree programmes in terms of societal needs. In 2001/2002 Tuning consultations had showed the disconnect between the expectations of society expressed by graduates and employers and the actual learning reflected by academic teachers. Over time, this gap has been largely bridged, which proved that the strategy of rising awareness – supported by offering a toolbox – did have an effect. In 2001 the concept of generic competences or transversal skills was not (really) applied in HE, today it is common practice and no longer challenged as it was at the beginning of the century. It was thought to be time for a new bold initiative, to be taken by informed and experienced academics.

Measuring and Comparing Achievements of learning outcomes in HE in Europe

Although the EC had not co-financed the AHELO feasibility study, this did not mean it had a negative attitude towards its basic idea. In the years immediately following the rapping up of the OECD initiative, discussions took place between representatives of DG Education and Culture, Tuning and Educational Testing Service (ETS). ETS had been a partner in the AHELO Consortium with a limited role. The basic arguments for having the discussions were that it had become common practice in the world to define and agree on qualifications (reference) frameworks, both at regional and national level. Frameworks that were not only inspired but also largely based on European initiatives; both at the overarching level and as a result of the Tuning initiative at the subject area level. They were and still are perceived as important means to formulate in statements/indicators intended and achieved learning. These combined with instruments for quality assurance being crucial for trust building and - as implication – (automatic) recognition of degrees and period of studies, offered a sound foundation for evidencing the quality of learning in comparative and international perspective. So, it seems highly logical if common ground could be defined, this should also allow for evidencing level of performance. Transnational comparative transnational diagnostic tests would according to the same logic allow for identifying strength and weaknesses but also shortcomings in degree programmes.

One can make the argument that HE in the world is mainly based on four educational philosophies developed in the 19^th and first half of the 20^th century in Europe, that is the Anglo-Saxon, Napoleonic, Humboldtian and Soviet models (Sam & Van der Sijde, 2014), allowing for global comparison, any initiative could also be limited to Europe offering comparable insights. This would make a new initiative more feasible. To be of value and also in response to earlier expressed criticisms, it was thought key for any initiative that it should focus on high-level competences and skills development, founded in an academic field. In addition, the approach should be forward-looking, making the outcomes relevant for the reform of the higher sector in general as well as its individual HE programmes.

As has been outlined in other publications this reflection resulted in the Erasmus + Forward-Looking Cooperation projects ‘Measuring and Comparing Achievements of Learning Outcomes in Higher Education in Europe’, in short CALOHEE (Wagenaar, 2018-2; Wagenaar, 2021). Projects, because it identified three steps with two initial steps before any comparative testing would makes sense. The first step is to identify common ground by defining sophisticated new reference frameworks at the disciplinary level based on a combination of the two existing European overarching qualifications frameworks. These frameworks - to be distinguished in Subject Area Qualifications Reference Frameworks and more detailed Assessment Reference Frameworks - should offer a robust basis for formulating programme learning outcomes and therefore (re)designing degree programmes. As the second step was foreseen, the development of diagnostic transnational comparative test formats plus items. Only the third step would be the actual testing of students, preferably at the end of the first cycle/bachelor studies.

In the period 2016–2023, frameworks have been prepared according to the model described for now eleven disciplinary fields, involving some 150 renown academics from 20+ countries, covering in practice all academic sectors, reaching from creative, performing arts and design to civil engineering. In a follow-up project, for five of these fields test formats and actual assessment examples were developed, that is civil engineering, history, nursing, physics and teacher education. Each of these steps having a value in itself for the modernisation of HE. The steps reflect progression in level of international alignment.

The choice for rolling out these projects in Europe was supported by the fact that in the period 1987 - the launch of the ERASMUS mobility programme - until 2020 nearly 12 million students and staff experienced a period of learning abroad (Erasmus for All partnership, 2023). No other world region can boost a comparable level of interaction. In addition, there was the aim - formulated in the Bologna Declaration of 1999 - to make HE programmes compatible and comparable, to result in an EHEA. To put the number of international students in context, in 2022 17.5 million students were enrolled at an EU university (European Education Area website, 2023).

As a direct outcome of its many bottom-up projects and initiatives - involving some thousand academics from tens of countries in Europe and close to 130 in the world -, Tuning was in a good position to reflect on best strategies and practices to align HE (programmes) further. Basing itself on deep insights and understanding, allowing for identifying communalities and differences as a result of intensive multi-national and cultural discourse by groups of senior academics and informed student representatives. These debates made transparent variety of missions, profiles, aspirations, orientations, etc. which should be taken into account. A strength for any new initiative proved to be also that many ‘Tuners’ became the best informed and experienced experts in the field, having a very international profile.

But there was more, that put the CALOHEE feasibility study in a far better position to evidence the quality, level and relevance of learning than the OECD-AHELO initiative. The role of HE is to prepare its students for a (leading) role in society. This implies that eventually graduates should not only be knowledgeable, but also skilled and be able to take responsibility for their actions and act autonomously. This is nowadays far better acknowledged than in the period AHELO was implemented. It is not by accident, that the OECD since the AHELO study has put emphasis on the importance of high-level skills and competences, critical and innovative thinking and creativity. One might add to this list intra- and entrepreneurship, stipulated by the EC as a consequence of a more flexible labour market. As a result of technical innovations, the role and importance of ‘knowledge ownership’ has been diminished, whilst positioning and analysing new knowledge and developing insight and understanding has grown in importance, requiring the high-level skills mentioned. Knowledge can easily be accessed on one’s smart phone; interpretation of the reliability of this knowledge requires skills and a critical mindset to be developed in an academic study.

Having all this in mind, resulted in the new CALOHEE model, intended to develop a state-of-the-art approach to not only allow for comparing and measuring learning, but also to define what should be learnt in present times to be relevant for society. At the same time the new model should solve a number of anomalies in the existing frameworks and approaches. Whilst other world regions developed one regional qualifications framework and/or national qualifications frameworks in Europe two overarching (competing) frameworks were developed, the already mentioned FQ for the EHEA, endorsed by the Bologna signatory countries and the EU EQF, as indicated, differentiating in design and purpose. The vast majority of European countries developed national qualification frameworks related to each of these – a rather confusing situation, in particular for practitioners. To add to this confusion, from 2008 the Tuning project published its Reference Points for the Design and Delivery of Degree Programme which applied its own approach and features again, in particularly by introducing, as explained, the concepts of generic and subject-specific competences as a basis for formulation of learning outcomes reflecting levels of intended and/or achieved learning.

Subject area qualifications reference frameworks: methodology applied

For defining the subject area-based Qualifications Reference Frameworks and related Assessment Reference Frameworks working groups were established of approximately 13 to 14 members, including one student representative. Using an open call, universities were invited to show their interest and to propose an informed expert for participation who should meet defined criteria included in the call. The European Student Union (ESU) was asked to suggest student members. It also applied a criteria-based call. The initial selection was done by highly experienced ‘Tuners’ and endorsed by an Advisory Board. These working groups, composed as a result, developed - on the basis of intense face-to-face debate - not only the frameworks but also collected and prepared examples of good practice of aligned teaching, learning and assessment approaches.

In short, to construct the CALOHEE tables of reference descriptors, a two-step approach was applied. The groups were asked first to identify so-called key dimensions, i.e. key components of a degree programme in one’s academic field to be one to one related to the descriptors of the FQ for the EHEA. The outcome of this step was a number of dimensions varying from five to eight depending on the academic field. The second step was to use the dimensions as the basis to organise and define the CALOHEE descriptors, making a distinction in knowledge, skills and autonomy and responsibility related ones by following the EQF model. See the example of the bachelor showed below. Of course, it was well understood that knowledge and skills are learnt in conjunction and are building upon one another. In other words, ‘existing knowledge’ - in terms of knowing and understanding – is deepened by practicing, applying subject-specific and generic skills. Given their construction, the tables offer a sound foundation for not only making the progression of levels of learning much more explicit but also allow for inclusion of current societal developments and challenges in each of the degree cycles. The outcome is Qualifications Reference Frameworks which express in much more detail what a learner is expected to learn today and tomorrow, to be relevant for society.

For all disciplinary fields included in the project more detailed Assessment or Learning Outcomes Reference Frameworks were made to break down the dimensions and descriptors in sub-dimensions and sub-descriptors. These were used to identify the most appropriate set of strategies, methods and approaches for aligned teaching, learning and assessment. Resulting in a full toolbox to design and deliver degree programmes. Draft tables were discussed with peers and representatives of professional organisations in case appropriate, to obtain feedback and to finalise the materials. The full set of tables can be downloaded from the CALOHEE website (CALOHEE website, n.d.).

Matching existing degree programmes and the frameworks: main findings

As an integral part of the CALOHEE projects one of the aims was to find out the state of affairs regarding the level in which the intended learning described in the frameworks was actually covered. Taking into account that the descriptors included in the CALOHEE tables are not only aspirational but also inspirational and that many programmes in the matching exercise predate the commonly agreed frameworks. They reflect both current intended learning outcomes but are also forward-looking. What to expect from a graduate in five years’ time? Both Bachelor and Master degrees were analysed. The exercise had a double aim. First to find out which degree learning outcomes could be identified and at which level, taking into account that programmes – based on their profiles – would offer more attention and pursued a higher level of learning for a particular (sub)dimension than another one. The second objective was to have a fresh look at the frameworks based on the findings which might as a result require adjustment. Civil Engineering, History and Nursing matched 8 EQF level 6 and 7 programmes (Bachelor and Master), Physics 5 of each from as many countries, except for History where 7 countries were represented and Teacher Education 15 first-cycle, 13 second-cycle and 2 long-cycle programmes covering together 19 countries. Although this number seems to be low to draw robust conclusions, it has to be understood that the programmes are a good representation of country situations. This made the exercise useful. The same exercise was executed in 2023 for an additional six academic fields.

It is fair to conclude that for each of the academic fields the matching exercise shows a variety of outcomes. These are outlined in the reports uploaded on the CALOHEE website (CALOHEE website, n.d.). However, more general conclusions can be drawn based on a critical analysis and overall findings. CALOHEE Reference Frameworks.

1. are able to account for a variety of HE programmes that exist in each subject area in Europe;

2. appear indeed to be instrumental for promoting relevance, recognition and quality of HE programmes across countries and types of programmes;

3. need to remain ‘living means’ in order to serve as benchmarks for determining and fostering relevance in HE programmes.

The exercise also shows that not for all degrees in Europe programme learning outcomes have been formulated. Where they exist, there is substantial variety in detail and use. In more than one country the course unit learning outcomes had to be analysed as part of the exercise. However, applying the descriptors as included in the qualifications reference frameworks will help to obtain more consistency in the way programme learning outcomes are formulated which will – in effect - facilitate the comparability and compatibility of intended and achieved learning and as a result trust building and recognition as well as the quality of learning.

Finally, it can be concluded that the matching exercise shows that international comparative assessment is (theoretical) possible. However, it has been highlighted that such assessment should be flexible and adaptable enough to account for a great variety and combination of elements to be assessed to allow for real and fair comparability of learning (CALOHEE, 2023-1).

Dissolving a loose end: state-of-the-art General Qualifications Reference Frameworks

When preparing the transnational comparative assessments and additional Qualifications Reference Frameworks for five individual academic fields from 2021–2023, it was noticed that the present key Bologna tools had aged and were no longer really fit for purpose, it was concluded that it would be very helpful to define also overarching Qualifications Reference Frameworks for all cycles, besides the Subject Area Qualifications Frameworks and Assessment Reference Frameworks developed for the academic fields of civil engineering, history, nursing, physics and teacher education from 2015–2018 and as a follow-up for the academic fields of business administration, creative and performing arts, medicine, information and communication technology (ICT) and international relations in the period 2021–2023.

It would offer, in result, consistency and a robust backbone for the subject area ones. For defining these overarching frameworks, a task force was established with representatives of the five working groups involved in developing the assessment formats. The initial outcomes of its work were discussed by all eleven CALOHEE working groups covering representatives from a very wide range of European countries. The key question to be answered: are they doing justice to your discipline? Obtaining this confirmation meant they are relevant and fair for all academic sectors and fields.

When constructing the frameworks, the merger approach was applied as in the case of the subject area ones, taking into account that these would not be meant to compete with or replacing the existing overarching frameworks, but only supplementing these with a state-of-the-art information. This approach avoided a complicated debate with the many HE stakeholders.

In more detail, the model is based on two legs, the vertical outlining the 5 or 6 dimensions/descriptors of the FQ of the EHEA and a horizontal, involving the three EQF descriptors. The basic philosophy applied is that in all learning there is always a knowledge/knowing/understanding part, a skills/application part and an autonomy and responsibility part, which also involves ‘attitude’. These three parts organise and define the progression of learning. ‘Knowledge’ being perceived as the foundation and ‘autonomy and responsibility’, expressing authority, being the most ambitious. When constructing the model there was full awareness that the concept of ‘knowledge’ has a variation of connotations in different educational cultures, national contexts and subject areas, such as (1) factual knowledge, (2) conceptual knowledge, (3) procedural knowledge and (4) metacognitive knowledge (Krathwohl, 2002) and many more according to scholarly literature. ‘Skills’ are distinguished in ‘subject specific’ or ‘technical’ and general/generic ones. As stated before, being fully aware that in real life, these are developed together, although always starting with knowledge, for each of these categories a verb’ was selected, respectively ‘demonstrate’, ‘evidence’ and ‘manifest’, which intend to do justice to the activities involved, but also reflect progression in evidencing personalised learning. The verb ‘demonstrate’ allows well for assessing ‘knowledge’ to be understood as ‘knowledge and understanding’, whilst ‘evidence’ is used for knowledge and skills application because it requires a measurable product prepared by the learner. ‘Manifest’ is related to personal competences, such as attitude, drive, motivation, action, leadership and the like, in workplace and societal contexts (CALOHEE, 2023-2).

The model resulting, is tailored to link sub-levels within cycles/EQF levels to course units and modules. These are instrumental to define pre-requisites and co-requisites for a learning unit or module to ensure readiness to undertake the forthcoming learning. Also, regarding the recognition of studies these offer transparency about the intended and achieved learning outcomes, that is the level of competence. For positioning electives, minors and most of all micro-credentials, that seems rather important. In addition, the model solves a robust response to the experienced problem that the present qualifications frameworks are not able to distinguish level of learning and outcomes in an Associated Degree of approximately 2 years, a Bachelor of 3 to 4 years, a Master of 1 to 2 years and a Doctorate/PhD programme of 3 to 4 years.

The structure allows not only for reflecting re-thinking the role of HE, but also for integrating new societal developments by integrating information collected from deep discussions as part of the CALOHEE exercise about five topical issues: multicultural society, political awareness, ethics and values, information society and sustainable society/ies/climate change integrating UN Sustainability Goals in all educational programmes. The identification of these items is based on the Rome Communiqué and the initiatives taken by the EC (see Table 1).

To offer insight how the General Tuning-CALOHEE Qualifications Reference Framework looks like, the Table for the Bachelor is presented here (Table by CALOHEE projects/author).

At present, these new easy to read state-of-the-art CALOHEE frameworks are disseminated amongst the HE stakeholders and discussed with key players, such as the quality assurance and accreditation organisations in Europe. This is thought important because the quality assurance mechanisms as reflected in the ESG are content wise depending on overarching European and national qualifications frameworks. Therefore, as part of the project, a Guideline for Applying Tuning-CALOHEE Qualifications Reference Frameworks for Higher Education in Europe the European Standards and Guidelines for Quality Assurance has been prepared (CALOHEE, 2023-3).

Together with quality instruments developed earlier by the Tuning Initiative, e.g. the Tuning Dynamic Quality Assurance Cycle, based on W. Edwards Deming Plan-Do-Check-Act / Plan-Do-Study-Act (PDCA/PDSA) cycle, it will help to ensure state-of-the-art, relevant and high-quality degree programmes meeting international standards key for alignment and building trust (Wagenaar, 2019, pp. 239–240).

Transnational comparative assessments in European HE: main findings

The Subject Area Qualifications Reference Frameworks and Assessment Reference Frameworks, supported by the General Qualifications Reference Frameworks outlined above, were used by five academic fields – Civil Engineering, History, Nursing, Teacher Education, Physics - to prepare assessment formats and actual test items to allow for measuring high-level competences and skills. For that purpose, the working groups were limited to 7 to 8 of the most active members of the initial working groups of 13 to 14 members. The others acted as consulting and peer references. The following procedure was implemented by all five disciplinary working groups:

1. Check whether the frameworks published in 2018 need adjustment.

2. Identify the (sub) descriptors best suitable for developing transnational comparative assessments

3. Identify/develop the most appropriate mode(s) of assessment and decide on its feasibility. Describe and document the choices made.

4. Identify for each of the items selected, the modalities for assessment:

   • teaching and learning required

   • best strategies for assessment

   • criteria for assessment (rubric)

   • document the rational for selecting a particular competence or combination of competences

   • describe the actual test

5. Ensure that the set of assessments developed reflects key parts of the descriptors as included in the assessment reference table and show there is variety of assessment formats.

Based on this procedure, the working groups prepared a report of findings. The key conclusion of all groups: transnational comparative assessments are feasible and have added value, but the groups also explain that the assignment had been complex. In particular, to find common ground ‘what to test’. The exercise proved to be easier for academic fields that had less to deal with national policies, i.e. physics and civil engineering. Rules and regulations proved to be challenging for both nursing and teacher education. For history common ground could not be found in content, given the endless variety of topics – national, regional, world history – chronological, that is ancient, medieval, modern and contemporary, and approaches gender, cultural, social-economic, political history, etc. All of these having their own features and skills sets. This implied that a higher level of abstraction had to be identified in the ‘historical mind-set’, developed on the basis of shared methodological and theoretical frameworks, by focussing on critical thinking, handling complex information and making judgement. This paper only allows for a short summary of findings. The complete reports of the working groups have been uploaded on the CALOHEE website.

Physics started with making an inventory of examples of tasks, approaches and criteria and analysed these in terms of their applicability in international context. It ordered the test in four categories: 1. Concept inventories and multiple-choice tests; 2. Tasks for evaluating ‘scientific abilities’; 3. Tasks for evaluating problem solving; 4. Assessments to integrate generic competences with physics core competences skills; 5. Assessments based on machine learning. The assessments presented are mainly based on capitalising and integrating best practices from literature and experts’ experience (Carli & Pantano, 2023).

Civil engineering noticed that it is important that degree programmes in Europe should adopt the CALOHEE or EUR-ACE framework and accreditation system standards to make comparative assessment feasible. Most programmes do not (yet) comply with the CALOHEE ones. As a foundation for developing/selecting assessments the working group identify the Alignment of Learning Outcomes and Assessment (ALOA) model using the web-tool TALOE (http://taloe.up.pt) developed for engineering courses. Each of the members identified one example of a sophisticated test already applied (Soeiro & Thomas, 2023).

Nursing applied a very systematic approach, stating of with formulating key principles to assure valid, reliable and timely assessments that allow for addressing the interrelated and complex nature of professional nursing practice. In international context it should be ascertained that relevant stakeholders would share a common understanding of the assessment task, its associated competences, learning outcomes, metrics/rubrics, content and performance standard. To develop assessments for each of the five dimensions of its framework it applied a six-step process: analysing Nursing competences, validating assessment tools, in-depth analysis of two competences within each dimension, scope of practice analysis and assessment literacy, development and final scenario development. This analysis resulted in an overview of examples of good practice to apply (Gobbi & Kaunonen, 2023).

The History group realised from the first moment it would have a difficult task to imagine a transnational assessment given the variety of focus points as outlined above. It found a solution by taking a holistic approach by raising the question ‘what it means to think, work and communicate like a historian’. Although only a small number of graduates nowadays become a professional historian, researcher, academic, what distinct history graduates form other graduates is their way of approaching, elaborating and communicating knowledge. Essential competences identified by the working group are:

1. Can the individual think chronologically? Contextually? Intersecting and reflecting on historical evidence? Gathering new evidence and relating it to existing knowledge and understanding in a new way?

2. Can the individual explain what they have accomplished? Can s/he communicate her/his conclusions in clear and convincing manner? Can he or she illustrate clearly and convincingly the premises, the activities carried out, the results and their significance?

3. Has the explanation been produced according to the narrative standards/styles appropriate for the discipline?

It was concluded that the most promising approach assessing these competences was to create ‘scenario’s or ‘situations’. Five examples were developed by the group which eliminated or at least attenuate the ‘national effect’ prominent in degree programmes. The scenarios produced are based on a common format: 1. description of a scenario, a context which could occur in the present in real life; 2. the scenario allows for adjusting to time, place and circumstances; 3. a role is assigned to the person/team being assessed; 4. tasks are defined to be accomplished as well as the target audience; 5. a rubric is provided for assessors with indicators related to the dimensions/sub-dimensions of the Assessment Reference Framework for History and descriptors of achievement (three levels: not yet passed, passed, passed with distinction) (Hálfdánarson, Isaacs, & Salvaterra, 2023).

Teacher Education was challenged by the variety in which the academic field is organised in European countries. It started it activities by making a detailed overview about which programmes were positioned at level 6 and which at level 7. It also constructed a table of comparison of assessment tasks in teacher education programmes offered by the HE institutions represented in the working group. The group followed an exploration process distinguishing three large phases: (1) choosing (sub)dimensions of the assessment reference framework, applying the outcomes of the matching exercise implemented (see below) to assure sufficient common ground; (2) collecting examples of assessment tasks and practices currently used to assess student development in these (sub)dimensions and (3) construction internationally applicable assessment tasks including an agreed assessment rubric of descriptors distinguishing three levels of achievement (González, 2023).

The working groups, basing their work on their assessment reference frameworks, all experienced that these frameworks needed adaptation, when turned into ‘real life’ assessments. Finally, all working groups concluded that technology development will impact future ways of assessment, but that further work is required to apply machine-based testing effectively. However, it is thought that this technology will be of great help in making international comparative assessments. The working groups identified a number of promising developments, both regarding academic field related knowledge and skills and high-level generic competences, ranging from sophisticated ‘progressive’ multiple-choice models, to the application and reflection on images and footage and scenario/simulations-based testing and game-based assessment (Buckley, Colosimo, Kantar, McCall, & Snow, 2021). As a result of the revolutionary development of artificial intelligence (AI) it will also be possible to analyse essay-based questions by computer in the near future. Something which was unthinkable in the days the AHELO study was implemented. However, the subject area groups concluded that further work needs to be established to tailor technology to the assessment formats and testing items to make these applicable in a transnational context and to allow for cost-effectiveness, technology development still being rather expensive. It is also obvious that the role of the academic keeps being a crucial one (OECD, 2021).

In conclusion

As has been highlighted in this paper many initiatives have been taken in Europe to align processes and further trust building with the overall aim in mind to enhance the quality of HE and the degree programmes offered to meet international standards. For this purpose, so-called Bologna tools were developed in the first decade of this century which deserves updating to reflect current societal challenges. Although the Bologna Declaration was signed a quarter of a century ago, (too) many countries are still struggling to meet the key Bologna commitments agreed, that is comparable degree structure, an agreed system for quality assurance and recognition of studies. It has been identified that lack of academic staff development and training is an important factor. In recent years, in particular the EC, but also associations representing universities and students, have taken a broad range of initiatives to promote change, in particular the implementation of the student-centred approach which is perceived as conditional for enhancing quality and relevance of the HE offered. The Erasmus Charter, but also the EC flagship programmes, in particular the European Universities Initiative, are meant as leverage for developing more consistency required to establish a credible EHEA.

Contrary to these initiatives, building on the AHELO feasibility study, the Tuning Educational Structure Initiative, choose the bottom-up approach again - as it had done at the beginning of the 21st Century -, by defining an infrastructure of frameworks and tests easy to apply. In addition to the more top-down approaches, these should help countries and HE institutions to make real progress now, having the models at hand. In its combination of initiatives, it might be hoped for that the urgency will be felt to align degree programmes much better to the (upcoming) needs of society to assure sustainability and prosperity. It is expected that the general and subject-specific reference frameworks will be of huge value here. They are meant to inspire and be exemplary for enhancement in terms of quality and relevance of the outcomes of the learning process. This is not different for the test formats that have been developed for five subject areas. Although, the actual testing based on these models have not taken place yet, they offer already a sound foundation for re-thinking present assessment models in Europe and beyond based on agreed common indicators of what might be expected to be learnt in the upcoming decade.

Table 1

TUNING-CALOHEE general qualifications reference framework for the first cycle

Publication: “New Opportunities and Challenges for the Alignment Process and Trust Building of Global Higher Education.”