Transforming e-participation: VR-dialogue – building and evaluating an AI-supported framework for next-gen VR-enabled e-participation research

Abstract

Purpose

The purpose of this study is to explore whether immersive virtual reality (VR) can complement e-participation and help alleviate some major obstacles that hinder effective communication and collaboration. Immersive virtual reality (VR) can complement e-participation and help alleviate some major obstacles hindering effective communication and collaboration. VR technologies boost discussion participants' sense of presence and immersion; however, studying emerging VR technologies for their applicability to e-participation is challenging because of the lack of affordable and accessible infrastructures. In this paper, the authors present a novel framework for analyzing serious social VR engagements in the context of e-participation.

Design/methodology/approach

The authors propose a novel approach for artificial intelligence (AI)-supported, data-driven analysis of group engagements in immersive VR environments as an enabler for next-gen e-participation research. The authors propose a machine-learning-based VR interactions log analytics infrastructure to identify behavioral patterns. This paper includes features engineering to classify VR collaboration scenarios in four simulated e-participation engagements and a quantitative evaluation of the proposed approach performance.

Findings

The authors link theoretical dimensions of e-participation online interactions with specific user-behavioral patterns in VR engagements. The AI-powered immersive VR analytics infrastructure demonstrated good performance in automatically classifying behavioral scenarios in simulated e-participation engagements and the authors showed novel insights into the importance of specific features to perform this classification. The authors argue that our framework can be extended with more features and can cover additional patterns to enable future e-participation immersive VR research.

Research limitations/implications

This research emphasizes technical means of supporting future e-participation research with a focus on immersive VR technologies as an enabler. This is the very first use-case for using this AI and data-driven infrastructure for real-time analytics in e-participation, and the authors plan to conduct more comprehensive studies using the same infrastructure.

Practical implications

The authors’ platform is ready to be used by researchers around the world. The authors have already received interest from researchers in the USA (Harvard University) and Israel and run collaborative online sessions.

Social implications

The authors enable easy cloud access and simultaneous research session hosting 24/7 anywhere in the world at a very limited cost to e-participation researchers.

Originality/value

To the best of the authors’ knowledge, this is the very first attempt at building a dedicated AI-driven VR analytics infrastructure to study online e-participation engagements.

Citation

Porwol, L., Garcia Pereira, A. and Dumas, C. (2023), "Transforming e-participation: VR-dialogue – building and evaluating an AI-supported framework for next-gen VR-enabled e-participation research", Transforming Government: People, Process and Policy, Vol. 17 No. 2, pp. 233-250. https://doi.org/10.1108/TG-12-2021-0205

1. Introduction

Literature defines e-participation as a technology-mediated dialogue between citizens and decision-makers (Sabo et al., 2008) that ensures dynamic interaction (Chadwick, 2003) while also introducing new, innovative channels for public participation (Van Dijk, 2000). The use of ICT showed to have a positive effect on e-participation (Khan and Krishnan, 2017; Hagen et al., 2015; Phang and Kankanhalli, 2008). Despite evident benefits, e-participation initiatives faced challenges with effective and constructive user engagement (Macintosh et al., 2009; Porwol et al., 2013). The efforts to address these issues with the late social media-powered e-participation exhibited limited success as largely polarized discussions faced low policy impact (Porwol and Ojo, 2017) and encountered challenges related to trust linked with specific modes of communication (Alharbi et al., 2015; Kim and Lee, 2012; Morgeson et al., 2010; Scherer and Wimmer, 2014; Tolbert and Mossberger, 2006; Vigoda-Gadot and Cohen, 2015). An important aspect is also the lack of translation between e-participation and the impact on policy (Bataineh and Abu-Shanab, 2016; Mustafa Kamal, 2009).

In our study, we focus on means of improving e-participation through better user engagement by technical means. E-participation researchers strive to identify and study factors that impact citizen engagement and interaction performance. Because of the volume of information to analyze, studying e-participation engagements has been largely focused on conceptual analysis, case studies and surveys or web statistics. In terms of conceptual analysis, researchers explored the factors of the sense of virtual community (Koh and Kim, 2014; Purarjomandlangrudi and Chen, 2020; Tsai et al., 2012) where citizens get involved using e-participation technologies, noting Koh and Kim distinguish three dimensions: membership, immersion and influence (Naranjo-Zolotov et al., 2019). Membership is the “feeling of belonging to a community” (Koh and Kim, 2014) and may have a strong impact on whether people will choose to use technologies that are being used by a community that is virtual (Cheung and Lee, 2012). Influence is “the degree to which a citizen perceives that (s)he can influence the other members in the e-participation community to share her/his view or goals” (Koh and Kim, 2014). In the context of e-participation, immersion is defined “as the state of flow the citizens may experience when using the system” (Koh and Kim, 2014). In their study, Naranjo-Zolotov et al. (2019) found all three of these constructs to be positive factors in the continued use of e-participation technologies by citizens. This understanding of immersion and membership is corroborated by Porwol et al. (2022) with a more communication-focused framework that is also applicable to immersive VR (Porwol and Ojo, 2021).

While the conceptual foundation for studying e-participation engagements is essential, it must be followed by corresponding qualitative and quantitative methods of analysis to achieve a better understanding of e-participation engagements. Medaglia (2012) argues that while surveys are the most common form used for e-participation research, the effectiveness of that approach is limited and action research is largely missing. On the other hand, considering more data-driven approaches, the web statistics for Web 2.0-based e-participation platforms became a popular source of quantitative data. Nevertheless, Grönlund (2011) points out that figures based on web statistics do not reflect well on the state of success of e-participation and can be artificially inflated.

The proliferation of artificial intelligence (AI) technologies allowed the e-participation researchers to reconsider data-driven approaches to e-participation analysis and go back to studying discussion content. Specifically, Teufl et al. (2009) used AI technologies to analyze e-participation messages. While that method is very efficient at dealing with large quantities of data, it deals only with the content of messages rather than actual participant behavior.

The methods discussed briefly are representative of the state-of-the-art in terms of classic e-participation research. Currently, the fast-changing technology space and progressing media convergence afford new types of communication that may prove transformational for the next-gen e-participation. Specifically, the emerging immersive social virtual reality (VR) platforms which are very distinct from traditional virtual environments offer new means of communication that can complement e-participation and help to overcome some of its online interactions' challenges (Porwol and Ojo, 2018a; Porwol and Ojo, 2018b). Through strong immersion and increased presence capabilities modern VR gets close to real experience (Loomis, 2016), which creates new opportunities for more trusted and more inclusive e-participation engagements. Despite positive premises for the use of VR in serious communication, there is a paucity of studies that would provide relevant frameworks and corresponding technical infrastructure to study immersive VR interactions from the perspective of applicability of the emerging technologies to e-participation. The next-gen e-participation requires new methods of investigation and our goal has been to investigate efficient and affordable ways of collecting VR engagement data and building an efficient, AI-supported data analytics pipeline. Our intention is to go beyond the communication content analysis (such as in classic e-participation studies) and study the actual behavior of participants as they engage in e-participation discussion.

Therefore, our main research questions include:

RQ1.

How can we analyze participant behavior in a next-gen immersive VR engagement mode of communication to provide a basis for VR-enabled e-participation?

RQ2.

How can AI-driven behavioral data analytics support live, automated feedback and advanced behavioral analytics for VR-enabled e-participation research?

The aim of our work is to better understand specific features of communication via immersive VR technologies. We present our approach to creating grassroots for future VR-meeting user experiments for serious communication use cases such as e-participation. We built a relevant data collection and AI-supported data processing infrastructure as a research platform for investigating different immersive VR online meeting scenarios and analysis of user interaction and collaboration patterns.

In section two, we discuss the background with relevant definitions along with an explanation for limiting our work to Immersive VR interactions. In section three, we describe our methodology and provide relevant acknowledgment of works and technical infrastructures used as a base for our studies. In section four, we describe our research infrastructure. In sections five and six, we present early evaluation results in several template behavioral scenarios. In section seven, we finish with discussion, conclusions and future work supporting next-gen e-participation research.

2. Background

Our focus has been to provide a relevant framework and corresponding technical infrastructure that would allow data-driven behavioral studies in immersive VR engagements for future e-participation studies. Because there is a paucity of articles on immersive VR and data-driven e-participation analysis, we had to turn to more general works. The research related to extracting social behavior data from online social virtual environment platforms dates back to the early 2000s with a focus on the once-popular online 3D (nonimmersive) social platform Second Life (SL). Friedman et al. (2007) created bots for SL to study spatial social behavior and found that users responded to bots by moving their avatars, thus suggesting the importance of proxemics, or the study of the space people need to put between them and others, in SL. La and Michiardi (2008) measured user mobility in SL by creating sensor tools and a crawler to collect position data finding that users gather around things of interest and do not move too far away from them. Additionally, using maximum likelihood estimation, La and Michiardi (2008) found the contact time with other users in the virtual world was similar to real-world observations. Varvello et al. (2008) study corroborated that observation as they created a crawler application to collect avatar behavior data in SL and found users in the virtual world interact in a similar manner as people in real life. These studies focused on social behavior in a virtual environment and were limited to collecting low-level data such as proximity and avatar gaze. Ullrich et al. (2008) integrated virtual agents in SL using Multimodal Presentation Language 3D. Similarly, Jan et al. (2009) created conversational and navigational agents in SL using the LIBUV library. The same library was leveraged by a study that explored collecting data and testing AI theories in SL (Ranathunga et al., 2012) and extracted high-level event data from the SL server such as animation, movement and messages.

In our work, we are focused only on human-to-human interactions, and we do not consider engagements via legacy virtual environments accessible through any type of standard display or monitor, laptop or mobile device (2D–3D), such as SL, as those experiences are not immersive and suffer from the “screen barrier” effect (Bricken, 1991). Bricken stresses the significant difference between viewing (on-screen) and inclusion (in immersive VR) by stating that in VR users interact directly with various information forms in an inclusive environment. The 2D–3D environments exhibit significantly lower levels of presence and immersion resulting in different participant behavior (Sanchez-vives and Slater, 2005) and therefore the studies conducted in those environments, while being an important base reference to our work are not applicable to next-gen VR. The base work for immersive VR e-participation engagements has been explored by Porwol and Ojo (2021) through a theoretical framework linking immersion and presence with more efficient dialogue for next-gen e-participation. Therefore, this work focuses only on immersive VR with simulated environments where participants are leveraging head-mounted displays (HMDs) with body tracking and dual controllers, offering a form of strong telepresence and copresence. In this setup, users are isolated from their surroundings in line with the definition of VR by Steuer (1992) and engage in collaborative discussions in virtual worlds as per the definition given by Bell (2008) presented as a synchronous, persistent network of people, represented as avatars, facilitated by networked computers. The newly emerged immersive VR solutions created a new opportunity to experiment widely with more advanced means of communication (Boas, 2013).

To date, limited direct data-driven research has been conducted on serious communication on fully immersive online VR platforms. To our knowledge, there is a paucity of data-driven studies tackling e-participation in immersive VR. The majority of related work in serious communication leverages the open-source platform, Mozilla Hubs, as the only off-the-shelf, technical infrastructure available to researchers of immersive VR. A study by Le et al. (2020) explored the potential for creating a “social environment” for participants to attend a conference remotely where the presentations were “live-streamed” on Mozilla Hubs. They collected demographic data from registration forms, logs from the Discord chat (Discord is a VoIP and instant messaging social platform), feedback questionnaires and qualitative data from observations and comments from the feedback surveys. Additionally, Le et al. (2020) conducted a virtual poster session and found that it improved the “social connectivity” between the participants. The attendees reported that the “overall experience” was “very satisfying” and observed an increased sense of presence (Le et al., 2020). Three studies involved student experiences of social VR using Mozilla Hubs in the context of COVID-19 and as an alternative to mainstream online communication tools. Holt et al. (2020) conducted virtual poster sessions with undergraduate ecology students presenting their final research projects in Mozilla Hubs. The students reflected that the benefits of presenting in the virtual environment were flexibility and less pressure and stress. Some students commented on how authentic the communication was using the avatars. Instructors said they felt a sense of community similar to what they would see in face-to-face poster sessions and saw improved engagement between the students. Yoshimura and Borst (2021) studied students’ experiences in remote lectures and presentations using Mozilla Hubs using both headsets and desktop computers. The surveys reported a higher sense of presence using the headset with positive scores, whereas desktop viewing was consistently negative. Gomes de Siqueira et al. (2021) used Mozilla Hubs to promote interaction and communication with students in a semester-long VR course at the University of Florida to help team formation. Data was collected by observation of and survey questionnaires to better understand design considerations of the rooms and learning activities and to promote interaction. Students preferred “high-detailed, closed spaces that resembled real life” (Gomes de Siqueira et al., 2021) and smaller rooms because they facilitated student interaction. Similar to the real world, students formed circles when talking in groups. All the studies elaborated here again focused on questionnaires and observations and did not use more advanced direct data extraction and analysis from the platform. In contrast, Williamson et al. (2021) created their own, extended instance of Mozilla Hubs to directly collect data, from infrastructure, on position measurement and orientation during workshop activities such as a keynote speaker, group breakout sessions and conversations in the hallways. Williamson et al. (2021) collected data through logs and complemented them with the classic approach driven by observations and semistructured interviews.

The discussed research cases are representative of the state-of-the-art of immersive VR group engagement research. To our knowledge, only the study by Williamson et al. went as far as extending the open-source platform with features essential for direct data extraction and analytics and went beyond qualitative observation and surveys for researching user behavior in VR. That study, in particular, informed our infrastructure development process described in the further part of this document.

3. Methodology

Our study builds upon desk research conducted in our past works on VR and e-participation. Based on elicited premises on immersive VR and e-participation, we apply a combination of an early quantitative study with the use of our technical infrastructure with qualitative observation in six basic scenarios. We limit our elaboration to immersive VR with specific highlights of differences between multiple aspects of the application of VR in online communications for group interaction in e-participation. We present our technical framework and corresponding AI-driven research infrastructure. Finally, we provide results of early evaluation of our infrastructure as means of preparation for future e-participation-related user studies.

3.1 Data collection and processing

The starting point for our infrastructure development was finding a platform allowing the extraction of VR interaction data. That was a challenge as the mainstream platforms do not offer access to data and there is a paucity of VR research platforms. Following the literature, we deployed our own instance of open-source, WebVR-based Mozilla Hubs on Amazon Web Services (AWS). We investigated options for the extraction of live interaction data, and we identified an existing open-source extension delivered by Li et al. (2021) and Williamson et al. (2021) for CHI 2020 workshop. The solution published on GitHub (Shamma, 2022) allows data collection via client applications of VR users. The component leverages Hubs architecture to extract basic parameters such as the position and direction of avatar’s body and head and voice volume level. We modified and extended that client application by including additional data parameters optimized the performance and coupled it with our own AI-driven analytics and feedback infrastructure deployed on a separate AWS cloud machine. We present the basic view of our design in Figure 1.

3.2 Virtual reality-dialogue ecosystem technical architecture

Our major contribution to the infrastructure is the data processing management and AI-supported analysis pipeline. The first key component allows controlling and managing parallel research sessions. Another element is a feedback dashboard that provides basic analytics information live, back to the moderators. To build those we created a relevant web interface, AI-Moderator, using mainstream technologies: JavaScript using REACT framework (Figure 2).

In Figure 3, we present the VR-dialogue ecosystem architecture: the VR-data-logger sends data to our self-designed VR-dialogue API. The VR-dialogue API has been implemented using NodeJS and related libraries such as Express and Sequelize. Express is used to handle the API endpoints and the associated requests, whereas Sequelize is used as an object-relational mapping for MySQL database.

Once data is stored in the database, we use our dedicated analytics module written in Python (using Scikit-Learn and PANDAS libraries) for AI-driven data analysis. The results from our module are sent back to the web interface dashboard as live feedback. The AI processing required us to apply relevant feature engineering that would help us to capture some important aspects of communication and collaboration in VR. We describe that process in Section 4 (Figure 4).

3.3 Data collection and pilot testing

To validate our infrastructure, we organized two research sessions, simulating e-participation engagements, with presentations and open discussion on August 3 (five participants) and August 8, 2021 (six participants) aiming at collecting general VR meeting behavioral data. Those general sessions were followed by structured, sample sessions where we followed a pattern leveraged for AI training. We have applied the following structure where individual segments took between 3 and 11 min each:

• a presentation session where the audience demonstrates focus without interaction with the presenters;

• a presentation session where the audience demonstrates focus and interacts with the presenters;

• a presentation session where the audience does not follow the presentation and demonstrates distraction;

• a presentation session where members of the audience do not follow the presentation and disregard the presenters by roaming around and talking;

• a conversation where participants gather and discuss; and

• collaboration – users discuss, interact and collaborate by drawing complex 3D shapes together.

That particular selection of different modes of discussion, used as an input for AI, reflects the key dimensions of e-participation from the perspective of the sense of community presented in Section 1. The first two modes relate to focus which is pivotal for the Immersion dimension, the further two relate to the aspect of followership, which matters to Influence, and the final two reflect on the sense of Membership through conversation and collaboration. Those relations between the dimensions of e-participation are presented in Figure 5.

The engagements were managed through our AI-Moderator dashboard. We trained AI algorithms in specific scenarios so that they would be able to recognize those types of engagements automatically based on data collected from users. We stress these are just basic examples of potential e-participation behavioral situations to be detected.

The structured sessions were organized on August 11th and 18th, with two different VR venues to test the transferability of our trained algorithms and a third session on August 22nd with participants repeating the exercise in two venues. The two venues are presented in Figure 5. While the first simulated venue was compact in space and did not allow much movement, the second venue was intentionally a large semiopen space for participants to roam around.

In each session, we had five participants with two taking the role of presenters and three of the audiences. All participants used exclusively six-degree freedom consumer VR headsets (Quest 2) to access the venue. After every session, we revised the set of parameters collected as well as generated new high-level features to ensure the strong accuracy of our clustering algorithms. In the following section, we describe in detail the feature engineering process applied.

4. Features engineering

In AI, supervised algorithms such as neural networks are given a feature vector as an input, and they output a prediction, in our case: focus (immersion), followership (influence) and membership (conversation and collaboration). The use of domain knowledge to select and transform variables from raw data or features into new features is called feature engineering. These engineered features can extend or replace existing features.

Before that step, the classification of participant interactions required the temporal alignment of streams of user data to describe a common temporal snapshot of the scene for further analysis. To address related challenges and satisfy our design choices, we created a framework to transform raw data into Scene High-level Features that can be used to train classification algorithms. Figure 6 presents the framework architecture. User-specific data transferred live from client applications are ordered and bulked in an analytical time-window according to the given timestamp in the database, as it arrives asynchronously through our API (Figure 7).

In Table 1, we provide a detailed list of the high-level features that we have elicited based on the specific data readings relating to the avatars body, head and individual hands position and direction, voice volume and use of pen for drawing.

We argue that efficient live tracking of those parameters can help estimate user involvement in e-participation as per Koh and Kim’s (2014) three dimensions: membership, immersion and influence elaborated in Section 1 of this paper. This type of analysis gives a new dimension to e-participation studies going beyond the content of exchanged messages and participant reports. In particular, as per our implemented scenarios, we believe that AI can process specific combinations of those parameters to determine whether the participant is focused (immersion), is followed by other participants (influence) and actively collaborates with other members (membership).

To identify the best combinations for AI training and better prediction, we evaluated the importance of the proposed features.

5. Evaluation

To evaluate the proposed features, we used random forest (RF) algorithm and its built-in feature importance identification method to assess each feature’s relevance (Weinmann et al., 2015). RF is an ensemble classification method that consists of a combination of tree predictors where each one depends on the values of a random vector sampled independently and with the same distribution for all trees in the forest (Pavlov, 2019). The key advantage is the accurate description of complex relationships among multiple variables (dos Reis et al., 2018). Once the trees in the forest output their decisions, a plurality vote is used to combine the final output using the same weight for each one (Chan and Paelinckx, 2008). RF uses a combination of k binary CART trees (Classification and Regression Trees). These trees are built without pruning, and at each node, a subset of randomly selected variables is used as input reducing the computational complexity of the algorithm and the correlation between the trees. One common value for splitting each RF node is the square root of the number of input variables (denoted by m). This recursive process will be repeated on each derived subset until a maximum depth (max_depth) is reached or when the number of samples at a node is less than a certain threshold (min_samples) (Belgiu and Drăgu, 2016). In our experiments, we used the parameters presented in Table 2.

6. Results

We present the results from running sessions on 18.08.21 and 22.08.21. We combined data sets coming from three dedicated engagement modes (presentation, conversation and collaboration) in two venues (small and large hall) and divided the set into training and testing 30% and 70% accordingly. In terms of transferability, we were able to demonstrate 56% accuracy when we attempted to transfer the learned AI classifiers from one virtual venue to another distinctively and over 90% when data from the venues were combined. With a full set of features included, we reached over 97% accuracy in predicting user focus (immersion), followership (influence), conversation and collaboration (membership) in our sample set. For better clarity, in this paper, we present only an aggregated overview of the features in Table 3 with a graphical representation in Figure 8.

As presented in Table 3, our engineered features showed to have a diverse impact on detecting simulated e-participation behavioral patterns elaborated in Section 3.3. This provides a strong premise (97% accuracy in our case) that a relevant combination of some of those features should also be a good instrument to detect other behaviors. We argue that despite a limited set of cases and features, our AI analytics infrastructure provides a solid first step toward supplying an advanced immersive VR research platform to researchers who wish to embark on experimentation with next-gen e-participation engagements. The extensible framework and flexible implementation allow the definition of new behaviors and provide near real-time results to participants and researchers alike. Specifically, we showed that elaborated in literature dimensions of e-participation from the perspective of online community engagement (immersion, influence and membership) can be effectively detected through means of AI analysis of user behavior.

7. Discussion and conclusion

To date, there is a paucity of research combining immersive VR and e-participation data-driven approaches. The related studies that investigate online 3D engagements deal largely with “on-the-screen VR” without emphasis on the immersive mode (using advanced HMDs) of communication for all the participants. To our knowledge, while multiple authors described data collection solutions and run user studies in VR, there is a paucity of efficient approaches to extending the VR interactions data extraction and integration with automated, near-real-time AI-driven analysis, especially in the context of e-participation.

To answer our first research question, we have delivered a flexible, expandable framework for serious VR meetings analysis and corresponding novel technical research data infrastructure as an enabler for next-gen VR-enabled e-participation research. To answer the second research question, we designed and deployed a dedicated, extensible AI-driven analytics engine over our data infrastructure and analyzed sample and simulated e-participation engagements.

Our results showed that the theoretical dimensions of online engagements in e-participation defined in the literature: immersion, influence and membership can be effectively tracked in VR meetings using a dedicated data extraction pipeline, relevant feature extraction and AI analytics engine. Specifically, the capacity of our framework corresponding infrastructure to detect participant focus (related to immersion), followership (influence) and conversation or collaboration (membership) provides a solid base for next-gen e-participation research inclusive of VR as a candidate technology.

While our work is limited to creating an infrastructure and running pilot evaluations between research contributors and awaiting larger user studies, our preliminary results provide essential validation of our tools. The instrument delivered allows studying immersive VR engagements from the perspective of applicability of VR to future e-participation research. The initial VR-engagement feature engineering for AI algorithm training and evaluation provides a basis for more advanced behavioral analytics in VR, going beyond directly observable phenomena.

7.1 Limitations

The work presented is intended to support next-gen e-participation research only through the prism of technical innovation as a future e-participation research enabler and focuses exclusively on the immersive VR approach. Our novel AI-driven analytics infrastructure offers basic insights into user behavior that can indicate greater immersion, influence or sense of membership and limited live feedback and should be considered only a template for more advanced, extensive feature engineering and fully fledged behavioral analytics.

7.2 Implications for e-participation research

The highly accessible cloud-based infrastructure is ready to host e-participation research sessions, online 24/7, and can be a model for similar affordable and highly accessible behavioral research infrastructures. The direct benefit for e-participation research is the possibility to go beyond the common, user-questionnaire-based participation performance analysis in favor of more structured and more discrete methods based on precise measurements in a controlled, simulated environment. Specifically, the capability to automatically identify and classify specific participant behaviors linked to key dimensions of participation online is of great importance to future next-gen e-participation stakeholders.

Our online software infrastructure approach has a clear edge over the solutions requiring expensive eye-tracked headsets and propriety analytics software at the individual participant level. We can deliver valuable insights including user gazing and focus with some meaningful approximation while using off-the-shelf VR headsets available. This lowers the entry barriers for researchers who wish to build upon our work and conduct more inquisitive research into user behavior in VR engagements in next-gen e-participation.

7.3 Context and future work

Our work stems from The Next Generation Internet Explorers (GA: 825183) research collaboration funded by an EU and USA university in the domain of VR communication analysis.

Future work will expand the delivered infrastructure to capture more data with an extended set of features to provide better insights into VR meeting dynamics. We intend to deliver a comprehensive set of tools allowing an in-depth data-driven view on serious engagements in immersive VR to support research in harnessing VR for e-participation. With relevant ethical approvals for our future user studies, we will include research sessions with groups of 10–20 participants engaging in specific online sessions using our deployed VR-dialogue infrastructure.

7.4 Additional applications

While our work has been conducted in the context of e-participation, our universal framework and expandable technical infrastructure for immersive VR engagement analysis have been applied to other domains and related studies on the application of VR to serious communication and collaboration.

Figures

Figure 1.

VR-dialogue infrastructure design

Figure 2.

Dashboard and AI-Moderator

Figure 3.

Ecosystem architecture

Figure 4.

VR-dialogue solution

Figure 5.

VR-dialogue support for e-participation dimensions

Figure 6.

VR-dialogue small hall venue and large hall venue

Figure 7.

Feature-engineering framework

Figure 8.

VR features importance

Table 1.

VR-dialogue high-level features

dist	Distance between two main clusters using rig (avatar’s body)
pov_dist	Distance between two main clusters using pov (avatar’s head)
percentage_people_drawing	Percentage people drawing in the time window
xyz_position_inertia	Sum of squared distances of user positions (based on rig) to their closest cluster center (in a 2 clusters classification) divided by the nr of users
xyz_quant_inertia	Sum of squared distances of user quants (based on rig) to their closest cluster center (in a 2 clusters classification) divided by the nr of users
xyz_direction_inertia	Sum of squared distances of user directions (based on rig) to their closest cluster center (in a 2 clusters classification) divided by the nr of users
xyzw_pov_position_inertia	Sum of squared distances of user positions (based on pov) to their closest cluster center (in a 2 clusters classification) divided by the nr of users
mean_right_hand_position_inertia	Average of all users’ average of the squared distances of each user’s right hand positions to the mean position
mean_left_hand_position_inertia	Average of all users’ average of the squared distances of each user’s left hand positions to the mean position
xyzw_quant_bigger_cluster_percentage_users	Percentage of users in the bigger quant cluster based on a kmeans partitioning of k = 2
xyzw_quant_smaller_cluster_percentage_users	Percentage of users in the smaller quant cluster based on a kmeans partitioning of k = 2
mean_right_hand_direction_inertia	Average of all users’ average of the squared distances of each user’s right hand directions to the mean direction
mean_left_hand_direction_inertia	Average of all users’ average of the squared distances of each user’s left hand directions to the mean direction
avarage_loudness	Average of all the volume records that fall in the time window
percentage_people_talking	Percentage of users (single UUIDS) that presented at least one volume record higher than 0.1 during the time window
xyzw_pov_position_bigger_cluster_percentage_users	Percentage of users in the bigger pov cluster based on a kmeans partitioning of k = 2
xyzw_pov_position_smaller_cluster_percentage_users	Percentage of users in the smaller pov cluster based on a kmeans partitioning of k = 2
xyz_direction_bigger_cluster_percentage_users	Percentage of users in the bigger direction cluster based on a kmeans partitioning of k = 2
xyz_direction_smaller_cluster_percentage_users	Percentage of users in the smaller direction cluster based on a kmeans partitioning of k = 2
xyz_position_smaller_cluster_percentage_	Percentage of users in the smaller position cluster based on a kmeans partitioning of k = 2
xyz_position_bigger_cluster_percentage_users	Percentage of users in the bigger position cluster based on a kmeans partitioning of k = 2

Table 2.

RF parameters

m	The square root of the number of features
max_depth	then nodes are expanded until all leaves are pure or until all leaves contain less than min_samples samples
min_samples	2
K (number of trees in the forest)	10,000

Table 3.

Results

Combined feature	Result
Avatar position and movement	dist ∼ 16% and xyz_position_inertia ∼9%
Position and movement of faces/heads	pov_dist ∼ 14% and xyzw_pov_position_inertia ∼ 7%
People drawing	percentage_people_drawing ∼13%
Gazing	xyz_quant_inertia ∼ 8% and xyz_direction_inertia ∼7 and xyz_quant_bigger_cluster_percentage_users 3% and xyz_quant_smaller_cluster_percentage_users 3%
People’s hands movement -	(mean_right_hand_position_inertia ∼4% and mean_left_hand_position_inertia 3%, mean_left_hand_direction_inertia ∼2% and mean_right_hand_direction_inertia ∼2%)
People talking	People talking/voice (percentage_people_talking 1.2% and average_loudness 2.8%)