The design, implementation and pilot application of an intelligent online proctoring system for online exams

Abstract

Purpose

The purpose of this study is to design and implement an intelligent online proctoring system (IOPS) by using the advantage of artificial intelligence technology in order to monitor the online exam, which is urgently needed in online learning settings worldwide. As a pilot application, the authors used this system in an authentic university online exam and checked the proctoring result.

Design/methodology/approach

The IOPS adopts the B/S (Browser/Server) architecture. The server side is implemented with programming language C and Python and stores the identification data of all examinees and their important behavior change status, including facial expression, eye and mouth movement and speech. The browser side collects and analyzes multimodal data of the examinee writing the online test locally and transfers the examinee’s most important behavior status change data to the server. Real-time face recognition and voice detection are implemented with the support of open-source software.

Findings

The system was integrated into a Web-based intelligent tutoring system for school mathematics education. As a pilot application, the system was also used for online proctored exam in an undergraduate seminar in Peking University during the epidemic period in 2020. The recorded log data show that all students concentrated themselves on the exam and did not leave the camera and did not speak.

Originality/value

During the epidemic period of the novel coronavirus outbreak, almost all educational institutions in the world use online learning as the best way to maintain the teaching and learning schedule for all students. However, current online instruction platforms lack the function to prevent the learners from cheating in online exams and cannot guarantee the integrity and equality for all examinees as in traditional classroom exams. The literature review shows that the online proctoring system should become an important component of online exams to tackle the growing online cheating problem. Although such proctoring systems have been developed and put on the market, the practical usage of such systems in authentic exams and its effect have not been reported. Those systems are heavyweight and commercial product and cannot be freely used in education. The light-weight IOPS developed by the authors could meet the need for online exam as a stable and practical approach and could contribute to the growing online learning and distance learning.

Citation

Jia, J. and He, Y. (2022), "The design, implementation and pilot application of an intelligent online proctoring system for online exams", Interactive Technology and Smart Education, Vol. 19 No. 1, pp. 112-120. https://doi.org/10.1108/ITSE-12-2020-0246

1. Introduction

During the epidemic period of coronavirus, almost every educational institution used online learning as the best way to maintain the teaching and learning schedule for all students. As known from UNESCO (2020), about 138 countries and 1.37 billion students around the world were studying online at home. However, current online instruction platforms lack the function to prevent the learners from cheating in online exams and cannot guarantee the fairness and equality for all examinees as in traditional classroom exams. Though synchronous video and audio capture technology can be used to record the face, body movement and speech of the examinee and his (her) surroundings, the required network bandwidth and speed, the laborious and instant one-to-one monitoring of the video and audio signal by the examiner make such capture technology very expensive and unfeasible.

In traditional test and exams, the supervision performed by the proctor can highly decrease the possibility of cheating and prevent students from cheating. But as for the students writing in distributed online exams, teachers can hardly tell a specific student’s status in an exam, not mention judging everyone’s learning status. Furthermore, with modern technology, students can use various ways to cheat, such as text messaging, googling or even plagiarizing.

The intelligent online proctoring system (IOPS) could be a stable and practical approach to meet the need for online exam and could be a valuable contribution to online and distance learning. Therefore, we designed and implemented an IOPS to facilitate the online exams. We introduce this online exam proctoring system and its pilot application in the next sections. In Section 2, some related works will be reviewed. Section 3 introduces the design method, the system architecture and function. Section 4 presents the pilot application. Finally, in Section 5, the conclusion and discussion will be made.

2. Related works

It is well acknowledged that some students cheat in exams in class (King and Case, 2014). Between 40% and 70% of all college students have reported cheating at some time during their academic careers in the USA (Aiken, 1991; Davis et al., 1992). Lin and Wen (2007) selected a sample of 2,068 college students throughout Taiwan, surveyed on four domains of academic dishonesty, including: cheating on test, cheating on assignment, plagiarism and falsifying documents, and found that the prevalence rate for all types of dishonesty behaviors among college students in Taiwan was 61.72%. No matter what the exam environment is, some students will try to cheat.

King and Case (2014) proposed that cheating using technology online is called “e-cheating.” Butler-Henderson and Crawford (2020) systematically reviewed 36 research papers on online invigilated exams and found cheating was the most prevalent area among all the nine themes identified. One reviewed paper, Sullivan (2016), argued that cheating in asynchronous, objective and online assessments was at unconscionable levels. The research on cheating focused mainly on technical challenges (e.g. hardware to support cheating), rather than ethical and social issues (e.g. behavioral development to curb future cheating behaviors).

To solve this great challenge to online learning, i.e. e-cheating, Atoum et al. (2017) argued that the ability to proctor online exam efficiently is an important factor to the scalability of the next stage in education. The founder of Kryterion Inc., Foster and Layman (2013), stated that “Online proctoring refers to proctors monitoring an exam over the internet through a webcam or other hardware devices.” The online proctoring should include every process of the exam, occur at a distance and offer the function of authenticating examinee.

The first proctoring system was introduced by Kryterion Inc., in 2006. At first, it relied on human to find cheating in an exam and began large-scale operations in 2008. On the basis of the previous experience and technology, the company provided a heavy-weight system for secure, Web-based, proctored exams (Foster et al., 2011).

Other proctoring systems came into birth following Kryterion. Securexam Remote Proctor, one of the autonomous proctoring systems, offers a sizeable rotation-free camera, which can record the students’ behavior when exercising (Tomasi et al., 2009). Webassessor uses actual proctors, combined with Web cameras and special software, that recognizes typing styles and whether the user pauses between certain letters. Acxiom Corporation reaches this goal in another way. It relies on the pretest, which means that if a person can pass the pretest of exam rules, he can get access to the exam. There is also a solution based on browser, named Lockdown Browsers, which prevents the user from escaping the present webpages. In addition to this browser, Respondus monitor suite is a fully automated proctoring solution that enables students to take online exams at any time without prescheduling and can be added to a mainstream learning management platform as a plugin. EZtest, as an integrated proctoring and exam system in China Mainland, owns some advanced features, such as video monitoring, forced full screen and IP restrictions. It can adopt the SaaS model to provide services. Customers can create and manage exams online and monitor the data changes in the exams in real time. Candidates can answer the questions via a browser or WeChat QR code. Some of the extensive standardized tests, such as recruitment exam, employee promotion exam and admission tests, are held on this platform.

However, the above mentioned proctoring systems were all heavy-weight systems, and their practical effect on exams and cheating-finding have not been reported. Hussein et al. (2020) evaluated eight online proctoring tools with a four-phased approach. At the last phrase, mock trials, only one system Proctorio was selected and evaluated by the staff and the students from a university. The evaluation result shows that the user experience was positive for students; the completion of test was positive, but the students’ overall experience was challenging and uncomfortable because this was the first time for them to verify themselves using Web cameras and their identification cards. Nonetheless, the practical usage and evaluation of such online proctoring tools were not introduced in this study.

Facial information, as one of the most accessible information in exam, can partly explain one’s emotion. Affective computing is “computing that relates to, arises from, or deliberately influences emotion or other affective phenomena” (Picard, 2000). In recent years, there have been many ways to study emotional computing technology, including skin activity (Yun et al., 2017), heart rate, blood pressure, Electroencephalogram (EEG), Electromyogram (EMG), Functional magnetic resonance imaging (fMRI) and facial expressions. Among them, facial expression information can be used as a key factor in analyzing emotions. Ekman and Keltner (1997) divided expressions into six basic types corresponding to related emotions: happiness, sadness, surprise, fear, anger and disgust. Facial expression can be used as an indicator of the examinee’s response status in the online exam process. But related works has not been found.

The above literature review indicates that the online proctoring system should become an important component of online exams to tackle the growing online cheating problem, and such proctoring systems have been developed and put on the market. However, the practical usage of such systems in authentic exams and its effect have not been reported. Those systems are heavyweight and commercial product and cannot be freely used in education.

Based on the literature review, the authors adopted the artificial intelligence technology including face identification, voice detection and facial expression recognition; designed and implemented a light-weight and easy-use IOPS; and applied it as a pilot study in a real-university exam setting.

3. System architecture and functions

Taking the advantage of artificial intelligence, for example, the multimodal interaction technology (Jia et al., 2020), we designed the architecture of IOPS. A multimodal human-computer interaction system “seeks to leverage natural human capabilities to communicate via speech, gesture, touch, facial expression, and other modalities, bringing more sophisticated pattern recognition and classification methods to human–computer interaction” (Turk, 2014). As a multimodal human–computer interaction system, the IOPS mainly captured the online user’s facially visual and audio signals to detect the user’s status during the exam. The IOPS uses the B/S (Browser/Server) architecture. The server side stores the identification data of all examinees and their important behavior status change, including facial expression, eye and mouth movement and speech. The browser side collects and analyzes multimodal data of the examinee writing the online test locally and transfers the examinee’s most important behavior change to the server.

The visual and audio signals are captured by the user’ hardware devices such as video cameras and microphones, which can be either internally integrated into the laptop or externally connected to the desktop computer. The signal capturing is real-time tracking. But the capture speed, i.e. the frames per second (FPS), is dependent on the capture hardware’s performance. Normally, the speed can be 30fps and the resolution of every frame image can be between 480p and 1080p in theory.

The video stream and audio stream are analyzed by the video monitor and audio monitor, respectively. Both monitors are mainly developed by JavaScript and integrated into the library Monitor.js, so that this system is lightweight and portable. The script can be a used as a plugin of any browser in the user’s client machine.

The video monitor is designed based on the following three components: dlib library, face-api.js and PyTorch. The dlib library is “a C++ toolkit containing machine learning algorithms and tools for creating complex software in C++ to solve real world problems” (http://dlib.net). face-api.js is “JavaScript face recognition API for the browser and nodejs implemented on top of tensorflow.js core” (https://github.com/justadudewhohacks/face-api.js). The PyTorch face recognition model is “An open source machine learning framework that accelerates the path from research prototyping to production deployment” (https://pytorch.org).

The face landmark model has been trained on a data set of 35,000 face images, labeled with 68 face landmark points. One user’s captured face can be identified with the features of the 68 landmark points. The Euclidean distance between two faces’ features is used to differentiate the two faces. If the Euclidean distance between two faces is greater than the threshold, for example, 0.4, the two faces are recognized as different ones. The threshold can be decreased for higher accuracy.

The proctoring system requires the function of secure login verification with facial identification, as shown in Figure 1. Before entering the exam system, the examinee will input the username and the password for the first identification. Because there is no reference face by the first login, the examinee is asked to take one snapshot within the captured facial video window. This snapshot is analyzed by the facial recognizer in the client and its features are uploaded into the server database as the face descriptors of the examinee. Of course, more snapshots can be taken so that more features can be stored in server. The mean of the set of features of one examinee will represent the examinee and be used as the reference of the examinee. The identification of one examinee can be more exact with more retrieved features from the facial snapshots.

After username and password identification, the examinee’s faces taken by the video camera are continuously analyzed and the correspondingly retrieved features are compared with the previous reference stored in the server database. If any one of the feature matched with the reference, the examinee passes the face validation and is allowed to enter the main exam page and can write the answers to the questions in the exams. Besides landmark feature identification, we add the function of face spoof detection to prevent someone who uses a still photo of some specific person from passing the validation. The face spoof detection is mainly implemented through mouth action analysis of the examinee so that the examinee is asked to open and close the mouth to pass the verification.

After secure login verification with facial identification, the system labels the examinee’s status as “recognized,” and the examinee enters the exam Web page and begins to write down the answers. The brief architecture of exam proctoring is shown in Figure 2. The metadata in the figure is defined as the collective data coming from a person's voice, face, gestures and other emotion-related activities.

Through the exam, the video and audio signal will be captured by hardware devices including cameras and microphones in the client machine.

The video monitor continuously analyzes the captured video image and labels the examinee’s status with one of the following three: absent, recurrent and multiple. If no human face is detected, i.e. no one examinee is found in front of the video camera, the system labels this status as “absent” of the examinee. If at next time one human face is detected, and this face is recognized as the examinee’s face by comparing the feature of the recognized face with the examinee’s mean feature retrieved from the server database, the system labels this status as “recurrent.” If more than one human face is detected, the system labels this status as “multiple.”

During the exam, it is not allowed to require the examinee to open and close the mouth for face spoof detection. Emotion recognition is used to check the dynamics of examinee’s face expression. Once the user’s face is identified with the examinee’s face, the facial expression is recognized as one of the six emotions defined by Ekman and Keltner (1997): happiness, sadness, surprise, fear, anger and disgust.

The audio monitor can both detect voice activity and recognize voice messages. The voice activity detection (VAD) can reveal whether there is any sound or not. The VAD is written in java script and can record the time of the sound start and the end with the label “start” and “end,” respectively. The voice recognition function should recognize the word the examinee has ever said and is implemented by using the open-source Julius system. Julius (https://github.com/julius-speech/julius) is an “Open-Source Large Vocabulary Continuous Speech Recognition Engine” and “as a high-performance, small-footprint large vocabulary continuous speech recognition (LVCSR) decoder software for speech-related researchers and developers, can realize accurate speech recognition.”

By proctoring the exam, a virtual proctor avatar was designed to represent the real proctor. The virtual proctor can be dragged anywhere in the screen. It can also speak through a text-to-speech script, for example, give warning to the examinee if the status “absent” or “multiple” is detected.

4. Pilot application

IOPS is designed as an easy-to-use, portable proctoring system and thus can be integrated into all kinds of online learning systems. We applied IOPS into “Lexue 100” (www.lexue100.com), with the Chinese meaning Happy Learning for 100%, a Web-based intelligent tutoring system for school mathematics, developed by Beijing Lexue 100 Online Education Co. Ltd., and equipped with the OLAI (Online Learning Activity Index) model proposed by the authors’ team (Jia and Yu, 2017; Zhang and Jia, 2017). Quiz writing is the main learning activity in this online learning system. However, our previous studies found the student might cheat by writing quizzes alone at home. To prevent cheating and to facilitate the independent learning online, the IOPS was integrated into the Lexue 100 system. The quiz-writing function performed well in the real production environment. Unfortunately, because of the data management regulation, the authors have received neither the log data nor the report about the usage of IOPS in this “Lexue 100” system.

As a pilot application, the authors also applied the IOPS in the exam in a university course. During the spring semester February 2020 to June 2020, all the courses in Peking University were held online due to the epidemic of COVID-19. It was no exception for the seminar “Education and Artificial Intelligence” given by the authors (Jia, 2018). In this seminar, the authors used and redesigned the popular course management system Moodle (www.moodle.org) to manage course resources and activities such as files, blogs, discussion forums, links, quizzes and so on. For the final exam, the authors integrated the IOPS into the course management system.

If the exam was regularly held in the university classroom, every student used his or her own laptop, tablet computer or smart phone to login to the course website with his or her own account and password and wrote the answers to the at least 10 questions, whose appearance order in the exam was randomly given by the server and different from each other. The exam should be completed within 10 min. If the examinee did not submit the answers within 10 min, the answers would be collected automatically by the client browser and be marked by the server. During the exam time, the student could read any document in papers and search in the computer or from the internet but could neither speak nor ask others for help. The instructor proctored the exam in the classroom. The tight exam time schedule, i.e. 10 questions within 10 min, the randomness of question orders and the importance of the exam grade for each student’s academic record, all lead to the independent completion of the exam by the students themselves. If one student left the computer or spoke to others, he or she would be labeled as cheating by the proctor.

The final exam in the spring semester 2020 was taken online. All students were distributed at different locations around China, and the instructor could not proctor the exam as in the classroom. The exam regulation was just the same as in the regular classroom exam. If one student left the computer or spoke to others, he or she would be labeled as cheating. This task of online cheating detection was undertaken by the IOPS.

All 16 students participated in the proctored exam, and their faces were recognized during writing the exam. If the examinee left the camera so that the camera could not detect the examinee’s face, the examinee’s status would be recorded as “absent” and stored in the server’s database. If the examinee came back to the screen and was recognized again, the examinee’s status would be recorded as “recurrent” and stored in the server’s database. If multiple faces were detected, the examinee’s status would be recorded as “multiple” and stored in the server’s database.

The log data with IOPS usage was retrieved and analyzed. For the face recognition, not any “multiple” status was found. Just five examinees had the record “absent” and then “recurrent.” The minimum record was once and the maximal record was 11. The interval time between the “recurrent” and last “absent” varied from 0 s, which means less than 1 s, to 17 s. It can be concluded that most students concentrated themselves on the screen and did not leave the screen and camera. Even if someone “left” the screen or was not recognized by the system for just no more than 17 s, this very short disappearance could not prove that the student went outside the sitting room for other’s help. So, no cheating was found during this pilot exam application.

For the audio activity detection, only 12 among all examinees had the record “start” and then “end.” For a specific examinee, the minimum record was once and the maximal record was 98. The interval time between the “end” and last “start” varied from 1 millisecond to 130 milliseconds. The voice activity duration sum varied from 2 milliseconds to 478 milliseconds. The voice activity duration mean varied from 2 milliseconds to 5 milliseconds, and the overall duration mean of all examinees was 3 milliseconds. For the sound with the duration no more than one 200 milliseconds, it is difficult to recognize the spoken words. This finding is also confirmed by the result from the null speech recognition result of Julius. It can be concluded that the 12 students concentrated themselves on the exam and did not speak. The detected sound was made by the keyboard typing or mouse clicking, which lasted for just no more than 200 milliseconds. The reason why the other 4 examinees did not save their audio record may be that they turned off the microphone.

Considering the findings from both the video and audio, we can conclude that no cheating was found during this pilot exam application.

We interviewed five examinees after the exam about the system’s usage. They felt this kind of proctored exam exciting, just as the real instructor proctored the exam. But they all admitted that they had neither time to ask others for help because of the tight exam schedule nor need to ask others because of the free usage of any document online or in print format during the exam. One examinee complained that the user login and face verification were too slow because his desktop computer was ten years old.

5. Conclusion and discussion

Based on the advancement of artificial intelligence technology such as multimodal interaction, we attempted to design, develop and test an IOPS to monitor the online exam, which is urgently needed in online learning settings worldwide. The IOPS uses the B/S architecture. The server side stores the identification data of all examinees and their important behavior status change, including facial expression, eye and mouth movement, and audio start and end. The browser side receives, collects and analyzes multimodal data of the examinee writing the online test locally and transfers the examinee’s most important behavior change to the server. Real-time face recognition and VAD are executed by the java script in the client browser. As a pilot study, the system was used for online proctored exam in an undergraduate seminar in Peking University during the epidemic period in 2020. The recorded log data show that all students concentrated themselves on the exam and did not leave the camera and speak, although some examinees’ client sounds were not recorded due to technical reason such as turning off the microphone.

This action research comprises the design, implementation and pilot application of one IOPS. It fills in the research gap that no practical application and result were reported about the online proctoring systems.

The IOPS is proved to be useful with acceptable accuracy and efficiency in terms of its performance. It is a practical approach to meet the need for online exam in terms of its usability. Compared with the proctoring systems studied or mentioned in the literature review, this light-weight system is based on open-source software and can be easily be integrated with other course management systems or learning management systems such as Moodle (www.moodle.org). Additionally, it can be used not only in the field of online exam but in the field of online learning. The monitor component in IOPS can be used in other intelligent learning environments, such as Computer simulation in educational communication Intelligent Companion (Jia, 2009), another product of our team, which is rendered as a simulated teacher in a Web server to help students study.

However, this system should be further improved if it is used in a much stricter online exam setting, which requires the independent answer writing of the student without the help of any document in paper or online, than that in our pilot application. The check of microphone connection in the examinee client by login should also be a prerequisite condition. More importantly, the system design should consider the bandwidth and connection speed if more examinees would take part in the exam. Such seemingly solely technical issues are also critical from the view point of educational equality: how can we guarantee all examinees who are located overall in China or around the world and come from various family background the equal hardware and software setting that are required by the IOPS?

Besides technical solution to prevent online cheating, management, ethical and social issues such as behavioral development to limit future cheating behaviors should also be stressed (Sullivan, 2016). Just similar to in our pilot application in the university exam, the cheating will seldom happen if the examinee has no time to ask for others’ help because of tight exam arrangement and has no chance to ask for others’ help because of questions’ randomness and uniqueness. Of course, the cheating will not be undertaken if the examinee is aware of the serious and legal effect on his or her life-long living, study and working.

Figures

Figure 1.

Entrance verification

Figure 2.

Brief architecture of exam proctoring