Activity recognition on smartphones using an AKNN based support vectors

M'hamed Bilal Abidine (Department of Electronics and Electrical Engineering, University of Sciences and Technology Houari Boumediene, Algiers, Algeria)
Mourad Oussalah (Faculty of Information Technology and Electrical Engineering, Oulun Yliopisto, Oulu, Finland)
Belkacem Fergani (Department of Electronics and Electrical Engineering, University of Sciences and Technology Houari Boumediene, Algiers, Algeria)
Hakim Lounis (Department of, Université du Québec à Montréal, Montréal, Canada)

Sensor Review

ISSN: 0260-2288

Article publication date: 17 May 2022

Issue publication date: 30 June 2022

677

Abstract

Purpose

Mobile phone-based human activity recognition (HAR) consists of inferring user’s activity type from the analysis of the inertial mobile sensor data. This paper aims to mainly introduce a new classification approach called adaptive k-nearest neighbors (AKNN) for intelligent HAR using smartphone inertial sensors with a potential real-time implementation on smartphone platform.

Design/methodology/approach

The proposed method puts forward several modification on AKNN baseline by using kernel discriminant analysis for feature reduction and hybridizing weighted support vector machines and KNN to tackle imbalanced class data set.

Findings

Extensive experiments on a five large scale daily activity recognition data set have been performed to demonstrate the effectiveness of the method in terms of error rate, recall, precision, F1-score and computational/memory resources, with several comparison with state-of-the art methods and other hybridization modes. The results showed that the proposed method can achieve more than 50% improvement in error rate metric and up to 5.6% in F1-score. The training phase is also shown to be reduced by a factor of six compared to baseline, which provides solid assets for smartphone implementation.

Practical implications

This work builds a bridge to already growing work in machine learning related to learning with small data set. Besides, the availability of systems that are able to perform on flight activity recognition on smartphone will have a significant impact in the field of pervasive health care, supporting a variety of practical applications such as elderly care, ambient assisted living and remote monitoring.

Originality/value

The purpose of this study is to build and test an accurate offline model by using only a compact training data that can reduce the computational and memory complexity of the system. This provides grounds for developing new innovative hybridization modes in the context of daily activity recognition and smartphone-based implementation. This study demonstrates that the new AKNN is able to classify the data without any training step because it does not use any model for fitting and only uses memory resources to store the corresponding support vectors.

Keywords

Citation

Abidine, M.B., Oussalah, M., Fergani, B. and Lounis, H. (2022), "Activity recognition on smartphones using an AKNN based support vectors", Sensor Review, Vol. 42 No. 4, pp. 384-401. https://doi.org/10.1108/SR-05-2021-0157

Publisher

:

Emerald Publishing Limited

Copyright © 2022, M'hamed Bilal Abidine, Mourad Oussalah, Belkacem Fergani and Hakim Lounis.

License

Published by Emerald Publishing Limited. This article is published under the Creative Commons Attribution (CC BY 4.0) licence. Anyone may reproduce, distribute, translate and create derivative works of this article (for both commercial & non-commercial purposes), subject to full attribution to the original publication and authors. The full terms of this licence may be seen at http://creativecommons.org/licences/by/4.0/legalcode


1. Introduction

Automatic human activity recognition (HAR) systems aim to capture the state of the user and its environment by exploiting heterogeneous sensors that are either attached to the subject’s body or placed at fixed locations in the environment, which enables continuous monitoring of numerous physiological signals. Equipped with a rich set of sensors and becoming an integral part of our daily life, smartphones are explored as an alternative platform to infer activities of the users (Voicu et al., 2019; Yuan et al., 2019). For instance, motion-related sensors, such as accelerometer and gyroscope, have been widely used in activity recognition systems as a wearable sensor (Horn et al., 2016; Randhawa et al., 2020) as well as identifying user’s behavior through series of observations (Shoaib et al., 2014). These activities range from simple full body motor activities such as walking, running and sitting to complex functional activities such as reading a book, playing a soccer and cooking a homemade dish. This can be immensely useful in health-care applications (Horn et al., 2016; Minetto, 2015; Ray et al., 2019) such as eHealth monitoring, fall detection as well as context-aware mobile applications, human survey system, home automation. Besides, the performance of such activities can be important indicators for designing and reshaping ingredients of the future aging society (Chen and Shen, 2017) where the existence of, e.g. reliable warning systems for accident occurrence such as falling down, can be crucial to save individual lives and seek cost-cutting solutions. It is therefore envisaged that such devices can seamlessly monitor and keep track of our daily activities, learnt from sensory analysis and assist regulators in their short-, medium- and long-term decisions.

Due to their mobility, battery performance, relatively low computational and memory costs and real-time implementation prospects, smartphone-based HAR offers an edge in future health-care systems. Roughly speaking, HAR involves a combination of sensor networks hand-in-hand with the data mining and machine learning (ML) approaches (Pei et al., 2012; Randhawa et al., 2020). In this context, a basic procedure for mobile activity recognition encompasses a data collection task from users that perform sample activities to be recognized, a classification model generation by using these collected data to train and test the suggested classification algorithms and a model deployment stage where the learnt model is used to assign a class label to a given action or activity. The overall process of smartphone-based HAR system is shown in Figure 1.

Sensor data can be processed either in real-time (Kose et al., 2012) or logged for offline analysis and evaluation. The model generation is usually conducted offline where the model is built and fine-tuned with optimal parameters on a server system and later transferred to the phone to infer the user’s activities according to these observations. Offline processing can be used for applications where online recognition is not necessary. For instance, if we are interested in monitoring the daily routine of a person, the sensors can collect the data during a day; the data will then be uploaded to a server at the end of the day, which, in turn, will be processed offline for classification purposes. While online processing is found useful in fitness-coach like application where the user is given a program with a set of activities, their duration and sequence of use and the interest is shifted toward what the user is currently doing. Likewise, in participatory-sensing applications, online processing arises when one might be interested in collecting information from users that are currently “walking” in a particular part of a city.

On the other hand, surveying the literature about smartphone inertial sensor-based activity recognition (Straczkiewicz et al., 2021) revealed the overwhelmed majority of these studies adopted a two-phases strategy. In the first phase, a large-scale training is performed offline to learn model’s parameters, followed by an online testing phase, which is embedded in smartphone memory resources. Besides, there is a strong consensus in this community that HAR training phase is typically very costly in terms of memory and computational resources as well as being impacted by the class-imbalance (because data set is not evenly distributed among various classes). Therefore, there is a general interest in providing activity recognition systems that overcome the class-imbalance dilemma and reduce the amount of training resources to enable its deployment into smartphone platform. This partly motivates the work undertaken in this paper where an adaptive k-nearest neighbors (AKNN) is put forward. More specifically, the paper advocates fourfold contributions:

  • We performed a concise review of the main works in HAR field with a focus on comparative results reported by the studies as well as the prospect of ensemble-classifier like approach.

  • We developed a new AKNN approach that accommodates smartphone-based implementation and hybridizes two ML methods. In essence, the developed AKNN approach is inferred from a novel weighted support vector machines (WSVM) issued from the application of kernel discriminant analysis (KDA) for the dimensional reduction prior to use of WSVM, while providing some adaptive scheme to KNN according to the extracted support vectors (SVs). This resulted in a light training scheme that can easily be accommodated to smartphone platform requirements and overcame class-imbalance.

  • We provided a sound algorithmic implementation of the developed AKNN and conducted a qualitative comparison with conventional KNN algorithm.

  • We provided an extensive quantitative evaluation of the proposal using five publicly data set, namely, HAR (Anguita et al., 2013), HAPT (Reyes-Ortiz et al., 2016), sensors activity recognition (SAR) (Shoaib et al., 2014), Wireless Sensor Data Mining (WISDM) (Kwapisz et al., 2011) and UniMiB SHAR (Micucci et al., 2017). The proposed AKNN approach is shown to outperform all the reviewed state-of-the-art approaches in the field with an important margin.

The remainder of this paper is organized as follows. Section 2 summarizes the related research in the past ten years. Section 3 introduces our automatic activity recognition approach and provides a comparison between KNN and the proposed method. The experimental validation of the proposed approach is given in Section 4 where the details of the publicly available activity recognition data set used in this work are also described in this section. Finally, Section 5 concludes the paper.

2. State of the art and related work

HAR has received a substantial interest recently. Currently, although there is a good prospect for collecting data with such smart devices, there is a limited capability in terms of automatic decision support capability and making sense out of this large data repository. There is an urgent need for new data mining and ML techniques to be developed to this end, such as SVM (Anguita et al., 2012), J48 (Kwapisz et al., 2011), logistic regression (Kwapisz et al., 2011), multilayer perceptron (Kwapisz et al., 2011), decision tree (Fan et al., 2013), hidden Markov model (HMM) (Lee and Cho, 2016), Long short-term memory (LSTM) recurrent neural networks (Ordóñez and Roggen, 2016), random forest (NJ and Kavitha, 2017), KNN (Mandong and Munir, 2018), artificial neural networks (ANN) (Suto and Oniga, 2018), bagging (Wu et al., 2019), boosting algorithm (Wu et al., 2019), convolutional neural networks (CNN) (Wu et al., 2019) and improved LSTM networks (Wan et al., 2020). Table 1 reports the comparison of different classification methods in the same framework when dealing with HAR (in Table 1, a > b means using classifier a resulted in a higher accuracy than classifier b).

Recently, several authors have demonstrated that two classifiers can be combined in different ways to improve the recognition performance in HAR tasks. For example, Ordóñez et al. (2012) showed that an ANN could be combined with HMMs to deal with activity recognition problems. Lester et al. (2006) developed a hybrid model that combined a modified version of AdaBoost with HMMs and demonstrated its effectiveness for recognizing various human activities using the embedded wearable devices. Anguita et al. (2012) proposed the multi-class SVM approach (MC-SVM), where the One-Vs-All approach is advocated because of its memory optimization compared to the One-Vs-One method. They have also introduced in Anguita et al. (2012), the concept of a multi-class hardware-friendly SVM (MC-HF-SVM) approach. This method was designed for binary classification problems by using a fixed-point arithmetic in the feed-forward phase of the SVM classifier, with the purpose of allowing its use for battery-constrained devices. However, such classifiers are not able to effectively distinguish very similar activities, such as going upstairs and going downstairs. Menhour et al. (2018) developed new schemes named principal component analysis (PCA)/KNN-SVM and linear discriminant analysis (LDA)/KNN-SVM, and they have demonstrated that LDA can outperform the traditional PCA for maximum discrimination between classes. A new approach for improving daily activity recognition combined with PCA, LDA and weighted-SVM (WSVM) has been addressed in Abidine et al. (2018) to overcome the problems of non-informative sequence features and class imbalance. The same authors have proposed in Abidine et al. (2019), an efficient classification model for physical activity recognition based on k-means clustering and SVM-HMM hybrid classification approach that uses labels outputting of SVM in HMM.

In the next section, we will present the principle of the proposed method called Adapted KNN (AKNN) approach. It allows a good representation and reduction of training set by using only the SVs.

3. Proposed adaptive k-nearest neighbors-based support vector

Our proposed approach has been motivated primarily by our desire to leverage the effects of imbalanced training data set, which is quite common in HAR problems where some activities are rather marginal, to very rare, with respect to other activities. For this purpose, in light of previous research in Abidine et al. (2018), Abidine et al. (2019), one may reasonably question the effectiveness of the WSVM, which has been widely populated in this field as well as KNN-based approach. Traditionally, WSVM has been applied to investigate the effect of overweighting the minority class on SVM modeling between the performed activities and deals with a “class-imbalance problem” (Fang et al., 2014). However, its high computational cost cannot be ignored neither. Observing that the high dimensional data, which is prevalent in HDR tasks, would rather make the handling of the imbalanced training data set possibly biased. Therefore, a cornerstone idea in our approach is to apply a dimensional reduction approach based on KDA prior to use of WSVM and providing some adaptive scheme to KNN according to the extracted support vector, which yields the new AKNN. The detailed architecture is highlighted in Figure 2.

Sensor data for different activities has been collected and stored using smartphone inertial sensors. Data has also been divided into two partitions: training-set and test-set. First, we reduce the number of features using KDA method using labeled data set to obtain the best discrimination between the classes in the new feature space. This is motivated by the fact that KDA has been shown to be more effective than the LDA due to separability criteria between classes in a high-dimensional implicit feature space (Kung, 2014), where the data are linearly separable. Second, we train the WSVM on the KDA features to generate the SVs that determine the boundary of activity data. The extracted SV that creates the new reduced balanced training data makes the classification process less complex. The ultimate classifier is performed by the Adapted KNN algorithm using the k samples limited only within the SVs such that when a new sample appears, it will be classified to its most similar class. An estimated label vector is generated by the AKNN classifier, and the system will output the recognized activity. These steps are briefly visualized in Figure 2. The following subsections will discuss in detail each step of the proposed approach.

3.1 Kernel discriminant analysis

To extract the nonlinear discriminant features, KDA (Tian et al., 2019), a non-linear discriminating method based on kernel trick (Kung, 2014), was developed. By introducing a kernel function which corresponds to the non-linear mapping Φ, all the computation can conveniently be carried out in the input space, see Figure 3 for a graphical illustration of the approach. More formally, the mapping is defined in the feature space (F) as:

(1) φ:RnF      xφ(x)

To get a nonlinear form of LDA, we simply apply a nonlinear kernel that should be symmetric positive-definite (Kung, 2014) as below:

(2) k(x,x)=Φ(x),Φ(x),x,xRn

Where 〈u; v〉 represents the dot product in the Hilbert space u,v F

Let SBϕ and Swϕ denote the in-between-class scatter matrix and the within-class scatter matrix in the feature space, respectively. In this new feature space, the objective function that needs to be maximized is as follows:

(3) Wopt=argmaxw|WTSBΦW||WTSWΦW|,such that
(4) SBϕ=i=1Nni(μiϕμϕ)(μiϕμϕ)T
(5) Swϕ=k=1Ni=1ni(ϕ(xi,,k)μiϕ)(ϕ(xi,,k)μiϕ)T

3.2 Weighted support vector machines training

Although SVMs often produce effective solution for balanced data sets, they are sensitive to the imbalance data sets which may yield a suboptimal model. The basic idea of weighted support vector training (WSVM) (Dzulkifli et al., 2019) is to handle the imbalance problem by assigning each data point a different weight assigning two misclassification costs C and C+ in the primal Lagrangian in the objective function [equation (6)] for the minority (yi =−1) and majority classes (yi = +1) by minimizing the following:

(6) mins,b,ζ12ww+C+i|yi=1m+ζi+Ci|yi=1mζisubjecttoyi(sΦ(yi)+b)1ζi,ζi0,i=1,,m.
Where ϕ is a nonlinear mapping and m+ (resp. m) are the number of positive (resp. negative) instances in the initial database (m + m+ = m). This is a nonlinear extension with a property of maximizing the margin between two classes.

The dual optimization problem of WSVM with different constraints on αi can be solved similarly to Fernández Hilario et al. (2018). The result of the above problem is solving the maximum of the following function:

(7) maxαi{i=1lαi12i=1lj=1lαiαjdidjK(yi,yj)}
subject to 0αi+C+, if di = +1, and

0αiC, if di = −1

(8) i=1lαidi=0,i=1,,l
Where αi+ and αi represent the Lagrangian multipliers of positive and negative instances, respectively. The corresponding data points are called SV.

If the training data gets more imbalanced, the ratio between the positive and negative SVs also becomes more imbalanced. Some authors (Abidine et al., 2018; Fernández Hilario et al., 2018; Yang et al., 2007), have proposed adjusting different cost parameters to solve the imbalanced problem. They put forward the corresponding solutions to deal with this problem in the SVM algorithm:

(9) C+C-=m-m+.
One way to deal with this problem is to increase the tradeoff C+ associated with the positive instances as in (Abidine et al., 2018) where different misclassification Ci per class were used to solve this problem. Especially, by taking C- =Ci and C+ = C, where m+ and mi stand for the number of samples of majority classes and the number of samples in the ith class, respectively. The main ratio cost value Ci for each activity can be obtained using:
(10) Ci=round(C×[m+/mi]),i=1,,N.

Where C is the common cost parameter of the WSVM.

3.3 An adaptive K-nearest neighbors classification using support vector

The K-nearest neighbor algorithm is amongst the simplest of all ML algorithms (Lu et al., 2018), and therefore easy to be implemented. The m training instances xRn are vectors in an n-dimensional feature space, each with a class label. In the KNN method, the result of a new query is classified based on the majority of the KNN category. This rule is usually called “voting KNN rule.” Such a classifier relies only on storage of feature vectors and class labels of the training instances whereas no model fitting is applied. They work based on the minimum distance (or similarity) from an unlabeled vector (a test point) to the training instances to determine the K-nearest neighbors. K (positive integer) is a user-defined constant. Usually Euclidean distance is used as the distance metric between x and the neighbor training xi represented by:

(11) d(x,xi)=(i,j=1n(|xxi|)2)12
One of the many issues that affect the performance of the KNN algorithm is the approach to combine the class labels. The simplest method is to take the majority vote, but this can generate a problem if the nearest neighbors vary widely in their distances. The intuition behind adaptive KNN is to calculate the contribution of k nearest neighbors, and to give more interest to the points which are nearby and less interest to those points which are farther away. Consequently, we start finding the k neighbors that have the smallest distances to the test data. In the second phase, we compute the contribution of each k nearest neighbors based on KNN algorithm by computing the adaptive distance (da) from observation of different categories.
(12) da=(1/|d|),with da0,
Finally, we set the classification results according to the distance values of various categories, where the distance (da) is higher in the decision when the existing observations (training set) are close to the new observation. We proposed in this paper the inverse of Euclidean distance as the adaptive distance da as follows. The prediction of the class for new test data (x) is computed by using different (da) for each nearest neighbor (NN) labeled (yNNi), with q as the total number of NN as follows:
(13) y˜(x)=dayNNi=1NNmdai,
Instance x is assigned to the class yNNi for which the adaptive distances of the representatives among the k nearest neighbors sum to the greatest value. More formally:
(14) y˜(x)=round(i=1NNqdaiyNNii=1NNmdai),
We label the data in the related window as the activity for which we have the maximum amount of data in the final K set. For instance, if K is 10 and the final list is “1 1 5 3” (1 vote for running, 1 vote for walking, 5 votes for sitting and 3 votes for standing) for the average feature, then the activity is labeled as sitting according to the average feature.

We show in Figure 4, the concept of Adaptive KNN using the balanced data set extracted from the SV. The training data has been significantly reduced by using the WSVM learning to deal with the imbalanced problem. The labeled SVs have become the new nearest neighbors. According to the proposed idea, the corresponding algorithm can be summarized by the pseudo-code described in Pseudo-code summarizing the proposed activity recognition approach.

Algorithm : Adaptive KNN (AKNN)
Procedure: Find class labels corresponding to activity classes
Require

DBtrain=(x1,,xm),

// DBTrain: Training Database
Labels_train= (Y1… Ym ) // The corresponding labels of DBtrain
DB test //DBtest: Test Dataset
DB={(xi,yi),i=1,,m}, DB=xi,yi,i=1,.m

% Number_classes = N
 Begin
// KDA algorithm applied to DB provided as input

1 Apply the RBF kernel according on DBtrain by computing the kernel matrix KRm×n, K Rm×m

2 Compute between -class scatter SBφ Sb //Equation (4)

3 Compute within -class scatter Swφ Sw//Equation (5)

4 Apply the Eigen-decomposition L = Eig((SWφ)-1SBφ) // Get the projection matrix

5 Rdtrain=LT DBtrain // Rd= Reduced data
6 Rdtest =LT DBtest
 //Multi-class WSVM training algorithm to extract SV
Training phase: (with N classes)
7 Solve N(N-1)/2 binary classifiers SVM on Rdtrain using the RBF kernel //Equation (6)
8 Extract and record the support vectors (SV) with LibSVM
9 Extract and record the Labels of SV
// AKNN test algorithm applied to SV input provided

10 Find kk th neighbor to the new observation x using the Euclidean distance function between x and the support

vectors (SV) as follow: d(i)=d(x, xvs(i)) //Equation (11)
11 Calculate the adaptive distance (da) //Equation (12)
12 Predict YAKNN the class label of observation x //Equation (14)
13 Return the activity classes Ci , with i = 1…N 
End

To illustrate the merits of the proposed concept, we summarized in Table 2 the advantages and disadvantages of the KNN compared to our introduced AKNN.

4. Experiment results

Our experiment used mainly some publicly available data set to examine the performance of our method. In this section, we start by describing the data set used in these experimental procedures, the learning algorithm implemented, the attribute extraction technique used to validate the construction of new features and the evaluation results. Nevertheless, our study skips data collection and preprocessing steps on purpose because we want to improve the HAR by providing better feature extraction and classification scheme.

4.1 Data sets

Publicly available and annotated data sets for activity recognition (Anguita et al., 2013; Kwapisz et al., 2011; Reyes-Ortiz et al., 2016; Shoaib et al., 2014, and Micucci et al., 2017) have been conducted to evaluate the performance of the proposed approach. We used four different data sets, using different physical activities (locomotion: walking, running, walking upstairs, walking downstairs), (postures: laying, sitting and standing) and (transitions: sit-to-stand, stand-to-sit). These data sets vary in their formats, type of sensors they are generated from and sampling frequencies. The data collection task was performed on an Android Samsung Galaxy SII phone. The first used data set is named Human Activity Recognition Dataset (HAR). The second data set (HAPT) with postural transitions is quite similar to the previous one, but it includes postural transitions such as sit to stand. The third data set is known as the SAR. The fourth data set is the data set created by the WISDM laboratory. Finally, the last data set, named University of Milano Bicocca Smartphone-based HAR (UniMiB SHAR), is grouped into two coarse grained classes: one containing the activities of daily living (ADL) and the other containing the types of falls (Figure 5). Some of these data sets included tri-axis angular velocity from the gyroscope measurements, in addition to the tri-axis accelerometer measurements common to all. A detailed description of each data set is presented in Table 3.

In contrast to other data sets, UniMib SHAR data set includes both ADL and Fall activities described in Figure 5. ADL consists nine activities: from standing to lying on a bed (LyingDownFS), from laying on the bed to standing (StandingUpFL), from standing to sitting on a chair (StandingUpFS), from standing to sitting on a chair (SittingDown), climb the stairs moderately (GoingDownS), down the stairs moderately (GoingUpS), normal walking (Walking), continuous jumping (Jumping), moderate running (Running). There are also 8 Fall scenarios: Generic fall backward from standing (FallingBack), Fall backward while trying to sit on a chair (FallingBackSC), Falls using compensation strategies to prevent the impact (FallingWithPS), Fall forward from standing, use of hands to dampen fall (FallingForw), Fall right from standing (FallingLeft), Fall right from standing (FallingRight), Falls with contact to an obstacle before hitting the ground (HittingObstacle), Getting unconscious (Syncope).

The number of observations of each activity in each data set is shown in Table 4. This enables us to visualize the disparity between activities in terms of number of observations (particularly for HAPT, e.g. “Walking” and “Sit to Stand”) and (WISDM data set, e.g. “Walking” and “Standing”). The HAPT data set has an imbalanced data distribution where the transiting activities are under-represented in comparison to non-transiting activities as shown Figure 6.

4.2 Evaluation measures

Five performance measures are used to test the proposed model, namely, accuracy, precision, recall, F1-score and error rate as defined below using evaluations of true positives, false positives and false negatives as follows:

(15) Accuracy=i=1NTPiTotal×100%
(16) Precision=1Ni=1N[TPiTPi+FPi]×100%
(17) Recall=1Ni=1NTPiTPi+FNi×100%
(18) Fscore=2.Precision.RecallPrecision+Recall×100%.
(19) Error Rate=100Accuracy
where N is the total number of instances. We adopt the accuracy, precision, recall, F-score and error rate as our evaluation metrics for fair comparison with some of the state-of-the-arts methods. In an extremely imbalanced data set, the overall classification accuracy is not regarded as an appropriate performance measure, but, instead, this measure was used to evaluate the accuracy of each activity class.

4.3 Results

4.3.1 Feature extraction analysis

It has been noticed that the performance of AKNN algorithm decreases with the existence of dependencies between features. Therefore, in this study, an approach based on KDA algorithm is conducted to eliminate the redundancy information from the segmented raw data. The kernel function used is the Gaussian kernel function, with a fixed σ = 1. Figure 7 shows clusters of data points class projection in the KDA and compared to LDA spaces, respectively, for WSDM data set. As it can be seen, each physical activity (six classes) is represented by a color in the projected 3D space using the first three feature vectors in WISDM data set. As it can be noticed, it is possible to obtain a good separation between the human activity classes using KDA method, better than the one generated by LDA method. This projection aims to maximize the inter-class variations between the different activities. Besides, the use of KDA features as a basis for training SVM model would inevitably to a substantial reduction of the training phase because the smaller number of KDA features that can be generated from the original input data.

The preceding indicates the usefulness of the chosen KDA feature decomposition and its ability to discriminate various classes, compared to the commonly used LDA technique.

To detail the LDA feature decomposition carried out in Figure 7, we represent in Figure 8 the LDA feature selection process for each dataset. Especially, the input dimensionality is reduced by selecting the number of extracted features equal to N-1 where N is the number of physical activities. This leads to a representation of the initial data on a N-1 feature space.

4.3.2 Classification results

To evaluate the performance of the proposed approach for automatic HAR, we have used the extracted features by KDA and the WSVM algorithm tested with a LibSVM implementation (Hsu et al., 2003).

4.3.2.1 Model parameters tuning.

Some of the classifier parameters are adjusted until we maximize the error rate through 10-cross validation technique. Results are computed by averaging the results obtained on each test fold. For the AKNN method, we have used an order q of similarity distance in a range of [1, 1.5, 2] and the k parameter in the range [1,3,5,7,9,11]. The hyper-parameter σ is estimated using a grid search method in the range [0.1–1]. The summary of the optimal values obtained with the proposed method are summarized in Table 5. We have used the radial-basis function (RBF): K(x, y) = exp (−0.5.‖x y2/σ2) where the width σ is specified as a priori by the user. For the WSVM classification, we optimize the cost parameter Ci adapted for each activity class by using the WSVM classifier with the common cost fixed parameter C = 1. The error weights assigned to various classes are set such that wi= m+/mi (or using the expression (10)). Values of wi for different data sets are given in Tables 6–10 using the training data set.

4.3.2.2 Performance evaluation and comparison.

We conducted several experiments to compare the results of the proposed method for all data sets. We have used 70% training and 30% testing on all data sets. As for activity recognition performance, we have used various combinations of possible features (KDA, LDA) and various SVM refinements when standard SVM, weighted SVM and K-NN selection approach for SVs in SVM training phase. Especially, we also considered what we refer as AKNN baseline, where neither KDA feature selection, nor WSVM were used in the input phase. In total the following methods were constructed for comparison purposes WSVM, AKNN-baseline and hybrid methods LDA/SVM-KNN, LDA/WSVM-KNN, KDA/SVM-KNN, KDA/WSVM-KNN, KDA/SVM-AKNN. In addition to the preceding, the results reported by some state-of-the-art methods that used the same data set (Abidine et al., 2019; Abidine and Fergani, 2020; Anguita et al., 2012; Bharathi and Bhuvana, 2020; Wan et al., 2020; Zainudin et al., 2017; Zheng et al., 2018; Li et al., 2018; Lv et al., 2020) are acknowledged and discussed as well. These findings are now summarized in Table 11. The latter clearly indicates that the proposed classifier achieved the highest scores in terms of F1-score, precision, recall and error-rate metrics. It improved significantly the performance of smartphone-based HAR, with a good classification accuracy in all data sets. For instance, with HAR data set, the classification error of our model is reduced to 0.8%, which corresponds to 50% improvement on the second-best state-of-the art method reported in the literature that used a multi-layer perceptron architecture (Bharathi and Bhuvana, 2020). Similarly, it achieved 2.2%, 2.3%, 1% improvement on F1-score, recall and precision metrics, respectively, when compared to the second best approach. The improvement is especially striking in error rate. While MC-SVM, multiclass hardware friendly SVM (MC-HF-SVM), decision trees and random forest methods present more than 10% of error rate, which indicates a more than ten times improvement of our method over these approaches. For the HAPT data set, the level of improvement of the developed method is 61.1%, 1.2%, −0.04%, 0.4%, with respect to error rate, recall, precision and F1-score, respectively, when compared to the second-best method.

Similarly, the level of improvement in case of SAR data set is 21.4%, 2.7%, 0.5% and 0.3%. For WiSDM data set, the improvement sequence is 34.7%, 1.5%, 3.6% and 2.7%. Finally, for UniMib SHAR data set, our method achieves 17.3%, 3.5%, 7.5% and 5.6% improvement with respect to error rate, recall, precision and F1-score, respectively, over the second-best performance. This testifies of robustness and efficiency of the proposed method for daily activity recognition tasks. Comparing all ML classifiers for each data set in this table shows a sensible performance improvement where F1-score is higher for the proposed method and equal to 0.98, 0.98, 0.99, 0.95, and 0.85 for HAR, HAPT, SAR, WISDM and UniMib SHAR data sets, respectively.

The analysis also sheds light on the relevance concept of hybridization of WSVM and AKNN classifiers to enhance recognition capabilities of daily living activities. This can be partly explained by the fact that WSVM is used as the training algorithm to select SVs for the AKNN classifier. We also notice from these results that WSVM significantly outperforms AKNN for recognizing imbalanced data activities across all data sets, particularly for HAPT, WSIDM and UniMib SHAR. This can be explained by the fact that in these data sets, some activities contain a large number of samples, whereas others are only assigned a very small number of samples. The consideration of postural transitions as individual imbalanced classes in HAPT data set can increase the degree of difficulty of the recognition task, and therefore, can decrease the recognition rates of basic activities. In addition, UniMib SHAR database contains a total of 11.771 samples not equally distributed across activity types: 7.759 samples describing ADLs and 4.192 samples describing falls.

Indeed, from the results highlighted Table 11, we notice that KDA/WSVM-KNN outperforms KDA/SVM-KNN. For instance, the error rate reduces from 7.9% for KDA/SVM-KNN to 4.1% for KDA/WSVM-KNN in case of HAR data set, and the same trend is almost observed for other performance metrics and other data sets. This is explained by the fact that WSVM is adapted for the imbalanced data set and consequently the obtained SVs regarded as the new training data set for different activity classes are balanced, unlike SVM where the number of SVs is imbalanced. The KDA features are found more relevant compared to the LDA features. This results from the improvements introduced by KDA/SVM-KNN, and KDA/WSVM-KNN compared to LDA/SVM-KNN, and LDA/WSVM-KNN.

Nevertheless, we notice that in SAR and WISDM data sets, the error rate is only slightly improved in the proposed method compared to HAR, HAPT and UniMib SHAR data sets. Also, the F1-score is lower for the proposed approach in the WISDM, and UniMib SHAR data sets using the accelerometer sensor while the activity was being performed. This is explained by the fact that the number of features (6) for both data sets is not sufficient when using KDA algorithm. A high dimension of features increases the complexity and has a negative effect on the final result. Hence, feature extraction using KDA in the proposed method becomes crucial for HAR, HAPT (561 features) and UniMib SHAR (453 features).

When using LDA features instead of KDA, the results do not confirm the superiority of LDA/WSVM-KNN over LDA/SVM-KNN across all data sets, although this occurred in the majority of the data sets.

In terms of the scope of the training phase, we can claim that the use of KDA features substantially reduces the scale and enhances the efficiency of the subsequent classifier task as KDA identifies the most relevant features for the training process. Therefore, the proposed method provides a better prediction of these activities when trained on the KDA features compared to other methods, particularly for HAR, HAPT and UniMib SHAR data sets as illustrated in Table 11.

4.3.2.3 Time complexity performance.

Meanwhile, we also evaluated the performance of proposed method in terms of execution times for training and testing performed from the original training set and the compact training set (SV). Table 12 summarizes the CPU execution time of our method compared to the AKNN baseline for both training and testing phases. The results show that the increase in size of training or testing data increases the execution time.

The computer used in this work for computing the execution time is equipped with an Intel Core i5-7700 CPU and 8 GB RAM.

As expected, the execution times of our approach are considerably reduced as the number of SV to compute the adapted distances of AKNN decreases. For example, in the case of UniMib SHAR data set, the execution times of testing decreases from 31.1 sec to 5.1 sec, by a factor of 6, while the training improves from 95.1 sec to 16.5 sec, by a factor close to 6 as well. This trend continues across all data sets.

It should be noted that despite the introduction of KDA and WSVM induces extra computational requirement as compared to baseline AKNN, this is widely compensated by the substantial reduction of subsequent SVs manipulation, which yields a sharp decrease in CPU execution time as compared to AKNN baseline.

4.3.2.4 Memory usage performance.

Finally, we investigated the performance of the developed approach in terms of the amount of memory resources required. We compared again the baseline AKNN method to our model that involves KDA and WSVM. The result of this process is highlighted in Table 13.

The preceding indicates that training of the developed approach consumes between 5 times and 8 times less memory resources for training the developed compared to the baseline AKNN method.

Therefore, with a substantial reduction in computational and memory resources of the training phase as compared to the AKNN baseline method, this provides a solid asset to implement the developed approach in mobile platform. Indeed, with a memory resources of less than or close to 1 GB, this is within reach of most nowadays smartphone devices, which are equipped with internal memory of more than 64 GB.

4.4 Discussion

Besides the results in the Table 11, to get a detailed knowledge of the performances on each current activity, we calculate the confusion matrices for the proposed method in the Tables 14, 15 and 16 by using the balanced HAR data set, imbalanced SAR and WISDM data sets, respectively, with six different user’s physical activities. These results show strong support for the effectiveness of our proposed method. Referring from Tables 14, 15 and 16, we see that in WISDM data set, with the exception for the dynamic activities “W. Upstairs” and “W. Downstairs,” the performance has high performance scores, whereas the score is higher and nearly consistent across all the activities in HAR and SAR data sets. This can be due to the highly imbalanced nature of WISDM where the percentage of data for “W. Upstairs” and “W. Downstairs” are about 12% and 10%, respectively, whereas for walking it is about 38%.

According to the Table 14, it can be observed that 99.3% of “W. Upstairs” activity instances are correctly recognized, while 0.3% goes into “W.Downstairs” and 0.2% are confused with “Walking” activity. The similar classes such as “Walking,” “W. Upstairs” and “W. Downstairs” show similar trend of sharing errors among each other. The reason is the similar status of smartphone when the user does these dynamic activities. The activities as “Sitting,” “Standing” and “Laying” could be classified more accurately. We explained this by the fact that the nature of data distribution is almost uniform among all classes. We can also notice that the static activities “Sitting,” “Standing,” and “Laying” share errors among each other. A total of 2.7% of “Standing” activity instances misclassified as “Sitting” activity and 1.4% of “Sitting” activity instances are misclassified as “Standing” activity. The main reason for this might be that both sitting and standing are still a static posture; hence the accelerometer readings are similar. However, the minority class “Laying” in terms of number of instances (792) is totally recognized using the proposed method.

In Table 15, the best accuracy was achieved for four activities (walking, sitting, standing and jogging). Walking was recognized with 99.4% accuracy. Sitting and standing showed accuracy of 100% and 99.7%, respectively. It appears the system is able to reliably recognize these two activities. The activities “Walking Upstairs” and “Walking Downstairs” are less recognizable compared to other activities. But a closer examination of this table indicates that all of the misclassified instances are recognized interchangeably between these two activities. When grouping these two activities as one (activity: stairs), the system was able to recognize it with 100% accuracy. The presented system was able to classify simple activities with a very good accuracy.

In Table 16, in most cases, we get a high level of accuracy. For the two most common activities in terms of number of samples, “Walking” and “Jogging,” we generally achieve accuracies above 97%. Both “Walking” and “Jogging” activities have significantly more samples than other activities, which would make the recognition results biased toward these activities. “Jogging” appears easier to identify than other activities, which seems to make sense, as jogging involves more extreme changes in acceleration. On the contrary, it appears much more difficult to identify the two stair climbing activities. (88.9% for “W. Upstairs” and 92.9% for “W. Downstairs”), but as we shall see shortly, that is because those two similar activities are often confused with one another. Indeed, when grouping these two activities as one (activity: stairs), the system was able to recognize it with 100% accuracy. For the static activities, we can note that there are very few instances of “Sitting” (306) and “Standing” (246), but we can still identify these activities quite well with the proposed method. Although some of the user’s activities recorded reflect somewhat insufficient performance as in WISDM dataset for “W. Upstairs” and “W. Downstairs,” we could state that our method is capable of producing a decent accuracy.

As mentioned above, the impact of gyroscope and accelerometer sensors were found to be sensitive to physical positions. Indeed, the gyroscope is not able to differentiate between similar activities like “Sitting” and “Standing.” On the other hand, the accelerometers perform badly with “W. Upstairs” and “W. Downstairs.” For the WISDM and UniMib SHAR data sets using an accelerometer sensor, we can note that the performances are decreasing comparatively to the other data sets. The results show that only one-axis acceleration is enough to classify simple full body motor activities. However, a sensor fusion-based strategy to collect the data sets would be useful for more accurate recognition performances.

5. Conclusion

In this work, we developed a novel activity recognizer model AKNN by revisiting the concept of extracting and handling the SVs used in standard SVM. The idea is to prone the training phase by using KDA features and hybridizing weighted SVM and KNN to enable handling of class imbalance data. The KDA dimension reduction technique selects the minimal number of discriminative and relevant features in the feature space. Extensive experimental evaluations using five publicly available data sets of daily activity recognition indicate a high classification performance compared to other ML classifiers. This is due to the ability of our approach to make the learning dataset more balanced and the high quality of the SVs generated by the combination of weighted SVM and KNN method. This deals with the class-boundary-skew problem and extracts useful SV to reduce the training dataset. The proposed method maintains the advantage of the WSVM in training phase. Additionally, the accuracy also tends to decrease when including few informative features to classify.

The purpose of this study is to build and test an accurate offline model using a compact training SVs from the original training data. This model reduces the computational and memory complexity of the system. Hence the real-time classification will be much faster using this light-weight approach. The proposed approach is promising to do real-time classification of activities, which can reduce the processing time to enable user-independent and operating system independent real-time recognition of the physical activities in Android OS smartphone. The comparison of the model with other state-of-the art approaches that used the same data set as well various hybridization modes of SVM, WSVM, LDA, KNN, KDA testify of the quality of the proposal. Besides, the substantial reduction of computational and memory resources of the training phase provides solid assets for real-time implementation of the developed model on conventional smartphone platform. More work is still needed so that this method generalizes well to more users and more complex activities. Moreover, it would be interesting to use transfer learning as a new approach to tackle the imbalanced class.

On the other hand, as an ultimate perspective work, one shall point the growing interest in online activity recognition. The starting idea is therefore to implement the proposed system in Android platform that performs in real time as follows:

Step 1: Establishing a connection between the data recording device (Smartphone user) and the controller (as a software running on a Windows computer or on a smart phone) using the User Datagram Protocol to transmit the data.

Step 2: Turn the control signal start gathering, pre-processing, feature extraction and training. The specific data to be arranged in a special format to comply with regulations on data formats. We stored the k SVs and the adaptive distances (di) for each nearest neighbor, in memory of controller to be used as a template at classification step.

Step 3: Recognition results will be sent to the smartphone user using the Transmission Control Protocol Connection. Due to the requirement of continuous activity recognition, the calculation should be done in parallel with the process of writing data, which is used for the next recognition.

Figures

Process of activity recognition

Figure 1

Process of activity recognition

Proposed smartphone-based activity recognition framework

Figure 2

Proposed smartphone-based activity recognition framework

KDA in the case of polynomial projection

Figure 3

KDA in the case of polynomial projection

AKNN classification

Figure 4

AKNN classification

Activity samples distribution for the UniMib SHAR data set

Figure 5

Activity samples distribution for the UniMib SHAR data set

The number of activities recorded per each of the 12 activity groups for the HAPT data set

Figure 6

The number of activities recorded per each of the 12 activity groups for the HAPT data set

Distribution of multi-activity patterns in (a) LDA, and (b) KDA spaces for WISDM data set

Figure 7

Distribution of multi-activity patterns in (a) LDA, and (b) KDA spaces for WISDM data set

Feature selection by applying LDA on the training data for each data set

Figure 8

Feature selection by applying LDA on the training data for each data set

Comparison of classification state-of-the-art methods in HAR

Ref. Method Comparison
Yang et al. (2008) Neural Networks > KNN
Yang (2009) Decision Tree > SVM > KNN > Naïve Bayes
Kwapisz et al. (2011) SVM > Random Forest > LMT > Simple Logistic > Logit Boost
Pei et al. (2012) LS-SVM > Decision Tree > Linear Discriminant Analysis > Quadratic Dis. Analysis > Bayesian Network-GMM
Mitchell et al. (2013) Naıve Bayes > Decision Tree > KNN > Neural Networks > SVM
Bayat et al. (2014) Neural Networks >Neural Networks > Decision Tree > Logistic Regression
Bulling et al. (2014) SVM > KNN > HMM > Naïve Bayes
Gao et al. (2014) Neural Networks > Decision Tree > KNN > SVM > Naïve Bayes
Fang et al. (2014) Neural Networks > Naïve Bayes > HMM
Capela et al. (2015) Decision Tree > Naïve Bayes > SVM
Catal et al. (2015) Neural Networks > Decision Tree > Logic Regression
Barua et al. (2019) Random Forest>KNN> CDNN>SVM
Menhour et al. (2019) K-SVM>SVM

Advantages and disadvantages of KNN and AKNN algorithms

Algorithms Advantages Disadvantages
K-Nearest Neighbors (KNN) - Easier to understand
- No training step because there is no model to build.
- Robust to noisy training data
-Robust to outliers in input space
- Handling of missing values
- Memory limitation
- For large training-set, computing distances can be expensive
- Classification by taking the majority vote
- Being a supervised learning lazy algorithm
- Performs linear classification
- Sensitive to lots of irrelevant attributes
(affect distance)
Adaptive K-Nearest Neighbors (AKNN) - All advantages of KNN
- No memory limitation using SV
- Adaptive computing distances
- Adapted for the imbalanced
dataset
- Performs nonlinear classification
- Robust to outliers in the input space
- Sensitive to several irrelevant attributes
- Require a training step using WSVM to build the model using SV

Overview of publicly available datasets of activity recognition used in the evaluation of the proposed approach. Accelerometer (a), gyroscope (G), magnetometer (M)

Datasets HAR (Anguita
et al., 2013)
HAPT (Reyes-Ortiz
et al., 2016)
SAR (Shoaib
et al., 2014)
WISDM (Kwapisz
et al., 2011)
UniMiB SHAR (Micucci
et al., 2017)
Age (years) [19–48] [19–48] [25–30] [18–60]
Nb. of subjects 30 30 10 29 30
Sampling rate (Hz) 50 50 50 20 50
Annotation Video Video PDA Graphical user interface Manual (Clap hands)
Features 561 561 9 6 453
Position Waist Waist Wrist Front leg pocket Thigh (Trouser pocket)
Sensors A and G A and G A, G and M A A
No. of Activities 6 (ADL) 12(ADL) 6(ADL) 6(ADL) 17 (9ADL + 8Fall)

Annotated list of physical activities

Activities Status HAR HAPT SAR WISDM
Walking Dynamic 1,012 1,722 31,751 2,081
Walking_
Upstairs
Dynamic 858 1544 21,903 632
Walking_
Downstairs
Dynamic 930 1,407 18,751 528
Sitting Static 1,123 1,801 30,000 306
Standing Static 1,029 1,979 30,000 246
Laying Static 792 1958
Jogging Dynamic 29,402 1,625
Stand to Sit Transition 70
Sit to Stand Transition 33
Sit to Lie Transition 107
Lie to Sit Transition 85
Stand to Lie Transition 139
Lie to Stand Transition 84

Optimal values of the proposed method

Parameters Method HAR HAPT SAR WISDM UniMiB SHAR
σ WSVM 0.8 0.8 1.0 0.6 0.8
k AKNN 11 9 5 7 7

Weights wi in HAR data set

ADL Walk Walk.Up Walk.Down Sit Stand Lay
wi 1 1 1 1 1 2

Weights wi in HAPT data set

ADL Walk Walk Up Walk Down Sit Stand Lay
wi 1 1 1 1 1 1
ADL St to sit Sit to St Sit to Lie Lie to sit St to Lie Lie to St
wi 30 62 19 24 16 25

Weights wi in SAR_belt data set

ADL Walk Walk Up Walk Down Sit Stand Jogg
wi 2 1 4 1 1 2

Weights wi in WISDM dataset

ADL Walk Walk Up Walk Down Sit Stand Jogg
wi 2 1 2 1 1 1

Weights wi in UniMib SHAR data set

ADL/Fall FallingLeft Syncope FallingBackSC FallingWithPS HittingObstacle FallingBack FallingRight FallingForw SittingDown
wi 3 3 4 4 2 4 4 4 10
ADL/Fall Lying Down FS GoingDownS Jumping GoingUpS Walking Running Standing UpFL StandingUpFS
wi 6 1 2 2 1 1 9 11

Comparison using the performance metrics (in percent) of the proposed model against the state-of-the-art methods

Dataset Approach Error Rate(%) Recall (%) Precision (%) F1-score(%)
HAR MC-SVM (Anguita et al., 2012) 10.7 89.6 89.9 89.7
MC-HF-SVM (Anguita et al., 2012) 11 89.3 89.2 89.2
Decision Trees (Bharathi and Bhuvana, 2020) 14.1 85.6 85.9 85.7
MLP (Bharathi and Bhuvana, 2020) 1.6 95.1 95.6 95.3
Random Forest (Bharathi and Bhuvana, 2020) 10.5 89.1 89.6 89.3
Logistic Regression (Bharathi and Bhuvana, 2020) 3.9 96.1 96.4 96.2
LSTM (Bharathi and Bhuvana, 2020) 5.5 94.5 94.6 94.5
CNN (Bharathi and Bhuvana, 2020) 5.9 91.9 94.0 92.9
CNN+LSTM (Bharathi and Bhuvana, 2020) 6.0 95.1 92.7 93.8
WSVM 6.6 95.8 93.5 94.6
AKNN-baseline 7.3 93.9 91.8 92.8
PCA/SVM-HMM (Mitchell et al., 2013) 94.1 93.3 93.7
PCA/WSVM-HMM (Abidine and Fergani, 2020) 5.1 94.0 96.7 95.3
LDA/SVM-KNN 3.4 93.1 91.9 92.5
LDA/WSVM-KNN 5.3 94.2 93.4 93.8
KDA/SVM-KNN 7.9 94.8 94.1 94.4
KDA/WSVM-KNN 4.1 96.7 95.7 96.2
Our Proposed Method 0.8 98.9 97.7 98.3
HAPT SVM (Zheng et al., 2018) 4.2 90.6 90.9 90.7
TASG-SVM (Zheng et al., 2018) 3.8 90.7 90.9 90.8
RF (Zheng et al., 2018) 5.7 87.4 88.3 87.8
TASG-RF (Zheng et al., 2018) 5.4 89.3 89.6 89.4
KNN (Zheng et al., 2018) 9.5 82.5 83.4 82.9
TASG-KNN (Zheng et al., 2018) 7.2 86.3 87.9 87.1
RNN (Zheng et al., 2018) 5.5 87.1 86.0 86.5
TASG-RNN (Zheng et al., 2018) 4.2 90.4 91.2 90.8
ANN (Wan et al., 2020) 10.9 65.3
DBN (Wan et al., 2020) 4.2 89.6
WSVM 3.2 95.1 96.0 95.5
AKNN-baseline) 5.1 90.7 92.9 91.8
PCA/WSVM-HMM (Abidine and Fergani, 2020) 3.2 97.3 99.0 98.1
LDA/SVM-KNN 2.9 92.8 94.2 93.5
LDA/WSVM-KNN 1.8 95.2 94.9 95.0
KDA/SVM-KNN 3.4 94.1 95.8 94.9
KDA/WSVM-KNN 2.8 97.3 96.6 96.9
Proposed 0.7 98.5 98.6 98.5
SAR Random forest (Zainudin et al., 2017) 4.8 95.2 95.1 95.1
CNN (San Buenaventura et al., 2018) 3.8
LSTM (San Buenaventura et al., 2018) 4.9
Rotation forest (Mohamed et al., 2018) 1.6 99.5 99.2 99.3
J48 (Mohamed et al., 2018) 3.9 97.0 98.1 97.5
SVM (Mohamed et al., 2018) 38.5 71.7 69.8 70.7
MLP (Mohamed et al., 2018) 26.6 81.3 79.8 80.5
PCA/SVM-HMM (Abidine et al., 2019) 4.1 94.5 95.7 95.1
WSVM 4.4 93.9 94.1 93.9
AKNN-baseline 96.9 94.7 95.8
LDA/SVM-KNN 1.9 94.9 95.8 95.3
LDA/WSVM-KNN) 3.4 96.4 97.8 97.1
KDA/SVM-KNN 1.4 95.9 95.2 95.5
KDA/WSVM-KNN) 2.1 98.1 97.4 97.7
Our Proposed Method 1.1 99.5 99.7 99.6
WISDM J48 (Kwapisz et al., 2011) 14.9 81.7 82.4 82.0
LogisticRegression (Kwapisz et al., 2011) 21.9 68.4 70.5 69.4
MultilayerPerceptron (Kwapisz et al., 2011) 8.3 80.4 84.6 82.4
SVM (Kwapisz et al., 2011) 6.5 81.6 82.4 81.9
BAGGING (Lu et al., 2018) 6.2 81.6 82.4 81.9
KNN (Lu et al., 2018) 8.0 75.1 76.4 75.7
CNN (Shakya et al., 2018) 7.8 90.0 90.1 90.0
WSVM 9.6 88.9 87.8 88.4
AKNN-baseline 11.3 89.5 85.9 87.7
PCA/WSVM-HMM (Abidine and Fergani, 2020) 7.7 91.1 79.8 85.1
LDA/SVM-KNN 8.9 89.8 90.2 89.9
LDA/WSVM-KNN 7.5 93.8 90.4 92.0
KDA/SVM-KNN 9.1 90.7 88.2 89.4
KDA/WSVM-KNN 8.8 94.0 91.9 92.8
Our Proposed Method 4.9 95.4 95.2 95.3
UniMiB-SHAR HC (Li et al., 2018) 67.9 13.7
CBH (Li et al., 2018) 24.8 60.0
CBS (Li et al., 2018) 22.9 63.2
MLP (Li et al., 2018) 28.4 59.9
CNN (Li et al., 2018) 25.0 64.6
LSTM (Li et al., 2018) 28.5 59.3
Hybrid (CNN+LSTM) (Li et al., 2018) 25.3 64.4
MLP-M (Lv et al., 2020) 26.0 61.5
CNN-M (Lv et al., 2020) 25.1 63.3
LSTM-M (Lv et al., 2020) 25.8 59.4
Hybrid-M (CNN+LSTM) (Lv et al., 2020) 22.1 65.3
WSVM 23.8 68.9 66.8 67.8
AKNN -baseline 19.8 61.8 59.1 60.4
PCA/WSVM-HMM 26.8 64.7 63.2 63.9
LDA/SVM-KNN 28.7 61.4 59.8 60.6
LDA/WSVM-KNN 21.7 78.7 79.8 79.2
KDA/SVM-KNN 17.8 79.8 77.7 78.7
KDA/WSVM-KNN 16.8 81.8 79.7 80.7
Our Proposed Method 13.9 84.7 85.8 85.2

CPU Time for the proposed activity classifier on HAR, HAPT, SAR, WISDM and UniMib SHAR datasets

Datasets Training (With all training data) AKNN Training (with SV) our method Testing (with all training data) AKNN Testing (with SV) our method
HAR 58.8 s 8.8 s 20.2 s 4.2 s
HAPT 73.2 s 11.3 s 29.5 s 7.5 s
SAR 128.3 s 50.7 s 45.8 s 9.4 s
WISDM 67.4 s 13.8 s 27.7 s 7.2 s
UniMib SHAR 95.1 s 16.8 s 31.1 s 5.8 s

Memory resources for the training phase of the proposed method on HAR, HAPT, SAR, WISDM and UniMib SHAR data sets, compared to AKNN baseline

Data set HAR HAPT SAR WiSDN UniMiB SHAR
AKNN-baseline -Training 4020 MB 7650 MB 113284 MB 3792 MB 8267 MB
Our method - Training 867 MB 994 MB 13654 MB 668 MB 1185 MB

Confusion matrix of proposed method on the HAR data set (percentage)

Activities Walking W. Upstairs W. Downstairs Sitting Standing Laying
Walking 98.2 1.2 0.6 0.0 0.0 0.0
Walking_Upstairs 0.2 99.3 0.3 0.0 0.0 0.2
Walking_Downstairs 0.8 0.4 98.8 0.0 0.0 0.0
Sitting 0.0 0.0 0.0 96.9 2.7 0.4
Standing 0.0 0.3 0.0 1.4 97.8 0.5
Laying 0.0 0.0 0.0 0.0 0.0 100.0

Confusion matrix of proposed method on the SAR dataset (percentage)

Activities Walking W. Upstairs W. Downstairs Sitting Standing Jogging
Walking 99.4 0.2 0.2 0.0 0.0 0.0
Walking. Upstairs 0.3 98.7 0.9 0.0 0.1 0.0
Walking. Downstairs 0.2 0.8 98.9 0.1 0.0 0.1
Sitting 0.0 0.0 0.0 100 0.0 0.0
Standing 0.0 0.0 0.0 0.2 99.7 0.1
Jogging 0.0 0.0 0.0 0.0 0.0 100

Confusion matrix of proposed method on the WISDM dataset (percentage)

Activities Walking W. Upstairs W. Downstairs Sitting Standing Jogging
Walking 97.9 0.0 0.0 0.9 0.9 0.3
Walking. Upstairs 0.6 88.9 6.1 0.4 0.0 4.0
Walking. Downstairs 0.2 6.8 92.9 0.0 0.0 0.1
Sitting 0.7 0.3 0.0 96.5 1.2 1.3
Standing 0.9 0.1 0.0 1.1 97.4 0.5
Jogging 0.3 0.0 0.0 0.0 0.6 99.1

References

Abidine, M.B. and Fergani, B. (2020), “Human activities recognition in android smartphone using WSVM-HMM classifier”, International Conference on Smart Homes and Health Telematics, Springer, Cham, pp. 386-394, doi: 10.1007/978-3-030-51517-1_35.

Abidine, M.B., Fergani, L., Fergani, B. and Oussalah, M. (2018), “The joint use of sequence features combination and modified weighted SVM for improving daily activity recognition”, Pattern Analysis and Applications, Vol. 21 No. 1, pp. 119-138, doi: 10.1007/s10044-016-0570-y. Springer,

Abidine, M.B., Fergani, B. and Menhour, I. (2019), “Activity recognition from smartphones using hybrid classifier PCA-SVM-HMM”, 2019 International Conference on Wireless Networks and Mobile Communications (WINCOM), IEEE, pp. 1-5, doi: 10.1109/WINCOM47513.2019.8942492.

Anguita, D., Ghio, A., Oneto, L., Parra, X. and Reyes-Ortiz, J.L. (2012), “Human activity recognition on smartphones using a multiclass Hardware-Friendly support vector machine”, in Bravo, J., Hervás, R. and Rodríguez, M. (Eds), Ambient Assisted Living and Home Care, Springer Berlin Heidelberg, pp. 216-223, doi: 10.1007/978-3-642-35395-6_30.

Anguita, D., Ghio, A., Oneto, L., Parra Perez, X. and Reyes Ortiz, J.L. (2013), “A public domain dataset for human activity recognition using smartphones”, 21th European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning, ESANN, Bruges, Belgium.

Barua, A., Masum, A.K.M., Hossain, M.E., Bahadur, E.H. and Alam, M.S. (2019), “A study on human activity recognition using gyroscope, accelerometer, temperature and humidity data”, 2019 international conference on electrical, computer and communication engineering (ECCE), pp. 1-6, doi: 10.1109/ECACE.2019.8679226.

Bayat, A., Pomplun, M. and Tran, D.A. (2014), “A study on human activity recognition using accelerometer data from smartphones”, Procedia Computer Science, Vol. 34, pp. 450-457, doi: 10.1016/j.procs.2014.07.009.

Bharathi, B. and Bhuvana, J. (2020), “Human activity recognition using deep and machine learning algorithms”, International Journal of Innovative Technology and Exploring Engineering (IJITEE), Vol. 9 No. 4, pp. 2460-2466, ISSN: 2278-3075.

Bulling, A., Blanke, U. and Schiele, B. (2014), “A tutorial on human activity recognition using body-worn inertial sensors”, ACM Computing Surveys, Vol. 46 No. 3, pp. 1-33., doi: 10.1145/2499621.

Capela, N.A., Lemaire, E.D. and Baddour, N. (2015), “Feature selection for wearable smartphone-based human activity recognition with able bodied, elderly, and stroke patients”, PloS One, Vol. 10 No. 4, pp. 1-18, doi: 10.1371/journal.pone.0124414, e0124414.

Catal, C., Tufekci, S., Pirmit, E. and Kocabag, G. (2015), “On the use of ensemble of classifiers for accelerometer-based activity recognition”, Applied Soft Computing, Vol. 37, pp. 1018-1022, doi: 10.1016/j.asoc.2015.01.025.

Chen, Y. and Shen, C. (2017), “Performance analysis of smartphone-sensor behavior for human activity recognition”, IEEE Access, Vol. 5, pp. 3095-3110, doi: 10.1109/ACCESS.2017.2676168.

Dzulkifli, S.A., Salleh, M.N.M. and Talpur, K.H. (2019), “Improved weighted learning support vector machines (SVM) for high accuracy”, Proceedings of the 2019 2nd International Conference on Computational Intelligence and Intelligent Systems, pp. 40-44, doi: 10.1145/3372422.3372432.

Fan, L., Wang, Z. and Wang, H. (2013), “Human activity recognition model based on decision tree”, 2013 International Conference on Advanced Cloud and Big Data, IEEE, Nanjing, pp. 64-68, doi: 10.1109/CBD.2013.19.

Fang, H., He, L., Si, H., Liu, P. and Xie, X. (2014), “Human activity recognition based on feature selection in smart home using back-propagation algorithm”, ISA Transactions, Vol. 53 No. 5, pp. 1629-1638, doi: 10.1016/j.isatra.2014.06.008.

Fernández, A., García, S., Galar, M., Prati, R.C., Krawczyk, B. and Herrera, F. (2018), Learning from Imbalanced Data Sets, Vol. 11, Springer, Berlin, pp. 1-377, doi: 10.1007/978-3-319-98074-4.

Gao, L., Bourke, A. and Nelson, J. (2014), “Evaluation of accelerometer based multi-sensor versus single-sensor activity recognition systems”, Medical Engineering & Physics, Vol. 36 No. 6, pp. 779-785, doi: 10.1016/j.medengphy.2014.02.012.

Horn, G., Eliassen, F., Taherkordi, A., Venticinque, S., Martino, B.D., Bücher, M. and Wood, L. (2016), “An architecture for using commodity devices and smart phones in health systems”, Proceedings of the 2016 IEEE Symposium on Computers and Communication (ISCC), Messina, pp. 255-260, doi: 10.1109/ISCC.2016.7543749

Hsu, C.W., Chang, C.C. and Lin, C.J. (2003), “A practical guide to support vector classification”, in Chang, C.-C. and Lin, C.-J. (Eds), LIBSVM: A library for support vector machines’. ACM Transactions on Intelligent Systems and Technology, Vol. 2, pp. 1-27.

Kose, M., Incel, O.D. and Ersoy, C. (2012), April), “Online human activity recognition on smart phones”, Workshop on Mobile Sensing: From Smartphones and Wearables to Big Data, Vol. 16, No. 2012, pp. 11-15.

Kung, S.Y. (2014), Kernel Methods and Machine Learning, Cambridge University Press, UK, USA, Australia, India and Singapore.

Kwapisz, J.R., Weiss, G.M. and Moore, S.A. (2011), “Activity recognition using cell phone accelerometers”, ACM SIGKDD Explorations Newsletter, Vol. 12 No. 2, pp. 74-82.

Lee, Y.S. and Cho, S.B. (2016), “Layered hidden Markov models to recognize activity with built-in sensors on android smartphone”, Pattern Analysis and Applications, Vol. 19 No. 4, pp. 1181-1193, doi: 10.1007/s10044-016-0549-8.

Lester, J., Choudhury, T. and Borriello, G. (2006), “A practical approach to recognizing physical activities”, Proceedings of the 4th International Conference on Pervasive Computing, Dublin, pp. 1-16, doi: 10.1007/11748625_1.

Li, F., Kimiaki, S., Muhammad, N., Lukas, K. and Marcin, G. (2018), “Comparison of feature learning methods for human activity recognition using wearable sensors”, Sensors, Vol. 18 No. 3, p. 679.

Lu, W., Fan, F., Chu, J., Jing, P. and Yuting, S. (2018), “Wearable computing for internet of things: a discriminant approach for human activity recognition”, IEEE Internet of Things Journal, Vol. 6 No. 2, pp. 2749-2759, doi: 10.1109/JIOT.2018.2873594.

Lv, T., Wang, X., Jin, L., Xiao, Y. and Song, M. (2020), “Margin-based deep learning networks for human activity recognition”, Sensors, Vol. 20 No. 7, p. 1871.

Mandong, A. and Munir, U. (2018), “Smartphone based activity recognition using K-Nearest neighbor algorithm”, Proceedings of the International Conference on Engineering Technologies, Konya, pp. 26-28.

Menhour, I., Abidine, M.B. and Fergani, B. (2018), “A new framework using PCA, LDA and KNN-SVM to activity recognition based SmartPhone’s sensors”, 2018 6th International Conference on Multimedia Computing and Systems (ICMCS), IEEE, pp. 1-5, doi: 10.1109/ICMCS.2018.8525987.

Menhour, I., Abidine, M.H. and Fergani, B. (2019), “A new activity classification method K-SVM using smartphone data”, 2019 International Conference on Advanced Electrical Engineering (ICAEE), IEEE, pp. 1-5, doi: 10.1109/ICAEE47123.2019.9014821.

Micucci, D., Mobilio, M. and Napoletano, P. (2017), “Unimib shar: a dataset for human activity recognition using acceleration data from smartphones”, Applied Sciences, Vol. 7 No. 10, p. 1101.

Minetto, L. (2015), “Machine learning approach to sport activity recognition from inertial data PhD thesis”, Università Degli Studi Di Padova.

Mitchell, E., Monaghan, D. and O’Connor, N.E. (2013), “Classification of sporting activities using smartphone accelerometers”, Sensors, Vol. 13 No. 4, pp. 5317-5337, doi: 10.3390/s130405317.

Mohamed, R., Zainudin, M.N.S., Sulaiman, M.N., Perumal, T. and Mustapha, N. (2018), “Multi-label classification for physical activity recognition from various accelerometer sensor positions”, Journal of Information and Communication Technology, Vol. 17 No. 2, pp. 209-231, doi: 10.32890/jict2018.17.2.3.

Nj, S.A. and Kavitha, R. (2017), “Human activity recognition from sensor data using random forest algorithm”, International Journal of Advanced Research in Computer Science, Vol. 8 No. 3, pp. 334-337, doi: 10.26483/ijarcs.v8i3.3009.

Ordóñez, F.J. and Roggen, D. (2016), “Deep convolutional and LSTM recurrent neural networks for multimodal wearable activity recognition”, Sensors, Vol. 16 No. 1, p. 115, doi: 10.3390/s16010115.

Ordóñez, F.J., Duque, A., de Toledo, P. and Sanchis, A. (2012), “A hybrid HMM/ANN model for activity recognition in the home using binary sensors”, Ambient Assisted Living and Home Care - 4th International Workshop, WAAL, Vitoria-Gasteiz, pp. 98-105, doi: 10.1007/978-3-642-35395-6_13.

Pei, L., Liu, J., Guinness, R., Chen, Y., Kuusniemi, H. and Chen, R. (2012), “Using LS-SVM based motion recognition for smartphone indoor wireless positioning”, Sensors, Vol. 12 No. 5, pp. 6155-6175, doi: 10.3390/s120506155.

Randhawa, P., Shanthagiri, V., Kumar, A. and Yadav, V. (2020), “Human activity detection using machine learning methods from wearable sensors”, Sensor Review, Vol. 40 No. 5, pp. 591-603, doi: 10.1108/SR-02-2020-0027.

Ray, P.P., Dash, D. and De, D. (2019), “Analysis and monitoring of IoT-assisted human physiological galvanic skin responsefactor for smart e-healthcare”, Sensor Review, Vol. 39 No. 4, pp. 525-541, doi: 10.1108/SR-07-2018-0181.

Reyes-Ortiz, J.L., Oneto, L., Sama, A., Parra, X. and Anguita, D. (2016), “Transition-Aware human activity recognition using smartphones”, Neurocomputing, Vol. 171, pp. 754-767.

San Buenaventura, C.V., Tiglao, N.M.C. and Atienza, R.O. (2018), “Deep learning for smartphone-based human activity recognition using multi-sensor fusion”, International Wireless Internet Conference, Springer, Cham, pp. 65-75, doi: 10.1007/978-3-030-06158-6_7.

Shakya, S.R., Zhang, C. and Zhou, Z. (2018), “Comparative study of machine learning and deep learning architecture for human activity recognition using accelerometer data”, International Journal of Machine Learning and Computing, Vol. 8, pp. 577-582, doi: 10.18178/ijmlc.2018.8.6.748.

Shoaib, M., Bosch, S., Incel, O.D., Scholten, H. and Havinga, P.J.M. (2014), “Fusion of smartphone motion sensors for physical activity recognition”, Sensors, Vol. 14 No. 6, pp. 10146-10176.

Straczkiewicz, M., James, P. and Onnela, J.P. (2021), “A systematic review of smartphone-based human activity recognition methods for health research”, NPJ Digital Medicine, Vol. 4 No. 1, p. 148, doi: 10.1038/s41746-021-00514-4.

Suto, J. and Oniga, S. (2018), “Efficiency investigation of artificial neural networks in human activity recognition”, Journal of Ambient Intelligence and Humanized Computing, Vol. 9 No. 4, pp. 1049-1060, doi: 10.1007/s12652-017-0513-5.

Tian, Y., Zhang, J., Chen, L., Geng, Y. and Wang, X. (2019), “Single wearable accelerometer-based human activity recognition via kernel discriminant analysis and QPSO-KELM classifier”, IEEE Access, Vol. 7, pp. 109216-109227, doi: 10.1109/ACCESS.2019.2933852.

Voicu, R.A., Dobre, C., Bajenaru, L. and Ciobanu, R.I. (2019), “Human physical activity recognition using smartphone sensors”, Sensors, Vol. 19 No. 3, p. 458, doi: 10.3390/s19030458.

Wan, S., Qi, L., Xu, X., Tong, C. and Gu, Z. (2020), “Deep learning models for real-time human activity recognition with smartphones”, Mobile Networks and Applications, Vol. 25 No. 2, pp. 743-755, doi: 10.1007/s11036-019-01445-x.

Wu, Y., Qi, S., Hu, F., Ma, S., Mao, W. and Li, W. (2019), “Recognizing activities of the elderly using wearable sensors: a comparison of ensemble algorithms based on boosting”, Sensor Review, Vol. 39 No. 6, pp. 743-751, doi: 10.1108/SR-11-2018-0309.

Yang, J. (2009), “Toward physical activity diary: motion recognition using simple acceleration features with mobile phones”, Proceedings of the 1st international workshop on Interactive multimedia for consumer electronics, ACM. pp. 1-10, doi: 10.1145/1631040.1631042.

Yang, J., Wang, J. and Chen, Y. (2008), “Using acceleration measurements for activity recognition: an effective learning algorithm for constructing neural classifiers”, Pattern Recognition Letters, Vol. 29 No. 16, pp. 2213-2220, doi: 10.1016/j.patrec.2008.08.002.

Yang, X., Song, Q. and Wang, Y. (2007), “A weighted support vector machine for data classification”, International Journal of Pattern Recognition and Artificial Intelligence, Vol. 21 No. 5, pp. 961-976, doi: 10.1142/S0218001407005703.

Yuan, G., Wang, Z., Meng, F., Yan, Q. and Xia, S. (2019), “An overview of human activity recognition based on smartphone”, Sensor Review, Vol. 39 No. 2, pp. 288-306, doi: 10.1108/SR-11-2017-0245.

Zainudin, M.S., Mohamed, R., Sulaiman, M.N., Perumal, T., Mustapha, N. and Nazri, A.S.A. (2017), “Multi-label classification using label combination to recognize human activity based on various sensor positions”, Proceedings of the 6th International Conference on Computing and Informatics, ICOCI’2017, Kuala Lumpur, Malaysia, doi: 10.32890/jict2018.17.2.3.

Zheng, Z., Du, J., Sun, L., Huo, M. and Chen, Y. (2018), “TASG: an augmented classification method for impersonal HAR”, Mobile Information Systems, Vol. 2018, pp. 1-10, doi: 10.1155/2018/6751363.

Further reading

Chetty, G., White, M. and Akther, F. (2015), “Smart phone based data mining for human activity recognition”, Elsevier Procedia Computer Science – ICICT, vol. 46, pp. 1181-1187, doi: 10.1016/j.procs.2015.01.031.

Wu, W. and Zhang, Y. (2019), “Activity recognition from mobile phone using deep CNN”, 2019 Chinese Control Conference (CCC), IEEE, pp. 7786-7790, doi: 10.23919/ChiCC.2019.8865142.

Acknowledgements

The research presented in this paper is partly funded by Algeria National and Development Program PRFU (Projets de Recherche Formation-Universitaire) with Key Research project (A25N01UN160420180003), Communication and Intelligent System Engineering Laboratory. The authors also acknowledge the Academy of Finland Project Profi5 (#326291) on DigiHealth.

Corresponding author

Mourad Oussalah can be contacted at: Mourad.Oussalah@oulu.fi

About the authors

M’hamed Bilal Abidine was born in Algiers, Algeria, in 1985. He received BS degree in Exact Sciences, Algiers, Algeria in 2002. Then the M.S, the Magister and PhD degrees from the Faculty of Electronics and Computer Sciences, University of Sciences and Technology Houari Boumediene, Algiers, Algeria, in 2007, 2010, and 2017, respectively. Currently, he is an Associate Professor, Researcher and he prepared the Post-Doc at the Faculty of Electronics and Computer Sciences, USTHB, Algiers, Algeria. His research focuses on pattern recognition and machine learning methods and activity recognition using wireless sensor networks and systems.

Mourad Oussalah is a Research Professor at University of Oulu, Faculty of Information Technology and Electrical Engineering where he is leading the social mining research group. His research interests include data mining, information retrieval, pattern recognition and computer vision where he published more than 200 technical papers, supervised dozen of PhD students and led several EU and national research projects in the field. He is a Fellow of Royal statistical society and a senior IEEE member. Prior to his current employment, he has academic position at University of Birmingham in UK (2003–2016), City University of London (2000–2003), KU Leuven (1998–2000).

Belkacem Fergani was born in Medea (80 km south Algiers) in 1963. After a BS on Mathematics in 1982, M.S and Magister degrees in electronics and Signal Processing, respectively, in 1987 and 1992, he gets a PhD in Signal Processing in 2007. He is currently a Full Professor at the Electronics and Computer Sciences Department at the University of Sciences and Technology of Algiers. His research focuses on pattern recognition and machine learning methods and activity recognition using wireless sensor networks and RFID sensors. He leads several PhD students and projects research on the activity daily living and pattern recognition.

Hakim Lounis is Professor at Université du Québec À Montréal, UQAM, Montréal, Canada, where he was also director of the doctoral program in cognitive computing, from 2014 to 2020. He received PhD degree in Computer Science from the University of Paris-Sud, Orsay, France, in 1994. Before joining UQAM in 2000, he was a researcher at Centre de Recherche Informatique de Montréal, CRIM, from 1996 to 1999, and principal researcher from 1999 to 2000. His research focuses on artificial intelligence, machine-learning, cognitive computing and also on software engineering and software quality.

Related articles