Search results

The purpose of this paper is to explain why ROC analysis is an inappropriate replacement for probative analysis in lineup research.

Taking as the medical example comparing two methods to detect the presence of a malignant tumor (Mickes et al., 2012), and operationally defining ROC analysis: radiologists are shown the results from two methods. Their confidence judgments create a graph of correct identifications by mistaken ones. The author can compare the methods on radiologists’ ability to differentiate sick from healthy. Lineup researchers create two distinct lineups. In target-present lineups, witnesses differentiate between the target and the foils, not the target and the innocent suspect. In target-absent lineups, witnesses cannot even differentiate between innocent suspects and foils, having seen none.

Eyewitness ROC curves are similar to probative analysis, but provide less useful information.

Researchers ware warned against using ROC when conducting lineup research.

Preventing inappropriate use of ROC analysis.

Receiver operating characteristic (ROC) analysis is a powerful tool to measure and specify the problems with diagnostic performance in medicine. Describes this analysis and the importance of the index area under ROC curve, using some examples to demonstrate its application. The study was conducted on two sets of new‐borns with very low birth weight, coming from neonatal intensive care units from Portuguese hospitals. The first application uses correlated samples, and aims to define which of the five indices of clinical seriousness can be considered the best to evaluate the risk of death for babies with very low birth weight. In the second application, regarding independent samples, compares four Portuguese hospitals, the aim being to identify the neonatal intensive care unit which presents the best performance in terms of care to the new‐borns, i.e. evaluated through the comparison of the clinical severity indices.

The purpose of this article is to introduce Receiver Operating Characteristic (ROC) surfaces and hyper‐surfaces within a banking context as natural generalizations of the ROC curve.

Nonparametric ROC analysis using U‐statistics theory was used.

Application of the proposed methodology on data from a small size Greek bank illustrates the usefulness of ROC analysis for scoring systems assessment. The area under the ROC curve and the volume under the ROC surface and hyper‐surface are useful diagnostic indices for the assessment of credit rating systems and scorecards. The notion of statistical significance is not adequate for the evaluation of the loan granting strategy of a financial institution.

This article will be of value to financial institutions during the process of evaluation/validation of rating models.

This study sheds light on the prediction of success using cutoff scores for student grades adopted for a required Physics pathway course for study in a health professions program at King Saud University in Saudi Arabia. Data on course grade and GPA for approximately 10,000 students enrolled in this course between 2008–2014, were analyzed. Receiver Operating Characteristic (ROC) curve analysis was used to determine cutoffs for course grades using ranges of GPA. This procedure has promise as a new method for quantitatively arriving at cutoff scores using an external criterion requiring less human judgment than most existing standard setting methods. The cutoff scores produced show that GPAs of students who complete the Physics course yield successive performance tiers that are lower than expected. In addition, the correlation between GPA and course grade for Physics is only 0.63 and therefore only 39% of the variation in GPA explains course grade. As a result of the findings of the study, the decision was made to maintain the existing standards thereby requiring higher grades in the Physics course for students seeking to enter a health professions course of study.

ﻧﺗﻟا ﺎﮭﺗردﻗو ﺔﯾدﺣﻟا تﺎﺟردﻟا ﻰﻠﻋ ءوﺿﻟا ﺔﯾﻟﺎﺣﻟا ﺔﺳاردﻟا طﻠﺳﺗ ﻲﻓ ﺔﺑﻠطﻟا حﺎﺟﻧ ﻰﻠﻋ ﺔﯾؤﺑﻲﻟوﻻا ءﺎﯾزﯾﻔﻟا ررﻘﻣ زﯾﻓ)145( ، دﻌﯾ يذﻟاوﻠﻋ ﺎﯾﺳﺎﺳا ﺎﺑﻠطﺗﻣكﻠﻣﻟا ﺔﻌﻣﺎﺟ ﻲﻓ ﺔﯾﺣﺻﻟا تﺎﺻﺻﺧﺗﻟا ﺔﺑﻠط ﻰ ﺔﻘﻠﻌﺗﻣﻟا تﺎﻧﺎﯾﺑﻟا ﻊﻣﺟ مﺗ دﻘﻓ ﺔﯾﻠﻋو ،ﺔﯾدوﻌﺳﻟا ﺔﯾﺑرﻌﻟا ﺔﻛﻠﻣﻣﻟﺎﺑ دوﻌﺳ نﻣ برﺎﻘﯾ ﺎﻣﻟ ررﻘﻣﻟا اذھ تﺎﺟردﺑ10000 ماوﻋﻻا نﯾﺑ ررﻘﻣﻟا اذﮭﺑ اوﻘﺣﺗﻟا نﯾذﻠﻟا ﺔﺑﻠطﻟا نﻣ2008 - 2014 .ﺔﯾﻣﻛارﺗﻟا مﮭﺗﻻدﻌﻣو ،م ﺗﻟو مادﺧﺗﺳا مﺗ دﻘﻓ ،تﺎﻧﺎﯾﺑﻟا هذھ لﯾﻠﺣ تﺎﯾﻠﻣﻌﻟا لﯾﻐﺷﺗ ﺔﯾﺻﺎﺧ ﻰﻧﺣﻧﻣReceiver Operating Characteristic (ROC) تﺎﺟردﻟا دﯾدﺣﺗﻟ نﻣ دﺣﻟاو ﺔﯾدﺣﻟا تﺎﺟردﻟا ﻰﻟا لوﺻوﻠﻟ ﺔﺛﯾدﺣﻟا ﺔﯾﻣﻛﻟا قرطﻟا نﻣ ﺔﻘﯾرطﻟا هذھ دﻌﺗ ثﯾﺣ ،ﺔﯾﻣﻛارﺗﻟا تﻻدﻌﻣﻟا نﻣ ﺔﻔﻠﺗﺧﻣ تﺎﻗﺎطﻧﻟ ﺔﯾدﺣﻟارﯾﺛﺄﺗﻟا .يرﺷﺑﻟا نﻣﺿ نﺎﻛ ررﻘﻣﻟا اذھ زﺎﺗﺟا نﻣﻟ ﺔﯾﻣﻛارﺗﻟا تﻻدﻌﻣﻟا نا ﻰﻟا ترﺎﺷا دﻗ ﺎﮭﯾﻠﻋ لوﺻﺣﻟا مﺗ ﻲﺗﻟا ﮫﯾدﺣﻟا تﺎﺟردﻟا نﺎﻓ ﺔﯾﻠﻋو تﻐﻠﺑ ﺔﺑﻠطﻟا تﺎﺟردو ﺔﯾﻣﻛارﺗﻟا تﻻدﻌﻣﻟا نﯾﺑ ﺔﯾطﺎﺑﺗرﻻا ﺔﻗﻼﻌﻟا نا ﺎﻣﻛ ،ﻊﻗوﺗﻣﻟا نﻣ لﻗا تﺎﻗﺎطﻧ0.63 ﻲﻧﻌﯾ ﺎﻣﻣ ، نا 39% نﻣتﺎﻧﯾﺎﺑﺗﻟا ﺔﯾﻣھا نﯾﺑﺗﯾ ﺞﺋﺎﺗﻧ نﻣ ﺔﯾﻠﻋ لوﺻﺣﻟا مﺗ ﺎﻣﻟ ﺎﻘﻓوو .رﻘﻣﻟا كﻟذ ﻲﻓ مﮭﺗﺎﺟرد رﯾﺳﻔﺗ ﻲﻓ مﮭﺳﺗ نا نﻛﻣﯾ ﺔﺑﻠطﻠﻟ ﺔﯾﻣﻛارﺗﻟا تﻻدﻌﻣﻟا ﻲﻓﺔظﻓﺎﺣﻣﻟا ﻰﻠﻋ ررﻘﻣﻟا رﯾﯾﺎﻌﻣﺔﯾﻟﺎﺣﻟا ﻊﻣدﯾﻛﺄﺗ لوﺻﺣﺔﺑﻠطﻟا تﺎﺻﺻﺧﺗﻟﺎﺑ قﺎﺣﺗﻟﻼﻟ نﯾﺑﻏارﻟاﺔﯾﺣﺻﻟا تﺎﺟرد ﻰﻠﻋﺔﻌﻔﺗرﻣ .ءﺎﯾزﯾﻔﻟا ررﻘﻣ ﻲﻓ

The purpose of this paper is to identify different job types in the Belgian electricity sector and their relations with employee outcomes such as work engagement and innovative work behaviour (IWB).

This paper uses a combination of latent profile analysis and relative operating characteristics (ROC) analysis.

Depending on the job resources and demands, five different job types are identified corresponding largely to the Karasek and Theorell (1990) job types. Their relation with the outcomes is not parallel with low-strain jobs performing best for work engagement, and active jobs for IWB.

The combination of methods used in this study increases significantly the ease of communication of the findings, yet an external benchmark for the ROC analysis would be preferable.

To foster engagement and IWB with employees one should focus on the job content and only increase demands if they are combined with sufficient resources.

This research is the first in its kind that relates latent job types with different employee outcomes using a combination of latent profile and ROC analysis.

Signal Detection Theory (SDT) has recently been used to evaluate the performance of imperfect inspectors. SDT model is based on a priori probabilities and perceived payoffs and penalties to study inspectors′ behaviour. In this article, Bayes′ theorem is used to compute posterior probabilities of the two types of inspection error. These posterior probabilities give rise to the definition of Receiver Analysis Curves (RAC), which depict the “after the facts” consequences of inspection error. A cost model is also developed that reflects the true benefits and costs of inspection accuracy to the organisation.

This paper aims to compare two diagnostic performance measures, i.e. signal‐to‐noise ratio (S/N ratio) and partial area under receiver operating characteristic curves (pAUC). It proposes the use of S/N ratio rather than pAUC for establishing optimal cut‐off point for diagnostic biomarkers.

This paper discusses the properties, uses, advantages and shortcomings of the two performance measures, namely the partial area under receiver operating characteristic curve (pAUC) and Taguchi's signal‐to‐noise (S/N) ratio. The benefits of S/N ratio have been illustrated in a sample of four biomarkers, each having five cut‐off points. The S/N ratio is compared to the pAUC index. The SAS software is employed to calculate pAUC and AUC.

This paper shows that S/N ratio can be used as a measure of diagnostic accuracy. The cut‐off point with the highest S/N ratio is the optimal cut‐off point for the biomarker. The proposed method has the advantages of being easier, more practical and less costly than that of pAUC.

This paper includes implications for the development of a more practical, equally powerful and less costly means of measuring clinical accuracy thereby reducing the costs and risks resulting from wrong selection of cut‐off point can be decreased.

This paper supports suggestions in the recent literature to replace pAUC with a new, more meaningful index.

The purpose of this paper is to assess the performance of competing methods and model selection, which are non‐trivial issues given the financial implications. Researchers have adopted various methods including statistical models and machine learning methods such as neural networks to assist decision making in direct marketing. However, due to the different performance criteria and validation techniques currently in practice, comparing different methods is often not straightforward.

This study compares the performance of neural networks with that of classification and regression tree, latent class models and logistic regression using three criteria – simple error rate, area under the receiver operating characteristic curve (AUROC), and cumulative lift – and two validation methods, i.e. bootstrap and stratified k‐fold cross‐validation. Systematic experiments are conducted to compare their performance.

The results suggest that these methods vary in performance across different criteria and validation methods. Overall, neural networks outperform the others in AUROC value and cumulative lifts, and the stratified ten‐fold cross‐validation produces more accurate results than bootstrap validation.

To select predictive models to support direct marketing decisions, researchers need to adopt appropriate performance criteria and validation procedures.

The study addresses the key issues in model selection, i.e. performance criteria and validation methods, and conducts systematic analyses to generate the findings and practical implications.

Warranty service plays a critical role in sustainability and service continuity and influences customer satisfaction. Considering the role of social networks in customer feedback channels, one of the essential sources to examine the reflection of a product/service is social media mining. This paper aims to identify the frequent product failures through social network mining. Focusing on social media data as a comprehensive and online source to detect warranty issues reveals opportunities for improvement, such as user problems and necessities. This model will detect the causes of defects and prioritize improving components in a product-service system based on FMEA results.

Ontology-based methods, text mining and sentiment analysis with machine learning methods are performed on social media data to investigate product defects, symptoms and the relationship between warranty plans and customer behaviour. Also, the authors have incorporated multi-source data collection to cover all the possibilities. Then the authors promote a decision support system to help the decision-makers using the FMEA process have a more comprehensive insight through customer feedback. Finally, to validate the accuracy and reliability of the results, the authors used the operational data of a LENOVO laptop from a warranty service centre and classifier performance metrics to compare the authors’ results.

This study confirms the validity of social media data in detecting customer sentiments and discovering the most defective components and failures of the products/services. In other words, the informative threads are derived through a data preparation process and then are based on analyzing the different features of a failure (issues, symptoms, causes, components, solutions). Using social media data helps gain more accurate online information due to the limitation of warranty periods. In other words, using social media data broadens the scope of data gathering and lets in all feedback from different sources to recognize improvement opportunities.

This work contributes a DSS model using multi-channel social media mining through supervised machine learning for warranty-service improvement based on defect-related discovery to unravel the potential aspects of social networks analysis to predict the most vulnerable components of a product and the main causes of failures that lead to the inputs for the FMEA process and then, a cost optimization. The authors have used social media channels like Twitter, Facebook, Reddit, LENOVO Forums, GitHub, Quora and XDA-Developers to gather data about the LENOVO laptop failures as a case study.

Software fault prediction is an important concept that can be applied at an early stage of the software life cycle. Effective prediction of faults may improve the reliability and testability of software systems. As service-oriented architecture (SOA)-based systems become more and more complex, the interaction between participating services increases frequently. The component services may generate enormous reports and fault information. Although considerable research has stressed on developing fault-proneness prediction models in service-oriented systems (SOS) using machine learning (ML) techniques, there has been little work on assessing how effective the source code metrics are for fault prediction. The paper aims to discuss this issue.

In this paper, the authors have proposed a fault prediction framework to investigate fault prediction in SOS using metrics of web services. The effectiveness of the model has been explored by applying six ML techniques, namely, Naïve Bayes, Artificial Networks (ANN), Adaptive Boosting (AdaBoost), decision tree, Random Forests and Support Vector Machine (SVM), along with five feature selection techniques to extract the essential metrics. The authors have explored accuracy, precision, recall, f-measure and receiver operating characteristic curves of the area under curve values as performance measures.

The experimental results show that the proposed system can classify the fault-proneness of web services, whether the service is faulty or non-faulty, as a binary-valued output automatically and effectively.

One possible threat to internal validity in the study is the unknown effects of undiscovered faults. Specifically, the authors have injected possible faults into the classes using Java C3.0 tool and only fixed faults are injected into the classes. However, considering the Java C3.0 community of development, testing and use, the authors can generalize that the undiscovered faults should be few and have less impact on the results presented in this study, and that the results may be limited to the investigated complexity metrics and the used ML techniques.

In the literature, only few studies have been observed to directly concentrate on metrics-based fault-proneness prediction of SOS using ML techniques. However, most of the contributions are regarding the fault prediction of the general systems rather than SOS. A majority of them have considered reliability, changeability, maintainability using a logging/history-based approach and mathematical modeling rather than fault prediction in SOS using metrics. Thus, the authors have extended the above contributions further by applying supervised ML techniques over web services metrics and measured their capability by employing fault injection methods.