Artificial intelligence in overcoming rifampicin resistant-screening challenges in Indonesia: a qualitative study on the user experience of CUHAS-ROBUST

Abstract

Purpose

The Chulalongkorn-Hasanuddin Rifampicin-Resistant Tuberculosis Screening Tool (CUHAS-ROBUST) is an artificial intelligence–based (AI–based) application for rifampicin-resistant tuberculosis (RR-TB) screening. This study aims to elaborate on the drug-resistant TB (DR-TB) problem and the impact of CUHAS-ROBUST implementation on RR-TB screening.

Design/methodology/approach

A qualitative approach with content analysis was performed from September 2020 to October 2020. Medical staff from the primary care center were invited online for application trials and in-depth video call interviews. Transcripts were derived as a data source. An inductive thematic data saturation technique was conducted. Descriptive data of participants, user experience and the impact on the health service were summarized

Findings

A total of 33 participants were selected from eight major islands in Indonesia. The findings show that DR-TB is a new threat, and its diagnosis faces obstacles particularly prolonged waiting time and inevitable delayed treatment. Despite overcoming the RR-TB screening problems with fast prediction, the dubious screening performance, and the reliability of data collection for input parameters were the main concerns of CUHAS-ROBUST. Nevertheless, this application increases the confidence in decision-making, promotes medical procedure compliance, active surveillance and enhancing a low-cost screening approach.

Originality/value

The CUHAS-ROBUST achieved its purpose as a tool for clinical decision-making in RR-TB screening. Moreover, this study demonstrates AI roles in enhancing health-care quality and boost public health efforts against tuberculosis.

Citation

Herman, B., Sirichokchatchawan, W., Nantasenamat, C. and Pongpanich, S. (2022), "Artificial intelligence in overcoming rifampicin resistant-screening challenges in Indonesia: a qualitative study on the user experience of CUHAS-ROBUST", Journal of Health Research, Vol. 36 No. 6, pp. 1018-1027. https://doi.org/10.1108/JHR-11-2020-0535

Introduction

Drug-resistant tuberculosis (DR-TB), and particularly multidrug-resistant tuberculosis (MDR-TB) is a public health threat. Several countries have reported that the incidences of MDR-TB accounted for 5.7% and projected 8.5–9.0% of total TB cases in 2040 [1]. Rifampicin-resistant tuberculosis (RR-TB) is associated with MDR-TB and requires similar treatment and management. Therefore, enhancing RR-TB screening should be prioritized.

Diagnosing DR-TB is essential, and the phenotypic drug-susceptibility test (DST) is the standard reference. It is based on the observed growth of bacteria in culture. Technical problems were revealed including that it is time-consuming, open to contamination, and that there are interpretation reliability issues [2]. A nucleic acid amplification technology (the GeneXpert MTB/RIF) (Cepheid) can detect the rpoB gene from various samples including sputum [3], pleural and pericardial fluid, and even urine. The line probe assay (LPA), (Hain Lifescience) is the preferred method for isoniazid resistance. But these are susceptible to procedure deviation and are inaccessible in some areas due to higher costs and complex procedures, which explains why only 46% of new TB patients and 83% of previously treated patients underwent DR-TB screening [4].

Artificial intelligence (AI) is widely adopted in medicine [5]. Pattern recognition and classifiers are some of the common purposes of machine learning and deep learning algorithms, and in some cases, these models provide robust classifier ability [6]. This model can be deployed into the application which can be accessed by health-care staff.

The CUHAS-ROBUST (Chulalongkorn-Hasanuddin Rifampicin-Resistant Tuberculosis Screening Tool) is an AI application based on an artificial neural network classifier to perform RR-TB. This model relies on clinical information of the patients and can be accessed at https://cuhasrobust.shinyapps.io/CUHASROBUST. This qualitative study aimed to address the situation of DR-TB diagnosis and management, as well as collecting the user experience and assessing the impact of the CUHAS-ROBUST application on the RR-TB screening and general health-care service.

Methodology

This qualitative study was conducted following the consolidated criteria for reporting qualitative research (COREQ) checklist [7].

Research team and reflexivity

The research team was made up of the interviewer and the assessment team. An in-depth interview was conducted with one author and two research assistants. Author 1 was a male medical doctor working for the public health office for five years, particularly in primary care service. This author had no specific work relationship with the participants, therefore, reducing the researcher bias arising from peer-relationship. Two research assistants, one (female) and two (male), medical doctors, were trained for research in public health including human subject protection. These two research assistants confirmed they had no mutual relationship with the participants and worked in pairs to ensure the quality of the interview (handling the bias arising from the questioning technique and interpreting the local context).

The assessment team consisted of four authors. Author 2 (male) was a professor in data science with expertise in integrating health data and medical technology. Author 3 (female) had a background in antimicrobial resistance, and author 4 (male) was a professor in health policy. Author 1 served as a bridge to this team and none of the authors had a direct relationship with the participants. This group interpreted the result and was responsible for manuscript writing and evaluation.

Study design

An in-depth interview was conducted from September to October 2020 with a thematic analysis approach that was stated to underpin the study and considering all information was important. This method is a superior form of content analysis as the manifest and the latent analysis could be performed simultaneously, yielding a deeper level of interpretation [8].

The participants for the in-depth interview were recruited from an online link invitation and manual referring (snowball technique). Participants had to be at least working in a primary care center for a minimum of six months. The link invitation provided information on in-depth interviews (the purpose of the study, access to application and the appointment setting). Implied consent was applied. Personal appointments could be done through texting. The participants had to prepare their DST-confirmed TB cases as their dataset for trial. Participants could retrieve the predictor information from the medical records as applying the CUHAS-ROBUST screening to the new suspected cases was not feasible due to the longer DST waiting period. The participants input the data to the CUHAS-ROBUST by themselves. The detailed data input into the application was not given to the investigators except the final prediction and actual DST results. The authors excluded participants who did not give any response exceeding two weeks after the appointment or participants who sent a request to refuse an interview or used the application to predict the suspected or unconfirmed cases. Retraction of the interview was possible if the participant decided to decline to participate after conducting the interview.

The interview was conducted through a video-call, in order to reduce coronavirus disease 2019 (COVID-19) transmission and to accommodate the voice-to-text transcriber. The interviewer informed the participants to elaborate on the DR-TB problem, their experience of predicting RR-TB using CUHAS-ROBUST and their overall impression on a numeric scale. The identification of the DR-TB problem was based on the variables of the MDR-TB mathematical dynamic model [9] and health-care service delivery model [10]. User experience assessment was analyzed based on the user engagement model for online intervention [11]. These models were used as preliminary themes and interview guidelines. The participants were required to provide their responses alone and without interference from others. A back-to-back Indonesian-English translation was applied to the interview transcripts. Repeated interviews were possible to overcome any technical or interpretation issue. No specific duration was set, as long as the themes were covered. The transcripts were not given to participants as they could rewatch the recorded video. An inductive thematic saturation was conducted for data saturation, focusing on the emergence of new codes or themes. Quantitative data collection was performed to assess three components of application usage experience during the interview. Three questions were asked in the range of 0–10 including the ease of use, clarity of information provided in the application and feasibility of the application to be implemented at the participant's workplace.

Domain 3: analysis

Participants’ characteristics were described along with quantitative assessments. The assessment of completeness and screening for codes in transcripts performed by one author and two research assistants were to be labeled with the data. Regeneration of the new themes was conducted while comparing the preliminary themes with the new themes until data saturation was reached. Final themes were named and written as a full summary. Google speech recognition and manual observation were performed to ensure the quality of the transcript. A final summary was then sent to the involved participants and a two-week period was given to receive any feedback.

Ethical issue

This study was approved by the Ethical Committee of the Faculty of Medicine, Hasanuddin University, Indonesia (279.UN.4.6.4.5.31/PP36/2020) as part of the AI RR-TB screening project.

Results

A total of 123 people replied to the invitation, but only 56 attended the interview. One interview was repeated due to unclear recording, three participants retracted their interviews (one person admitted that her answer was irrelevant, and two people did not utilize the application properly for RR-TB screening, but for pneumonia screening). The authors rejected 20 participants who used the application for unconfirmed cases, leaving 33 participants for analysis.

A summary was sent to the participants, but no feedback was received. A total of 56 (6 MDR, 2 RR) participants' data were provided for application testing. Patient diagnosed with Multidrug-Resistant Tuberculosis (MDR) were also included, as these patients also have rifampicin resistance, along with isoniazid resistance. The participant's characteristics and major findings are presented in Table 1.

Tuberculosis care in the primary health center remains challenging

Despite the new drug-sensitive tuberculosis incidence is steady, prevalence of Chronic and Drug-Resistant TB increase in Primary care. These findings are reflected as variable tγ in the dynamic model of TB-To MDR TB transformation. The higher burden of primary care and inadequate facility may contribute to DR-TB development. Diagnosis and treatment of tuberculosis are provided free of charge in primary care which gives a higher influx of patients, particularly in rural areas. The private clinic charges more service fees and medical costs which is an unpopular option for those with financial hardship [12]. This creates an overcrowded health-care facility.

We receive more tuberculosis patients compared to other centers, and right next to us is the private clinic. Even they refer to us some of their patients as those people could not afford treatment (JY, Head of primary care, 40 years old)

The subsequent impact of the overcrowded health-care facility is increasing the work burden and a higher risk of getting infected.

We receive more than 100 visits per day, even now we have an outpatient clinic for TB, and it's never vacant. Even though I am not the TB manager, I feel like I need to help S (the TB Manager) to deal with her patients (AH, Nurse, 36 years old).

With many people in primary care, and less awareness to prevent infection such as wearing a mask, we have no idea how risky the situation is, and this is a threat not for the healthy visitors but also the staff who stay longer in primary care. Who knows that the healthy patient may end up infected by TB after coming here? (IS, Nurse, 24 years old).

Managing the patients as well as educating them to practice infection prevention is challenging in the primary health center and it is essential to confine the β or the unisolated infected person. The primary care building is not as big as the hospital; therefore, specific adjustments cannot be implemented.

If we take a look at the origin of DR-TB, most of the cases were originating from unfinished previous therapy. It is really hard to convince people to adhere to the longer multiple regimes (GG, Medical doctor, 30 years old).

The new theme “policy” addressed the link between overcrowded health-care facilities, unfinished treatment (which related to variable c or the successful cure rate in the dynamic model), and the occurrence of DR-TB. A study in primary care shows that poor surveillance systems are associated with altered tuberculosis management [13]. And this is related to the increased burden and overcrowded health-care facilities.

As a nurse, I have to take care of the outpatient clinic, and still have to find some spare time to do surveillance and home visits. I admit that it is really hard to observe the therapy by myself. Even though there are some health volunteers, but again, the voluntary basis won't ensure the quality of their work (WH, TB Manager, 46 years old).

The inadequate facility for diagnosis has been addressed, and this is the problem that is symbolized with f (the TB detection rate) in the dynamic model. An argument illustrates this:

I know that sputum or radiology are important tools for tuberculosis diagnosis, but we don't have the radiology modality in primary care. In the case of a clinically suspected patient but without supporting sputum results, we refer this patient to the hospital, but again this is referring the burden from primary care to hospital, and not all the patients will come back to primary care for treatment (NS, Medical doctor, 28 years old).

Only 20% of patients were able to encounter the expected diagnosis procedure in the first place they sought for care, particularly in primary care [14], and there is always a conflicting procedure between the primary care and hospital. Some of the primary care facilities try to provide supporting modalities but the problem is not about the availability of the tools but also the technician [15].

Our primary care has X-rays but this device needs specific treatment. Moreover, some of the staff are questioning the safety of the radiation. As the referral hospital is not that far, and the running cost may affect our general expenditure, we prefer to refer the patient instead (AY, Head of the primary care, 50 years old).

In summary, tuberculosis and DR-TB remain challenging in terms of diagnosis and care delivery including monitoring of therapy. Furthermore, unclear referral systems and inadequate health facilities contribute indirectly to higher tuberculosis cases and health burden.

Diagnosis of drug-resistant tuberculosis is time-consuming, costly and inaccessible

The DR-TB diagnosis management of primary care is crucial. A health-care service delivery model was used to identify the problem in DR-TB diagnosis. The study revealed that the rate of sputum examination in Turkey was around 71.6% and the DST was around 25.8% [16]. However, these statements reveal more extensive problems:

I admit that performing the microscopic examination is exhausting as we have to look at and count the number of bacilli in continuous high exposure of light. Sometimes I missed the number of bacilli and it happened in my quality control assessment (FS, Laboratory technician, 23 years old).

I will ask my doctor to know whether a patient is a clinically suspected tuberculosis or not, so I will put more attention to those cases. But it does not mean I ignore those unsuspected cases (JK, Laboratory technician, 36 years old).

The reliability issue of sputum smears in tuberculosis has been identified. Many cases with negative or scanty results but clinically suspected were referred for radiology examination, thus extends the diagnosis time and affects the overall quality of care. If there is a reliability issue of sputum examination, those suspected DR-TB cases will be misdiagnosed.

The gold standard of drug-DR-TB is DST, but with GeneXpert, it is more convenient to confirm any cases with unsupportive sputum results, particularly cases with negative or scanty sputum results. But with the scarcity of these tools, I think we could only rely on a reliable sputum smear test (NS, Medical doctor, 28 years old).

The DST is time-consuming and costly. The GeneXpert (a nucleic acid amplification test) has been provided in some centers, but the accessibility issue arises.

It's a daunting experience to wait for the culture (DST) because the center is too far and needs a longer time. But now GeneXpert exists and it's faster. Still, the patient should go to the hospital for further checking. All of which seems complicated according to the patient experience (SA, TB Manager 29 years old).

Now is hard to request GeneXpert as it is now allocated for COVID 19 as well (JK, Laboratory technician, 36 years old).

The uncertainty in diagnosis leads to delayed treatment and inappropriate medication despite patient-centered factors still being the major underlying reason for delayed treatment [17]. But some of the physicians rely on the existing guidelines to solve the problem.

I refer to a pulmonologist for a better diagnosis and see what happens. The preferred treatment is usually written on a referral form. Indeed, it still delays the treatment but appropriate examination at least performed to those cases (HJ, Medical doctor, 32 years old).

While waiting for the diagnosis, I will refer to RR/MDR TB as a prompt treatment (JB, Medical doctor 27 years old).

The difficulties in diagnosis are worsened by the reliability issue, the insufficient provision of rapid and DST tests and prolonged waiting time which lead to delayed treatment.

Integrating artificial intelligence to rifampicin-resistant tuberculosis screening

The CUHAST-ROBUST was implemented in the participant's workplace, and it was assessed whether the implementation may alleviate the problems using the user engagement model (UEM). In terms of the persuasive design domain of the UEM, participants stated that the user interface of the application is crucial (including accessibility, functional tabs, installation and instant results).

The “ease of use” means score was 8.18/10. Layout changes were observed in some users when switching the apps from computer to mobile phone affecting the input process. This indicates a problem in the personal relevance factor of the UEM. Thus, the authors as the developer should consider re-adjusting the input mechanism by making an easier input button and flexible layout for mobile and computer view.

Regarding the clarity of information, most of the participants agreed that the variables' explanation and interpretation of the results were clear, shown by an average score of 8.27/10.

The authors explored some new themes which elucidated the possible impact of CUHAS-ROBUST and some problems that need to be resolved. These new themes are related to the credibility and usability factor in the UEM.

I have to say, it is really helpful than using the criteria for suspecting DR-TB. It increases my confidence in decision-making and referring the case or initiating treatment (YW, Medical doctor, 36 years old)

AI has been known to boost clinical decision-making, thus, leading to appropriate and immediate management of the disease [18]. But some issues need to be solved, including extensive diagnostic performance evaluation, interactive explanation on how the application works and preferring the real parameter over the estimated parameter.

“This application is good but as I don't know exactly how this application works, I feel like it needs more studies in terms of accuracy, sensitivity, and specificity” (BC, Medical Doctor, 24 years old)

The parameters embedded in the application are feasible. But as for Hba1c, I think it is better if we can obtain the real results rather than estimate it for more accurate results (NA, Medical doctor, 34 years old)

Furthermore, another impact was observed including the data collection process, and efforts to provide supporting circumstances. These factors are associated with the environment and individual domain in the UEM.

The real challenge perhaps is not the application. It is on how precisely we collect the information as most of the parameters come from history taking. With limited time to do history taking and physical examination, I believe that we might miss essential information (YM, Medical doctor, 43 years old)

History taking is limited due to time restrictions. But sometimes, gaining empathy plays a pivotal role to obtain the deepest and more honest answer from the patient. This application has possible future impacts such as increasing compliance of providing medical procedures and could lead to a discovery of advanced surveillance techniques.

It enforced me to do one thing. I should focus on examining the sputum smear as it seems an essential predictor. If I can't provide the correct answer, then the AI result is incorrect (EG, Laboratory technician, 30 years old)

As the head of primary care, I think I need to enhance the provision of supporting modalities to ensure that my staff will be able to provide the required information for the application. The application itself is free (JI, Head of primary care, 30 years old)

I think I can use this for active surveillance and home visits so more suspected patients will be screened and they don't need to come and expose other healthy people in primary care. (AM, TB Manager 36 years old)

Discussion

This study deciphered problems according to the dynamic model of MDR-TB, including the rising of DR-TB and the inadequacy of the health system to provide health service which leads to loose monitoring of therapy and staggering disease transmission. Similar findings were found in Vietnam where a lack of TB screening capacity was observed in district hospitals and conflicting policies appear due to miscommunication between the policymakers and the executive officers. A discrepancy of private and public healthcare is seen, specifically in MDR-TB reporting where the private sectors tend to disobey the surveillance protocol. In detailed key findings, Vietnam faces failure to identify presumptive MDR cases [19]. The current guideline in Indonesia for presumptive DR-TB is based on a list of the condition including positive contact with DR-TB, HIV-reactive or chronic TB, and it is a very broad criterion to narrow down the eligible people for further testing which is available in limited numbers.

This study emphasized the problem of DR-TB to be a diagnosis problem. According to the user engagement model, the CUHAS-ROBUST possesses good engagement and can tackle some of the RR-TB screening problems. It also achieved its purpose by providing a fast clinical decision-making [18] which overcomes the delayed diagnosis and demonstrates more than 80% accuracy to predict MDR + RR-TB. Another impact addressed in this study includes the enhancement of clinical procedure compliance (proper history taking and laboratory examination), as well as enhancing plans for supporting tools provision. Embedding CUHAS-ROBUST into active surveillance is the new concept expressed by the participant, expanding the possibility of AI integration to further aspects of healthcare.

Implementation of AI in tuberculosis screening is performed in other countries despite no official acknowledgment given by the government for nationwide use due to inconsistent screening performance (sensitivity and specificity) and supportive modalities. The computer-aided diagnosis method (mainly using the radiology features) faces a challenge when tested with real data as the radiological features of TB vary between individuals. The commercial TB tool was implemented in some countries (Zambia, Tanzania, South Africa, Pakistan and Bangladesh), but a lower area under curve (AUC) value was observed (0.71–0.84) [20]. Furthermore, this tool does not detect the possibility of RR-TB, which the CUHAS-ROBUST offers.

This study was conducted with a rigorous qualitative method where several models were embedded to explore the problem and depict the engagement of CUHAS-ROBUST, resulting in a scientific study, rather than a testimony. The authors accommodate various professions from different provinces, hence, increasing the generalizability of the finding. Also, the authors have a linear background to interpret the results and understand the working situation of the participants. Nevertheless, the participants were unaware of the preliminary performance of CUHAS-ROBUST. Participants’ expectations of certain diagnostic performance may affect the user experience. Moreover, the authors did not measure the digital literacy of the participants which might affect their experience.

Conclusion

CUHAS-ROBUST, an AI-based screening tool, introduces benefits for RR-TB screening and is relevant to what the private sectors are expected to achieve [21]. The authors projected that implementing this screening tool also affects other aspects of the health-care system, including the quality of care improvement and enhancing other TB-related policies.

Conflict of Interest: None

Table 1

Participants’ characteristics

No	Variables	Statistic (n = 33)
1	Age (years)	Mean (min–max) ± SD	33.12 (23–50) ± 7.57
2	Years of working in primary care	Mean (min–max) ± SD	5.72 (1–19) ± 5.00
		Group	Frequency
3	Gender	Male	14
		Female	19
4	Occupation	Medical doctor	12
		Head of primary care	4
		Tuberculosis manager	6
		Laboratory technician	7
		Nurse	4
5	Education	Diploma	13
		Bachelor	17
		Postgraduate	3
6	Islands of domicile	Bali + Nusa Tenggara	3
		Java	12
		Kalimantan	4
		Maluku and Papua	3
		Sulawesi	5
		Sumatera	6
7	The total cases for trial	Sensitive	48
		RR/TB	2
		MDR/TB	6

Application perception		Value
1	Ease of use	Mean (range) ± SD	8.18 (7–10) ± 0.57
2	Clarity of information		8.27 (7–10) ± 0.56
3	Feasibility to be implemented		8.06 (7–9) ± 0.34
4	Diagnostic performance	Sensitivity MDR	66% (4/6)
		Sensitivity MDR + RR	75% (6/8)
		Specificity drug sensitive	83% (40/48)
		Accuracy MDR + RR vs drug sensitive	82% (46/56)

Note(s): MDR = multidrug-resistant tuberculosis RR = rifampicin resistant tuberculosis