An Evaluation of Safety and Usability of an Integrated Electronic Medication Management Systems in Specialised Settings

Abstract

Background In 2017, the World Health Organisation (WHO) initiated the third WHO Global Patient Safety Challenge: Medication Without Harm, with two key themes being medication safety and transitions of care [1]. It outlines that improving medication safety requires strengthening the systems for reducing medication errors and avoidable medication- related harm [1]. The Institute of Medicine Quality of Health Care reported that as many as 98,000 Americans die each year from preventable medical mistakes, they experience during hospitalisations [2, 3]. The single leading type of error are medication errors, with estimates ranging from 4% to 20% of all hospitalised patients encountering medication errors [2]. In the Australian context, this is estimated to range from 2.5% to 11% of hospitalised patients [4, 5].

The potential for Electronic Medical Records (EMR) and Electronic Medication Management Systems (EMMS)(the part of the EMR where medications are managed) to vastly decrease the number of preventable medical errors in the health care system [6] is the most important reason to justify the shift away from paper medical records. They have become the focus of efforts for providing information for integrated care that is well coordinated across health and social care settings. Coordinated care is particularly essential when there are intra-hospital transfers of care (TOC) between specialised wards such as the Intensive Care Unit and Oncology Unit and general wards.

However, EMMS have the potential to incur unintended consequences. A literature review by Campbell et al described the frequency of the types of unintended consequences associated with EMMS implementation as: (1) extra work for clinicians (19.8%), (2) unfavourable workflow complexities (17.8%), and (3) the persistence of paper (10.8%) [7]. These unanticipated consequences can have downstream effects on the medication management system, user-acceptance, workload and negatively impact patient care.

To minimise the risk of these unintended consequences, the evaluation of EMMS has been widely encouraged to measure the systems’ success post-implementation. At the Global eHealth Evaluation Meeting in 2011, the WHO announced that “To improve health and reduce health inequalities, rigorous evaluation of eHealth is necessary to generate evidence and promote the appropriate integration and use of technologies” [8]. Whilst comprehensive EMMS evaluation may be difficult and complex, it is essential to ensure system aligns with work processes. In specialised settings, such the Intensive Care Unit (ICU) or Oncology Unit, patient management is extremely complex and involves interdisciplinary patient care. Their unique workflows, ratios of staff-to-patient and medication prescribing patterns and day-to-day tasks are often the reason for implementing a ‘Best-of-Breed’ system. However, the literature shows that a single vendor hospital wide integrated system decreases the need for interfaces for the applications to communicate with one another [9] . Despite the call for evaluation by the WHO, we found little focus on the evaluation of the safety and usability of an integrated EMMS in specialised settings which have implemented the same EMMS hospital wide. In New South Wales (NSW) Australia, efforts to develop and implement Electronic Medical Records have been part of the eHealth Strategy for NSW Health 2016-2026 [10]. With the aim of improving integration across the hospital, in 2018, a medium sized urban teaching hospital in Sydney New South Wales decided to implement a comprehensive integrated EMMS in the ICU and Oncology unit, to test its large-scale clinical applicability. The integrated EMMS aligned with the same EMMS that had been implemented hospital wide, years earlier. However, there is limited evidence in specialised settings that an integrated system will not introduce unanticipated consequences and can in fact benefit medication safety, usability or workload. Due to the complexity of the EMMS, measuring its success should be multifaceted, encompassing elements of usability, safety, and impact on workload. Despite this, there has been little focus on the evaluation of usability and safety of an integrated EMMS in specialised settings. The overall aim of this thesis was to evaluate the safety, usability and impact on workload of an integrated EMMS in specialised settings. To achieve this, methodological pluralism including a mix of a historical case control study, qualitative and quantitative research was conducted.

Methods:

Part A: Electronic Medication Management Systems and Medication Safety

The aim of part A was to evaluate the effect of a hospital wide integrated EMMS on medication error rates during ICU admission and at transitions of care. A 6-month historical control study was performed before and after implementation of an integrated hospital wide EMMS in the ICU. The study ran between February and July 2017 (Pre-EMMS) and June and November 2018 (Post-EMMS). All the patients admitted to the ICU and reviewed by a pharmacist during the study period were included. Prescribing errors detected by a pharmacist were entered in a dedicated Research Electronic Data Capture Tool (REDCap) under the intervention collection form. Pharmacy services were provided by a full-time pharmacist 8 hours a day Monday to Friday and rotating pharmacists 4 hours a day on a Saturday and Sunday. Prescribing errors detected by pharmacists in the study period were divided into phase 1, (pre-EMMS, 6 months), phase 2 (3 months post implementation after shakedown stage) and phase 3 (next 3 months of post implementation after phase 2). They were first categorised as either a system or clinical error, then the error type category. Two pharmacists from the research team independently classified the errors using the same classification system, then compared results. Inter-rater reliability tests produced a κ score of 0.823. When disagreements arose, a third pharmacist reviewed the differences and classified the error. Descriptive analysis of patients demographics were conducted. Chi Square statistics were used to compare the proportion of patients who had an error at TOC during each phase and to compare the proportion of patients with a system and/or clinical error during their ICU admission. Interrupted time series (ITS) analysis was performed across the three phases of the study. Error rates were plotted over time to examine the data visually, and autoregressive integrated moving average ITS techniques were used to study the effect of the intervention. Logistic regressions were used to determine the relationship between the dependent (error type) and the independent variable (study phase) for errors that occurred during ICU admission and TOC for both clinical and system errors. The odds ratio of a specific error type/mechanism was given as the odds of an error occurring in phase 1, compared to phase 2 or 3. Logistics

regression was also used to describe the relationship between phase of the study and the severity of the error. The level of significance was set at 5% for all tests. All analyses were performed using Statistical Package for the Social Sciences 24 (SPSS) (IBM Corp. Released 2016, Version 24.0. Armonk, NY: IBM Corp).

Part B: Electronic Medication Management Systems and Usability

A survey was developed by the research team; designed against the “Unified Theory of Acceptance and Use of Technology (UTAUT)” [11] framework to evaluate the usability and acceptance of the hospital wide EMMS in both the ICU and Oncology Unit . The UTAUT, was chosen as the framework as it has been widely applied and empirically tested to investigate factors that could influence individuals to adopt and use technology in various environments. The survey used a 7-point Likert scale (1 – strongly disagree; 7 – strongly agree). It consisted of 69 questions capturing a range of user feedback questions, with 33 questions specifically relevant to the UTAUT framework. The same survey was used to evaluate the usability and acceptability in both the ICU (setting 1) and Oncology Unit (setting 2). In both settings the evaluators distributed the surveys to the three main user clinical groups (nurses, doctors and pharmacists). Recruitment was via email and distributed hard copies of the survey on the wards. Preliminary analyses of survey results were performed to assess normality. For data analysis, mean, median and distributions for each item were examined. User satisfaction/agreement and the differences between user types (e.g., role type, years of experience,) were analysed with the Chi-Square statistic. Mean scores were calculated for each of the items assessing levels of agreement/satisfaction pertaining to EMMS functionality. The level of significance was set at 5% for all tests. All analyses were performed using SPSS (IBM Corp. Released 2016, Version 24.0. Armonk, NY: IBM Corp). Semi-structured interviews and focus groups were also conducted in the same ICU. Staff external to the ICU who cared for ICU transferred patients were also interviewed to determine the hospital wide effect of transitioning from a hybrid prescribing hospital environment of paper-based and an EMMS to a homogenous one. It also allowed for the comparison of the impact on teams involved in the direct use of the system with those on the receiving end of patients transitioning in and out

of the ICU. Semi-structured interviews and focus groups were also conducted in the same Oncology unit as above. The interview guide was designed against the UTAUT framework. The aim was to extrapolate and build on the findings of the survey. In both settings, purposive maximum diversity sampling was used to recruit staff. The same researcher (a pharmacist research student) interviewed all user groups. Transcripts were thematically analysed via inductive and deductive methods with the assistance of NVivo software (QSR International. Released 2018, Version 12.0. Melbourne, Australia).

Part C: The relationship between system usability and workload

Part C evaluated the impact of a hospital wide integrated EMMS in the ICU and Oncology unit on workload. The NASA-TLX [12] was used to measure overall workload as well as the dimensions of workload in the three user groups in the ICU and oncology setting. The NASA-TLX measures six dimensions to assess perceived workload and provides a score from 0 to 100 for each scale. The assumption of the instrument is that the combination of these 6 dimensions is likely to represent Overall Workload (OW) experienced by operators [12]. The NASA-TLX was distributed to the same three settings and EMMS users as in part C, ICU (setting A), non-ICU (setting B) and Oncology (setting C) unit’s doctors, nurses and pharmacists. The NASA-TLX was attached to the original quantitative survey described in part C to maximise user- response in the one instance. Due to time constraints before EMMS implementation, participants were asked to report on the elements of workload for pre- and post-EMMS implementation in the one instance. Preliminary analyses of survey results were performed to ensure there was no violation of the assumption of normality and linearity between the constructs. Dimensionality was evaluated by testing several factor models. Statistical Package for the Social Sciences 24 (SPSS) (IBM Corp -Released 2016, Version 24.0. Armonk, NY: IBM Corp) was used to conduct Exploratory Factor Analysis (EFA) to assess construct convergent and discriminant validity. Principal Component Analysis (PCA) was used to identify and compute composite scores. To assess the validity of the 6-factor mode, the factorability of the six NASA-TLX items were analysed using principal component analysis with Varimax (orthogonal) Rotation Factors [13]. The

Kaiser-Meyer-Olkin was tested to measure the sampling adequacy and the significance of the Bartlett’s test of sphericity was tested. Factors were screened and extracted if they had an eigenvalue of greater than 1.0 [14]. Convergent validity was then assessed within factors. Confirmatory Factor Analysis was conducted using Structural Equation Modelling (SEM). This was used to test the models hypothesised based on existing literature. All models were tested for fit, using empirically validated fit indices [15]. Models were also tested for convergent validity and composite reliability using Analysis of a Moment Structures (AMOS) (IBM Corp -Released 2020, Version 27.0. Armonk, NY: IBM Corp).

The impact of the EMMS on workload between each of the user groups survey results was compared between settings. A two-level multivariate analysis with repeated measures examined differences in workload before and after EMMS implementation for each of the settings. The OW was compared before and after EMMS implementation and between user groups. A calculation of the change in mean difference in OW and the individual elements of OW were used to compare pre- and post-EMMS implementation. The unstandardized correlations were used to calculate the impact on effort and performance for each user group. These were compared before and after EMMS implementation using two-level multivariate analysis.

Results:

Part B: Electronic Medication Management Systems and Medication Safety

A total of 762 patients were reviewed by a pharmacist over 6-months in phase 1, 276 patients over 3-months in phase 2 and 271 patients over 3-months in phase 3 and included in the study analyses. Overall, there were 351 (phase 1), 296 (phase 2) and 123 (phase 3) system and clinical errors recorded throughout the study. System related errors occurred during transition of care (TOC) in 42%, 64% and 19% of patients in phase 1, 2 and 3 respectively. There was a significant decline in the proportion of patients with an error between phase 1 and 3 (p < 0.01). During phase 1, the number of patients with an error at TOC were increasing by approximately 4.6 patients per month over the 6-months (p< 0.01). After an initial increase in phase 2, error rates fell by 20% (95%CI= - 31.4 to 7.8, p = 0.20) by the end of phase 2. Errors rates had a further significant reduction of 95% (95%CI= - 103.5 to –46.7, p < 0.01) in phase 3. Of the ten possible system related error types during TOC, two system error

categories ‘wrong rate/frequency’ and ‘drug omission’ showed a significant decrease between phase 1 and 3, and no significant change for the eight others. During a patient’s ICU admission, at least one medication error occurred in 28.3%, 62.6% and 25.1% in phase 1, 2 and 3 respectively. Besides procedural errors, the likelihood of a system-related error occurring during an ICU admission was greatest in phase 1, compared to phase 2 and 3 across all system-related error categories. Although there was an increase in the proportion of four clinical errors categories during ICU admission, these had a lower clinical severity than errors identified before the implementation of the integrated EMMS and two clinical error types reduced to zero.

Part C: Evaluating Usability

The survey was completed by a total of 74 respondents (63% response rate) across both the ICU (setting 1) and oncology (setting 2) settings. Of those, 43 respondents were from setting 1, (response rate: 47%), and 29 respondents from setting 2 (response rate: 79%). Across both settings, there were a total of 19 doctors, 39 nurses and 9 pharmacists. No responders worked in both settings. Results of the survey illustrated that there were no significant overall grouped differences in satisfaction between setting 1 and 2. When comparing individual user groups, nurses in both settings reported highest levels of satisfaction across all UTAUT constructs, compared with doctors and pharmacists. On the other hand, doctors were generally more satisfied in setting 2 compared to setting 1, which contrasts with pharmacists who were generally more satisfied in setting 1. Interviews with ICU (n=18) and non-ICU (n=7) staff revealed multiple reasons that affected clinician’s satisfaction with elements of the EMMS. Across all groups, both within and external to the ICU, there was satisfaction with clarity and legibility of the MAR, and the standardisation of practice via embedded system protocols. Overall, non-ICU staff generally perceived the system as improving effort and performance expectancy. Contrasting with ICU doctors and pharmacists who reported less satisfaction and concern for the increased effort and decreased performance expectancy to achieve end-user satisfaction. Interviews with Oncology staff (n=27) illustrated that doctors and pharmacists were generally satisfied with the facilitating conditions (hardware and training), but had

divergent perceptions of performance (automation, standardised protocols and communication and documented) and effort (mental and temporal demand) expectancy. In counterpoint, nurses were generally satisfied across all constructs. Prior experience using an alternative EMMS influenced performance and effort expectancy and was related to early dissatisfaction with the EMMS. Furthermore, whilst not originally designed for the healthcare setting, the flexibility of the UTAUT allowed for translation to the hospital environment.

Part D: Evaluating workload

Results of the EFA suggested two clear patterns, 5-items loading onto OW, with PE as a separate factor. The model in which OW is measured by five-items (mental, physical, temporal demand, frustration and effort) fit the data best. The model showed that frustration is an innate characteristic of overall workload and has a direct impact on performance. This study also found that both the theoretical and dimensionality aligned. The model assessing the dual relationship of FR with OW and PE fit the data well (CFI =0.99, TLI=0.99, RMSEA =0.04, SRMR=0.05). Composite reliability (0.84) and AVE (0.52) were both above the acceptable thresholds. It showed there is a significant negative direct effect of frustration on performance (path coefficient = -0.31, p<0.05). Using the modified NASA-TLX model, survey respondent in setting B (non-ICU), revealed a significant decrease in OW for both doctors and nurses following EMMS implementation. Setting B doctors reported a mean decrease in OW of 17, (t (10) =4.56, p=0.001) and nurses reported a mean decrease of 15 (t (31) =4.53, p= 0.00) (table 2). Across all other user groups in settings A (ICU) and C (Oncology Unit), there were no significant changes to the OW following EMMS implementation. On the other hand, all other user-groups reported a non-significant mean increase in OW. Setting A doctors reporting the highest mean increase of 18, t (8) =-1.96, p=0.086, followed by setting A pharmacists. Across all groups setting A pharmacists also reported the highest OW (M=71, SD= 8.31) followed by setting C nurses (M= 66, SD=4.82).

Conclusion:

The research described in this thesis revealed both parallels in requirements between specialised settings as well as unique requirements, following implementation of an integrated hospital wide EMMS. Elements of safety, usability, and workflows surfaced as unique to each setting. It revealed that the EMMS is both a tool that can improve medication safety, communication and coordinated care, yet can hinder workflows, workload and increase effort required to perform tasks in specialised settings. The rigorous evaluation found both facilitators and barriers to acceptance of an integrated hospital wide EMMS. The historical case-control study in the ICU, confirmed the proposition that EMMS can vastly decrease the number of preventable medical errors and is the most important reason to justify the shift away from paper medical records or disconnected electronic records. Despite the concerns around the drawbacks of implementing an integrated EMMS in a specialised setting, system- related medication error rates significantly declined during ICU admission and at patient transitions of care. Despite identified drawbacks of an integrated EMMS, user groups across both settings also identified benefits to their practice. Facilitators of system acceptance include standardisation of protocols and order sentences which promoted patient safety. As the primary aim of the implementation of the EMMS was to improve patient safety, optimising the drawbacks identified could result in better alignment between patient system and usability. Despite the benefits to medication safety and satisfaction with elements of the EMMS, the evaluation found poor user-satisfaction and usability across doctors and pharmacists in ICU and Oncology unit. This dissatisfaction was predominantly attributed to a misalignment between the wards needs and the integrated EMMS. In the ICU, there were concerns with the increased effort and time required for information processing, potentially impacting on performance. Concerns with time and effort could be attributed to the unique nature of ICU patients. As treatment of critically ill patients requires pertinent physiologic and medication data to be readily available for doctors so that quick and accurate decisions can be made in life-threatening situations. Similarly, within the ICU, specific concerns with a lack of flexibility to amend ‘order sets’ as needed for complex Oncology regiments, resulted in user frustration.

Furthermore, across all settings, dissatisfaction with poor usability, resulted in an increase in overall workload. Settings and user groups which reported the greatest dissatisfaction, consequently reported the highest increase in overall workload and frustration. Highlighting the interrelated nature of both aspects following implementation, and the need for healthcare organisations to address contributing factors to reduce the risk of user resistance.