Development of standard indicators to assess use of electronic health record systems implemented in low-and medium-income countries

The primary goal of these implementations is to support HIV care and treatment, with funding coming from programs such as the US President's Emergency Plan for AIDS Relief (PEPFAR). Several countries, including Rwanda, Uganda, Mozambique, and Kenya, have rolled out large-scale national EHRs implementations in government-run public facilities. However, tracking the status of each implementation is challenging, particularly in standalone systems distributed over large geographical areas. A core consideration is the extent to which the EHRs implemented are actually in use to support patient care, program monitoring, and reporting. Without robust evidence of use, it becomes difficult to justify continued financial support of these systems in resource-constrained settings and to realize the anticipated benefits. Implementation of EHRs within a clinical setting does not automatically translate to use of the system. Studies have revealed critical challenges to realizing the benefits of EHRs, including poor infrastructure, inadequate technical support, limited computer skills and training, and limited funding. Additionally, implementation of EHRs is complex and can be highly disruptive to conventional workflows, affecting its acceptance and use. The use of EHRs can also be affected by data quality issues, such as completeness, accuracy, and timeliness. This is a particular risk in LMICs given the lack of adequate infrastructure, human capacity, and EHRs interoperability across healthcare facilities. Success of EHRs implementation depends on numerous factors, and these often go beyond simple consideration of the technology used. To address these challenges, a systematic approach was employed to develop high-quality indicators for evaluating actual usage of EHRs in LMICs. An initial set of 14 candidate indicators was developed by adapting the Human Immunodeficiency Virus (HIV) Monitoring, Evaluation, and Reporting indicators format. A multidisciplinary team of 10 experts was convened in a two-day nominal group technique (NGT) workshop in Kenya to systematically evaluate, rate, prioritize, refine, and identify new indicators. The NGT participants settled on 15 final indicators, categorized as system use (4), data quality (3), system interoperability (3), and reporting (5). Data entry statistics, systems uptime, and EHRs variable concordance indicators were rated highest. The developed indicators are intended to provide a multidimensional tool for assessing the success of EHRs implementations in LMICs, allowing for standardized aggregation of performance across locations and countries. The indicators are designed to be relevant, meet the EHRs monitoring needs, and not be too burdensome to accurately collect. Carefully developed EHRs use indicators and metrics are needed to regularly monitor the status of the EHRs implementations, identify and rectify challenges, and advance effective use. A common set of EHRs indicators and metrics would allow for standardized aggregation of performance of implementations across locations and countries, similar to the systems currently in use for monitoring the success of HIV care and treatment through a standard set of HIV Monitoring, Evaluation and Reporting (MER) indicators.