Telepharmacy is defined in the strategic framework of the MAPEX project of the Spanish Society of Hospital Pharmacy (SEFH) as remote pharmaceutical care based on information and communication technologies (ICTs), with four primary scopes of application: pharmacotherapeutic follow-up; home drug delivery; patient education and information; and care team coordination1. Although some experiences have been previously reported, the use of Telepharmacy has widespread as a result of the SARS-CoV-2 pandemic, which highlighted the need for technology tools that guaranteed continuity in pharmaceutical care2–5. A range of initiatives was launched to provide remote dispensing and home drug delivery, and ensure coordination between the pharmacist, healthcare care professionals and patient associations, with very good patient satisfaction6.

Once implemented, an assessment of the quality, effectiveness, safety and activity of Telepharmacy, understood as a complementary tool to face-to-face pharmaceutical care, should be performed. A quality assurance plan involving a continuous review and evaluation of Telepharmacy quality criteria, standards or indicators is necessary to measure results, identify opportunities for improvement, and ensure continuous quality improvement7.

In 2015, the Pan American Health Organization (PAHO) set a scorecard of Telemedicine quality indicators8. The SEFH7, American Society of Health-System Pharmacists (ASHP)9 and other scientific societies10,11 have issued supportive positioning statements in relation to Telepharmacy. Although there are some descriptive studies on the structure of Telepharmacy, there is scarcity of national or international studies reporting results12. The limited clinical evidence on Telepharmacy and its cost-performance warrants further research to set specific quality, activity and effectiveness indicators, in order to measure the impact of Telepharmacy.

The primary goal of this project is to propose a scorecard of activity, efficacy and quality indicators that provides an overview of the situation and progress of Telepharmacy and assist decision-makers in improving healthcare quality. The secondary objective is to design an indicator measurement tool and provide recommendations for implementation.

The method for designing a scorecard of indicators in Telepharmacy was developed between April and December 2021 by a working group (WG) composed of hospital pharmacy specialists from Spain. WG members were selected based on their professional and academic experience in Telepharmacy.

The project was conducted in four stages: literature review; preliminary set of quality standards and indicators; assessment of indicators; definition of priority indicators; design of descriptive cards, and design of a scorecard. In parallel and aligned with the methodological support documents developed within the framework of the SEFH Strategy for the Development and Spread of Telepharmacy in Spain.

In the first stage, a literature review was carried to identify potential quality indicartors and standards and indicators in Telepharmacy. To such purpose, the WG performed a review of documents of reference in the field of Telemedicine13,14, Telepharmacy, Hospital Pharmacy1,7,9–12,15–18 and quality indicators8,19–28. The WG conducted a search on Pubmed and WOS (Web Of Science) of literature published in the last five years (February 2017-February 2022), without language restrictions, using the following search terms to screen titles and abstracts: Pubmed: (“TELEMEDICINE”[Title/Abstract] OR “TELEPHARMACY”[Title/Abstract] OR “EHEALTH”[Title/Abstract] OR “TELEHEALTH”[Title/Abstract]) AND “INDICATORS”[Title/Abstract] AND “QUALITY”[Title/Abstract]; WOS: ((AB=(telemedicine OR Telepharmacy OR telehealth OR ehealth)) AND AB=(quality)) AND AB=(indicators). The search was complemented with a screening of evaluation reports and documents available at the websites of healthcare assessment agencies and institutions. Finally, a search of documents of quality assessment agencies of reference was conducted (Joint Commission on Accreditation of Healthcare Organizations, European Fórum for Quality Management, ISO Certification).

In the second stage, we established quality indicators and standards for the development and monitoring of a Telepharmacy service in a Hospital Pharmacy. The WG developed a preliminary set of activity, effectiveness and quality indicators. Quality criteria were categorized into five groups: a general group and four specific groups for each scope of application; pharmacotherapeutic follow-up; Home drug delivery; Patient education and information; and Care team coordination. A scope of assessment was assigned to each criterion. These scopes influence the implementation of a Telepharmacy program or are influenced by this activity, namely: Organization; Regulation and Ethics; Healthcare activity; Patient/professional experience; Education and information; Human resources; Cost-effectiveness; Quality; Clinical effectiveness, and Logistics. The preliminary scorecard of indicators was aligned with these quality criteria and with the documents of reference for Telemedicine and healtcare quality indicators.

In the third stage, an assessment of the preliminary indicators was carried out to identify priority indicators. Preliminary criteria and quality indicators were validated. Thus, indicators were assessed separately according to their suitability, utility/relevance, and feasibility. Suitability was rated on a 5-point scale based on the scientific validity or ability of the indicator to measure the attainment of the goal/expected results, where 1 indicated very low suitability (low scientific validity) and 5 indicated very high suitability (high scientific validity). Utility/relevance was assessed on a 5-point scale according to the ability of the indicator to guide decision-making in relation to the management of Telepharmacy in a hospital pharmacy service or to contribute to education, where 1 indicated very low utility/relevance (scarcely useful) and 5 indicated very high utility/relevance (very useful). Feasibility was also evaluated on a 5-point scale based on whether measurement of the indicator was easy or not due to the availability of data, whether estimation was complex, or progress could be monitored easily, where 1 indicates low feasibility (measuring the indicator is difficult) and 5 means high feasibility (measuring the indicator is easy). Once individual assessments were completed, and based on the results obtained, the definitive activity, performance and quality indicartors were established, and priority indicators were identified and determined. A priority indicator was defined as an indicator considered essential to measure the development of a Telepharmacy program in a hospital pharmacy service due to its suitability, utility and feasibility.

In the fourth stage, the WG designed descriptive cards of the indicators including the quality criterion, scope, formula, priority and definition of terms, population, type, regularity, standard, source of data, and comments on each indicator. A repository of variables and a glossary of terms were also developed to expedite the collection of the quantitative variables to be assessed in the scorecard and facilitate the understanding of the terms employed. In addition, to develop the scorecard, a spreadsheet was designed to better identify the variables to be measured.

A methodology support document was developed of the scorecard of activity, performance and quality indicators including recommendations for proper use in real practice. The scorecard of indicators in Telepharmacy was evaluated by the SEFH through 22 experts in Telepharmacy from the three WGs that developed the methodological support documents in Telepharmacy. Subsequently, the scorecard was evaluated by the Patient Committee and members of the SEFH. The scorecard was approved by the SEFH's Board of Directors.

The scorecard of indicators in Telepharmacy consists of 50 activity, effectiveness and quality indicators (31 of them being priority indicators), grouped into five scopes of application. This scorecard makes it possible to assess progress in the implementation of the Strategic Plan of Telepharmacy at organizational and clinical level and in terms of user experience an economic impact on the hospital pharmacy service (Annex 1).

The general scope of application of Telepharmacy includes 18 transversal items. The purpose of these indicators is to assist the hospital pharmacy service in monitoring progress in the implementation of Telepharmacy, as well as in healthcare activity, user experience, and economic impact of the interventions.

Pharmacotherapeutic follow-up includes 12 indicators. These indicators help the Hospital pharmacy service to evaluate and register the patients included, teleconsultations performed, claims and suggestions, and measure pharmacotherapeutic targets and associated PROMs (Patient-Reported Outcome Measure) and PREMs (Patient Reported Experience).

Fifteen indicators were established for home drug delivery. These indicators provide a record of patients, teleconsultations and dispensations performed, and claims and suggestions received in a Telepharmacy program. Finally, they also help measure pharmacotherapeutic targets, PROMs and PREMs.

Patient information and education include two indicators related to healthcare activity and are employed to assess patients and keep a record of Telepharmacy users.

With regard to the scope of care team coordination, three indicators related to healthcare activity were established. These indicators are used to evaluate and keep a record of cross-consultations to the care team recorded on the medical record of the patient.

Of the 50 indicators of quality, activity and performance of the scorecard for Telepharmacy, 31 were considered by the WG as priority or essential indicators for assessing progress in a Telepharmacy program (Figure 1). Most of these priority indicators are general (15) or correspond to pharmacotherapeutic follow-up (6) or home drug delivery. Other indicators are related to information and education (1) and care team coordination (2). Annex 1 contains the scorecard of indicators in Telepharmacy, where priority indicators are identified as [P]. Recommendations for the implementation of the scorecard are included in Annex 2. The full-text document including the Repository of Variables and Glossary of Terms and Indicator Descriptive Cards are available at MAPEX website (https://www.sefh.es/mapex/images/telepharmacy-indicators-scorecard.pdf).

Figure 1.

Indicators of quality, activity and performance of the scorecard for Telepharmacy.

(0.25MB).

The tool designed to facilitate data collection is available at: https:// www.sefh.es/mapex/cmo-oportunidad.php.

The spread of Telepharmacy as a complementary activity to face-to-face pharmaceutical care in the last years, boosted by the pandemic, makes it necessary that a strategic plan is available at each hospital pharmacy service. This plan should lead to the implementation of a scorecard of activity, efficacy, and quality indicators that helps the Hospital pharmacy service assess the initial situation, monitor progress, measure quality and effectiveness, facilitate decision-making, and establish an improvement plan. The strategic plan must be aligned with the strategic plan of the institutions and regional health systems of each autonomous community.

Health innovations, such as Telemedicine, must be proven to preserve or improve healthcare quality29. No studies on indicators in Telepharmacy were found during the literature search. Therefore, it is necessary to develop and assess indicators for measuring safety, quality, and outcomes in Telepharmacy1,9 in accordance with national and international recommendations20,21 for the implementation of a Telemedicine program and the SEFH “Manual of Certification of Outpatient Hospital Pharmacy Units”15.

The scorecard of indicators of Telepharmacy includes 50 indicators. The high number of indicators led the WG establish a set of priority indicators for a realistic implementation and evaluation. Thus, a total of 31 indicators were identified as essential or priority based on their suitability, utility, relevance and feasibility. We recommend to first use priority indicators and, in the long-term, expand the number of indicators employed. It is worth mentioning that these indicators are dynamic guidelines that each hospital pharmacy service can adapt to its own characteristics.

The methods employed have been previously employed by other working groups on healthcare quality indicators25,26,28,30. This method consist of a literature search; establishing a preliminary set of indicators based on a previous analysis of Telepharmacy activity and what it is intended to be measured. This analysis is based on recommendations on Telepharmacy provided by the SEFH, the relevant scientific literature, and the expertise and experience of a panel of experts in Telepharmacy. Each expert individually assessed the suitability and feasibility of the indicators for subsequent selection and prioritization. The indicators selected were defined using the desirable characteristics of healthcare quality indicators22 in terms of measurability and feasibility, validity, timeliness, reproducibility, sustainability, relevance and importance, and understandability. Healthcare indicators are generally developed based on the model designed by Donabedian31 in 1966, who established three basic categories for measuring medical care quality25,30. The author recommended assessing quality by evaluating the healthcare structure, process, and outcomes resulting from the care provided. To assess implementation, it is necessary to employ indicators that evaluate the service provided (monitoring, evaluation and optimization)20. In addition, indicators should be henceforth developed considering the results of a recent literature review for the optimal integration of e-health into the health system. According to this review, it is essential that the role of the care receiver is incorporated into the organizational structure, technology is attuned to the organizational structure, and the human resources deployed are aligned with the desired end results32. In relation to the evaluation of Telemedicine, the Institute of Medicine of USA suggested that the approach to and classical tools for health technology assessment should be adapted to the specific context of Telemedicine. This strategy is aimed at assessing quality, accessibility, acceptability and costs. This approach was described by Serrano Aguilar et al. in “Guía de diseño, evaluación e implantación de servicios de salud basados en Telemedicina” (2006)21.

In this document, we established a set of indicators to assess the implementation, structure, process and results of Telepharmacy based on quality, accessibility, acceptability and cost criteria. Although our scorecard of indicators was designed following the Donabedian model, the WG did not categorize indicators as suggested by the author. This contrasts with other proposals such as the indicators of pharmaceutical care for HIV and AIDS25, chronic lymphocytic leukemia26 or indicators of oncology practice28. The WG decided to employ a more intuitive classification to ensure a good understanding of the indicators according to the scopes of application of Telepharmacy designed in the SEFH positioning statement7, which includes pharmacotherapeutic follow-up, and informed home drug delivery, patient education and information, and care team coordination, adding a fifth group of general indicators. These scopes form the structure of the scorecard of indicators in Telepharmacy. In our literature search, we found a diversity of classifications of health indicators related or not to telehealh, without none of them having prevailed over the other26,28.

Each activity, performance and quality indicator include several sections and a detailed description to minimize observer bias and optimize reliability. Each indicator includes a rationale or quality indicator, dimension, formula, priority, definition of terms, population, type (structure, process, outcomes), regularity, standard, source of data, and a comments section. Based on this information, a descriptive card was designed for each indicator. The WG found in the literature descriptive cards for indicators structured into a similar number and type of sections, as the eight sections of the medical care quality indicators for patients with HIV24, the 11 pharmaceutical care indicators for HIV and AIDS patients25, and the nine indicators proposed for oncology practice28, all including sections for the rationale or quality criterion, formula, population, type, standard and source of data.

With regard to the standard of desirable level of the indicator, a standard of 100% was set for dichotomous indicators. As to the remainder indicators, each hospital pharmacy service must establish them according to their characteristics, data history, and quality goals. This is due to the absence of previous information or similar studies that would have guided the identification of standards, and the novel and heterogeneous nature of Telepharmacy.

The indicators selected by the WG include patient satisfaction, one of the pillars of telehealth. Patients are the only source of information to assess whether the treatment received met patient care expectations. If patients are not satisfied with Telemedicine, the service is not useful and expensive33. This type of indicators is not generally incorporated as a telehealth indicator, as shown in a recent review published by Hellfritz in August 202134. However, patient satisfaction was already included in the 2006 Guide for the design, assessment and implantation of Telemedicine21 and, more recently, in the 2020 Manual for implementation of telehealth of the American Medical Association13.

The quality of Telepharmacy must be consistent with that of face-to-face care, which makes further research on the impact of Telepharmacy on clinical outcomes necessary. Some studies have been conducted in intensive care35, emergency care36, neurology37, geriatrics38, inflammatory bowel disease39 and endocrinology-diabetes40. However, there is scarcity of data on the clinical effectiveness and impact on patient management, organization, and costs of Telemedicine20. In this line, a systematic review demonstrated the poor scientific quality of evaluations of the impact of Telepharmacy on patient care12. The impact of Telepharmacy should be evaluated in quality multicentric studies in the near future once the scorecard of indicators has been incorporated into practice.

This is the first scorecard proposed in pharmaceutical care, which is also framed within the SEFH Strategy for the development and expansion of Telepharmacy in Spain. It is a dynamic scorecard that can be adapted by the Hospital pharmacy service based on their vision of Telepharmacy, which incorporates the experiences of patients and professionals through satisfaction surveys and improvement suggestions. This added to the recent evidence published could increase the quality of the service. This scorecard will be initially applied at local level, but with the shared goal of improved national pharmacy care standards. The common structure of the scorecard will provide a global vision and reduce disparities. This proposal also includes recommendations for use of the scorecard from a practical approach, in order to facilitate its incorporation and use. Each pharmacy service, as a function of the data available, quality strategy and Telepharmacy plan can incorporate the indicators selected into their local scorecard using the tools designed to facilitate structured data collection.

A limitation of this study is that the indicators proposed were not tested in a pilot study to verify their reliability and feasibility. The Working Group will perform a pilot inter- and multicentric study. It is worthy of note that this scorecard was developed from the pharmacist's perspective and, although it was evaluated by the SEFH Patient Committee, this Committee was not a member of the Working Group. More indicators suggested by patients will be incorporated in the future, which will be related to quality of life, course of disease, and communication with the health system. Another limitation of the study is the lack of quantitative standards for the reasons explained above.

The scorecard of indicators in Telepharmacy will serve to assess the initial situation, monitor the implementation, assess quality and effectiveness, facilitate decision-making, and develop an improvement plan. In the light that Telepharmacy is dynamic evolving process, these indicators may change based on patients’ and professionals’ opinion, experiences and new evidence made available.

No funding.

We thank all members of the Working Group on Telepharmacy and the SEFH Patient Committee.

To Ascendo Consulting for its consulting and advisory services in the development of this document.

No conflict of interest.