Health research is an essential element for the success of any strategy aimed at improving the health of citizens. The integration of research and clinical practice ensures higher quality healthcare, better and faster translation of scientific advances into prevention, diagnosis, and treatment of disease, and more ethical and efficient patient care.1

Although research is a recognised activity for specialist pharmacists, historically, this role has been limited to supporting clinical trials or to individual efforts. It is only in recent years that pharmacists have become principal investigators in high-level research projects or heads of research groups. Thus, research is now a relevant activity in hospital pharmacy services (HPS), competing for resources and, in some cases, integrated with other hospital services with a stronger research tradition. For example, between 2014 and 2018, the number of physicians in HPSs increased by 35%.1

The Spanish Society of Hospital Pharmacists (SEFH) is fully committed to research activities. In fact, one of its 5 strategic lines is to “actively promote the research and innovation activities of hospital pharmacists as scientific healthcare professionals”.2 Currently, most Spanish HPSs have implemented the recommendations of the American College of Clinical Pharmacy regarding its advanced vision of pharmacy-directed research by 2030.3

Articles and guidelines on research in HPSs have focused more on the process itself than on systematic evaluation using objective standardised indicators, which is the approach adopted for HPS healthcare activities, where a focus on objective metrics, such as relative value units (RVUs), is more common.4

Several qualitative and quantitative indicators are available to assess research activity.5 A total of 80 different indicators were identified in a systematic review of methods for assessing the impact of healthcare research.6 However, there is currently no consensus on which indicators to use and how often to measure them. Some countries have developed initiatives to objectively and comparatively assess research quality, such as the Research Excellence Framework (REF) in the UK.7 The REF assesses the quality of research in universities and research centres, with the aim of informing the allocation of funding and promoting excellence in academic research. However, it has not been adapted to the health sector, and therefore excludes HPSs. In Spain, the Instituto de Salud Carlos III has developed a dashboard for monitoring accredited Health Research Institutes.8

The aim of this study was to develop, by consensus, a dashboard model to standardise and promote the evaluation of research activity in Spanish HPSs. Through this process, we developed a consensus model that lists the variables that need to be measured and how often this should be done.

This study was conducted according to the modified Delphi method9 and developed in 5 phases: formation of the coordinating group; development of a list of scenarios; selection of participating centres; evaluation of the list of scenarios; and analysis of the results.

This study defined a dashboard for assessing research activity in HPSs, making it possible to define objectives, set challenges, and improve the impact of HPSs in generating knowledge. Firstly, the 16 participants in the Delphi consultation process—all from the Spanish HPSs with the highest number of publications—agreed that HPSs should have available a research dashboard with a basic structure and a common minimum dataset. A total of 40 indicators were identified to evaluate the research activity of the HPSs, and consensus (≥75%) was reached on their need and feasibility of measurement. These indicators assess publications (13 indicators), human resources (12 indicators), research projects (9 indicators), doctoral theses (4 indicators), and patents and intellectual property registrations (2 indicators). To date, this is the first published research evaluation dashboard adapted to the reality of Spanish HPSs.

It should be noted that no consensus was reached on the need to evaluate communications at conferences. Currently, they have little curricular value in the academic and professional sphere. Therefore, research groups should consider such communications as a way of making preliminary or partial results visible. The final availability of such communications to the scientific community should be in the form of scientific articles in biomedical journals. This perspective, which is quite widespread, does not yet seem to have been incorporated into the research culture of Spanish HPSs.

One of the main achievements of this study is to have reached consensus on distinguishing HPS-led research projects from those led by other groups in which HPSs participate and to have approved a definition of this criterion according to whether the projects are single-centre or multicentre. From now on, it will be possible to compare both types of project conducted in Spanish HPSs using a common framework. Unfortunately, consensus was not reached on the issue of publications. The panel were inconsistent in that they agreed that HPS-led publications should be distinguished from those led by other groups in which HPSs participate, but failed to reach consensus on a definition of these aspects. Future work will be needed to standardise the quantitative figures that may eventually be collected in Spain on the number of articles led by the 2 types of group.

A consensus was reached on excluding from the dashboard results stratified by individual researcher or by research group, if there is more than one group in the HPS. This aspect is reasonable in reference to publicly shared dashboards or if results are integrated into the dashboard system of a scientific society. However, in reference to the HPS internal dashboards, we suggest that it could be useful to replicate all the approved indicators for each research group in the HPSs, so that their performance can be assessed comparatively. Similarly, individual indicators, such as the h-index, have a strategic function in the internal dashboards of HPSs to measure researcher performance and growth. This information can be used to customise recognition and incentive programs to acknowledge and motivate researchers at the organisational level.

No consensus was reached on how to measure research infrastructure, as there are no accepted standards and it is difficult to assess in hospital departments or units.

A dashboard enhances efficiency in managing research, enables effective monitoring of the results, and supports strategic reflection on the role of each HPS in research activity. Previous studies have focussed on the definition or analysis of the best indicators to be included in such dashboards. Research Centers in Minority Insitutions funded by the National Institute of Health (NIH) worked on defining a global dashboard for all centres included.10 Indicators were established based on 4 main targets: increasing scientific productivity, fostering scientific collaborations, promoting professional growth, and expanding research resources. On the other hand, a systematic review found that the most common research performance indicators in 50 studies were as follows: number of publications (n=38), number of citations (n=27), impact factor (n=15), research funding (n=10), degree of co-authorship (n=9), and h-index (n=5).5 Overall, this study found that there was limited research on feasibility, validity, reliability, and acceptability. In 2017, another systematic review on assessing the impact of health research identified 5 broad categories: (1) direct research-related impact; (2) influence on policy-making; (3) health and health systems impact; (4) health-realted and societal impact; and (5) broader economic impact.6 The first category was the most common scenario and addresses the assessment of new knowledge generation and dissemination, capacity building, training, and leadership, and the development of research networks. Most of the indicators included in our dashboard are from the first category (i.e., direct research-related impact), such as the number of publications, impact factor, research projects, and so on. This result may be due to the fact that these indicators are well established, objective, relatively easy to obtain, and are in line with the scales used by most research funding agencies and institutions. The availability of such a dashboard is the first step in further exploring research results in a more holistic way, thereby facilitating data-driven and evidence-based decision-making.

However, as the scientific landscape evolves, new forms of evaluation are emerging. Any such dashboard should therefore be dynamic and be able to adapt to new evidence and trends in research assessment. There is ongoing debate on identifying indicators that better reflect how research results can change or improve clinical practice and their real impact on society, as traditional metrics are geared toward measuring the dissemination of knowledge in the scientific world rather than its real-world impact.11 It is important to understand the advantages and disadvantages of these indicators and to consider other qualitative and contextual factors when evaluating scientific research.12 The Leiden Manifesto and the San Francisco Declaration are 2 initiatives that seek to improve the assessment of the quality and impact of scientific results.12–14 The Leiden Manifesto emphasises the need for a more balanced and accountable evaluation that goes beyond traditional metrics. It aims to avoid an over-reliance on quantitative indicators and to encourage a more comprehensive and contextualised evaluation of research.12 The San Francisco Declaration, also known as the Declaration on Research Assessment (DORA), challenges the overuse of the impact factor and promotes a more holistic assessment that emphasises the quality, originality, and real impact of research, rather than relying solely on bibliometric indicators.13,14 Large investments in research have sparked debate on the importance of incentivising researchers and organisations to conduct responsible research15 that has a tangible impact on society, provides a return on investment, and saves costs.16

The development of dashboards to assess responsible research is essential to promote change at the institutional level. The European Commission's report Next-Generation Metrics: Responsible Metrics and Evaluation for Open Science provides a detailed analysis of alternative metrics for research evaluation and advocates for a responsible and equitable approach to the development and implementation of next-generation metrics.17 The proposed framework highlights plurality of metrics, contextualisation, transparency, qualitative assessment, and equity as key principles for improving the evaluation of open science. The report stresses that the European Commission should encourage academic publishers across Europe to reduce the importance of journal impact factors as a promotional tool and to use them only in the context of a variety of metrics that provide a more complete picture. However, methodological and temporal difficulties in assessing the real impact of research mean that traditional indicators are still the most widely used metrics for assessing research performance and securing funding.10 In the field of hospital pharmacy, the Granada Declaration on improving the quality of publications and advancing research paradigms in clinical and social pharmacy practice recommends that decision-makers should consider broader bases and not just journal-based metrics for assessing quality and achievements in the disciplines.18

On the other hand, the collation of the main lines of research in HPSs shows a notable degree of homogeneity in the fields where research is being conducted, which is expected to facilitate the creation of synergies between different services. Pharmacists involved in networks and/or collaborative research groups have a greater chance of producing more impactful research than those working individually. In 2020 and 2021, the 3 most common areas addressed in the articles published in Farmacia Hospitalaria were as follows: management and organisation of pharmacy services (n=28), pharmacogenetics and pharmacogenomics (n=18), and oncology (n=18).19 Nevertheless, it should be noted that during this period, there were 2 special issues on COVID-19 crisis management and on personalised medicine.

The main limitation of this study is that the creation of the dashboard was based on expert opinion, as no previously published evidence on the composition of HPS dashboards was found. In addition, a repository designed for archiving research activity might be misconstrued as a system fostering undue competition. There is also a risk of that the ultimate goal would be to achieve high scores on the dashboard metrics, rather than to pursue research with a real impact on the health system and society. Finally, the dashboard needs to be dynamic and able to adapt to new forms of evaluation, especially by incorporating a qualitative assessment of research activity.

In conclusion, this study has developed by consensus the first dashboard model for assessing HPS research activity. This dashboard will facilitate the systematic and continuous analysis of productivity and research impact in this setting. It can also be used to compare different HPSs in Spain, to create synergies, and to identify trends, patterns, and challenges. A further step would be to create a system for collecting and sharing these data among hospitals under the coordination of the SEFH.

None declared.

Bernardo Santos and Ana Herranz proposed the study and coordinated its implementation. Bernardo Santos designed the study method. All authors participated in drafting the proposed dashboard scenarios, selecting the experts, and in project presentation meetings. Ana Belén Guisado and Vicente Escudero analysed the data obtained from the Delphi consultation process. Vicente Escudero wrote the first draft of the manuscript which was reviewed and approved by all authors.

All the authors accept the responsibilites defined by the International Committee of Medical Journal Editors (available at: http://www.icmje.org/).

In the event of publication, the authors grant exclusive rights of reproduction, distribution, translation, and public communication (by any means or sound, audiovisual or electronic support) of their work to Farmacia Hospitalaria and, by extension, to the SEFH. To this end, a letter of assignment of rights will be signed at the time of sending the paper through the online manuscript management system.

Vicente Escudero-Vilaplana: Data curation, Formal analysis, Investigation, Project administration, Writing – original draft, Writing – review & editing. Ana Belén Guisado-Gil: Data curation, Formal analysis, Investigation, Project administration, Writing – review & editing. Bernardo Santos-Ramos: Conceptualization, Investigation, Methodology, Project administration, Writing – review & editing. Ana Herranz: Conceptualization, Investigation, Methodology, Project administration, Writing – review & editing.