Insights from using an outcomes measurement properties search filter and conducting citation searches to locate psychometric articles of tools used to measure context attributes
BMC Research Notes volume 16, Article number: 34 (2023)
To describe our experience with using a methodological outcomes measurement search filter (precise and sensitive versions of a filter designed to locate articles that report on psychometric properties of measurement tools) and citation searches to locate psychometric articles for tools that can be used to measure context attributes. To compare the precise filter when used alone and with reference list checking to citation searching according to number of records found, precision, and sensitivity.
Using the precise filter, we located 130 of 150 (86.6%) psychometric articles related to 22 of 31 (71.0%) tools that potentially measured an attribute of context. In a subset of six tools, the precise filter alone was more precise than searching with the precise filter combined with reference list searching, or citation searching alone. The precise filter combined with reference list checking was the most sensitive search method examined. Overall, we found the precise filter helpful for our project as it decreased record screening time. For non-patient reported outcomes tools, we had less success with locating psychometric articles using the precise filter because some psychometric articles were not indexed in PubMed. More research that systematically evaluates database searching methods is needed to validate our findings.
Evaluation of context is important for the translation of research into policy and practice [1,2,3,4]. Evaluation is the process by which implementation strategies, innovation, or context are tested using valid and reliable tools  to determine their ability to influence research uptake or outcomes . Systematic identification of psychometric articles can be resource intensive and time-consuming . Psychometrics are properties of measurement tools that dictate their ability to capture the concept they are intended to measure, minimize measurement error and detect change over time  Many validated search filters  have been developed to mitigate the workload associated with searching the extensive amount of published literature indexed in bibliographic databases. Methodological search filters are a combination of search terms that ensure the retrieval of a particular type of article . Terwee and colleagues  developed a methodological search filter to identify psychometric articles indexed in PubMed for tools that measure patient-reported outcomes (PROM) (tools measuring health outcomes and daily functions reported by patients [10, 11]) and potentially non-PROM measurement tools. This outcome measurement properties (OMP) filter has a sensitive and a precise version, hereafter referred to as the “sensitive filter” or the “precise filter”. The sensitive filter was designed for systematic reviews and the precise filter for gathering psychometric articles without the need for a comprehensive search . It is important to validate search filters to ensure that there is a balance between sensitivity (ability to identify relevant articles) and precision (minimize number of records) [10, 12].
We completed a series of three large international studies to develop the Implementation in CONtext (ICON) framework [1,2,3]: (1) a concept analysis of context as reported in the international published literature (n = 70 articles) ; (2) an analysis of 145 interviews conducted with a wide range of healthcare providers in multiple healthcare settings for elements of context relevant to research use in clinical practice ; (3) a qualitative interview study with 39 health system stakeholder (individuals responsible for change in a healthcare organization) in 4 countries to elicit their perceptions of important elements of context for implementation . The findings from these studies were then triangulated to produce ICON, a meta-framework where context is conceptualized in three levels (micro, meso, and macro) that are divided into five domains, 22 core attributes and 108 features . We aim to create a repository of tools that measure ICON attributes, and to summarize the psychometric properties of these tools. However, the heterogeneity in the constructs and populations associated with ICON attributes made it difficult to construct a search strategy, guided by traditional systematic review methods [14, 15], that had a good balance between sensitivity and precision. We used Terwee and colleagues’  filter (both versions) to refine our search for psychometric articles. Since a tool name was required to locate psychometric articles of specific tools, we performed citation searches [16, 17] to locate articles for unnamed tools (Additional File 1: search method definitions).
The purpose of this research note is to describe our experience of using an OMP filter and citation searching to identify psychometric articles for 31 tools [18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48] that measure ICON attributes. Our objectives are to: (1) provide an overview of our experience of using an OMP filter (both versions) and citation searching; (2) compare the precise filter with or without reference checking with citation searching according to the number of records located and time required to complete article selection, precision, and sensitivity; (3) discuss the benefits of and the insights garnered from using an OMP filter and citation searching.
In consultation with a health information specialist (TR), we performed searches in PubMed from inception to July/December 2021 using the precise filter combined with the tool’s name (Additional File 2: search terms). We used the sensitive filter only if the search using the precise filter produced zero results. We performed citation searches using an indexed development article when: (1) tools had no name; or (2) the OMP (both versions) filter search did not locate a relevant article. Citation searching was conducted in Scopus and Web of Science. We checked the reference lists of included articles to supplement our database searches [17, 49]. We (LKC, MG, WJS or VW) screened records and full text articles independently and in duplicate. Discrepancies were resolved through consensus or in consultation with a senior researcher. We included articles that aimed to evaluate the psychometric properties of any of the 31 tools [18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48].
Nineteen of 31 (61.3%) tools were non-PROM [22, 24,25,26,27, 29,30,31, 33, 34, 36, 38, 39, 41,42,43,44, 46, 47], and the remaining 12 (38.7%) were PROM [18,19,20,21, 23, 28, 32, 35, 37, 40, 45, 48]. We conducted a search on PubMed using an OMP filter for 29 of 31 (93.5%) tools which had names; 12 PROM [18,19,20,21, 23, 28, 32, 35, 37, 40, 45, 48] and 17 non-PROM [22, 25,26,27, 29,30,31, 33, 34, 36, 38, 39, 41,42,43,44, 46]. We used the precise filter for 28 of 29 named tools [18,19,20,21,22,23, 25,26,27,28,29,30,31,32,33,34,35, 37,38,39,40,41,42,43,44,45,46, 48], and the sensitive filter for one tool  (precise version found zero records). We performed a citation search on 2 of 31 tools (6.5%) (both non-PROM [24, 47]) because they were unnamed.
Methods for sub-analysis
We compared the search results of the precise filter (alone or combined with reference checking), with citation searching in a sub-analysis of six randomly selected tools (three PROM [28, 35, 40] and three nonPROM [26, 31, 43]). We compared the search methods according to number of records identified, time to complete record screening, precision, and sensitivity. One researcher appraised the inclusion of articles for the citation searches. Precision is the total number of included articles divided by the total number of records identified by a search method . Sensitivity is the number of included articles divided by the total number of unique articles included across the three search methods .
We found at least one relevant article for 27 of 31 (87.1%) tools [20,21,22,23,24,25,26,27,28, 30,31,32,33, 35,36,37,38,39,40,41,42,43,44,45,46,47,48] (Table 1: searches and search results). Across all search methods, we screened 10,303 records, appraised 546 full-text articles for eligibility and included 150 articles (Additional file 3 and Additional file 4: PRISMA diagram and search results for each tool respectively).
OMP filter (N = 29 named tools) and reference checks (N = 22 tools)
Our searches using the precise filter successfully identified 130 relevant articles across 22 of 29 (75.9%) named tools [20,21,22,23, 26,27,28, 30,31,32,33, 35, 37, 39,40,41,42,43,44,45,46, 48]. We identified a relevant article for a slightly higher proportion of PROMS (10 of 12; 83.3%) [20, 21, 23, 28, 32, 35, 37, 40, 45, 48] than non-PROMS (12 of 17; 70.6%) [22, 26, 27, 30, 31, 33, 39, 41,42,43,44, 46]. Our OMP (both versions) filter search did not locate a relevant article for 7 of 29 tools (24.1%), a larger proportion of non-PROM (5 of 17; 41.7%) [25, 29, 34, 36, 38] than PROM (2 of 12; 16.7%) [18, 19].
By searching the reference lists of included psychometric articles, we found 12 additional psychometric articles [43, 50,51,52,53,54,55,56,57,58,59,60] for 5 (two PROM [21, 45] and three non-PROM [30, 39, 43]) of the 22 (22.7%) tools identified using the precise filter. Ten of the 12 (83.3%) articles (one article  for a PROM  and nine articles [43, 51,52,53, 56,57,58,59,60] across three non-PROMs [30, 39, 43]) were not indexed in PubMed. The remaining two articles [54, 55] (for one PROM ) were indexed in PubMed but not captured by the precise filter search.
Citation searches (N = 9 tools) and reference checks (N = 5 tools)
Citation searches were performed using the index tool development article for 9 of 31 (29.0%) tools (two PROM [18, 19] and seven non-PROM [24, 25, 29, 34, 36, 38, 47]). Citation searching retrieved eight psychometric articles across 5 of 9 (55.6%) [24, 25, 36, 38, 47] tools. We did not locate additional articles from reference checking. We were unable to perform a citation search for 3 of 9 (33.3%) tools (two PROM [18, 19] and one non-PROM ) because the development study was in a book or an online report and not indexed on Scopus or Web of Science. With citation searching we did not locate a psychometric article for the non-PROM  tool.
Amongst a subset of six tools, the precise filter alone identified 500 records, while the precise filter search combined with reference checks located 1882 records, and citation searching found 2340 records (Table 2: detailed sub-analysis results). The precise filter search alone was also the most time-efficient method in our sub-analysis. We completed screening of records and full-text articles for the six tools four times faster when using the precise filter alone, compared to citation searches (~ 350.5 min versus 1386 min, respectively). The addition of reference checks to the precise filter search increased the time required for article selection (~ 1079.8 min) but was still more time efficient than citation searching.
The precise filter combined with reference checks located the most included articles (n = 56 (96.6%)). The precise filter when combined with reference checking was the most sensitive search method (mean, standard deviation = 96.7%, 8.2%) and the precise filter alone was the most precise search method (precision = 10.8%). The precise filter (alone or combined with reference checks) was more sensitive and precise when searching for psychometric articles for the three PROM [28, 35, 40] than for the three non-PROM [26, 31, 43] tools.
The precise filter in combination with reference checks identified nine articles, across four tools [26, 28, 40, 43]; all these articles were previously unidentified through the citation search (Additional file 5: missed articles and reasons they were missed by one of the methods). One  of 9 (11.1%) articles relating to one tool  was not identified by the citation search because an article  other than the development article was cited. Two of 9 (22.2%) articles [51, 63] relating to two tools [40, 43] were missed by our citation search because they were not indexed as a “cited” article in Scopus or Web of Science even though the authors referenced the article we used for citation searching. Two of 9 (22.2%) articles [64, 65] relating to two tools [28, 43] were not identified by citation searching because they were earlier reports about the tools’ development and were not indexed in either Scopus or Web of Science. Four of 9 (44.4%) articles [66,67,68,69] relating to two tools [28, 40] were missed by citation searching because the authors cited a different article reporting on the tool’s development from the one we used in our citation search. Additional citation searches were performed for these two tools [28, 40] using the newly discovered development articles [65, 70].
In summary, the additional searches did not change the precision of citation searching across the six tools (2.1%) but did increase the overall mean sensitivity of citation searching from 93.3% to 94.1% (Additional file 6: sub-analysis results with the additional citation searches considered). In contrast, the citation search captured two articles [71, 72], relating to one tool , that were not located by the precise filter search because they were not indexed in PubMed.
Our study provides a greater understanding of the benefits of using the precise filter and conducting citation searches (see Table 3).
The most important benefit of searching using the precise filter was the decreased time required to screen records. Based on our sub-analysis of six tools, searching with the precise filter alone proved to be the most precise method, and the combination of the precise filter with reference checks was the most sensitive search method. However, before using the OMP filter (both versions), the following should be considered: 1) the OMP filter cannot locate articles that are not indexed in PubMed; 2) the search for psychometric articles of particular tools is dependent on knowing the name of the tool; and 3) the unknown implications of recent changes to the PubMed interface [73, 74] (detailed in Table 3) on the OMP filter.
We found that the search conducted with the precise filter successfully identified more psychometric articles for PROM than non-PROM. However, further systematic research comparing the efficacy of the OMP filter at locating psychometric articles for PROM and non-PROM is needed to confirm our findings. Systematic testing of the OMP filter in the context of recent developments in the field of measurement is an avenue for future research (e.g. how the addition of search terms related to content validity (relevance, comprehensiveness, comprehensibility [75, 76]) and pragmatic properties (usefulness, compatibility, acceptability, etc. ) affect sensitivity and specificity).
Citation searching was helpful in identifying psychometric articles of tools that had not been named or tools for which the search with the OMP filter (both versions) did not locate a psychometric article (the majority being non-PROMs). When planning a citation search to locate psychometric studies, the following should be considered: (1) tool development study might be reported in multiple articles; (2) multiple citation searches on several development articles may be more sensitive but increase the number of retrieved records; (3) citation searching is limited to the articles indexed in the utilized database(s); (4) citation searching may identify more records than the precise filter.
We did not intend to conduct a systematic evaluation of the OMP filter or citation searching. Furthermore, we did not evaluate other search methods that are used in measurement reviews (e.g., a traditional search strategy). Our sub-analysis, comparing the precise filter and citation searching, was based on only six tools. The references we used for citation searching were originally retrieved using the OMP filter and only one researcher screened records retrieved by citation searching. Considering these limitations, our findings may reflect trends in each methods’ search results, precision and sensitivity that require further investigation.
The precise filter decreased the number of records required for screening for our project. Based on a sample of six tools, searching with the precise filter alone was more precise, sensitive, and efficient than citation searching. More systematic research is required to evaluate the usefulness of OMP filters and other search methods in measurement reviews, especially reviews on tools that are developed/used across multiple disciplines and specialties and address a diverse range of theoretical constructs.
Availability of data and materials
The definition of search methods, search strategy, PRISMA diagram, the search results for each tool, reasons for discrepancies between search methods during the sub-analysis, changes in results with additional citation searches for the sub-analysis are provided as additional files. The datasets used and/or analyzed during the current study are available from the corresponding author on reasonable request.
Squires JE, Graham I, Bashir K, et al. Understanding context: a concept analysis. J Adv Nurs. 2019;75(12):3448–70.
Squires JE, Aloisio LD, Grimshaw JM, et al. Attributes of context relevant to healthcare professionals’ use of research evidence in clinical practice: a multi-study analysis. Implement Sci. 2019;14(1):52.
Squires JE, Hutchinson AM, Coughlin M, et al. Stakeholder perspectives of attributes and features of context relevant to knowledge translation in health settings: a multi-country analysis. Int J Health Plann Manage. 2021;11(8):1373.
Kivimäki M, Elovainio M. A short version of the team climate inventory: development and psychometric properties. J Occup Organ Psychol. 1999;72(2):241–6.
Bowen S. A Guide to evaluation in health research. 2012; https://cihr-irsc.gc.ca/e/45336.html#a4.4.1. Accessed July 27. 2022.
Straus SETJ, Bhattacharyya O, Zwarenstein M, Graham ID. Chapter 3 5 Monitoring knowledge useand evaluating outcomes. In: Straus SETJ, Graham ID, editors. Knowledge translation in health care: moving from evidence to practice. Toronto: Wiley; 2013.
Polanin JR, Pigott TD, Espelage DL, Grotpeter JK. Best practice guidelines for abstract screening large-evidence systematic reviews and meta-analyses. Res Synth Methods. 2019;10(3):330–42.
Mokkink LB, Prinsen C, Patrick DL, et al. COSMIN methodology for systematic reviews of patient-reported outcome measures (PROMs). User Manual. 2018;78(1):63.
The InterTASC Information specialists' sub-Group. ISSG search filter resource. 2006. https://sites.google.com/a/york.ac.uk/issg-search-filters-resource/home?authuser=0. Accessed Sep 7 2022.
Terwee CB, Jansma EP, Riphagen II, de Vet HC. Development of a methodological pubmed search filter for finding studies on measurement properties of measurement instruments. Qual Life Res. 2009;18(8):1115–23.
Quittner AL, Nicolais CJ, Saez-Flores E. Integrating patient-reported outcomes into research and clinical practice. Kendig’s Disord Respir Tract Children. 2019;231–240:233.
Durao S, Kredo T, Volmink J. Validation of a search strategy to identify nutrition trials in pubMed using the relative recall method. J Clin Epidemiol. 2015;68(6):610–6.
Squires JE, Brehaut J, Curran J, et al. Understanding context in knowledge translation: development and consensus of a conceptual framework (phase I). In: Canadian Institutes of Health Research 2014–2015.
Lefebvre C, Glanville J, Briscoe S, et al. Searching for and selecting studies. Cochrane Handbook Syst Rev Interv. 2019;23:67–107.
Aromataris E, Munn Z. Chapter 1: JBI Systematic Reviews. JBI Manual for Evidence Synthesis 2020. https://synthesismanual.jbi.global. https://0-doi-org.brum.beds.ac.uk/10.46658/JBIMES-20-02. Accessed 2 Oct 2022
Wright K, Golder S, Rodriguez-Lopez R. Citation searching: a systematic review case study of multiple risk behaviour interventions. BMC Med Res Methodol. 2014;14(1):1–8.
Lefebvre C, Glanville J, Briscoe S, et al. Technical supplement to chapter 4: searching for and selecting studies. cochrane handbook for systematic reviews of interventions version 6.3 (updated February 2022) 2022. www.training.cochrane.org/handbook. Accessed 15 Aug 2022.
Webster G. Final report on the patient satisfaction questionnaire project. 1989.
Weiss K. Measuring the reliability and validity of the art of medicine survey. Denver: Denver (CO). HealthCare Research Inc 2001.
Coleman EA, Mahoney E, Parry C. Assessing the quality of preparation for posthospital care from the patient’s perspective: the care transitions measure. Med Care. 2005;43(3):246–55.
Barr PJ, Thompson R, Walsh T, Grande SW, Ozanne EM, Elwyn G. The psychometric properties of collaborate: a fast and frugal patient-reported measure of the shared decision-making process. J Med Internet Res. 2014;6(1):e2.
King G, Servais M, Kertoy M, et al. A measure of community members’ perceptions of the impacts of research partnerships in health and social services. Eval Program Plann. 2009;32(3):289–99.
Campbell J, Narayanan A, Burford B, Greco M. Validation of a multi-source feedback tool for use in general practice. Educ Prim Care. 2010;21(3):165–79.
Ehrhart MG. Leadership and procedural justice climate as antecedents of unit-level organizational citizenship behavior. Pers Psychol. 2004;57(1):61–94.
Carless SA, Wearing AJ, Mann L. A short measure of transformational leadership. J Bus Psychol. 2000;14(3):389–405.
Aarons GA, Ehrhart MG, Farahnak LR. The implementation leadership scale (ILS): development of a brief measure of unit level implementation leadership. Implement Sci. 2014;9(1):45.
Körner M, Wirtz MA. Development and psychometric properties of a scale for measuring internal participation from a patient and health care professional perspective. BMC Health Serv Res. 2013;13(1):1–11.
Stewart AL, Nápoles-Springer AM, Gregorich SE, Santoyo-Olsson J. Interpersonal processes of care survey: patient-reported measures for diverse groups. Health Serv Res. 2007;42(3 Pt 1):1235–56.
Stogdill RM. Manual for the leadership behaviour development questionnaire: form XII Columbus: Bureau of business research. Ohio: Ohio State University; 1963.
Bass BM, Avolio BJ. Transformational leadership development: Manual for the multifactor leadership questionnaire. Palo Alto: Consulting Psychologists Press; 1990.
Kenaszchuk C, Reeves S, Nicholas D, Zwarenstein M. Validity and reliability of a multiple-group measurement scale for interprofessional collaboration. BMC Health Serv Res. 2010. https://0-doi-org.brum.beds.ac.uk/10.1186/1472-6963-10-83.
Frank C, Asp M, Fridlund B, Baigi A. Questionnaire for patient participation in emergency departments: development and psychometric testing. J Adv Nurs. 2011;67(3):643–51.
Nahm E, Resnick B, Mills ME. Development and pilot-testing of the perceived health web site usability questionnaire (PHWSUQ) for older adults. Stud Health Technol Inform. 2006;122:38.
Hatcher M. Survey of acute care hospitals in the United States relative to technology usage and technology transfer. J Med Syst. 1997;21(5):323–36.
Risser NL, Batey V. Development of an instrument to measure patient satisfaction with nurses and nursing care in primary care settings. Nurs Res. 1975;24(1):45–52.
Welbourne TM, Johnson DE, Erez A. The role-based performance scale: Validity analysis of a theory-based measure. Acad Manag J. 1998;41(5):540–55.
Scholl I, Kriston L, Dirmaier J, Buchholz A, Härter M. Development and psychometric properties of the shared decision making questionnaire—physician version (SDM-Q-Doc). Patient Educ Couns. 2012;88(2):284–90.
Barbuto JE Jr, Wheeler DW. Scale development and construct clarification of servant leadership. Group Organ Manag. 2006;31(3):300–26.
van Dierendonck D, Nuijten I. The servant leadership survey: development and validation of a multidimensional measure. J Bus Psychol. 2011;26(3):249–67.
Kriston L, Scholl I, Hölzel L, Simon D, Loh A, Härter M, The 9-item Shared Decision Making Questionnaire (SDM-Q-9). Development and psychometric properties in a primary care sample. Patient Educ Couns. 2010;80(1):94–9.
Singer S, Meterko M, Baker L, Gaba D, Falwell A, Rosen A. Workforce perceptions of hospital safety culture: development and validation of the patient safety climate in healthcare organizations survey. Health Serv Res. 2007;42(5):1999–2021.
Schutz AL, Counte MA, Meurer S. Development of a patient safety culture measurement tool for ambulatory health care settings: analysis of content validity. Health Care Manag Sci. 2007;10(2):139–49.
Anderson NR, West MA. Measuring climate for work group innovation: Development and validation of the team climate inventory. J Organ Behav. 1998;19(3):235–58.
Berk RA, Berg J, Mortimer R, Walton-Moss B, Yeo TP. Measuring the effectiveness of faculty mentoring relationships. Acad Med. 2005;80(1):66–71.
Glasgow RE, Wagner EH, Schaefer J, Mahoney LD, Reid RJ, Greene SM. Development and validation of the patient assessment of chronic illness care (PACIC). Med Care. 2005;43(5):436–44.
Jones J, Barry M. Developing a scale to measure trust in health promotion partnerships. Health Promot Int. 2011;26(4):484–91.
Costa AC, Anderson N. Measuring trust in teams: development and validation of a multifaceted measure of formative and reflective indicators of team trust. Eur J Work Organ Psychol. 2011;20(1):119–54.
O’Cathain A, Knowles E, Nicholl J. Measuring patients’ experiences and views of the emergency and urgent care system: psychometric testing of the urgent care system questionnaire. BMJ Qual Saf. 2011;20(2):134–40.
Horsley T, Dingwall O, Sampson M. Checking reference lists to find additional studies for systematic reviews. Cochrane Database Syst Rev. 2011. https://0-doi-org.brum.beds.ac.uk/10.1002/14651858.MR000026.pub2.
Cramm JM, Nieboer AP. Development and validation of the older patient assessment of chronic illness care (O-PACIC) scale after hospitalization. Soc Indic Res. 2014;116(3):959–69.
Kivimäki M, George K, Elovainio M, Thomson L, Kalliomäki-Levanto T, Heikkilä A. The team climate inventory (TCI)—four or five factors? Testing the structure of TCI in samples of low and high complexity jobs. J Occup Organ Psychol. 1997;70(4):375–89.
Kivimaki M, Elovainio M. A short version of the team climate inventory: development and psychometric properties. J Occup Organ Psychol. 1998;72:241.
Bobbio A, Dierendonck DV, Manganelli AM. Servant leadership in Italy and its relation to organizational variables. Leadership. 2012;8(3):229–43.
Elwyn G, Barr PJ, Grande SW, Thompson R, Walsh T, Ozanne EM. Developing CollaboRATE: a fast and frugal patient-reported measure of shared decision making in clinical encounters. Patient Educ Couns. 2013;93(1):102–7.
Forcino RC, Bustamante N, Thompson R, et al. Developing and pilot testing a Spanish translation of collaboRATE for use in the United States. PLoS ONE. 2016;11(12): e0168538.
Bycio P, Hackett RD, Allen JS. Further assessments of Bass’s (1985) conceptualization of transactional and transformational leadership. J Appl Psychol. 1995;80(4):468.
Den Hartog DN, Van Muijen JJ, Koopman PL. Transactional versus transformational leadership: an analysis of the MLQ. J Occup Organ Psychol. 1997;70(1):19–34.
Garman AN, Davis-Lenane D, Corrigan PW. Factor structure of the transformational leadership model in human service teams. J Organ Behav. 2003;24(6):803–12.
Tejeda MJ, Scandura TA, Pillai R. The MLQ revisited: Psychometric properties and recommendations. Leadersh Q. 2001;12(1):31–52.
Vandenberghe C, Stordeur S, D’hoore W. Transactional and transformational leadership in nursing: structural validity and substantive relationships. Eur J Psychol Assess. 2002;18(1):16.
Lyon AR, Cook CR, Brown EC, et al. Assessing organizational implementation context in the education sector: confirmatory factor analysis of measures of implementation leadership, climate, and citizenship. Implement Sci. 2018;13(1):1–14.
Finn NK, Torres EM, Ehrhart MG, Roesch SC, Aarons GA. Cross-validation of the implementation leadership scale (ILS) in child welfare service organizations. Child Maltreat. 2016;21(3):250–5.
Goto Y, Miura H, Son D, et al. Psychometric evaluation of the Japanese 9-item shared decision-making questionnaire and its association with decision conflict and patient factors in Japanese primary care. JMA journal. 2020;3(3):208–15.
Anderson N, West MA. The team climate inventory: development of the TCI and its applications in teambuilding for innovativeness. Eur J Work Organ Psychol. 1996;5(1):53–66.
Stewart AL, Nápoles-Springer A, Pérez-Stable EJ. Interpersonal processes of care in diverse populations. Milbank Q. 1999;77(3):305–39.
Kasper J, Heesen C, Köpke S, Fulcher G, Geiger F. Patients’ and observers’ perceptions of involvement differ validation study on inter-relating measures for shared decision making. PLoS One. 2011;6(10):26255.
Beaulieu MD, Haggerty JL, Beaulieu C, et al. Interpersonal communication from the patient perspective: comparison of primary healthcare evaluation instruments. Healthc Policy. 2011;7:108–23.
Haggerty JL, Beaulieu C, Lawson B, Santor DA, Fournier M, Burge F. What patients tell us about primary healthcare evaluation instruments: response formats, bad questions and missing pieces. Healthc Policy. 2011;7:66–78.
Nápoles-Springer AM, Santoyo-Olsson J, O’Brien H, Stewart AL. Using cognitive interviews to develop surveys in diverse populations. Med Care. 2006;44(11 Suppl 3):S21-30.
Simon D, Schorr G, Wirtz M, et al. Development and first validation of the shared decision-making questionnaire (SDM-Q). Patient Educ Couns. 2006;63(3):319–27.
Antino M, Gil-Rodriguez F, Martí M, Barrasa A, Borzillo S. Development and validation of the Spanish version of the team climate inventory: a measurement invariance test. Annu Rev Psychol. 2014;30(2):597–607.
Tseng H-M, Liu F-C, West MA. The team climate inventory (TCI) a psychometric test on a Taiwanese sample of work groups. Small Group Res. 2009;40(4):465–82.
National Library of Medicine. New pubmed to replace legacy PubMed in Mid-May. NLM Tech Bull. 2020. https://www.nlm.nih.gov/pubs/techbull/ma20/ma20_pubmed_default.htmlnational. Accessed 15 Aug 2022.
National Library of Medicine. New pubmed transition FAQs. 2021. https://support.nlm.nih.gov/knowledgebase/article/KA-05275/en-us. Accessed 14 Aug 2022.
Terwee CB, Prinsen CA, Chiarotto A, et al. COSMIN methodology for evaluating the content validity of patient-reported outcome measures: a Delphi study. Qual Life Res. 2018;27(5):1159–70.
Terwee CB, Prinsen C, Chiarotto A, et al. COSMIN methodology for assessing the content validity of PROMs–user manual. Amsterdam: VU University Medical Center; 2018.
Lewis CC, Mettert KD, Stanick CF, et al. The psychometric and pragmatic evidence rating scale (PAPERS) for measure development and evaluation. Implement Res Pract. 2021;2:26334895211037390.
Bakshi AB, Wee SL, Tay C, et al. Validation of the care transition measure in multi-ethnic South-East Asia in Singapore. BMC Health Serv Res. 2012;12:256.
Cao X, Chen L, Diao Y, Tian L, Liu W, Jiang X. Validity and reliability of the Chinese version of the care transition measure. PLoS ONE. 2015;10(5): e0127403.
Coleman EA, Smith JD, Frank JC, Eilertsen TB, Thiare JN, Kramer AM. Development and testing of a measure designed to assess the quality of care transitions. Int J Integr Care. 2002;2: e02.
Shadmi E, Zisberg A, Coleman EA. Translation and validation of the care transition measure into Hebrew and Arabic. Int J Qual Health Care. 2009;21(2):97–102.
Charalambous A. Validation and test-retest reliability of the Risser patient satisfaction scale in Cyprus. J Nurs Manag. 2010;18(1):61–9.
Charalambous A, Adamakidou T. Risser patient satisfaction scale: a validation study in Greek cancer patients. BMC Nurs. 2012;11:27.
Ventura MR, Fox RN, Corley MC, Mercurio SM. A patient satisfaction measure as a criterion to evaluate primary nursing. Nurs Res. 1982;31(4):226–30.
Biering P, Becker H, Calvin A, Grobe SJ. Casting light on the concept of patient satisfaction by studying the construct validity and the sensitivity of a questionnaire. Int J Health Care Qual Assur Inc Leadersh Health Serv. 2006;19(2–3):246–58.
Yellen E, Davis GC, Ricard R. The measurement of patient satisfaction. J Nurs Care Qual. 2002;16(4):23–9.
Stanick CF, Halko HM, Nolen EA, et al. Pragmatic measures for implementation research: development of the psychometric and pragmatic evidence rating scale. Transl Behav Med. 2021;11(1):11–20.
JES holds a University Research Chair in Health Evidence Implementation. AMH holds a Chair in Nursing at the Centre for Quality and Patient Safety Research at Barwon Health Partnership and is the Director for the Centre for Quality and Patient Safety Research (QPS). IDG holds a CIHR Foundation Grant (FDN# 143237) in Integrated Knowledge Translation.
This research was supported by a Canadian Institutes of Health Research (CIHR) Project Grant (Award #: 426122). The CIHR had no role in the study’s design, conduct and reporting.
Ethics approval and consent to participate
Consent for publication
The authors declare that they have no competing interests.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Definition of search methods.
Tool names and search terms combined with terwee filter.
PRISMA diagram of identified records screened and full texts appraised.
Search results breakdown for each tool.
Reasons for discrepancies between the precise outcomes measurement properties filter search and citation sear.
Sub-group analysis of terwee filter (with or without reference checking) and citation search.
About this article
Cite this article
Santos, W.J., Hutchinson, A.M., Rader, T. et al. Insights from using an outcomes measurement properties search filter and conducting citation searches to locate psychometric articles of tools used to measure context attributes. BMC Res Notes 16, 34 (2023). https://0-doi-org.brum.beds.ac.uk/10.1186/s13104-023-06294-2