Development of the quality of teen trauma acute care patient and parent-reported experience measure

Abstract

Objective

Patient-Reported Experience Measures (PREMs) provide valuable patient feedback on quality of care and have been associated with clinical outcomes. We aimed to test the reliability of a modified adult trauma care PREM instrument delivered to adolescents admitted to hospital for traumatic injuries, and their parents. Modifications included addition of questions reflecting teen-focused constructs on education supports, social network maintenance and family accommodation.

Results

Forty adolescent patients and 40 parents participated. Test-retest reliability was assessed using Cohen’s kappa, weighted kappa, and percent agreement between responses. Directionality of changed responses was noted. Most of the study ran during the COVID-19 pandemic. We established good reliability of questions related to in-hospital and post-discharge communication, clinical and ancillary care and family accommodation. We identified poorer reliability among constructs reflecting experiences that varied from the norm during the pandemic, which included “maintenance of social networks”, “education supports”, “scheduling clinical follow-ups” and “post-discharge supports”. Parents, but not patients, demonstrated more directionality of change of responses by responding with more negative in-hospital and more positive post-discharge experiences over time between the test and retest periods, suggesting risk of recall bias. Situational factors due to the COVID-19 pandemic and potential risks of recall bias may have limited the reliability of some parts of the survey.

Introduction

Patient-reported experience measures (PREMs) and outcome measures (PROMs) can guide alignment of care practices with patient and family preferences, needs, and values [1]. Positive patient and family experiences in medical and surgical care have been shown to improve clinical outcomes, including attention to medication regiments, adverse event reporting, rehabilitative therapy outcomes, and even length of stay (LOS) [2,3,4]. Patient and family experiences can also influence clinical decision-making, care coordination, and communication, enabling improved clinical outcomes [5]. Previous research highlights the importance of patient experience research in adolescents, who may fall between the patient management approaches of pediatric versus adult centers. For example, adolescents may find themselves in a position where they recognize the need to understand their own healthcare information but are not given the opportunity to do so [6]. As such, several PREMs and PROMs instruments have been developed for children and/or their families that evaluate general experiences of healthcare or focus on specific fields such a juvenile arthritis, pediatric allergies, inflammatory bowel disease, epilepsy, diabetes, cancer, emergency care and mental health [7,8,9]. PREMs targeting adolescents specifically are uncommon but have been reported for substance use treatment [10].

Despite extensive research in clinical management and outcomes of traumatic injuries in children and adolescents, there has been little research in this growing field of patient and family experience with regards to management of traumatic injuries. We sought to develop an instrument to measure PREMs/PROMs of adolescents and their parents for a proposed comparative analysis of adolescents managed at adult versus pediatric trauma centers (unpublished data). Development and Reliability testing of a new Quality of Teen Trauma Acute Care Patient-Reported Experience Measure (QTTAC-PREM), patient and parent versions, is reported here.

Main text

Methods

QTTAC-PREM development

We chose to adapt the Short-form version of the Quality of Trauma Acute Care Patient-Reported Experience Measure (QTAC-PREM), developed by Bobrovitz, which focuses on in-hospital and post-discharge phases of adult (16y +) trauma care, and was derived within the same regional trauma system as the proposed comparative study [11, 12]. The QTAC-PREM was developed through an iterative process involving focus groups, cognitive interviews, and pilot testing and found to have moderate to excellent test-retest reliability [11,12,13]. Several questions were added to the short-form QTAC-PREM to target constructs known to influence adolescent health and wellness. Consultation with adolescent medicine experts and employment of the HEADSS (home, education and employment, activities, drugs, sexuality, suicide/depression) framework identified areas of adolescent health not addressed in the original measure including questions focusing on “Educational supports”, “Social supports” and “Family accommodation” [14]. Questions focusing on post-discharge pain control and receipt of prescriptions for opiates, were included to explore increasing risk of opioid dependency in adolescence [15]. To target parent-specific needs, we queried family accommodation opportunities and use during their child’s hospital admission, adapting from work done previously by Franck [16]. Questions targeting school supports were adapted from the Pediatric Integrated Care Survey, a previously validated PREM examining family experiences of care integration within pediatrics [17]. Questions on maintenance of social support were adapted from the Social Support Questionnaire for Children [18]. Questions were formulated to address the adolescent patient in the patient version and the caregiver in the parent version.

Modifications resulted in 24 and 25 new in-hospital and post-discharge care questions for the patient and parent surveys, respectively. The new survey tool was piloted for readability with a group of adolescents recruited from the Family and Community Resource Centre of the Alberta Children’s Hospital and teen family members of the study team. This informed further modifications to question wording and survey logic, framing questions to be understood by adolescents.

Copies of the Teen and Parent-versions of the QTTAC-PREM are available as supplementary files at Additional files 1 and 2.

Reliability testing

Participants for reliability testing of the QTTAC-PREM were recruited as part of a larger study comparing adolescent patient and parent experience at the regional pediatric or adult trauma center (unpublished data). Adolescents (15–17 years) admitted with an injury diagnosis between 01/01/2020 and 31/05/2021 were invited to participate, along with one of their parents. We excluded patients injured from self-inflicted events to avoid confounding from psychological factors that could bias experiences relative to those sustaining non-self-inflicted injuries. We excluded patient whose primary nurse felt they and/or their parent would be unable to provide informed consent or complete the survey due to cognitive impairments (e.g. brain injury or pre-existing deficits) or language barriers (the survey was only available in English). If more than one guardian was with the patient during their hospital stay, the guardian who self-identified as being with the patient the most was chosen for the study.

Based on the proposed sample size of 75 from previous reliability testing of the original QTAC-PREM, we aimed for 80 participants, enrolling the first 40 patients and 40 parents recruited into the larger study agreeing to participate in this retest process [13].

Using emails provided, patients and parents were sent an electronic link to the patient or parent version of the QTTAC-PREM 8 weeks post-discharge, accommodating the assumption that most patients would have had their follow-up care by then. Surveys were completed online using the Qualtrics platform of the University of Calgary. Retesting was done by emailing the same survey 7–10 days after initial survey completion. Participants were offered a $25 gift card for each test and retest survey completed.

Data analysis

Cohen’s kappa [with 95% confidence intervals (CI)] and percentage agreement were calculated to determine the level of agreement between test and retest survey responses for each participant. A weighted kappa was used for questions assessed on an ordinal scale. All data analysis was completed in SPSS version 25.

We used Cohen’s suggested values for interpretation, with kappa scores 0.01–0.20 reflecting slight agreement, 0.21–0.40 fair, 0.41–0.60 moderate, 0.61–0.80 substantial, and 0.81–1.00 near-perfect [19]. Though kappa remains a validated and well-used measure of reliability, we observed limitations for questions where responses tended to be uni-modal, with low kappa scores despite very high percentage agreement. Because agreement remained significantly greater than chance and CIs for these questions tended to be large, percentage agreement was chosen to allow for inclusion of such questions, provided their CIs also suggested at least moderate reliability. As such, questions were deemed reliable if they reached a kappa of ≥ 0.41 (moderate or greater agreement) or a percentage agreement of > 80% if kappa CIs exceeded 0.40 as suggested by Miles [19, 20]. Where applicable, descriptive comparisons of kappa values were made with those of the original QTAC-PREM [12].

Directionality of change of responses between the test and the retest phases (reported experience being more negative or positive) was recorded to glean if responses may have changed over time due to changes in overall perceptions of experience, which would be suggested if responses generally changed in one direction versus randomly in either direction. Questions where a greater proportion of retest responses becoming more negative than positive were categorized as “became more negative”; those where a greater proportion became mor positive were categorized as “became more positive”; those with equal proportions were categorized as “no change”.

Results

Forty adolescents and 40 parents completed the QTTAC-PREM test-retest process, representing 77% and 71%, respectively, of the larger study recruitment. The test-retest reliability scores for individual questions are highlighted in Table 1.

Table 1 Survey Test-Retest Kappa and Agreement Percentage

Kappa scores of adolescents ranged between 0.18 and 1.0. with 30/46 questions from the patient version reaching moderate- to near-perfect reliability. A further seven questions were considered reliable with response agreements being > 80% and kappa 95% CIs spanning > 0.40.

Amongst adolescents, nine questions with poor reliability predominated in the PREM constructs of “Social network supports”, “Education supports”, “Discharge and post-discharge information and communication” and “Post-discharge supports”. Isolated questions from each of “In-hospital information and communication” and “Clinical and ancillary care” constructs also tested as unreliable, although appeared reliable with patients > 15 years old in the original QTAC-PREM [12]. For both in-patient and post-discharge care, questions about healthcare responses to expression of frustrations were unreliable. All other constructs, global ratings and isolated questions tested as reliable.

Amongst parents, seven questions with poor reliability predominated in the PREM constructs of “Social network supports” and “Scheduling follow-ups”. Isolated questions from each of “Clinical and ancillary care” and “Post-discharge supports” also tested as unreliable, although appeared reliable with patients > 15 years old in the original QTAC-PREM [12]. As with adolescents, questions asking about healthcare responses to expression of frustrations whether in-hospital or post-discharge were unreliable. All other constructs and global rating questions tested as reliable.

There was little consistency of directionality of changes in responses between the test and retest phases among adolescents (Fig. 1). Amongst parents, however, there tended to be an increase of reporting negative in-hospital experiences and positive post-discharge experiences over the test-retest timeframe.

Fig. 1

Direction of change of responses between test and retest phases

Discussion

Variable reliability of components of the QTTAC-PREM was observed when applied to hospitalized injured adolescent patients and their parents. These observations can be attributed to situational and survey administration factors.

Many patients only stayed 1 or 2 days in hospital making the need for education supports inconsequential. In addition, 13 of the 17 months over which this study was run was during the SARS-CoV-2 (COVID-19) pandemic, when on-line schooling prevailed for many schools in the trauma centers’ catchment areas. These two situations likely resulted in questions related to school support needs less relevant to all respondents; superficial reflection on these could thereby make them prone to poor test-retest results. Alternatively, needs for outside supports for post-discharge issues such a homecare, and dealing with frustration and school performance issues, may have been compounded by limited access to supports because of the pandemic, confounding responses between the test and retest phases. Similarly, poor reliability on questions related to social network supports may have affected questions related to having friends visit. During the pandemic, visitors were extremely limited in acute care settings, likely rendering these questions inconsequential and subject to inference rather than true experience.

Asking questions pertaining to in-hospital care, eight weeks post-discharge, may also have decreased their reliability due to respondent recall, Reliability of acute care questions in the original (unmodified) QTAC-PREM was relatively strong, with reliability coefficients ranging between 0.44 and 0.88 [12]. Unlike our study, these were tested at the end of patients’ hospital stays, albeit with only a 24-h lag between the test and re-test processes [11]. This original PREM instrument primarily focused on only the two constructs of clinical and ancillary care and information and communication. Although not as strong as the original PREM instrument, these two constructs still remailed mostly reliable in our study, despite the 8-week lag post-discharge, suggesting that the 8-week lag post-discharge may only partially explain any recall bias.

Amongst parent responses, a stronger directionality, namely in-hospital experiences being reported as more negative and post-discharge experiences as more positive suggest that overall perceptions of these two phases may have changed somewhat overtime, with a resultant overall influence on reliability in the retesting phase of parents. On the contrary, the lack of clear overall directional change of responses over time with the adolescent group may suggest more randomness of their retest responses. Similarly, poor reliability of questions pertaining to post-discharge supports and information/communication may be related to timing of survey administration if some patient completed the first survey prior to follow-up with their health care practitioner and the re-test after follow-up.

Recommendations and future directions

Given the limitations outlined above and, notably, the situational factors that may have influenced the test-retest process (COVID-19 pandemic), we would still recommend use of the QTTAC-PREM for examining adolescent and parent experiences during in-hospital and post-discharge care of adolescents hospitalized for traumatic injuries. Retesting of reliability of sections pertaining to maintenance of social networks and schooling reliability and supports needed post-discharge, when there is no concurrent pandemic, may be of value. We also recommend it be delivered in two phases temporal to the in-hospital and post-discharge phases of care, to avoid recall bias.

Limitations

The COVID-19 pandemic likely influenced the interpretation of some of the questions related to supports for schooling and maintenance of social networks, suggesting need for repeat reliability testing outside the pandemic.

The potential for recall bias must also be acknowledged as the survey inquired about in-hospital care but was administered at eight weeks post-discharge. This 8-week period was chosen with an assumption that most patients would have had their follow-up care within this period. However, the effects of the pandemic on scheduling follow-up appointments may have made this assumption invalid and we would suggest future querying of post-discharge experiences include a screening item to exclude respondents who have not yet had follow-up care. Finally, administering survey questions relating to in-hospital versus post-discharge care at time frames more temporal to these phases of care should be considered.

Our reliability study may also have been limited by our sample size. We note the original study examining the reliability of the QTAC-PREM SF included 117 retest cases, with a calculated sample size requirement of 75 [12, 13]. We were only able to recruit 40 patients and 40 parents and although there were similarities in reliability results between the two respondent groups, variations in directionality suggest that they may not be comparable.