Table 2 Twelve interrelated studies of accreditation--overview of approaches and methods

1. Accreditation models

What are the relative strengths and consequences of different accreditation models?

Undertake a multi-method evaluation of three accreditation models: those of the ACHS, AGPAL, and ACSAA

▪ Interview key stakeholders in three accreditation agencies (n = 18)

▪ Conduct a web-based questionnaire survey of acute health services, general practices and aged-care facilities (~n = 300)

2. Critical elements of accreditation

What are the critical elements of the accreditation process that stimulate improvement? What drives behaviour change in provider organizations and clinicians?

Assess each element (e.g., self-assessment, clinical indicators, patient data, surveyor visits and accreditation reports) and describe its role in promoting improvement

▪ Run focus groups of stakeholders drawn from accreditation agencies (n = 6 focus groups) and jurisdictional health departments (n = 8 focus groups), and 15 randomized focus groups from accredited general practices (n = 5), acute-care health-care organizations (n = 5), and aged-care providers (n = 5)

▪ Conduct a ranking exercise by surveying large samples of staff from accredited organizations across acute, general practice, and aged-care settings (~n = 600) to ascertain the relative importance of the accreditation elements

▪ From this sample, gather perspectives on and examples of how the respective accreditation elements drive change

3. Standards and their impact

How are standards developed and used? How do standards incorporate evidence, and influence the expertise of clinicians, managers, and policy makers? How does the application of standards promote change in organizational performance and clinical practice?

Examine the development of standards and their application using widespread observational activities and surveys across different accreditation programmes, selecting and investigating a sample of standards during their development phase to determine the sources of the standards (e.g., public inquiries, adverse events, international guidelines), how they should be developed, and how they should be applied

▪ Undertake ethnographic observations of the development of standards, assessing their use of evidence and the engagement of stakeholder groups

▪ Conduct a survey of accredited organizations, investigating how standards are applied and how they promote change (~n = 600 respondents). From these data, identify for detailed analysis case study sites in which standards have promoted measurable change

▪ Conduct case studies (n = 5) of specified key standards (evaluation of care, documented policies, the quality improvement system, health records, infection control). Use these case studies to identify factors related to organizational change. Obtain organizational data to quantify the extent of this change

4. Key new standard for consumer participation

Can we develop and trial a standard for consumer participation?

Use the Delphi method to create and field test a standard for consumer participation in acute settings, general practice, and aged care

▪ Systematically review instances of consumer participation cited in the accreditation literature

▪ Assess the review's evidence as the basis for the standard

▪ Consult with stakeholders, using the Delphi approach, to secure agreement on the standard

▪ Apply the standard in the field (n = 30)

▪ Evaluate its use and efficacy with survey and qualitative methods

5. The patient experience

How do patient experiences vary across a range of settings with differing accreditation results?

Compare the ethnographic mapping results for a range of patient experiences in different accreditation settings against positive and negative accreditation results

▪ Extend the research into patient journeys to a larger trial that includes all three accreditation domains

▪ Review 60 randomly selected complex patient journeys in depth

▪ Compare these against the accreditation results from each of the participating organizations in which patient journeys were taken.

6. Cost-benefit analysis of accreditation

What are the benefits and costs of accreditation and the different accreditation models?

Design and apply a model for a health-economics cost-benefit analysis (CBA) of accreditation, including an examination of the factors that affect (e.g., drive or inhibit) costs and benefits. Assess the benefits and costs over time, modelled on Brent [42]

▪ Conduct a detailed analysis of the cost effectiveness of accreditation in the three domains, examining the extent to which the benefits outweigh the costs

▪ Express the benefits and costs in nett present value terms, and adjust for the time value of money

▪ Seek cost estimates from the partner agencies, and benefit estimates from a randomized sample (n = 30) of their member organizations

▪ Execute CBA modelling on this basis

7. A natural experiment examining what the ACSQHC does to transform accreditation

What changes have ensued from the initiatives of the ACSQHC?

Conduct a formative evaluation of the impact of ACSQHC's transformation of accreditation, particularly the impact of the comprehensive set of National Safety and Quality Healthcare Standards applied to high-risk services

▪ Evaluate this progress using formative evaluation techniques, in a partnership arrangement with ACSQHC

▪ Review the gains made in establishing a national co-ordinating body to implement accreditation reforms, including standards development, piloting, implementation, expanding accreditation into high-risk services, and co-ordination with regulating bodies

8. Public disclosure of accreditation results

Is it possible to develop and test an effective model for the public disclosure of accreditation results?

Examine extant methods of public disclosure of information in international contexts, and their relative impacts

▪ Identify in a literature review the different models of public disclosure (e.g., types, formats, and approaches) and compare web-based reports, newsletters to health-care organizations, local newspaper reports, and community meetings

▪ Undertake focus groups with members of the public to investigate views and strategies (n = 10 focus groups)

▪ Conduct trials, with each accreditation domain, of the three most-relevant disclosure models (n = 30 enrollees)

9. Patient journey methodology

What is the effect of the application of the patient journey methodology?

Map the use of the patient journey methodology under various circumstances

▪ Evaluate the utilization of the patient journey technique using ethnographic observations of four accreditation survey teams in each of the three accreditation domains (n = 12 survey teams)

▪ Apply this knowledge to these domains, extending earlier tests of the patient journey method [30]

▪ Triangulate the results and compare and contrast the patient journey method against survey outcomes using standards

10. Short-notice surveys

What is the effect of the application of short-notice surveys?

Examine the use of short-notice surveys under various circumstances, including variables such as points in the accreditation cycle and service type

▪ Evaluate ACSAA's experience of short-notice surveys using key informant interviews with ACSAA staff (n = 10) and randomly selected aged-care facilities (n = 15)

▪ Apply this knowledge to trials with ACHS and AGPAL, extending earlier tests of short-notice surveys [29]

▪ Evaluate the ACHS and AGPAL trials and triangulate the data with previous results

11. Surveyors and their place in accreditation

What are the roles, effectiveness, and reliability of surveyors?

Conduct an ethnographic analysis of surveyors and surveying processes, with a comparative analysis of the roles, effectiveness and reliability of surveyors in the three accreditation domains

▪ Analyse existing accreditation databases to assess the relationships between the judgements and survey outcomes of accreditation teams, to quantify the variation between the teams and surveyors

▪ Undertake experiments, using test scenarios, with the methods developed in earlier research to assess the surveyors' reactions to and consistency in differing accreditation situations [31]

12. Differentiate effective and ineffective uses of accreditation processes and methods to promote change

How do effective and less-effective organizations use accreditation levers to improve performance?

Undertake a comparative, randomized, stratified examination of effective and less-effective organizations and the ways they use accreditation to promote performance improvement, drawing upon the results of study 3.

▪ Examine randomly selected organizations, 20 in each accreditation domain (n = 60 organizations)

▪ Separate these organizations into a split sample of effective and less-effective organizations, judged by stakeholders' attributions and external organizational performance criteria

▪ Apply detailed case study methods to derive both quantitative and qualitative indicators; assess how the two samples use accreditation to improve performance and promote change