An example of governance for AI in health services from Aotearoa New Zealand

Framework development

The governance framework was developed through background research on international guidance and best practices, patient surveys, analysis of internal data and software systems, integration with Māori data sovereignty principles²¹, and establishment of a representative governance group. The framework was further tested and refined by reviewing AI proposals.

Background research was conducted by the Te Whatu Ora Waitematā Institute for Innovation and Improvement (i3). This involved a review of the current state of AI implementation in clinical services internationally and recommendations around partnerships and contractual arrangements. National and international documents that were relevant in informing our processes are shown in Table 1.

We conducted a cross-sectional survey and in-depth interviews to understand the perspectives of our healthcare service users regarding the secondary use of their personal health information for purposes such as AI. Inpatients and outpatients (n = 1377) were surveyed about what they expected the organisation was already doing with their health information and what their level of comfort was with secondary use including aggregation of data for service improvement and for the benefit of others²⁰. The vast majority were comfortable with the aggregation of their data with others for the purposes of improving health care services for the future, with the conditions that it would produce benefit for others, their privacy would be maintained, data would still be secure, and appropriate governance or approvals were in place. The survey showed that generally people were comfortable with contributing the use of their health information for the greater good of the population, although better communication about this was requested. This includes transparency on projects undertaken and clarity around governance, confidentiality, and data security processes within our healthcare District. There were a small proportion of people uncomfortable with the use of their health information which was commonly linked to negative experiences with the health service.

The second phase of this research involved scenario based in-depth interviews (n = 12) including a scenario around the secondary use of data for AI development²⁷. Participants reported conditional support for their health information being used for this purpose as it was advancing science and for the greater good. Participants reported that they needed to be able to trust their health service to respect these conditions. Conditions included adequate security and protection of the data, that the data was adequately de-identified and their privacy protected, that there was no potential for secondary harms, that there was good governance and clinical oversight, and lastly that the health information remained in the health system and was not shared with outside organisations or commercial companies. Where there was potential for commercial gains from the development of the AI, comfort levels decreased; participants described that in this case the intent might no longer be for public benefit.

A software engineering review was then conducted on the local database and software management systems. This review pointed out points of potential risk in terms of software development and maintenance, which are summarised in Table 2.

A software framework was developed internally based on a previous exemplar framework to conceptualise big data through the lens of tikanga and matauranga Māori (Māori ethical principles and philosophies)²⁸ and adapted for the AI context (Fig. 2). This framework provides a methodological process for evaluating AI, highlights potential areas of negative implications of AI for Māori, and creates the expectation that AI will be developed consistently with the key tenets of tikanga (ethical principles). In particular, it emphasises the obligations for layers of engagement with Māori necessary for a Te Tiriti honouring process, from concept to development, implementation and monitoring. This accountability to Māori has been expressed in the use of the takarangi (double helix spiral), building on the work of Te Ara Tika¹⁸ and He Matapihi ki te Mana Raraunga²⁸, where one follows around the circumference asking the necessary questions to “make the road by walking”²⁹. The notion of a spiral signifies that the questions are continually asked and reflected upon throughout the lifetime of the AI. For example, the second time the questions are asked will reveal a deeper understanding and further develop the capabilities of the stakeholders.

The next step was establishing a new AI Governance Group (AIGG) for Te Whatu Ora Waitematā. The Terms of Reference state that the purpose of the AIGG is to provide oversight and expert advice about the appropriateness, safety, effectiveness, ethics and ongoing improvement of any AI research, development, projects, partnerships, contracts, or implementation at Waitematā. It was considered vital that the following areas of representation be included: consumers, clinical governance, data and digital governance, privacy and security, legal, Māori health, Pacific Island health, research, analytics, innovation and improvement (supporting implementation), and external expertise in AI and machine learning.

The final step was adapting all of the above into a checklist for use by the AIGG when considering new proposals for access to data or clinicians at any stage of the AI development workflow from development to implementation (Fig. 3).

Initial testing of the framework

Following development of the framework, testing was undertaken by the AIGG reviewing proposals for AI tools intended to be used within the health service. These included a tool for identifying the potential early signs of diabetic retinopathy on retinal screening images, and the early development of a COVID risk of hospitalisation score. Through initial testing, the AIGG identified particular issues to be addressed which allowed us to further refine the checklist. Some examples of considerations added after the first iteration include conflicts of interest (of clinicians who are also developers/ entrepreneurs), ongoing monitoring and accountability, responsiveness to potential future changes in ownership and accountability, and the ability to share benefits with the public health system on behalf of the Waitematā population.

Māori terms

A number of Māori language terms are included in this description of methods. These are explained here:

• Tino rangatiratanga—the sovereign right for Māori to be in charge of their own resources and aspirations, acting with authority and independence over their own affairs.

• Tikanga—Māori customary practices and behaviours.

• Taonga—an object or resource which is viewed by Māori as a treasured possession.

• Kaitiakitanga—describes guardianship and protection based upon the Māori world view.

Reporting summary

Further information on research design is available in the Nature Research Reporting Summary linked to this article.