Part 2: Governing AI Agents in Healthcare: Least Privilege, Identity, and Human Oversight

In Part 1, I explained why security models built for people do not fit AI agents. This post covers what to put in place.
The New Principle: Least Privilege for AI
Security leaders have long embraced the principle of least privilege. AI agents should be no exception.
Healthcare organizations should resist the temptation to grant broad access simply to accelerate implementation. An agent that can do everything can eventually do the wrong thing everywhere.
Instead, organizations should focus on:
Purpose-Built Permissions
Every agent should have a clearly defined mission and only the data access required to complete that mission.
Segmented Trust Boundaries
Agent access should be isolated by workflow, department, and data sensitivity.
Continuous Monitoring
Agent activity should be logged, reviewed, and analyzed with the same rigor as privileged human users.
Human Oversight
Critical clinical, financial, and compliance-related decisions should maintain human review points until governance maturity is proven.
The reality is that we do not yet have decades of operational experience managing autonomous agents in healthcare environments. Therefore, proceeding with caution is recommended.
The Hidden Risk: Identity Sprawl
One of the security topics receiving less attention than it deserves is AI identity management. Organizations already struggle with human identities, service accounts, contractor accounts, and privileged access management.
Now imagine managing hundreds or thousands of AI agents. Each agent may require:
- Authentication
- Authorization
- Credential management
- API access
- Activity logging
- Lifecycle management
Without strong controls, healthcare organizations risk creating a new category of exposure and technical debt: unmanaged agent identities.
In many ways, AI security may become as much an identity challenge as a cybersecurity challenge.
Why Governance Must Move Faster Than Adoption
My own journey with AI started well before enterprise adoption became mainstream. Like many technology leaders, I began independently studying emerging AI capabilities, experimenting with publicly available tools, and evaluating how these technologies could reshape business operations. Initially, the focus was on content generation, productivity gains, and efficiency improvements. Over time, however, it became clear that the most important conversations were not about capability. They were about control, accountability, and trust.
The more time I have spent evaluating AI, the more convinced I have become that governance cannot be an afterthought.
The organizations that succeed with agentic AI will not necessarily be those that deploy the most agents.
They will be the organizations that deploy them responsibly. That means establishing:
- AI governance boards
- Security review processes
- Risk classification frameworks
- Agent approval workflows
- Monitoring and audit standards
- Incident response procedures specifically for AI systems
Healthcare has always balanced innovation with patient safety. AI should be treated no differently.
Trust Will Define the Next Phase of AI Adoption
The next generation of healthcare innovation will not be defined solely by larger models or more sophisticated agents. It will be defined by how effectively organizations balance autonomy with accountability.
Platforms such as Epic’s Agent Factory demonstrate the tremendous potential of AI agents to improve clinical, operational, and patient-facing workflows. At the same time, industry approaches such as Kyndryl’s emphasis on policy-driven orchestration, governance, and secure deployment highlight an equally important reality: AI systems must be trusted before they can be scaled.
Patients must trust that their information remains protected. Clinicians must trust that recommendations are explainable. Leaders must trust that their governance structures can effectively manage autonomous systems operating across critical business and clinical processes.
Long-term success will not be measured solely by how many workflows are automated. It will be measured by trust.
As healthcare leaders, our responsibility is not simply to deploy AI faster. Our responsibility is to ensure that AI operates securely, transparently, and in alignment with the mission of improving patient outcomes.
That is how we unlock the promise of agentic AI while protecting the people, data, and outcomes that matter most.
Let's Connect
Talk with Healthcare IT Leaders about governing AI agents in your health system. We can help you set up access, oversight and support.
Healthcare only works with Humans in the Loop.