Connect
Imagine three everyday scenarios-A clinician uses an AI scribe to document a patient visit. A nurse asks a public AI assistant to simplify discharge instructions and an administrator uploads a spreadsheet to identify scheduling patterns. Using AI in these scenarios may save time.Each one can also send patient information to a tool that nobody in IT or compliance has reviewed.
AI is already part of clinical and administrative work. To protect patient privacy, healthcare organizations need to know which AI tools are in use, what data those tools can reach, and who or what is acting on that data.
Where AI Puts Patient Information At Risk
Patient information can enter an AI tool through a prompt, an uploaded file, an image, or a connected app. A one-line prompt can hold a patient’s name, diagnosis, and medications. An ambient scribe, an AI tool that listens to a visit and drafts the clinical note, can capture the whole conversation.
These actions feel lighter than working in an electronic health record. The privacy questions are the same. Where does the data go? How long is it kept? Who can reach it?
The biggest gap is shadow AI: AI apps, assistants, and agents that staff use without IT’s knowledge or approval. It usually starts with a good reason. A clinician wants to spend less time on notes. A care coordinator needs a quick translation. But if IT doesn’t know a tool exists, no one can check how it signs users in, what it connects to, or what the vendor does with the data.
Shadow AI shows up in browsers, browser extensions, meeting apps, and developer tools. It can also arrive through Model Context Protocol (MCP) connections, which let AI tools plug straight into other apps and data sources. Learn why IT teams need to discover, govern, and secure shadow AI
AI also creates new access paths. An employee can move data from an approved system into a public tool within seconds. A transcription assistant can join a meeting through a calendar integration. An AI system may connect to an application and take action on a user’s behalf. Each step can sit outside the controls built around traditional healthcare systems.
What Are the Main AI Patient Privacy Risks?
AI privacy failures rarely come from one weakness. They emerge when gaps in visibility, consent, access, device management, and vendor oversight overlap. The risks vary by workflow, but several patterns appear across healthcare environments:
Unapproved Tools and Unclear Data Practices
Employees may enter protected health information into services that have not been approved for that use. The visible application may also depend on model providers, cloud infrastructure, analytics services, and other subprocessors. Organizations need to know which parties receive patient information, how it is used, and how long it is retained.
Inadequate Patient Notice or Consent
AI scribes can capture highly sensitive conversations. Healthcare organizations must determine which notice and consent requirements apply in each jurisdiction and provide an alternative when a patient declines. Patients should understand when AI is being used, what it captures, and how the resulting information is handled.
Excessive Access and Weak Accountability
An AI system may receive broader access than its task requires, especially when teams reuse employee accounts or grant broadly scoped tokens. Access should match the purpose and duration of the work. Organizations should also be able to distinguish between activity performed by a person and activity performed by an automated system.
Unmanaged Devices and Incomplete Offboarding
Personal devices can store meeting invitations, transcripts, credentials, and AI configurations. Access may persist when an employee changes roles or leaves. Offboarding should cover calendars, recurring meetings, shared resources, connected services, tokens, and devices as well as core applications.
Incomplete Deletion and Investigation Records
Deleting a local file may not remove copies held by a provider or subprocessor. Organizations need a process for requesting deletion and confirming the outcome. They also need records that connect relevant activity to the users, devices, applications, and automated systems involved.
Re-Identification of De-Identified Data
Removing names and other direct identifiers does not always eliminate privacy risk. AI models may infer sensitive attributes or help link remaining clinical details to an individual when other information is available. A 2026 study using clinical notes from one health system found that language models could predict attributes and re-identify patients from de-identified notes at rates above its comparison baseline. The authors also note important limits to how broadly the findings can be applied. The research examines how information retained in clinical narratives can contribute to re-identification risk.
Healthcare organizations should not assume that de-identified data is risk-free. Before using it for AI training, testing, or analysis, teams should assess the possibility of re-identification, the availability of other data that could be linked to it, and whether the proposed use justifies the remaining risk.
Which Compliance Frameworks Apply to Healthcare AI?
Healthcare AI does not fall under one universal regulation. Requirements depend on the organization, location, data, system, and intended use.
HIPAA and the Proposed Security Rule Update
HIPAA does not prohibit AI or certify individual products as universally compliant. The way a covered entity or business associate implements a technology determines how HIPAA applies.
The HIPAA Security Rule requires regulated organizations to use appropriate administrative, physical, and technical safeguards to protect the confidentiality, integrity, and availability of electronic protected health information. HHS provides an overview of the current Security Rule.
An AI vendor handling ePHI on behalf of a regulated organization may require an appropriate business associate agreement. That agreement does not replace access controls, device security, workforce training, or risk management.
HHS describes risk analysis as a foundational part of Security Rule compliance and advises organizations to revisit risk when introducing new technology or changing business operations. AI workflows that create, receive, maintain, or transmit ePHI should therefore be considered within the organization’s risk analysis. HHS explains the role of risk analysis in Security Rule compliance.
HHS also issued a proposed update to the HIPAA Security Rule in December 2024. The proposal seeks to strengthen cybersecurity protections and provide more specific direction for regulated entities and business associates. It is not a final rule, and the current Security Rule remains in effect. HHS maintains the official Security Rule NPRM page.
The NIST AI Risk Management Framework
The voluntary NIST AI Risk Management Framework organizes AI risk management around four functions known as Govern, Map, Measure, and Manage. It helps organizations establish accountability, understand how AI operates, assess risks, and address them over time. The NIST AI RMF Playbook provides suggested actions for each function.
Healthcare organizations can use the framework to connect privacy, security, clinical, operational, and business concerns. It supports a repeatable governance process but does not replace HIPAA or other legal requirements.
The EU AI Act and Other Requirements
Healthcare organizations operating in or serving people in the European Union may also need to consider the EU AI Act. Certain systems associated with health, safety, fundamental rights, or regulated products can be classified as high risk. Depending on the system and the organization’s role, obligations may cover risk management, documentation, logging, human oversight, cybersecurity, and accuracy.
Not every healthcare AI tool is automatically high risk. Each use case must be assessed according to its actual purpose and impact. The implementation timeline has also changed, so organizations should use current official guidance. The European Commission maintains an updated overview of the AI Act.
State health privacy laws, recording and consent laws, consumer protection requirements, and medical device regulations may also apply. An administrative chatbot, ambient scribe, and AI-enabled medical device present different legal and operational questions.
Also read JumpCloud’s analysis of what the EU AI Act means for AI agents explores the operational questions around human oversight, ownership, access, monitoring, and audit records.
How to Protect Patient Privacy in AI Workflows
No single product or policy can address every risk but A practical program brings these six steps together:
- Find the AI already in use. Keep an inventory of AI apps, assistants, agents, and connections. Note who owns each one, what data it touches, and what it connects to. Update it as use changes.
- Approve tools and make requests easy. Tell staff which tools they can use and what data each one may handle. Use real clinical examples. Give people a fast way to request new tools so they don’t work around the rules.
- Vet vendors and plan for consent. Ask how each vendor collects, stores, keeps, and deletes data, and whether it trains models on customer data. For ambient scribes, decide when consent is needed, how it’s recorded, and what happens when a patient says no.
- Control the data itself. Data protection tools can flag PHI in prompts and uploads, block risky submissions, or mask sensitive values. These controls govern what data passes through AI. Identity controls govern who or what can use it. Most organizations need both.
- Lock down identities, devices, and access. Require approved accounts and managed devices for healthcare work. Use strong authentication. Give people and AI agents only the access a task needs, for only as long as they need it. Make offboarding cover calendars, connected apps, tokens, and devices.
- Keep records and plan for incidents. Log activity so you can tie it to a user, device, app, or agent. Plan for PHI pasted into an unapproved tool or a visit recorded without consent. When a workflow ends, revoke its tokens and integrations.
How JumpCloud Helps Govern AI Access in Healthcare
JumpCloud covers the identity, device, and access layer of this work. It manages the people, devices, and AI agents that reach sensitive systems from one platform.
JumpCloud Agentic IAM extends that control to AI agents. It helps IT teams:
- Discover shadow AI across browsers and managed devices
- Give each agent its own identity, tied to a named human owner
- Block AI tools from reaching resources on unmanaged or untrusted devices
- Manage agent access through its full lifecycle, from onboarding to removal
JumpCloud works alongside your data protection, consent, vendor review, and compliance controls. Together, they help move healthcare AI from hidden activity to adoption you can see, govern, and audit.
See how JumpCloud Agentic IAM brings AI agents under the same identity controls as the rest of your workforce.
Frequently Asked Questions
What Are the Main Patient Privacy Risks of AI in Healthcare?
The main risks include PHI being entered into unapproved tools, patient conversations being recorded without appropriate notice or consent, excessive access to healthcare systems, unmanaged vendors, and incomplete activity records. Shadow AI makes these risks harder to address because privacy and security teams may not know which tools are processing patient information.
Can AI Be HIPAA Compliant?
AI can be used within a HIPAA-compliant environment, but no product makes an organization or workflow compliant by itself. Compliance depends on the data processed, the vendor relationship, required agreements, security safeguards, access controls, risk management, and how employees use the technology.
Can Healthcare Organizations Use AI With PHI?
Healthcare organizations may be able to use AI with PHI when the workflow satisfies applicable privacy and security requirements. Before introducing PHI, they should assess the vendor, data flow, retention, subprocessors, access controls, security measures, and contractual requirements.
How Can Healthcare Organizations Protect Patient Privacy When Using AI?
Organizations can reduce risk by discovering approved and unapproved AI use, defining which data may be used with each tool, reviewing vendors, protecting sensitive information, addressing consent, and securing the identities and devices involved. These measures should be supported by least-privilege access, useful records, employee training, human oversight, and an incident response process that covers AI tools.
