Shadow AI in Healthcare

Healthcare AI Security

Finding the AI in Your Hospital Nobody Told Security About

Shadow AI in mid-market hospitals is more pervasive than most security teams estimate. SaaS vendors embed AI features into tools clinicians already use, departments adopt AI-augmented tooling without IT or compliance review, and clinical staff paste PHI into public LLM interfaces. Armorstack’s shadow-AI discovery work — Pillar 1 of the AI Adoption Security Framework — produces a complete, healthcare-specific inventory in 30 days.

The 50-Word Answer

Shadow AI in hospitals shows up in four places: SaaS-embedded AI features, departmental tooling adopted outside IT review, clinical staff pasting PHI into public LLMs, and third-party clinical AI vendors integrated through Epic or Oracle Health. Discovery combines API, network, and endpoint telemetry with a clinically-calibrated staff survey — and typically finds two to four times more AI in use than IT estimated.

The Four Places It Hides

Where Shadow AI Hides in Mid-Market Healthcare

Shadow AI in hospitals appears in four distinct places, and each carries a different discovery challenge and a different HIPAA exposure profile.

1. SaaS-Embedded AI Features

AI features vendors quietly enable inside tools already licensed — Microsoft 365 Copilot, Google Workspace Gemini, Slack AI, Notion AI, Zoom AI Companion, Webex AI Assistant — often touching PHI and often invisible in standard SaaS admin consoles.

2. Departmental AI Tooling

Clinical departments adopting AI scribing tools, marketing adopting AI content generation, HR adopting AI resume screening, finance adopting AI invoice processing — each potentially touching PHI in workflows security never reviewed.

3. Public LLM Use by Clinical Staff

ChatGPT, Claude, Gemini, Perplexity, and similar tools on work or personal devices. Staff paste discharge summaries for summarization, documentation for editing help, and patient questions for AI-drafted responses.

4. Third-Party Clinical AI Vendors

AI vendors integrated into Epic or Oracle Health (Cerner) via FHIR APIs or EHR partner programs, each processing PHI under BAA terms that may not contemplate the vendor’s AI feature set.

Pillar 1: Discovery

What Healthcare-Specific Shadow-AI Discovery Looks Like

Pillar 1 discovery work, applied to mid-market hospitals, combines four signal sources into a single inventory.

1

API-Based Discovery

Against the major SaaS administrative consoles the organization uses — Microsoft 365, Google Workspace, Slack, Zoom, and the specific clinical SaaS in the environment.

2

Network Telemetry Analysis

Against known AI service domains, identifying traffic to public LLM endpoints and AI-vendor APIs from organizational devices.

3

Endpoint Telemetry

Browser extension and application usage data that identifies AI tools in active use across clinical and administrative devices.

4

A Structured Staff Survey

Calibrated to clinical operations — not a generic “do you use AI” question, but a workflow-specific battery that surfaces use cases staff don’t consciously categorize as “AI.”

The output is an inventory classified by department, by data type (PHI / non-PHI), by vendor authorization status, and by clinical workflow involvement. Mid-market hospital discovery exercises typically find an inventory two to four times larger than what the IT team estimated before the exercise began.

FAQ

Frequently Asked Questions — Shadow AI in Healthcare

How does shadow AI become a HIPAA issue?
Three primary paths: (1) PHI is disclosed to an unauthorized AI service that is not under a Business Associate Agreement, becoming an impermissible disclosure under the Privacy Rule and potentially a reportable breach; (2) PHI processed by an AI vendor is used to train the vendor’s models without explicit authorization, raising §164.502 use questions; (3) AI-generated content based on PHI is shared inappropriately, replicating disclosure paths the security team’s existing DLP rules don’t recognize because they were built for human action patterns, not AI output patterns.
Can we just ban AI use by clinical staff?
Bans rarely work in practice and often create worse outcomes than governance. Clinical staff under workflow pressure will find ways to use AI to accelerate documentation and decision-making; if approved tools are banned, staff use unapproved tools on personal devices, removing all visibility. The more durable approach is governance — an Acceptable Use Policy that defines what AI use is permitted, with what data, on what devices — combined with visibility into actual use.
Will the discovery work identify our clinical AI vendors?
Yes. Pillar 1 discovery is explicitly scoped to enumerate vendor-supplied AI in addition to staff-use AI. The discovery work surfaces vendor AI features that may have been enabled without explicit organizational authorization and produces the inventory of clinical AI vendor relationships the governance committee can review.
How does shadow-AI discovery feed into Joint Commission preparation?
Joint Commission information management standards require organizations to know how clinical information is being used and protected. The shadow-AI inventory becomes documentary evidence that the organization knows what AI is touching clinical information, classified appropriately, with governance documented. Pillar 4’s deliverables feed into Joint Commission survey preparation directly.
What happens with the inventory after the assessment?
The inventory is yours. Some organizations operate it as a living artifact maintained quarterly. Others integrate it into their existing IT asset inventory or vendor risk management program. Armorstack can operate the inventory as a managed service for organizations that prefer ongoing managed shadow-AI discovery, but the assessment itself produces the inventory as a deliverable regardless of whether the relationship continues.

See What AI Is Actually Inside Your Hospital

Apply for the free 30-day AI Risk Assessment. Pillar 1 discovery alone is worth the engagement — a complete, healthcare-specific shadow-AI inventory, classified by department, data type, and vendor authorization status.

Not in healthcare? Read the general Shadow AI Detection guide or the full Shadow AI Governance Framework.