The Observability Gap
Enterprise AI adoption is outpacing the visibility, governance, and monitoring required to make it safe. This is the single most important framework a mid-market security leader can hold today — it explains why the same teams that have spent a decade maturing their cybersecurity programs are watching their boards pour budget into AI initiatives that have effectively zero security telemetry.
The gap is not theoretical. By the time most mid-market organizations recognize they have an AI security problem, three things have already happened:
Employees are using public LLMs (ChatGPT, Claude, Gemini) for work tasks — often pasting sensitive data into prompts as part of normal workflows.
Vendors and SaaS platforms have shipped LLM features that touch the organization’s data without separate security review.
The organization has launched at least one internal AI initiative — chatbot, copilot, agent, RAG application — without a documented threat model.
What the security team can see: little to none of the above. What the security team is asked: “Are we safe?”
This guide is the resource we wish existed when CISOs and CIOs first started asking us that question. It covers the full taxonomy of AI security risks, the regulatory landscape, the practical detection and governance patterns, and how the work fits with the security program you already have.
What This Guide Covers
The AI Threat Taxonomy
We organize AI security risks into eight categories. The categorization matters because it dictates which tools, which controls, and which team owns each one.
1. Input-side attacks
Prompt injection — adversarial inputs that manipulate model behavior, the most common AI-specific attack.
Indirect injection and jailbreaking round out this category.
2. Output-side attacks
Inference exfiltration — responses leaking sensitive data the model had access to.
Hallucination-induced harm — confident but wrong outputs trusted downstream.
3. Model-side attacks
Model inversion — reconstructing training data from outputs.
Membership inference and model theft via API queries.
4. Training-side attacks
Data poisoning — corrupting training data to install backdoors.
Supply chain compromise of the model dependency tree.
5. Operational risks
Shadow AI — unauthorized tools touching corporate data.
Vendor AI compromise and compute cost runaway.
6. Compliance risks
NIST AI RMF gaps — missing governance documentation.
EU AI Act and sector rules: HIPAA, GLBA, GDPR + AI.
7. Adversarial AI
AI used against you: deepfakes, voice clones, and generated phishing.
LLM-augmented social engineering and automated recon.
8. Governance gaps
Missing AI use policy; no documented decision rights.
Absent or untested AI incident response procedures.
The first seven categories are technical. The eighth is organizational — and it’s the one mid-market firms underweight most.
Shadow AI: The Most Common Starting Point
Most organizations start their AI security program by trying to answer: “What AI tools are our people actually using?” The answer is almost always more than you think.
A representative mid-market organization (250–1,000 employees) typically looks like this:
Detecting shadow AI requires three telemetry sources working together:
1. Network egress monitoring — DNS and TLS visibility into AI provider domains (openai.com, anthropic.com, claude.ai, gemini.google.com)
2. DLP / CASB — content inspection of outbound API calls and web traffic
3. Endpoint detection — process-level visibility into AI client applications
Once detected, the operational question becomes: block, govern, or allow with monitoring? Most mid-market firms land on a tiered model — public LLMs blocked except via approved enterprise gateway; sanctioned tools (Microsoft Copilot, enterprise ChatGPT) governed with logging; internal applications monitored at the model layer.Read the deep dive on Shadow AI Detection →
The Seven AI Attack Patterns Every Security Team Should Know
These are not theoretical. Each has been demonstrated against production systems within the last 24 months.
Prompt injection
Adversarial input embedded in a prompt or retrieved content that overrides the model’s intended behavior. Example: an email forwarded to a summarization assistant contains hidden text instructing it to forward all subsequent emails to an attacker.
Detection: input/output anomaly detection, prompt-pattern matching, response-action gating.Full coverage →
Indirect prompt injection
Same mechanism as direct injection, but the malicious payload arrives via a third-party source — a document, web page, email. Example: a support agent ingests an attacker-controlled ticket whose body instructs it to escalate the attacker’s permissions.
Detection: content sandboxing, retrieval-source allowlisting, agent-action review gates.
Model inversion
Querying the model in ways that reconstruct training data — most dangerous when fine-tuned on sensitive data. Example: a healthcare-tuned LLM reproducing patient note fragments verbatim under specific prompts.
Detection: training data audit, output filtering, differential privacy in fine-tuning.
Data poisoning
Inserting corrupted data into the training set to install backdoors or degrade model performance. Example: an attacker contributes to a public dataset later used in fine-tuning, embedding triggers for misbehavior on specific inputs.
Detection: training data provenance, anomaly detection in training samples, post-training behavioral testing.
Training-data extraction
A specialized form of model inversion that aims to reproduce specific training records verbatim.
Detection: memorization audits, output filtering for known sensitive strings, query-rate limiting.
Supply chain compromise
Attacks on the model’s dependency tree — base models, datasets, fine-tuning data, plugins, retrieval indexes. Example: a compromised Hugging Face repository delivers a malicious base model used in production.
Detection: signed model artifacts, hash verification, dependency provenance.
Inference exfiltration
Model outputs leak information the model has access to but should not disclose — system prompts, retrieved private documents, other users’ conversation history in multi-tenant deployments. Example: a customer-facing chatbot reveals its system prompt or another customer’s data when prompted with a specific construction.
Detection: output filtering, prompt-secret separation, multi-tenancy isolation testing.
Regulatory Landscape
Three frameworks define the current AI compliance perimeter for most US mid-market organizations.
NIST AI RMF 1.0
Voluntary in most contexts, but rapidly becoming the de facto US standard. Four functions: Govern, Map, Measure, Manage.
The natural anchoring framework for mid-market firms — complements NIST CSF 2.0 and provides the structure auditors increasingly expect.
EU AI Act
Applies regardless of where headquartered. Tiers systems by risk: Unacceptable (banned), High-risk (strict requirements), Limited, Minimal.
Phased enforcement runs through 2027. If you sell to or operate in the EU, High-risk obligations are the compliance work to scope.
Sector frameworks
Healthcare: HIPAA + AI guidance from HHS.
Financial services: OCC/Fed model risk guidance (SR 11-7), extended to GenAI.
Defense: CMMC 2.0 implications for AI use with CUI.
Building an AI Security Program: The Four-Phase Maturity Model
We use this four-phase model when standing up AI security programs for mid-market clients. Each phase has specific deliverables and a graduation gate. Most firms start in Phase 1 and reach Phase 3 within 6 months under a focused engagement.
Discovery
• Shadow AI inventory
• Data flow mapping
• Use case inventory
• Initial risk register
Gate: inventory + risk register reviewed by execs
Governance
• AI use policy
• Decision rights matrix
• Vendor AI assessment
• AI incident response plan + training
Gate: policies signed, training delivered, IR plan tested
Controls
• Network controls (egress, DLP, CASB)
• Identity controls (SSO + MFA)
• Application controls
• Audit controls
Gate: controls deployed, logging in SIEM, tabletop tested
Observability
• Continuous monitoring + threat detection
• Periodic red team exercises
• Quarterly compliance attestations
• Roadmap iteration
Gate: this phase is steady state, not an end
Where AI Security Fits Your Existing CISO Program
AI security is not a separate program. It is an extension of the security program you already have, with five specific additions:
Shadow AI detection added to your existing shadow IT discovery process
AI use policy added to your existing acceptable use policies
AI vendor assessment added to your third-party risk management workflow
AI-aware incident response added to your existing IR runbooks
NIST AI RMF maturity added to your existing NIST CSF 2.0 reporting
If your security team is telling you AI security requires hiring entirely new specialists, slow down — they may be selling a more complex picture than reality. The fundamentals are extensions, not replacements. The specialization is in the controls layer (Phase 3) and the observability cadence (Phase 4) — and even there, mid-market firms can usually run with existing CISO + SOC capacity supplemented by an external advisory partnership.
Frequently Asked Questions
How do I know if my organization has an AI security problem?
Three diagnostic questions: (1) Can you list every AI tool in use across your business? (2) Do you have a documented AI use policy communicated to employees? (3) Has your incident response plan been updated for AI-specific threats? If the answer to any is no, you have a program gap. If all three are no, you have a Phase 1 starting point.
We don’t use ChatGPT or build AI internally. Are we exposed?
Almost certainly yes — via vendor AI features. SaaS platforms across HR, finance, sales, support, marketing, and operations have shipped LLM features that touch your data. Microsoft Copilot, Salesforce Einstein, Notion AI, Slack AI, and Zoom AI Companion are now ambient in mid-market technology stacks.
We’re not in healthcare or finance. Do we still need NIST AI RMF?
NIST AI RMF is voluntary in most contexts. We recommend it because it integrates cleanly with NIST CSF 2.0, regulators and auditors increasingly use it as a benchmark, and it gives you structured language for board-level AI risk reporting. Adoption typically takes 4–12 weeks of advisory effort.
How does AI security relate to traditional cybersecurity?
Most AI security problems are still cybersecurity problems with new attack surfaces. Prompt injection is an injection attack. Inference exfiltration is a data exfiltration attack. Shadow AI is shadow IT. The novelty is the attack surface (model behavior) and the detection telemetry — the fundamentals of defense in depth, least privilege, logging, and IR apply unchanged.
Should we govern AI use before deploying any AI?
In most mid-market contexts, governance and deployment proceed in parallel. Trying to gate all AI usage behind a finished governance program produces shadow AI faster than it produces governance. Better: discover what’s in use today, write rapid-iteration policy that legitimizes safe use cases, govern the high-risk ones, build out controls in Phase 3.
Is the EU AI Act relevant if we don’t sell to the EU?
Possibly. The Act applies if your organization is established in the EU, the output of your AI system is used in the EU, or you use an AI system that affects EU residents. Many US mid-market firms are caught by that second criterion without realizing it.
What’s the smallest first step?
A single-day Shadow AI Discovery — what AI tools are touching corporate data right now. This produces an inventory, surfaces the highest-priority risks, and gives the executive team enough to commission Phase 1 if the inventory warrants it. Armorstack runs this as a fixed-scope engagement.
How Armorstack Can Help
Armorstack runs AI security programs for mid-market clients across healthcare, financial services, manufacturing, defense contracting, and education.
Shadow AI Discovery
Single-day fixed-scope engagement
AI Risk Assessment
Phase 1 program scoping (4 weeks)
vCISO + AI Security
Embedded executive partnership
NIST AI RMF Implementation
Phase 2 + 3 work (12–24 weeks)
AI Security Observability
SENTRY portfolio, continuous monitoring