An AI gateway is a single entry point between your applications and employees on one side and model providers on the other. Every request passes through it, so routing, budgets, logging, and policy live in one place. A well-built gateway also finds sensitive data in each prompt, masks it before the request leaves your network, and restores the real values in the reply.

What an AI gateway does

Without a gateway, each team tends to wire its own tools to model providers: an API key in a backend service, another in an IDE plugin, and a third in a browser chat someone expensed. The full picture of which data went where, and what it cost, lives nowhere. An AI gateway, also called an LLM gateway, replaces those direct connections with one API that apps and tools call instead of the provider’s.

Most gateways cover four jobs:

  • Routing. Each request goes to a cloud or local open-weight model based on policy, cost, or data type, with fallback when a provider is down. Apps stay the same when the model changes.
  • Budgets and rate limits. Limits per team and per user. The OWASP Top 10 for LLM Applications treats Unbounded Consumption as a separate risk, including “Denial of Wallet” attacks that exploit pay-per-use pricing, and recommends rate limiting and user quotas.
  • Logging. Who sent what, when, to which model, and at what cost.
  • Keys and access. Provider keys stay in the gateway. People sign in as themselves, so access is revoked per person.

Those controls govern who calls which model, not what’s inside the prompt, which is the part that leaves.

Why the prompt is the risk

People paste whatever makes the task faster: a customer email thread, a contract clause, or a stack trace with a connection string. OWASP’s entry on Sensitive Information Disclosure names the categories at stake: personal data, financial details, health records, confidential business data, security credentials, and legal documents. Once a prompt reaches a provider, its retention and handling follow the provider’s terms.

What counts as sensitive also depends on the business. NIST SP 800-122 gives context-based guidance for identifying PII rather than a fixed list, and the same logic applies to a company’s own identifiers: customer IDs, contract numbers, and deal code names. A gateway that only knows emails and card numbers misses the identifiers that matter most.

How a gateway detects sensitive data

PII redaction stands or falls on detection. Production setups usually layer three families of detectors.

Pattern rules with validation

Regular expressions find candidates, and validators throw out lookalikes. Card validators check the Luhn digit, a check-digit method patented by Hans P. Luhn in 1960: the open-source Presidio project’s credit card recognizer pairs a regex with a Luhn checksum. That lets a rule reject most order numbers of the same length that merely look like a card.

Where no checksum exists, context words help. Presidio’s list of supported entities detects US Social Security numbers by pattern plus context, and IBANs by pattern, context, and checksum. Rules are fast, deterministic, and explainable, and they run on a CPU. Their weak spot is anything without a fixed shape: names, addresses, and facts in free text.

NER models

Named entity recognition (NER) models label spans such as person, organization, and location from context, without a list of names. Presidio, for example, combines NER with regular expressions, rule-based logic, and checksums. NER handles names far better than rules, but it was trained on someone else’s text. Lowercase chat, mixed languages, and product names that look like surnames shift its error rate, and its confidence threshold trades misses against over-masking.

A local LLM as detector

An open-weight model running inside the network reads a prompt and returns the sensitive spans with their types. It catches what the other two miss: “the patient in bed 4”, an unreleased product described without its name, and a code name listed in its instructions. The costs are a GPU, extra latency per request, and nondeterministic output.

Two safeguards make it usable. Every span the model returns is checked against the input verbatim, since a paraphrased span masks the wrong text. And because the detector reads untrusted text, a pasted document sometimes carries instructions aimed at it (indirect prompt injection), so rules stay underneath to catch the card number a fooled model lets through.

Approach Catches well Weak spots Runs on Deterministic
Pattern rules with validators Cards, IBANs, emails, API keys, internal ID formats Names, addresses, free text CPU Yes
NER model Names, organizations, locations Domain terms, odd casing, untrained languages CPU or GPU Yes, for a fixed model
Local LLM detector Indirect, context-dependent references Unpredictable errors, injected instructions GPU No

Masking, blocking, and restoring values

Mask, block, or route locally

Detection produces spans, and policy decides what happens to them. Masking replaces a value with a placeholder and still sends the request, which fits when the model doesn’t need the value: summarizing a complaint doesn’t require the customer’s email. Blocking fits categories that must never leave, such as credentials and private keys. The third option is routing to a local model when the task needs the real value.

# Illustrative policy: one action per data type
policies:
  - types: [SECRET, PRIVATE_KEY]
    action: block            # never leaves
  - types: [CARD]
    action: mask
    restore: last4           # last four digits only
  - types: [PERSON, EMAIL, PHONE, CUSTOMER_ID]
    action: mask
    restore: full
  - types: [HEALTH]
    teams: [support]
    action: route_local      # local model answers

Consistent placeholders and restoration

Gateways that keep answers useful replace each distinct value with a typed, numbered placeholder and reuse it. Every mention of Jane Cooper becomes [PERSON_1], her email becomes [EMAIL_1], and a later bare “Jane” maps to [PERSON_1] as well. Consistency lets the model track who did what across a long thread, while a generic [REDACTED] strips out the structure it needs.

The gateway keeps a per-conversation mapping table from placeholders to real values and swaps them back before the user sees the reply. With streamed replies, it buffers text until a placeholder closes, because [PERSON_1] sometimes arrives split across two chunks.

Restoration doesn’t have to be complete. A support agent drafting a reply rarely needs more than the card’s last four digits. The PCI Security Standards Council treats the first six and last four digits as the baseline maximum to retain after truncation.

Restoration also works best scoped tightly: only placeholders issued in this conversation, only to the user who sent it. If injected text makes the model emit a placeholder from another session, it stays a placeholder.

Our home page has a small masking demo that runs in the browser. It masks a name, an email, and a card number, then restores them in a simulated reply, card cut to its last four digits.

Masking is not anonymization

The text around a placeholder still carries clues about the person. NIST IR 8053 summarizes research showing that de-identified data can sometimes be re-identified. Placeholder masking with a mapping table is pseudonymization, and the European Data Protection Board states that pseudonymised data is still personal data. Masking reduces what leaves, but for whoever holds the mapping table, the data stays in privacy law’s scope.

Measuring detection quality on your own data

No detector catches everything. Presidio’s own documentation warns that there is no guarantee it will find all sensitive information. The useful question is how much a gateway misses on the company’s own data, by type.

Recall is the share of real sensitive values the gateway caught, so it measures leakage. Precision is the share of masked spans that were actually sensitive. Low precision means over-masking, which degrades answers and pushes people to bypass the gateway. A score published for a model was measured on its benchmark text, not on internal tickets.

We recommend building the evaluation set before choosing tools. A useful set has five parts:

  1. A few hundred real texts of the kind people send to AI: tickets, emails, contract excerpts, and logs.
  2. Every sensitive span labeled by type, including internal identifiers.
  3. Negative controls: texts with no sensitive data, and lookalikes such as order numbers.
  4. Recall per type rather than one average, which hides a type the detector barely catches.
  5. A rerun after every rule, threshold, or model change.

Teams that measure well also compare answers with and without masking on the same tasks to see what masking costs in quality. We cover that discipline in LLM evaluation for product teams.

Deployment, logging, and prompt injection

Where the gateway runs

Detection has to happen before data crosses the boundary that matters. When masking runs in a third party’s cloud, the raw prompt leaves the network before anything is masked. We recommend running the gateway on your servers or in your cloud account, with the mapping table and logs kept there.

Rules and most NER models run on a CPU. An LLM detector needs a GPU. Teams that get full coverage also block direct egress to provider APIs, since otherwise people route around the gateway.

Logging

An audit log of raw prompts is a second copy of every sensitive value, often with looser access rules. The OWASP Logging Cheat Sheet says access tokens, sensitive personal data, and payment card holder data should usually not be recorded directly in logs, but removed, masked, sanitized, hashed, or encrypted instead. We recommend logging the masked prompt plus metadata (user, time, model, cost, and which types were masked) and keeping the mapping table out of logs.

Prompt injection

A gateway doesn’t solve prompt injection. OWASP’s LLM01 entry says it is unclear whether fool-proof prevention methods exist, and that indirect injection arrives through external sources such as websites and files. What a gateway does offer is one place to apply OWASP’s recommended input and output filtering, keep restoration session-scoped, and record enough to investigate.

Checklist for evaluating or building an AI gateway

Area What to check Red flag
Deployment Runs in your network or cloud account Masking in a third party’s cloud
Detection Validated rules, plus NER or a local model where needed Regex without validators or measured recall, or an LLM only
Quality Recall and precision per type on your samples Only a published benchmark
Policy Mask, block, or route locally, per type and team One on/off switch
Placeholders Typed, numbered, consistent per conversation [REDACTED] everywhere
Restoration Session-scoped, partial for cards, streaming-safe Restores any placeholder in any reply
Logging Masked prompts and metadata, SIEM export Raw prompts or mapping table in logs
Budgets Per-team and per-user limits, alerts One shared provider key
Coverage Chat, IDE assistants, and apps all routed Direct egress to providers still open

Where to start

We recommend this order:

  1. An inventory of current use. Proxy or DNS logs show which provider domains get traffic, and a short survey covers the tools teams use.
  2. A policy table. For each data type: mask, block, or route locally, plus who approves exceptions.
  3. The evaluation set. Labeled samples with negative controls, built before any tool is chosen.
  4. Rules first. Validators for structured identifiers and a block on credentials, with NER or a local model added where recall falls short.
  5. One tool first, then the side doors. An internal chat goes first, then IDE assistants and apps, and finally direct egress is blocked.

If you’d rather not build this yourself, our Private AI Gateway service deploys it in your network or cloud account in 4–6 weeks, with detection measured on your own samples before go-live.