Every prompt you send to an AI tool is built on a decision about where your data goes: not "is AI safe?" but instead "which AI should see which data?"
I answer this with three tiers and a router:
- Tier 1 is public frontier models suited for low-risk work and anything you've obfuscated first.
- Tier 2 is a private tenant where you add observability and controls on what can leave.
- Tier 3's for local open-weight models handling your most sensitive work.
Your router sends each request to the right tier based on how your system classifies the data - in short, it determines what data you're working with and uses the correct tier based on that. (Some companies also call this an AI gateway.)
Reminder: no data is safe once it leaves your control.
Everything you're about to read works off this thought.
Why a yes or no answer fails
A single rule can't cover company data, because it isn't one thing … some of it's harmless to share, but other data's critical and should never be exposed.
Every time data leaves your environment, that's exfiltration: you've moved it from a place you control to a place you don't (doesn't matter if it's a vendor portal vs. an AI agent).
Once you put data out, you might as well tack it on the church bulletin board … that's the level of trust to have with data in the world.
So match the decision to how sensitive the data you're working with is. Banning every tool isn't the answer - that stalls the work AI does well. Classify data internally (you can build a system for this) and route each request to a tier that fits.
Tier 1: public frontier models
Public frontier models are the right place for low-risk work and for anything you've redacted or obfuscated first. If the data is still sensitive after you've scrubbed it, don't use these - move it up a tier.
For instance, let's say you're uploading sales transcripts from customer conversations; you'd want to remove any identifying information for what companies are being discussed / represented and anything that might represent a competitive advantage from the discussions. If a customer's spilling the beans on their top priorities and problems they're facing, you don't want a competitor to have a way to find that! But once redacted, you can have a frontier model work through this information and provide an analysis.
Are public AI tools like ChatGPT private?
Not in the way most people assume.
Anything typed into a public AI tool leaves your environment, and whether it's kept or used to improve a model depends on your plan and the tool's settings.
Those terms differ between personal and business accounts and change over time, so read the current terms for your exact plan - a paid personal subscription doesn't automatically change this!
What data should stay out of public AI tools?
Anything regulated or identifying stays out of a public tool unless it's been redacted or obfuscated first. That includes patient and student records, nonpublic financial information, customer data covered by a contract, and absolutely anything like a login credential. Don't paste a medical record into a chat window you pay twenty dollars a month for.
A better way: Ask the question without sending everything - swap names, dates, account numbers, or other identifiers for placeholders. You need to be very careful with this, because an unusual job title can still point to one person.
Tier 2: private tenant
A private tenant means running AI inside an environment you control, with monitoring on what goes in and limits on what can leave. An example might be summarizing 90 days of support calls and tickets to find patterns, but the transcripts contain account details which are necessary for spotting patterns across customer types - this lands here because you need to keep customer information out of public-facing models, but it's not so sensitive it needs a fully-local instance.
What AI software is private?
Privacy comes from where the model runs and what controls surround it. No AI product is private out of the box!
A private setup usually has three parts:
- Observability and auditing on every request, so you can see what was sent and what came back.
- Egress controls, so you decide where data can go out and what can come back in.
- Anomaly detection that flags unusual patterns of use.
A private tenant limits and records what leaves your system, but doesn't guarantee nothing leaves.
Some AI models are hosted by the cloud provider while others can be deployed on infrastructure you manage, so the hosting also can differ significantly. You have to confirm what this entails for each model you're using before you describe a setup as private.
Tier 3: local open-weight models
This is for your most sensitive work. You control the entire infrastructure so the data in it isn't going anywhere.
I run several open models on my own machine today and the best of them are closing the gap with frontier models quickly. Now, this doesn't mean you'll replace frontier models fully because those are where you're getting the latest developments … they're worth paying for!
But a local model is now a realistic home for requests too sensitive to send out, and most companies serious about using AI for a competitive advantage will end up with a blend of tiers 1-3. That blend is most attainable for mid-market and enterprise organizations with staff who can build and maintain it, because you'll own the hosting, updates, testing, and debugging (one reason the AI Talent Gap is proving problematic).
The router: sending each request to the right tier
An AI router (or AI gateway) is the layer that decides where each request goes; it looks at how your data's classified and sends the request to the public, private, or local tier that fits.
The routing can be automatic or a choice the user makes. Asking "could a competitor see this information?" or, for other scenarios, "how much compliance risk is there if someone sees this?" usually makes it easy to make a safer choice.
This is different than an API gateway - that manages traffic between applications and services, an AI gateway adds awareness of the AI itself that gauges a prompt vs. your classification rules and help pick which model answers (plus log what's happened - observability is always important!).
As CIO Dive reported from Gartner's IT Symposium keynote, a starting point is investing in an AI gateway that routes requests to the best model or platform. Establish who owns the platforms, tools, and models in the organization; I'd add data classification as one of the criteria needed for that routing.
Guardrails for bots that talk to the public
A bot that answers people you haven't authenticated, like a chat assistant on your website, is a different risk from an internal tool: anyone can type anything into it, including attempts to trick it into revealing data or taking actions it shouldn't. Put hard guardrails at the gateway for these bots: limit which data they can reach, filter what goes in and what comes out, and log every exchange.
What auditors want to see
Regulators ask to see evidence of a process: that you thought through how data is managed, check compliance, and then double-check your watchdog processes. Yes, I'm simplifying and on purpose; HIPAA, FERPA, SEC rules, and SOC reports each add their own specific requirements. Treat evidence as the common thread and your compliance team's list as the rest.
We've written about why pilots stall without enforceable governance and observability, and evidence is the same idea seen from the audit side. As organizations grapple with settling who owns AI and is accountable for it, this need only grows.
A gateway logs each routing decision, and that log is the evidence trail auditors ask for. Regulated sectors already lean this way: in our analysis of 187,567 U.S. job postings, financial services posted 240% more data infrastructure roles than AI roles, and healthcare 164% more. That fits with sectors that have long had to produce audit trails building their data foundations before they experiment with AI.
Where to start on deciding what AI model gets what data
Start by classifying your data, then match each class to a tier. Do NOT start by buying a tool - that's backwards! Only buy something once you've mapped out where data should go, and ideally run it through a four Rs test to ensure you're working with what's useful instead of noise.
So, to recap:
- Classify how sensitive data is and which rules govern it
- Redact or obfuscate anything regulated before it goes to a public tool.
- Put a private tenant around work that needs more: observability on every request, egress controls, and anomaly detection.
- Send your most sensitive requests to a local model so they never leave your infrastructure.
- Keep the evidence: document how you manage data, watch whether the process is followed, and audit that watching.
- Put hard guardrails at the gateway for any bot that answers people you haven't authenticated.
Need help? We do this work today in highly-regulated industries - I'd be happy to chat about our AI Readiness work or try the 844 Snapshot for a first read on a domain of your data.
Frequently asked questions
Is ChatGPT private?
A public tool is never fully under your control, and your plan also determines some levels of privacy. Anything you type leaves your environment and whether it's retained and/or used to improve a model's training and performance varies between personal and business accounts … and it changes over time. Always be sure to check the current terms for your current plan before pasting in anything sensitive.
What shouldn't employees put into AI tools at work?
Employees shouldn't put anything regulated or identifying into their AI tools, such as patient / student records, nonpublic financial data, customer data under contract, login credentials, and the list goes on. If you have to use AI to help, redact or obfuscate this first so the request carries no identifiers; ideally, you'd send it to a private or local tier built for this data.
How do you keep company data safe when using AI?
Safety begins by classifying your data and routing each request to the tier that fits it. Use public frontier models for low-risk work, a private tenant with observability and egress controls for protected data, and local open-weight models for your most sensitive needs. Keep a record of what went where at all times - auditors ask for evidence, so audit trails are a must.
What does a private AI setup look like?
A private setup runs AI models inside an environment you control - including your own cloud account, monitoring on every request, controls over what can leave and come back, and alerts for unusual activity. If you aren't sure, it's best to first confirm where each model runs.
What is an AI gateway, and how is it different from an API gateway?
An AI gateway is the layer between your people or applications and your AI models that decides where each request goes. An API gateway manages general traffic between services, such as access and rate limits, while an AI gateway also understands the prompt, so it can apply classification rules, redact or block sensitive content, pick a model, and log what happened.
What do auditors want to see for AI governance?
Evidence of a process: that you thought through how data is managed, that you observe whether the process is followed, and that you audit that observation. Rules such as HIPAA, FERPA, SEC requirements, and SOC reports add their own specifics. Gartner's September 2026 predictions also say that by 2030, 80% of Global 500 companies will contractually make their CIO or chief AI officer the "evidence custodian" for AI accountability, as TechTarget reported.
