URL Category Guardrails for AI Agents That Browse the Web
An AI agent with a browser tool can open hundreds of pages to finish one task. Nobody reviews each link it follows. The question is no longer whether a person should visit a site, but whether your agent should.
This guide is for teams building browsing agents, agent platforms and research assistants. It shows how URL categories turn "browse the web" into a policy you can explain, test and audit.
What can go wrong when an agent browses freely
Each risk below has happened in some form with browsing automation. Category controls reduce most of them, either by keeping the agent away or by forcing a human decision first.
Hostile instructions in pages
Pages can contain text written to steer the agent. Low-trust categories such as forums, personal blogs and parked pages carry more of this risk.
Data leaving through uploads
An agent that can post to paste sites, file hosts or form builders can be tricked into sending private data out.
Unwanted transactions
Shopping, banking, gambling and crypto sites let an agent spend money or move funds if nothing stops it.
Malware and phishing
Agents follow search results and links without suspicion, including to fresh malicious domains and fake login pages.
Content you never want touched
Adult, hate and extremist content in an agent's context or output creates legal, brand and safety problems.
Controls bypassed by proxies
Web proxies and anonymizers hide the real destination, which defeats every other rule.
Three tiers: allow, ask, deny
Most agent builders end up with three tiers of categories. Read access and write actions are treated separately, because posting data is riskier than reading a page.
Allow
The agent may read freely. These are the categories most tasks need.
- News/Media, Science, Education/Reference
- Government & Legal, Health & Medicine
- Developer Tools / Code Hosting
- Business, Travel, Search Engines & Platforms
Ask a human
The agent pauses and asks before acting. Reading may be fine; submitting forms is not.
- Shopping, Marketplaces, Online Banking
- Financial Services, Cryptocurrency
- Email and Messaging, Social Networking
- Cloud Storage / File Hosting (any upload)
Deny
Never opened, whatever the task or the page text says.
- Malware / Phishing / Suspicious
- Adult, Hate/Discrimination, Illegal Drugs
- VPN / Proxy / Anonymizers
- Gambling, Parked / For-Sale Domains
Which categories matter, and why
The table covers the categories agent builders discuss most. Read and write columns differ because many categories are safe to read but risky to act on. Definitions for all 59 are on the category list.
| Category | Agent risk | Read | Write or submit |
|---|---|---|---|
| Malware / Phishing / Suspicious | Payloads, credential traps, hostile page text | Deny | Deny |
| VPN / Proxy / Anonymizers | Hides the real destination, defeats policy | Deny | Deny |
| Cloud Storage / File Hosting | Main route for data leaving the agent | Allow approved hosts | Ask or deny |
| Web-based Applications | Forms and paste tools can receive data | Allow | Ask |
| Shopping, Marketplaces | Purchases and account actions | Allow | Ask |
| Online Banking, Financial Services | Money movement, sensitive accounts | Ask | Deny or ask |
| Forums / Message Boards, Blogs/Personal Sites | Untrusted text, a common place for hostile instructions | Allow with low trust | Deny posting |
| Social Networking | Posting in a user's name, untrusted content | Allow with low trust | Ask |
| AI / LLM Tools | Agents passing data to other AI services | Approved tools only | Approved tools only |
| Parked / For-Sale Domains | No useful content, often ad redirects | Deny | Deny |
| URL Shorteners | Unknown destination | Resolve, then apply target's tier | Resolve first |
| Adult, Hate/Discrimination | Unsafe content in context and output | Deny | Deny |
Where to enforce category guardrails
A guardrail that lives only in the agent's prompt is a suggestion. Real enforcement sits in code the agent cannot rewrite. Most designs use at least two of these layers.
Tool layer
The browse or fetch tool checks the category before it navigates and returns a clear refusal the agent can reason about.
Browser automation
The headless browser checks every navigation and redirect, including those started by page scripts.
Egress proxy
All traffic from the agent sandbox goes through a proxy or DNS filter with the same policy. It catches what layers 1 and 2 miss.
Approval service
"Ask" decisions go to a human through your app, with the domain, category and intended action shown.
Local database, lookup API, or both
Agents visit more unfamiliar domains than people do, because they follow long chains of search results and citations. That makes the miss path more important than in a normal filter.
Inside your agent runtime
- Fast checks on every navigation
- No browsing data sent anywhere
- Works in isolated or air-gapped runtimes
- Needs a rule for unknown domains
Called from the tool layer
- Full coverage without a data pipeline
- Quick to add to a prototype agent
- Single and batch endpoints
- Adds a call to each new domain
Local first, API on a miss
- Most visited domains answered locally
- Rare domains resolved through the API
- Batch-check search results before the agent picks one
- Cache answers across agent sessions
Update cadence: regular refreshes
Attackers who target agents use fresh domains, just as they do for people. The update stream should reach every agent runtime without redeploying the agent, so keep data in a shared store or a mounted file rather than the image.
Daily delta files
About 300,000 newly registered domains are categorized every day. Apply them to every runtime's local copy and to your egress proxy.
Rechecks of new domains
Each new domain is checked again after a week and after a month, because many sites change after launch. Updated labels arrive in the deltas.
Monthly refresh
All other domains are refreshed monthly. Reload the full file on that cycle in each runtime image or shared store.
AI tools set, daily
The 16,000+ AI tool domains are updated daily, so approved-tool rules stay current as new AI services launch.
Integration steps
These steps add category guardrails to an existing browsing agent. They work for single agents and for platforms that host many customers' agents. Steps 1 to 3 give you a working guardrail in days; the rest make it hold up under pressure.
Write the tier policy
List allow, ask and deny categories for reading and for writing. Get sign-off from security and legal before coding.
Load the data
Import the CSV, Parquet or JSON file into a store your runtimes can read with low latency. Schedule the daily delta job.
Wrap the browse tool
Before each fetch or navigation, normalize the URL, look up the domain, apply the tier, and return a clear result to the agent.
Hook the browser
Intercept every navigation and redirect in the automated browser, including script-driven ones and new tabs.
Add the egress layer
Route sandbox traffic through a proxy or DNS filter using the same policy, so a bypass in code still meets a wall.
Build the approval flow
Send "ask" decisions to the user with domain, category and intended action. Time out to deny, never to allow.
Filter search results
Batch-check result domains and remove denied ones before the agent chooses where to go.
Log and review
Record every decision and review denied and approved actions weekly to tune the tiers.
Audit trails and customer controls
Buyers of agent platforms ask the same question security teams ask of any automation: "Show me what it did." Categories turn a long navigation log into something a reviewer can read.
Decision log per run
For every navigation: time, domain, categories returned, tier applied and outcome. Store it with the agent run so reviewers see context.
Category summary
A short summary per run, such as "42 pages read: 60% Developer Tools, 25% News/Media, 2 denied", makes reviews fast.
Customer-set tiers
On a platform, let each customer adjust tiers within your locked deny list, and add their own allow and deny domains.
Exception requests
When an agent is denied a site the user needs, let the user request an exception that an admin can approve for that domain only.
Guardrail defaults by agent type
Different agents need different maps of the web. These defaults show how the tiers usually shift by agent type. Start from the closest row and adjust.
| Agent type | Allow widely | Ask first | Deny |
|---|---|---|---|
| Research assistant | News, science, education, government, reference | Any form submission, any login | Uploads to file hosts, proxies, malware, adult |
| Coding agent | Developer tools, code hosting, technical documentation | Package publishing, account settings | Paste sites for code upload, proxies, malware |
| Shopping or booking assistant | Shopping, travel, restaurants, marketplaces for reading | Checkout, payment, account creation | Gambling, crypto exchanges, parked domains |
| Customer support agent | Your own domains, approved partner sites | Anything outside the approved list | Social posting, messaging, file uploads |
| Operations or IT agent | Approved web-based applications and admin consoles | Remote access tools, changes in production systems | Unknown domains, proxies, file hosting uploads |
Testing your guardrails
Agents are creative about reaching their goals. Test the guardrails the way an agent would try to get around them, and repeat the tests after each data load and each policy change.
Redirect chains
Start on an allowed site that redirects to a denied one. The final destination must be blocked, not just the first hop.
Short links
Ask the agent to open a short link to a denied category. The shortener should resolve and the target's tier should apply.
Upload attempts
Give the agent a task that tempts it to paste data into a file host or form. The write action must stop at the tier check.
Fresh domains
Use domains from the latest daily delta in tests, so you know new sites reach every runtime the same day.
Direct fetches
Try to reach a denied domain through a code tool or raw HTTP call. The egress layer must catch it.
Approval timeouts
Leave an "ask" request unanswered. The action must fail closed, and the agent should report why.
Licensing category data for agent guardrails
How you licence depends on who runs the agents and whether customers see the categories. The OEM licensing page covers embedded use in detail.
OEM licence
Guardrails run inside an agent platform or product offered to customers. They see categories and decisions, not the raw file.
Internal-use licence
Agents used only inside your own company, for research, support or operations.
From $999
Starter database tiers from $999; enterprise and OEM licences are quoted per deployment. See pricing. Resale or publication of the raw data is not allowed.
AI agent browsing guardrails FAQ
Why do browsing AI agents need URL category controls?
An agent follows links and search results on its own, often faster than any person could review. Category controls let the builder decide in advance which kinds of sites the agent may read, which actions need human approval, and which sites are always off limits.
Where should the category check run in an agent?
Before every navigation or fetch: in the tool layer that the agent calls to browse, and again at an egress proxy around the agent's sandbox. Two checks mean a bug in one layer does not open the whole web.
Can category controls stop prompt injection?
Not on their own. They reduce exposure by keeping agents away from untrusted categories and limit damage by blocking upload and paste destinations. They should sit alongside content handling and permission controls in the agent design.
Should the agent use a local database or an API?
Both. A local copy gives fast answers on the hot path for the most visited domains, and the included lookup API covers rare domains. Agents visit many unfamiliar sites, so the miss path matters more than in human browsing.
How fresh is the category data?
About 300,000 newly registered domains are categorized each day as daily delta files and rechecked after a week and after a month. All other domains get regular monthly refreshes, and the AI tools set is updated daily.
What licence does an agent platform need?
If the guardrails run inside a product or platform you offer to customers, an OEM licence. For agents used only inside your own company, an internal-use licence. Resale or publication of the raw data is not allowed.
What should an agent do when a site is denied?
Return a clear, structured refusal to the agent that names the category and says the site is off limits. A good agent then looks for another source or tells the user, instead of retrying the same site in a different way.
How should agents treat domains with no category yet?
Treat unknown domains as low trust. Allow reading only if the task needs it, never allow write actions, and resolve the domain through the lookup API so the next visit has an answer.
Do multiple categories per domain help agents?
Yes. A domain can carry several categories ranked by how much of the site they describe. Deny rules should match any label, while allow rules can rely on the top-ranked one, which keeps mixed sites from slipping through.
Can the same policy cover agents and human users?
Often it can share the same data and deny list, but agents usually need stricter write rules and an approval step that people do not. Keep one category source and separate policies for each.
Does this work for agents running on customer infrastructure?
Yes. The database is delivered as files that run inside your own systems, including on-premise and air-gapped environments, so guardrails work without outside calls where customers require it.
Give your agents a map of the web before they explore it
Tell us how your agents browse, where they run and who uses them. We will suggest a data layout and licence for your guardrails.
Talk to us OEM licensing