What is URL filtering, in one paragraph
URL filtering is a control that checks every web address requested on a network or device against a policy. The filter finds out what kind of site the address belongs to, compares that with the rules for the user, and then allows, blocks, warns or logs.
Almost every modern filter decides by category rather than by individual address. Nobody can list billions of pages by hand, but a policy that says "block Gambling" works for every gambling site the database knows.
Four terms people mix up
URL filtering
Deciding by web address: domain, subdomain and, where visible, the path.
Web filtering
The umbrella term for controlling web access, whatever the method.
Content filtering
Broader again: can include page text, files, search results and email.
Web filter
The product or service that enforces the policy, in a firewall, resolver, proxy, browser or agent.
So what is a web filter in practice? It is any enforcement point that turns a written internet use policy into allow and block decisions. The decisions are only as good as the category data the filter consults.
What is content filtering beyond the URL?
Page content
Some gateways read the page itself and react to words or images, at a cost in speed.
Search results
Safe search settings hide explicit results before a user clicks anything.
Downloads
File type rules stop executables or archives from risky categories.
Email links
Mail filters check link domains against the same categories before delivery.
How URL filtering works, step by step
The details differ between products, but the decision path is nearly always the same five steps. The whole path has to finish in milliseconds, which shapes how the category data is stored.
Request
A browser or app asks for a URL or resolves a domain.
Intercept
The filter sees the domain, and the full URL if it can.
Look up
Local cache, then the category database or API.
Decide
Categories are compared with the rules for this user and time.
Act
Allow, block page, warn, throttle or log.
A URL filtering example
A secondary school blocks Gambling, Adult and VPN / Proxy / Anonymizers, and warns on Social Networking. Here is how five requests play out.
| Request | Categories found | Result | Why |
|---|---|---|---|
betting-site.example | Gambling | Blocked | Primary category is on the block list |
news.example/gambling-law | News/Media | Allowed | The domain is a news site; the topic of one article does not change it |
free-proxy.example | VPN / Proxy / Anonymizers | Blocked | Bypass tools are blocked so the rest of the policy holds |
social.example | Social Networking | Warned | A warning page, then access, with the visit logged |
brand-new.example (registered today) | Shopping | Allowed | Categorized on its first day, so it is not left in an unknown bucket |
Categories vs blocklists vs reputation
Three kinds of data can drive a URL filter. They answer different questions, and good filters combine them.
Category database
What is this site about?- Covers
- The whole web: millions of domains in topics such as News, Gaming or Adult
- Strength
- One rule handles every site in a topic, including ones you have never heard of
- Watch for
- Coverage gaps and slow updates on new domains
Blocklist
Is this exact domain on a list?- Covers
- A fixed set of domains, often for one purpose such as ads or malware
- Strength
- Simple, fast and easy to explain; good for mandated lists
- Watch for
- Lists go stale and cannot express allow-by-topic policies
Reputation score
How risky is this site?- Covers
- A risk rating, often based on age, behaviour and history
- Strength
- Catches suspicious sites before anyone labels them
- Watch for
- Says nothing about topic, so it cannot enforce a content policy
A web content filtering policy needs topics, because policies are written in topics: "no gambling, no adult content, social media at lunch". Blocklists and reputation sit on top as extra layers. See blocklist formats for how category data is exported as lists.
Where the category database fits
Every web filter has three parts: something that sees traffic, something that holds the policy, and something that knows what sites are. Vendors often build the first two and license the third.
Enforcement point
- Firewall or gateway
- DNS resolver
- Proxy or browser extension
- Device agent
Category database
- Domain to category labels
- Several ranked labels per domain
- Daily new-domain updates
- Local file plus lookup API
Policy engine
- Rules per user or group
- Time and location conditions
- Exceptions and overrides
- Reports and alerts
File or API?
A local file answers in microseconds and keeps working offline, which suits resolvers and gateways under heavy load. An API suits low-volume lookups and the long tail of rare domains. Most products use both: a file for the bulk, the API for misses.
Our web filtering database ships as CSV, Parquet or JSON for on-premise or air-gapped use, with a lookup API included. Read more about the domain categorization API and real-time URL classification.
Where a web filter can run
URL filtering is a function, not a box. It can live at several points between the user and the site, and each point sees a different amount of the request.
| Enforcement point | What it sees | Good for | Trade-off |
|---|---|---|---|
| Next-generation firewall | Domain always; full URL with inspection | Offices and campuses with one exit point | Devices off the network are not covered |
| DNS resolver | Domain names only | Fast, wide rollout for networks, ISPs and MSPs | Cannot act on paths; can be bypassed by other resolvers |
| Secure web gateway or proxy | Full URL and content | Detailed policies and reporting | Heavier to run; needs certificates for HTTPS |
| Browser extension | Full URL and page | Managed browsers, such as school laptops | Covers only that browser |
| Device agent | Domain or full URL, depending on design | Remote staff and take-home devices | Must be installed and kept running on every device |
| Router or ISP filter | Usually domain names | Homes, small sites, opt-in family filters | Coarse control per network, not per person |
All of these can use the same category data. That is why one licensed database can power a vendor's firewall, DNS service and agent at once, with the same labels in every report.
Writing a web content filtering policy
A filter enforces a policy; it does not write one. These steps keep the policy short, defensible and easy to translate into categories.
State the purpose
Safety of minors, security, legal risk, bandwidth or productivity. Each purpose points to different categories.
Define the groups
Students by age, staff, guests, servers. Each group gets its own rule set.
Pick categories, not sites
Choose block, warn and allow categories per group. Keep individual sites for exceptions only.
Plan exceptions
Decide who can ask for one, who approves it, and when it is reviewed.
Tell people
A clear block page and a short acceptable use notice prevent most complaints.
Review yearly
New kinds of sites, such as AI tools, change what a sensible policy looks like.
Common misconceptions about URL filtering
"More categories is better"
Policies need a manageable set. Fine detail helps reports, but 59 well-defined categories cover real policy needs.
"HTTPS makes filters useless"
The domain is still visible in almost every case, and domain categories handle most policies.
"A blocklist is enough"
Lists cover what someone already found. Categories cover sites nobody has reported yet.
"Unknown sites are rare"
Hundreds of thousands of domains are registered every day. Without day-one labels they pile up fast.
For vendors: building URL filtering into a product
If you are adding URL filtering to a firewall, DNS service, browser or parental control app, the data layer is where most of the hard work hides. This checklist covers the parts teams most often underestimate.
- Normalise host names before lookup: lower case, trailing dots, internationalised names
- Match the longest known suffix first, so a subdomain can differ from its parent
- Keep a local copy for speed, with the API as a fallback for misses
- Apply daily deltas without restarting the service
- Store several categories per domain and let policies use the ranking
- Map your categories to the labels customers know from other vendors
- Log the category with every decision, so support can explain blocks
- Decide what happens to unknown domains before customers ask
Web content filtering policy by use case
The same category data supports very different policies. These are typical starting points; every organisation adjusts them.
Schools and libraries
CIPA asks for a measure that blocks obscene and harmful-to-minors images on computers used by minors. Most schools go further.
Business offices
Focus on security and legal risk, with light productivity rules and logging for HR questions.
Guest Wi-Fi
Protect the brand and the bandwidth while keeping guests happy.
Regulated sectors
Finance and health add data-leak controls: unapproved storage and AI tools.
Families
Parental controls block by age band and allow by schedule.
ISPs and telcos
Opt-in family filters and legally required blocks for whole customer bases.
For deeper guidance see CIPA web filtering, web filtering software for business and our list of content filtering categories.
What separates good category data from weak data
Two filters with the same features can behave very differently because of the data underneath. When you evaluate a filter or a database, test these four things.
Coverage
How much of real traffic gets a category, not how many domains are listed.
- Test with your own traffic logs
- Count the share that comes back unknown
- Include non-English sites
Freshness
How fast new and changed sites get labels.
- Ask how new domains are handled on day one
- Ask when they are checked again
- Ask how often the rest is refreshed
Multi-category
Real sites mix topics. A single label forces a bad choice.
- Look for several labels per domain
- Look for a ranking between them
- Check how mixed portals are handled
Accuracy
Measured, not claimed. Wrong labels cause both leaks and complaints.
- Ask how accuracy is measured
- Ask for validation reports
- Spot-check sites you know well
How our database answers those four
- 120 million categorized domains
- The 20 million most visited cover 99.99% of web traffic and usage
- About 300,000 new domains categorized every day
- New domains checked again after a week and after a month
- All other domains refreshed monthly, delivered as daily deltas
- Several categories per domain, ranked by share of the site
- 59 filtering categories plus the IAB Content Taxonomy
- Regular manual validations, with reports for licence holders
You can try any domain in our free domain lookup tool, or browse the full taxonomy.
Limits of URL filtering, and how teams cover them
URL filtering is the base layer of web security, not the whole of it. Knowing the limits helps you place it correctly.
| Limit | What happens | Common answer |
|---|---|---|
| Encrypted traffic | Without inspection only the domain is visible, not the path | Decide by domain category, inspect where policy allows |
| Bypass tools | VPNs and proxies tunnel around the filter | Block the VPN / Proxy / Anonymizers category and encrypted DNS resolvers |
| Mixed platforms | One domain hosts both good and bad content | Ranked multiple categories, plus path rules where visible |
| Brand-new sites | Not yet labelled anywhere | Day-one categorization and a rule for very young domains |
| Off-network devices | Laptops leave the office filter behind | Agent or cloud DNS filter using the same categories |
The comparison between resolver-level and proxy-level filters is covered in our guide to DNS filtering vs web filtering.
A short glossary of web filtering terms
These terms come up in every product sheet and request for proposal. Here is what each one means in practice.
| Term | Plain meaning |
|---|---|
| Category | A topic label such as News/Media or Gambling, attached to a domain or URL |
| Taxonomy | The full set of categories a product uses, with a definition for each |
| Primary category | The label that best describes most of a site; secondary labels describe the rest |
| Uncategorized | A domain the database has no label for yet; the policy must decide what to do |
| Override | A local decision that replaces the database label for one site |
| Block page | The page shown instead of the site, ideally naming the category and how to ask for access |
| Safe search | A setting that forces search engines to hide explicit results, often enforced by the filter |
| TLS inspection | Decrypting HTTPS at a gateway so the full URL and content are visible |
| Newly registered domain | A domain created in the last days or weeks; a common starting point for scams |
| Delta file | A daily file with only the domains that are new or changed since the last update |
| DNS RPZ | Response policy zone: a standard way to give a DNS resolver a list of names to block or redirect |
Frequently asked questions
What is URL filtering?
A control that checks each requested web address against a policy and allows, blocks, warns or logs. Most URL filters decide by category, using a database that labels domains by topic.
What is the difference between URL filtering, web filtering and content filtering?
Web filtering is the umbrella term. URL filtering is its most common method, deciding by address. Content filtering is broader and can include page content, files, search results and email.
What is a web filter?
Software or a service that applies a web access policy. It can live in a firewall, a DNS resolver, a proxy, a browser extension or an agent on the device.
Is URL filtering the same as a firewall?
No. A classic firewall decides by address and port. URL filtering is often a feature inside a next-generation firewall, but it decides by what the website is.
Can URL filtering see HTTPS pages?
It can always see the domain. It sees the full path only if the traffic is inspected or the filter runs in the browser or on the device.
Does URL filtering slow down browsing?
Not noticeably when it is built well. Lookups against a local copy of the database take a tiny fraction of a page load, and results are cached. Slow filters are usually waiting on a remote lookup for every request.
How do I check how a site is categorized?
Most filters have a lookup page in their console. To see how our database labels a site, including secondary categories, use the free web filter lookup or the URL checker.
Where does the category database come from?
Filter makers build one or license one. Ours is categorized from site content, reviewed and validated, and delivered as files. Starter database tiers start from $999; enterprise and OEM licences are quoted per deployment.
Build your filter on solid category data
120 million domains, 59 categories plus the IAB taxonomy, daily new-domain deltas, and a lookup API. See also OEM licensing.