URL category data for AI training data: filter, balance and document a corpus
Teams that build training sets from the web have to answer the same questions every time: what to leave out, what to treat with care, how balanced the result is, and how to describe it afterwards. This guide shows how domain categories answer those questions cheaply, before heavier document-level filters run.
Five jobs category data does in a data pipeline
Every crawled document has a source domain. Joining it to the domain's categories is one lookup, and it serves five purposes at once, from the first filtering pass to the final report on what the corpus contains.
Exclusion
Drop whole categories that should never be in the data, such as adult content.
Quality
Remove parked pages, link shorteners and ad-heavy sources that add noise.
Sensitivity
Flag sources dense in personal data, such as dating sites and forums, for scrubbing.
Balance
See and steer the mix of topics, from science to sport.
Documentation
Describe the corpus by source category for governance and disclosure.
Join your crawl to 120 million categorized domains in one pass
Our database delivers web filtering categories and IAB Content Taxonomy labels for each domain as files you load next to your crawl. One join on the registrable domain gives every document its source categories, ready for exclusion rules, balancing and reporting.
- 58 filtering categories, several ranked per domain.
- IAB labels to tier 4 for topic balance.
- A dated version for every delivery, for reproducible corpora.
A starting matrix: what to do with each category
Teams set these rules differently for different models. A reasonable starting point for a general-purpose corpus:
| Category | Typical action | Reason |
|---|---|---|
| Adult | Exclude | Unwanted content and frequent personal images |
| Parked / For-Sale Domains | Exclude | Template text, ads and no real content |
| URL Shorteners | Exclude | Redirects, not content |
| Hate/Discrimination | Review | Exclude for most uses; keep labelled for safety research |
| Dating | Exclude | Dense in personal profiles |
| Advertising/Marketing | Down-weight | Repetitive promotional text |
| Gambling | Down-weight | Template-heavy affiliate pages |
| Forums / Message Boards | Review | Valuable dialogue, high personal-data load: scrub first |
| Health & Medicine | Review | Useful but sensitive; check for patient stories |
| Education/Reference, Science, Government & Legal | Keep | Often under-represented relative to commercial sites |
Use any-match for exclusions: a domain with Adult anywhere in its categories counts as adult. For down-weighting, the main category, the first in the row, is usually the better guide.
Domain categories and document filters work together
Domain categories decide at the level of the site; document filters decide page by page. Each is good at what the other is not, and the strongest pipelines use both in a fixed order.
Domain categories are best at
- Removing whole sources cheaply, before any text is processed
- Consistent decisions for every page of a site
- Balancing and describing the corpus by source type
- Catching sources whose pages look harmless one by one
Document filters are best at
- Language detection and text quality
- Toxicity and personal data inside otherwise good sites
- Duplicate and near-duplicate removal
- Mixed sites, such as a news site with a comment section
Running the domain step first shrinks the volume the expensive document filters have to handle, which saves compute on every pass.
Five sources that show up in nearly every web corpus
Real rows from our data. The categories suggest how each source might be treated, before anyone reads a single page.
| Domain | Categories in our data | A typical curation choice |
|---|---|---|
| stackoverflow.com | Developer Tools / Code Hosting, Education/Reference, Forums / Message Boards | Keep, scrub user details |
| quora.com | Forums / Message Boards, Community Sites | Review for quality and personal data |
| medium.com | Blogs/Personal Sites | Keep, with document quality filters |
| fandom.com | Community Sites, Gaming | Keep, watch topic balance |
| archive.org | Education/Reference | Review per collection and rights |
Balancing a corpus against the web itself
A crawl is not a neutral sample. Comparing its category mix with the web's own mix shows where it is skewed. From our count of the 20 million most visited domains:
Business
The largest category on the web. A corpus far above this share is probably heavy on company boilerplate.
Shopping
Product pages repeat themselves. Many teams cap this share well below its web share.
Science
Small on the web and often worth up-weighting for reasoning and knowledge tasks.
The full set of shares, by popularity band and top-level domain, is in what the web is made of. IAB labels in the same file give a second, finer topic view.
Wiring it into the pipeline
The join itself is simple and runs in the same framework as the rest of the pipeline. Four details make it reliable at web scale.
# one pass: attach source categories to each document
docs = load("crawl_shard_*.parquet") # url, text, ...
cats = load("web_filtering_database.parquet") # domain, categories
docs["domain"] = registrable_domain(docs["url"])
docs = docs.join(cats, on="domain", how="left")
EXCLUDE = {"Adult", "Parked / For-Sale Domains", "URL Shorteners", "Dating"}
keep = ~docs["categories"].str.split("|").apply(lambda c: bool(set(c or []) & EXCLUDE))
docs = docs[keep]
# store the delivery version with the shard for reproducibility
How the licensed file supports repeatable corpus builds
Every delivery of the database is identified by its version, and refreshes arrive on a schedule you choose, from quarterly to daily. A team can rebuild an older corpus with the categories it used then, or move to the newest delivery deliberately.
Documenting what the corpus contains
Governance teams, partners and regulators increasingly ask what a model was trained on. In the EU, providers of general-purpose AI models are expected to publish a summary of their training content. Category breakdowns make such descriptions concrete.
Share by category
Documents and tokens per source category, before and after filtering.
What was removed
Which categories were excluded or down-weighted, and why.
Top source domains
The largest contributors in each category, for review.
Data versions
The category data version used for each build.
This is general information, not legal advice. Disclosure duties depend on the model, the provider and the market; check the current guidance and templates that apply to you.
What category data does not settle
Three questions stay separate from categorization, and a good pipeline records each of them on its own. Keeping them apart also makes audits easier, because each decision has one clear source.
| Question | Why categories do not answer it | Where categories still help |
|---|---|---|
| May we use this source? | Licences and site opt-out signals are per site, whatever its category | Prioritising which categories to check first, such as news publishers |
| Is this page's text good? | Quality varies page by page inside one site | Skipping whole low-value categories before text filters run |
| Does this page contain personal data? | Any site can contain it | Flagging categories where it is dense, such as forums and dating |
Different models, different category rules
The matrix above suits a general model. Special-purpose corpora move the lines, and category data makes each variant a configuration change rather than a new pipeline.
Code models
Keep Developer Tools / Code Hosting and technical forums; down-weight most consumer categories.
Safety classifiers
Keep Adult and Hate/Discrimination sources, labelled and stored apart, as training examples for the classifier itself.
Education products for children
Exclude the whole safety set, including Dating, Gambling, Weapons and Alcohol, and keep Education/Reference.
Domain-specific models
For a medical model, keep Health & Medicine, Science and Government; review patient forums with care.
For multilingual corpora, combine categories with the country fields in the same file to see whether one region's commercial sites crowd out everything else.
Curating sources with text classifiers alone, or with domain categories first
The same crawl, with and without a domain step.
| Task | Text classifiers alone | Domain categories first |
|---|---|---|
| Removing adult and parked sources | Every page scored one by one | Whole sources dropped by one lookup each |
| Compute spent on filtering | Full volume through every filter | Less volume reaches the expensive filters |
| Describing the corpus | Topic labels per page, hard to summarise | Clear shares by source category |
| Reproducing a build | Model versions to track | A versioned delivery to rebuild from |
| New sources in the next crawl | Scored again from scratch | About 300,000 new domains categorized daily |
Category data for training corpora: frequently asked questions
How can domain categories help curate training data?
By joining every crawled document to the category of its domain, a pipeline can exclude unsafe or low-value sources, flag sensitive ones for review, balance the topic mix and describe the composition of the final corpus.
Are domain categories enough to filter a corpus?
No. They work at the level of the site and are best used together with document-level filters for language, quality, toxicity and personal data. Domain categories are cheap and run first; document filters handle what is left.
Which categories are usually excluded from training data?
Adult, Parked / For-Sale Domains and URL Shorteners are common exclusions, with Hate/Discrimination sent to review. Dating sites are often excluded because they are dense in personal data.
Can category data help with training data documentation?
Yes. Shares of documents and tokens by source category make a composition summary concrete, which helps with internal governance and with public summaries such as those expected from general-purpose AI model providers in the EU.
Do categories replace checks on rights and opt-outs?
No. Site opt-out signals and licence terms are separate questions. Categories can help prioritise where to look, for example news publishers, but they do not answer whether a source may be used.
Can we use IAB labels as well as filtering categories?
Yes. The same file carries IAB Content Taxonomy v2 and v3 labels down to tier 4, which give a finer topic view for balancing, next to the web filtering categories used for exclusion.
How are new domains in a fresh crawl handled?
About 300,000 newly registered domains are categorized every day and delivered as daily delta files on plans that include them. The included lookup API answers for domains not yet in your copy.
How do we record which data version we used?
Each delivery of the database is identified by its version, so a pipeline can store that version next to every filtering decision and reproduce the corpus later.
Database licences start at $999 for the 1 million most popular domains and go up to the full 120 million; licences for use inside AI products are quoted per deployment. Tell us the size of your crawl and how often you rebuild, and we will suggest the tier and refresh schedule.
Know what every source in your corpus is
Tell us about your crawl, your exclusion rules and your documentation needs. We will propose the categories, files and licence that fit the pipeline.
Domain names in the tables identify sources for illustration only. Web Filtering Database is not affiliated with, sponsored or endorsed by their owners.