Research release v1.0.0, CC BY 4.0

49,000 domains, categorized for research

A documented sample from the Web Filtering Database: 49,000 unique domains across six popularity bands, every one of the 58 categories represented.

49,000Unique domains
58Categories, all present
41.4%Carry several labels
6Popularity bands
When 49,000 rows are not enough

Move from a research set to the whole category database

The download on this page is a fixed snapshot for study. If your project needs every domain, new ones as they appear and a feed you can ship, the full database is the next step.

  • Take the data as CSV or JSON files, or call a real-time API.
  • Run it inside your own systems instead of calling out.
  • Use the same 58 categories you are already studying here.
Rows of domains with their category labels from the full database
58
categories, the same taxonomy as this set
120M+
domains in the complete database
~300,000
newly categorized domains added daily
Explore the full databaseLicensing for productsCheck a domain of your own

What it is for

A static sample for research and testing. It is not a blocklist and not a substitute for continuously updated filtering data.

Taxonomy research

See how multi-purpose domains carry several labels, and how the category mix changes from popular to long-tail sites.

Integration testing

Test joins between DNS or proxy logs and a category reference before connecting a live feed.

Teaching

Concrete, labelled examples for courses on content filtering, classification and network policy.

Labels that overlap

Well-known sites that carry more than one label

Taxonomy work is mostly about domains that fit several categories at once. These six, looked up in our data, show how it plays out.

DomainLabels recorded
reddit.comSocial Networking, Forums / Message Boards, Community Sites
kaggle.comAI / LLM Tools, Developer Tools / Code Hosting
github.comDeveloper Tools / Code Hosting
huggingface.coAI / LLM Tools
zenodo.orgEducation/Reference
data.govGovernment & Legal
See the labels for another domain

Sample design

Domains are drawn from six bands of source rank. Small categories are protected within each band so they still appear.

Rank 1,001 to 10,000
4,000
Rank 10,001 to 100,000
6,000
Rank 100,001 to 1,000,000
9,000
Rank 1,000,001 to 5,000,000
10,000
Rank 5,000,001 to 10,000,000
10,000
Rank 10,000,001 and later
10,000

The 49,000-domain download compared with the complete database

The research set answers questions about the taxonomy, while the database is built to run in a product.

AspectResearch downloadFull database
Number of domains49,000120M+
Categories58-category taxonomy58 web filtering categories
UpdatesOne fixed versionAbout 300,000 new domains categorized every day
Intended useStudy, teaching, testing joinsFilters, firewalls, DNS and security products
FormatsCSV and JSON LinesCSV or JSON files, real-time API, offline database
TermsCC BY 4.0Plans from $999, OEM licences from $10,000 per year
See database plansRead about OEM licensing

Download

Version 1.0.0. Also on Figshare.

web-filtering-research-sample-49000.csvThe sample, CSV, 4.9 MB
Download
web-filtering-research-sample-49000.jsonlThe sample, JSON Lines, 10.6 MB
Download
category-summary.csvLabel and primary-category counts for all 58 categories, 2 KB
Download
web-filtering-taxonomy.csvThe 58 categories with descriptions (CSV), 16 KB
Download
web-filtering-taxonomy.jsonThe 58 categories with descriptions (JSON), 28 KB
Download
dataset-analysis.jsonBand allocations, stratum quotas and summary statistics, 15 KB
Download
README.mdDataset description, 4 KB
Download
DATA_DICTIONARY.mdField definitions, 2 KB
Download
SAMPLING_METHODOLOGY.mdHow the sample was drawn, 4 KB
Download
CITATION.txtHow to cite, 1 KB
Download
LICENSE.txtCC BY 4.0 licence, 1 KB
Download
CHANGELOG.mdVersion history, 1 KB
Download
manifest.jsonFile sizes and checksums (JSON), 2 KB
Download
SHA256SUMSChecksums, 1 KB
Download

Cite this dataset

Alpha Quantum (2026). Web Filtering Research Dataset: 49,000 Categorized Domains (Version 1.0.0). Web Filtering Database. Figshare.

Related open data: AI Agent Page-Type Statistics (page types across 40.8 million domains, for AI agent egress research); AI Tools Taxonomy and Statistics.

The complete database covers up to 120 million domains with scheduled refreshes, API access and OEM licensing. See the full web filtering database.

Database plans begin at $999 and cover the 1 million most popular domains. If you want to build the data into your own product, an OEM licence starts at $10,000 per year.