# Web Filtering Research Dataset: 49,000 Categorized Domains

Version 1.0.0, September 2026.

49,000 unique domains labelled with the 59-category Web Filtering Database taxonomy. The domains are sampled across six popularity bands, from source rank 1,001 to beyond rank 10 million, with every category represented. The most popular 1,000 domains are not included.

The full [enterprise web filtering database](https://www.webfilteringdatabase.com/web-filtering-database.php) covers up to 120 million domains. This release is a static research sample, not a blocklist and not a substitute for continuously updated filtering data.

## At a glance

| Measure | Value |
|---|---:|
| Domains | 49,000 |
| Categories | 59, all present |
| Domains with more than one category | 20,292 (41.4%) |
| Popularity bands | 6 |

## Files

| File | Content |
|---|---|
| `web-filtering-research-sample-49000.csv` | The sample: domain, categories, primary category, popularity band, source rank, classification version |
| `web-filtering-research-sample-49000.jsonl` | The same records as JSON Lines |
| `category-summary.csv` | Label occurrences and primary-category counts for all 59 categories |
| `web-filtering-taxonomy.csv`, `web-filtering-taxonomy.json` | The [education web filtering taxonomy](https://www.webfilteringdatabase.com/taxonomy.php) and its descriptions |
| `dataset-analysis.json` | Band allocations, stratum quotas and summary statistics |
| `DATA_DICTIONARY.md`, `SAMPLING_METHODOLOGY.md` | Field definitions and how the sample was drawn |
| `CITATION.txt`, `LICENSE.txt`, `CHANGELOG.md` | Citation, licence and history |
| `manifest.json`, `SHA256SUMS` | File sizes and checksums |

## Sample design

| Popularity band (source rank) | Domains |
|---|---:|
| 1,001 to 10,000 | 4,000 |
| 10,001 to 100,000 | 6,000 |
| 100,001 to 1,000,000 | 9,000 |
| 1,000,001 to 5,000,000 | 10,000 |
| 5,000,001 to 10,000,000 | 10,000 |
| 10,000,001 and later | 10,000 |

Within each band, small categories are protected so they still appear in the sample. Details are in `SAMPLING_METHODOLOGY.md`.

## Suggested uses

- Studying how multi-purpose domains carry several labels.
- Comparing category mixes between popular and long-tail domains.
- Testing joins between DNS or proxy logs and a category reference.
- Teaching domain classification and content-filtering policy.
- Integration testing for filtering, analytics and security software.

Unweighted counts describe this sample, not the whole internet, because some rank ranges and small categories are deliberately oversampled.

## Related domain datasets

- **AI tools.** General categories file most AI products under technology. For a finer view, see this [list of AI URLs to block](https://www.aitoolsblocklist.com/) and its AI-specific taxonomy.
- **Unsanctioned AI use.** Joining logs to categories is also how organisations measure AI adoption; see the [difference between shadow AI and shadow IT](https://www.shadowaitools.com/detection-methodology.php).
- **Content topics.** A [domain enrichment API](https://www.websitecategorizationapi.com/api-docs.php) returns IAB content topics for domains and pages.
- **Bulk category files.** Projects that need to [categorize websites](https://www.urlcategorizationdatabase.com/) in bulk can use a downloadable URL category database.
- **Autonomous agents.** The [AI incident database](https://www.aiagentallowlist.com/ai-agent-incidents-report.php) for agents documents cases where page-level controls mattered.
- **Schools and enterprises.** Packages for [content filtering software for schools](https://www.webfilteringdatabase.com/pricing.php), businesses and OEM partners are on the product site.

## Limitations

- A static snapshot. Domains change purpose and ownership, so labels can go out of date.
- The unit is a domain, not a URL. Large domains can host very different content.
- Categories describe content; each organisation still decides what to allow or block.

## Citation

Alpha Quantum (2026). *Web Filtering Research Dataset: 49,000 Categorized Domains* (Version 1.0.0). Web Filtering Database. Figshare.

Include the DOI shown on the Figshare record.

## Licence

CC BY 4.0. See `LICENSE.txt`. The licence covers this sample only, not the complete commercial database.