# Sampling Methodology

## Release objective

The release provides broad, documented coverage for structured analysis: domains from across the popularity range, with every category represented. Domains ranked in the top 1,000 of the source extract are not included.

## Source

The sample was drawn from a snapshot of 20,128,938 classified domains, ordered by popularity. Each record has a domain, its ordered category labels and a classification version. All rows used in this release carry `version_2026_03_15`. The complete commercial database covers up to 120 million domains and the same 59 categories.

## Stratified sample: 49,000 domains

The sample is divided among six source-rank bands, starting after rank 1,000:

| Popularity band | Sample allocation |
|---|---:|
| Source ranks 1,001-10,000 | 4,000 |
| Source ranks 10,001-100,000 | 6,000 |
| Source ranks 100,001-1,000,000 | 9,000 |
| Source ranks 1,000,001-5,000,000 | 10,000 |
| Source ranks 5,000,001-10,000,000 | 10,000 |
| Source ranks 10,000,001 and later | 10,000 |
| **Total** | **49,000** |

Within each popularity band, the first ordered category label is used as the mutually exclusive `primary_category` stratum. This does not discard the other labels; all original labels remain in the `categories` field.

For every available primary-category stratum in a band, the allocation first protects up to 25 observations. The remaining band capacity is distributed proportionally to the remaining category-stratum sizes using largest remainders, subject to the number of available rows. This approach improves the visibility of smaller categories without turning the sample into an equal-size category benchmark.

Rows are then selected uniformly at random inside every band/category stratum, using the seed `20260922`.

## Duplicate handling

The release contains unique domains. If the same normalized domain is selected more than once, the earliest selected occurrence is preserved. The later duplicate is replaced by an eligible unique domain from the same popularity-band and primary-category stratum. The released build required three such replacements.

## Multi-label treatment

The source can assign several ordered labels to one domain. The complete ordered list is retained in `categories`, separated by `|` in CSV and stored as the same delimited string in JSONL for parity. The first label is copied to `primary_category` solely to produce mutually exclusive sampling strata and simple grouped analyses.

Researchers studying the semantic coverage of a domain should use all labels. Researchers requiring one grouping variable may use `primary_category`, provided they state that it is the first ordered source label rather than an independently produced single-label classification.

## Reproducibility

The seed, band allocations and per-stratum quotas are recorded in `dataset-analysis.json`. The commercial source file is not included, so reproducing the exact sample requires authorized access to the same source snapshot.

## Interpretation boundaries

The popularity bands are based on the ordering of the source extract. `source_rank` is suitable for reproducing that ordering and comparing bands inside this release. It should not be described as a current traffic count or as a rank from an external measurement provider.

The sample intentionally oversamples some rank ranges and protects small category strata. Unweighted totals therefore describe the released research collection, not the complete source database or the internet. Researchers estimating source-wide prevalence would require the full stratum population sizes and appropriate weights.

Domain-level labels describe a registrable destination at the granularity supplied in the source. They do not guarantee that every URL under that domain has the same subject or policy relevance. Categories are descriptive inputs for filtering systems; final allow/block decisions depend on the deployment's users, policy, jurisdiction, and tolerance for risk.