URL Category Data for Threat Research and Intelligence Teams
Threat researchers spend much of their time deciding which domains deserve a closer look. A reliable category for every domain answers the first question fast: what kind of site is this, and is it normal to see it here?
This guide is for research and intelligence teams inside security vendors and service providers. It covers where category data fits in research workflows, which categories carry signal, and how to keep the data current.
login-verify-portal.examplePhishingquiet-harbor-domain.exampleParkedfiles.storage-host.exampleFile Hostingtiny-link.example/9FqURL ShortenerWhere category data fits in threat research
Research teams use category data at almost every stage, from first triage to the report that goes out. These are the places it most often plugs in.
Indicator enrichment
Every domain indicator gets ranked categories, so analysts see at a glance whether it is a file host, a parked page or a known bad site.
Noise filtering
Established business, news and government sites drop out of large domain sets, leaving the unusual ones to study.
Infrastructure pivots
Categories on related domains show whether a cluster is hosting, parking, redirects or live lures.
Campaign clustering
Groups of domains that move from parked to phishing on similar dates often belong to one operation.
Sandbox triage
Contacted domains from detonations are labelled, so analysts skip the update servers and focus on the odd calls.
Detection model training
Categories act as features or weak labels for models that score new domains and URLs inside your products.
Hunting queries
Hunts can ask for rare categories in a customer's traffic, such as remote access tools or anonymizers.
Trend research
Category counts over time show which site types attackers favour, from file hosts to AI-themed lures.
Published findings
Aggregate category statistics give threat reports a clear, repeatable way to describe infrastructure.
Cutting a large domain set down to what matters
A typical research question starts with far too many domains, for example every domain seen in a week of telemetry. Category data removes most of them before an analyst opens a single record; the proportions below are illustrative.
Categories that carry signal for researchers
Most categories describe what a site is about, and only a few are risk labels. For research, the context categories often matter as much as the direct ones; definitions for all 59 are on the category list, and the mapping to IAB is in the taxonomy.
| Category | Role | Why researchers care |
|---|---|---|
| Malware / Phishing / Suspicious | Risk label | Direct confirmation or a starting point for pivots to sibling domains. |
| Parked / For-Sale Domains | Staging context | Many attack domains sit parked before use. A switch from parked to live content is a classic early warning. |
| CDN / Hosting / Infrastructure | Shared platform | Tells you a domain serves many tenants, so the verdict depends on the path and the account, not the domain. |
| Cloud Storage / File Hosting | Shared platform | Frequent payload and lure hosting on trusted names. Flags where content level analysis is needed. |
| URL Shorteners | Redirect context | Hides the true target. Researchers expand and score the final hop. |
| Developer Tools / Code Hosting | Shared platform | Code hosts and paste services are used to stage scripts and configuration for malware. |
| Remote Access Tools | Tradecraft context | Legitimate tools that show up in support scams and in hands-on intrusions. |
| VPN / Proxy / Anonymizers | Tradecraft context | Points to evasion, data exfiltration routes or policy bypass inside a network. |
| Cryptocurrency | Theme context | A frequent lure theme for investment scams and wallet drainers. |
| Web-based Applications | Shared platform | Form builders and site builders host many credential harvesting pages. |
| AI / LLM Tools | Theme context | AI-themed lures and fake AI tool downloads are a growing research topic. |
Pivoting across attacker infrastructure
Categories are most useful when they change. Watching a group of related domains move through categories tells a story about how an operation prepares and runs.
Before use
- Fresh registrations in bulk
- Parked or for-sale pages
- Similar naming patterns
- No content or template content
During use
- Category changes to phishing or suspicious
- Redirects through shorteners
- Payloads on file hosts
- Brand themes in content
After use
- Content removed or parked again
- Domain left to expire
- New siblings appear
- Same pattern, new names
Training detection models with category data
Research teams increasingly build models that score domains and URLs inside their products. Category data helps in three ways, and has limits worth keeping in mind.
How it helps
- Features: the primary and secondary categories add strong context to lexical and network features
- Weak labels: benign categories on established sites give a large, varied negative set
- Stratified evaluation: measure precision per category, so a model that over-flags file hosts is easy to spot
- Drift checks: watch the category mix of new detections over time
Watch out for
- Benign categories on shared platforms do not make every page benign
- Very new domains may not have content yet, so rely on rechecks
- Leakage, when the category itself encodes the label you want to predict
- Licence scope: training for your own products is fine, publishing the data is not
Delivery: data lake files, API and real-time lookup
Research work mixes huge batch jobs with quick single lookups. The data comes in forms that suit both, and most teams use all three; details of files and fields are on the web filtering database page.
Files in your data lake
CSV, Parquet or JSON, loaded beside your telemetry.
- Join billions of events to categories
- Keep your own dated copies for retro analysis
- Runs fully inside your environment
Lookup API
Included with licences for notebooks, platforms and analyst tools.
- Single and batch requests
- Fills gaps for domains not in your copy
- See the API documentation
Real-time enrichment
A local index inside sandboxes and pipelines that label as events arrive.
- Labels contacted domains during detonation
- Daily deltas keep the index fresh
- See streaming classification
Coverage and freshness in numbers
Research depends on two things: that the domain you care about has a label, and that the label is recent. These are the figures that matter for both.
categorized domains in the database
of web traffic and usage covered by the 20 million most visited domains
newly registered domains categorized every day
web filtering categories plus the IAB Content Taxonomy in one file
Research examples
These examples show typical research questions and how category data shortens the path to an answer. Each one combines categories with evidence the team already holds.
Lure hosting survey
Campaign analysisA team collects the landing domains from a month of phishing reports. Grouping them by category shows how many sat on file hosts, site builders or fresh domains.
The result guides which detection features to improve next quarter.
Sleeping domains
Early warningResearchers track a set of lookalike domains that are parked today. When a recheck moves one to a live login page, the team publishes a detection before the first wave of mail.
Sandbox noise
Malware triageDetonations contact dozens of domains, most of them update, analytics and CDN services. Labelling them leaves two odd domains in a rare category for an analyst to study.
False positive review
Detection qualityA new detection rule fires too often. Splitting its hits by category shows most come from one shared platform, and the rule is narrowed to the path pattern instead.
Remote access in intrusions
Incident researchAcross many incident cases, the team counts how often Remote Access Tools domains appear before data theft. The finding becomes a hunting query shipped to customers.
AI-themed lures
Trend researchA team compares fake AI tool download sites with the real AI services in the AI / LLM Tools category. Lookalike names that sit outside the known set become the starting point for a new detection.
Questions a category answers during an investigation
Analysts ask the same handful of questions about every unfamiliar domain. The table shows which part of the category record answers each one, and what to check next.
| Analyst question | What the record shows | Next step |
|---|---|---|
| What is this site? | The primary category, plus secondary categories ranked by how much of the site they describe. | Compare with what the indicator was doing in your telemetry. |
| Is it a shared platform? | Hosting, file hosting, code hosting, site builder or shortener categories. | Move the analysis from the domain to the path or account. |
| Is it live or waiting? | A parked or for-sale label, or no content category yet. | Add it to a watch list and check after the next recheck. |
| Has it changed recently? | A different category in today's delta than in your last copy. | Diff your dated copies and review the change. |
| Does it fit this network? | Category compared with what is normal for the customer's sector. | Rare categories for that customer go to the top of the queue. |
| What content theme does it use? | IAB Content Taxonomy labels such as personal finance or travel. | Group lures by theme for campaign tracking. |
Common mistakes with category data in research
Category data is easy to misuse when deadlines are tight. These are the errors we see most often in research pipelines, with a simple fix for each.
Allowlisting by category
Treating every Business site as safe hides compromised small business sites. Use benign categories to lower priority, not to close cases.
Stale snapshots
A copy loaded once and never updated misses most new attack domains. Schedule the daily delta job before anything else.
Ignoring the host level
A subdomain on a shared platform can differ from its parent. Look up the full host as well as the registered domain.
Reading "unknown" as "safe"
A domain without a category is often new or empty. Treat it as a reason to look closer and recheck soon.
Mixing taxonomies
Web filtering categories and IAB labels answer different questions. Keep them in separate fields in your schema.
One label, one verdict
Sites with several categories need all of them. A news site with a file sharing section behaves differently from a pure news site.
Getting started
Research teams usually begin with a batch load into the data lake and add real-time enrichment once the value is clear. These steps cover a first production setup.
Load
Import the Parquet file and key it by registered domain and host.
Join
Join telemetry, sandbox output and indicator stores to categories.
Filter
Write saved views that drop established benign sites and isolate rare categories.
Track
Apply daily deltas and alert on category changes for domains you watch.
Enrich
Show categories in your analyst console and threat platform records.
Model
Add categories as features in detection models and evaluate per category.
Ship
Move proven research into product detections under an OEM licence.
Report
Publish aggregate findings that describe infrastructure by category.
Licence scope for research teams
Research work often starts internal and ends up inside a product. Choosing the right licence early saves a second negotiation later; terms for embedding are on the OEM licensing page.
Internal use licence
- Enrichment, hunting and analysis by your own team
- Training internal models and tools
- Aggregate statistics in published reports
- Lookup API included
OEM or group-wide licence
- Categories shipped inside detections and products
- Labels shown to customers in consoles
- Several product lines under one agreement
- No resale or publication of raw data
Threat research FAQ
How do threat researchers use URL category data?
They enrich indicators, filter benign noise out of large domain sets, pivot across infrastructure, cluster campaigns and label training data for detection models.
Is a URL category the same as a threat verdict?
No, most categories describe what a site is about, while Malware / Phishing / Suspicious is a direct risk label. Context categories such as parked, hosting or shorteners are combined with your own evidence.
Can category data reduce false positives in detection research?
Yes. Removing established benign sites leaves a much smaller set to review, and flags shared platforms where a benign domain hosts abused content.
Which delivery format suits a research data lake?
Parquet loads directly into most analytics engines and data lakes. CSV and JSON are also available, and the lookup API included with licences covers domains your copy does not hold.
How fresh is the data for tracking new campaigns?
About 300,000 newly registered domains are categorized every day as daily delta files. Each is checked again after a week and a month, and all other domains are refreshed monthly.
Can we use the categories to label training data for machine learning?
Yes, within your licence. Categories are common features or weak labels for detection models inside your own products, while publishing or reselling the raw data is not allowed.
Can we cite category statistics in published threat reports?
Aggregate findings, such as the share of a campaign's domains that were parked before use, are normally fine. Publishing the underlying records is not, so check your licence terms.
Does the data cover more than one taxonomy?
Yes. Each domain carries the 59 web filtering categories and the IAB Content Taxonomy in one file, with several categories ranked by how much of the site they describe.
Should we look up the full host or only the registered domain?
Both. Shared platforms and large sites can carry different categories on different hosts, so look up the full host first and fall back to the registered domain when the host has no record of its own.
What should a researcher do with a domain that has no category?
Treat it as a signal rather than a blank. Uncategorized domains are often new, empty or short-lived, so add them to a watch list, query the API for a fresh result and check again after the next daily delta.
Put a category on every domain you study
Tell us about your research pipeline, data lake and the products your findings feed. We will suggest the data layout and licence that fit.
License the database OEM licensing