webfilteringdatabase.com
Login Try Free Lookup
Home Find Your Solution
Features
Domain Categorization API Real-Time Classification 59 Filtering Categories Offline Database (100M) ML Classification Content Classification
Industries
K-12 Schools Corporate Healthcare Government ISPs
Tools
Domain Lookup Bulk Categorization Category Explorer
Resources
Pricing API Documentation Login / Sign Up
Knowledge Base

Machine Learning Classification

Inside the NLP transformers, computer vision models, graph neural networks, and ensemble methods that classify 100 million domains across 59 categories

Explore ML-Powered API
NLP Transformers
Computer Vision Models
Graph Neural Networks
Metadata Signals
Learned Ensemble
Continuous Learning
Architecture

Multi-Model Classification Architecture

Classifying the web accurately requires more than a single machine learning model. Different types of web content present different signals:

  • A news article is best identified by its text
  • A restricted content page by its visual similarity to a legitimate brand
  • A newly created botnet domain by its network neighborhood

Our architecture deploys specialized models for each signal type and combines their outputs through a learned ensemble. Each model produces independent category probability distributions that are fused by the ensemble layer.

These same web signals can indicate business owner retirement patterns when analyzed through a commercial lens.
The system consists of four core model families:

Multilingual NLP Transformer

For text analysis

Convolutional Neural Network

For visual content classification

Graph Neural Network

For structural web-graph analysis

Feature-Based Gradient Boosted Model

For metadata signals like domain age, DNS configuration, and traffic patterns

97.3% Top-1 accuracy — this multi-model approach on our benchmark
89.1% Text-only
91.4% Best single-model baseline

The ensemble also provides calibrated uncertainty estimates, flagging domains where model disagreement suggests the need for human review as part of our real-time classification pipeline.

Model Families

Core Model Architectures

Four specialized model families work together to analyze every dimension of web content

01

NLP Transformer (Text Analysis)

Our text classification model is a fine-tuned multilingual transformer with 110 million parameters, trained on a curated corpus of 12 million labeled web pages across 12 languages.

  • The model processes page titles, meta descriptions, heading text, body content, and anchor text to produce a 58-dimensional probability vector over content categories.
  • Attention mechanisms allow the model to focus on the most discriminative phrases: a single mention of "checkout" and "add to cart" strongly activates the Shopping category even in a page dominated by informational text.
  • The model handles code-switching (mixed-language content) and achieves 96.8% accuracy on English content and 94.2% on non-English content.
02

Convolutional Neural Network (Visual Analysis)

A ResNet-50 backbone fine-tuned on 8 million labeled website screenshots powers visual classification.

  • The model learns to recognize visual patterns that strongly correlate with categories: product grid layouts for Shopping, article typography for News, video player interfaces for Streaming, and brand-impersonation layouts for restricted content.
  • Transfer learning from ImageNet provides strong feature extraction for general visual concepts, while domain-specific fine-tuning teaches the model web-specific patterns.
  • The visual model is particularly valuable for adult content detection (98.9% recall) and restricted content detection where attackers clone legitimate page visuals.
03

Graph Neural Network (Structural Analysis)

The web is a graph, and a domain's neighbors reveal its nature.

  • Our GraphSAGE-based model operates on a web graph of 500 million edges connecting domains through hyperlinks, shared hosting IPs, shared SSL certificates, and common JavaScript library fingerprints.
  • The GNN propagates category information through 3 hops of message passing, allowing newly observed domains to inherit category signals from their established neighbors. A domain that links to and is linked from established news sites is very likely itself a news site.
  • The GNN achieves 92.1% accuracy as a standalone model and provides the strongest signal for newly created domains with minimal text content.
04

Gradient Boosted Metadata Model

A LightGBM model processes over 200 engineered features extracted from domain metadata:

WHOIS registration age registrar reputation score nameserver configuration entropy hosting ASN reputation TLS certificate attributes HTTP header patterns DNS record types Alexa/Tranco popularity rank geographic hosting location historical category stability
  • This model excels at identifying web-based risks where metadata patterns are more reliable than content analysis (a newly registered domain on a bulletproof host with a free DV certificate is suspicious regardless of its content).
  • These metadata features also power analyses such as retirement-age business owner statistics drawn from web infrastructure patterns.
110M Parameters

Multilingual transformer trained on 12M labeled web pages in 12 languages

8M Training Screenshots

ResNet-50 visual classifier fine-tuned on labeled website screenshots

500M Graph Edges

GraphSAGE model operating on hyperlink and infrastructure connections

Fusion Layer

Ensemble Fusion and Calibration

The ensemble layer is not a simple average of model outputs. It is a learned meta-model (a small neural network) that takes the 59-dimensional probability vectors from all four base models along with their internal uncertainty estimates, and produces the final fused classification.

The meta-model learns which base models to trust for which categories:
  • The text model dominates for News and Education
  • The visual model dominates for Adult Content and restricted content
  • The graph model dominates for newly registered domains
  • The metadata model dominates for security content risk categories

Confidence calibration ensures that when the ensemble reports 90% confidence, approximately 90% of those predictions are actually correct. We use temperature scaling and Platt calibration, validated on a held-out calibration set of 100,000 domains.

Well-calibrated classification details are critical because downstream filtering policies use confidence thresholds: a miscalibrated model that reports inflated confidence would cause either over-blocking or under-blocking.

The ensemble also produces a model disagreement score. When the NLP model says "News" but the visual model says "Adult Content," the disagreement signal triggers routing to a human analyst queue. This active learning strategy focuses human effort on the most informative edge cases, continuously improving model accuracy where it matters most.

Continuous Learning

Training Data and Continuous Learning

How models are trained, validated, and improved through production feedback loops

Training Pipeline

Model training begins with a labeled dataset of 20 million domains curated through a combination of expert human annotation, semi-supervised labeling from high-confidence model predictions, and web directory ground truth.

The dataset is stratified to ensure adequate representation across all 59 categories, with oversampling for rare categories like Cryptomining and Dynamic DNS.
We train models on a distributed GPU cluster using mixed-precision training for efficiency.
  • The NLP transformer is fine-tuned from a pre-trained multilingual checkpoint, requiring approximately 72 GPU-hours per training run.
  • The CNN model trains in 48 GPU-hours, the GNN in 24 GPU-hours, and the metadata model trains in under 1 hour on CPU.
  • The ensemble meta-model trains in minutes once base model predictions are computed.

Weekly Model Updates

The gradient boosted metadata model and ensemble layer are retrained weekly to incorporate new labeled data from human review and customer feedback.

Monthly Deep Model Updates

The NLP transformer and CNN are retrained monthly with expanded datasets, capturing evolving web content patterns and new website design trends.

Quarterly GNN Updates

The graph neural network is retrained quarterly as the web graph structure evolves, with new edges from web crawling and DNS observation.

Continuous Active Learning

Uncertain predictions are routed to human analysts daily. Their corrections flow into the training set, creating a feedback loop that targets the model's weakest areas.

Benchmarks

Performance and Accuracy Benchmarks

Measured on a held-out benchmark of 500,000 human-labeled domains refreshed quarterly

97.3% Top-1 Accuracy

The ensemble model's primary category prediction matches the human label 97.3% of the time. This represents a 8.2 percentage point improvement over the best single-model baseline.

99.1% Top-3 Accuracy

The correct category appears in the model's top three predictions 99.1% of the time, making multi-category filtering policies extremely reliable.

99.8% content risk Recall

For security-critical categories (unsafe content, C2), the combined ML and content classification system achieves 99.8% recall with a 0.1% false positive rate.

94.2% Multilingual Accuracy

Non-English content classification accuracy across 12 supported languages, within 2.6 percentage points of English-language performance.

45ms Inference Latency

End-to-end model inference time on GPU from feature extraction to ensemble output. Distilled models achieve 12ms for latency-critical paths.

+1.2% Annual Improvement

Continuous learning from production feedback improves accuracy by approximately 1.2 percentage points per year, compounding as the training set grows.

Leverage State-of-the-Art ML Classification

Access our ML-powered classification engine through the API and benefit from 97.3% accuracy across 59 categories with continuous improvement.

Start Using the API