How unknown URLs are analyzed, classified, and scored in under 500 milliseconds using parallel ML pipelines, content analysis, and enriched metadata
Try Live ClassificationOur pre-classified database covers over 100 million domains, but the web grows by tens of thousands of new domains every day. When a query arrives for a domain not yet in our database, the real-time classification pipeline activates.
Once classified, the result is written into the permanent database and becomes available for instant cache-hit responses on all subsequent queries. This means the database is continuously growing, learning from every new domain encountered in production traffic.
Real-time classification of never-before-seen domains is central to how K-12 schools implement content filtering that keeps pace with new sites students discover daily.
Six parallel and sequential stages transform an unknown URL into a fully classified entry
The first stage resolves the domain's DNS records (A, AAAA, MX, TXT, NS) and queries WHOIS data. This reveals the registrar, registration date, hosting provider, nameserver configuration, and geographic location.
A headless Chromium browser fetches the URL with a standard user agent, executing JavaScript to capture dynamically loaded content. The page is rendered for 3 seconds to allow single-page applications and lazy-loaded elements to appear.
Extracted text content is processed through a multilingual transformer model fine-tuned on web content classification.
The page screenshot is processed by a convolutional neural network trained on millions of labeled website screenshots.
The domain's position within the web graph provides powerful classification signals.
Signals from all five preceding stages are fed into a gradient-boosted ensemble model that produces the final category assignments and classification details.
Real-time classification performance measured across production traffic
Transformer models extract topic signals from page content in 12 languages
CNN classifiers evaluate screenshots, logos, and layout patterns
GNN maps domain relationships via hyperlinks and shared infrastructure
Classification accuracy is measured continuously against a held-out validation set of 500,000 human-labeled domains that is refreshed quarterly.
Every classification produced by the real-time pipeline enters a feedback loop, and the accumulated data is valuable for identifying patterns in owner-operated businesses approaching retirement.
When users submit category corrections through the API suggestion endpoint, corrections are reviewed and incorporated into the training dataset.
We retrain the ensemble model weekly and the deep learning components monthly, so accuracy improves continuously as the system encounters new content patterns.
Active learning selects the most informative uncertain classifications for human review, focusing analyst time on edge cases where model improvement is greatest. This hybrid human-in-the-loop approach ensures our 59 content categories maintain consistent quality even as the web evolves.
How we achieve sub-500ms classification for unknown domains at global scale
DNS lookup, WHOIS query, and content fetch launch simultaneously. NLP and vision models run in parallel on GPU-accelerated inference servers. Total latency is bounded by the slowest stage (content rendering) rather than the sum of all stages.
Production models are distilled from larger teacher models into compact student networks optimized for inference speed. Our NLP model achieves 95% of the teacher's accuracy in one-tenth the inference time using knowledge distillation and quantization.
Classification infrastructure is deployed at 15 global edge locations. Content is fetched from the geographically nearest node to the target server, minimizing network round-trip time for the content rendering stage.
If high-confidence signals emerge from early pipeline stages (e.g., DNS matches a known unsafe content nameserver), the system returns a classification immediately without waiting for slower stages like content rendering.
Certificate Transparency logs and newly observed domains from passive DNS are pre-classified in batch during off-peak hours, reducing the chance that a domain requires real-time analysis when first queried by a customer.
Each pipeline stage has a configurable timeout. If content rendering times out at 3 seconds, the system proceeds with whatever signals are available. Partial-signal classifications are flagged for asynchronous re-analysis later.
Leverage our real-time classification pipeline to filter unknown domains with 97.3% accuracy and sub-500ms latency across 59 content categories — the real-time layer behind modern enterprise web protection.
Start Classifying Now