Carrier-Grade Domain Categorization for Clean-Pipe Services, Subscriber Management, and Regulatory Compliance Across Millions of Concurrent Users
ISPs operate at a scale where every millisecond matters. With tens of billions of DNS queries per day across millions of subscribers, the web filtering database delivers sub-millisecond categorization through in-memory local mirrors and 15-minute incremental delta updates.
Transform web filtering from a cost center into a revenue-generating subscriber service
Clean-pipe services represent one of the most compelling revenue opportunities for ISPs.
A typical clean-pipe offering includes three tiers.
The base tier, often included free with the connection, blocks unsafe content, and command-and-control domains -- protecting the ISP's network and reducing support burden.
The family tier adds parental control categories including adult content, gambling, violence, and social media restrictions.
The premium tier provides full category control through a subscriber portal, allowing customization of all 59 content categories with time-based scheduling and per-device policies.
Premium filtering tiers add $2-5/month ARPU with minimal infrastructure cost
unsafe content blocking reduces infection-related support calls by up to 40%
Family safety features reduce churn by giving parents a reason to stay
Per-subscriber policy management at scale with self-service portals
ISP-scale filtering requires the ability to maintain individual policies for millions of subscribers simultaneously. Each household may have different filtering preferences, and within a household, different devices may require different policies. A teenager's phone should have stricter filtering than a parent's laptop.
A white-labeled subscriber portal reduces support costs by enabling customers to manage their own filtering settings.
Carrier-grade technical considerations for reliable, scalable content filtering
Use BGP announcements to redirect DNS traffic from subscriber-facing routers to the filtering DNS cluster.
Carrier-Grade NAT (CGNAT) complicates subscriber identification because multiple subscribers share a single public IP.
The filtering cluster scales horizontally by adding DNS resolver nodes behind the anycast address.
ISP filtering infrastructure must achieve 99.999% availability because DNS failures affect all subscriber internet access.
Each filtering node handles 200,000+ queries per second with p99 latency under 1ms using in-memory database lookups. A four-node cluster provides 800K+ QPS with N+1 redundancy, sufficient for an ISP serving 2-3 million subscribers.
The full 90-million-domain database requires approximately 8GB of RAM when loaded into an optimized trie structure. Combined with resolver cache and operating system overhead, plan for 32GB per filtering node to ensure headroom for growth.
Incremental database updates (IXFR) propagate to all nodes within 15 minutes. Emergency content risk updates can be pushed in under 5 minutes through an out-of-band notification channel. Nodes apply updates atomically without service interruption.
ISPs face unique regulatory obligations around content filtering that differ significantly from enterprise requirements. These obligations vary by jurisdiction but share common themes: protecting minors, blocking illegal content, and providing transparency to subscribers.
Requires ISPs to implement measures to prevent children from accessing pornographic content online. The Act mandates age verification mechanisms and content filtering as part of a layered approach to child safety. ISPs that fail to comply face significant fines from Ofcom.
The eSafety Commissioner can issue blocking notices requiring ISPs to prevent access to specific domains hosting illegal content. The web filtering database supports rapid integration of these blocking notices into the filtering infrastructure.
While not mandating filtering directly, the DSA requires ISPs to act on valid takedown notices and implement measures against the dissemination of illegal content. Proactive filtering demonstrates good-faith compliance and reduces the volume of reactive takedown requests.
ISPs that serve American schools and libraries must be able to support CIPA and E-Rate compliant filtering for school districts, since those customers are legally required to filter internet access as a condition of their federal funding.
Content filtering must be implemented transparently and with subscriber consent. Subscribers must be informed about what categories are filtered, have the ability to opt out (where legally permitted), and filtering must not discriminate based on the source of content or favor the ISP's own services.
Integrate carrier-grade domain categorization into your subscriber management pipeline
Classify subscriber DNS requests against 59 content categories and enforce per-subscriber clean-pipe policies in real time.
import requests API_URL = "https://webfilteringdatabase.com/api/moderate.php" API_KEY = "YOUR_API_KEY" # Subscriber tier policies TIERS = { "base": {"Malware", "Phishing", "Botnet"}, "family": {"Malware", "Phishing", "Adult Content", "Gambling", "Violence", "Drugs"}, "premium": {"Malware", "Phishing", "Adult Content", "Gambling", "Violence", "Drugs", "Social Media", "Streaming"} } def filter_subscriber_request(domain, tier): result = requests.post(API_URL, json={ "api_key": API_KEY, "query": domain }).json() categories = set(result.get("categories", [])) blocked = TIERS.get(tier, TIERS["base"]) violations = categories & blocked if violations: return "BLOCK", f"Policy: {tier} | {', '.join(violations)}" return "ALLOW", result.get("primary_category", "Uncategorized") # Simulate subscriber DNS requests domains = [ "streaming-service.com", "adult-site.xxx", "news.com", "social-media.com" ] for d in domains: action, reason = filter_subscriber_request(d, "family") print(f"{action}: {d} — {reason}")
const API_URL = 'https://webfilteringdatabase.com/api/moderate.php'; const API_KEY = 'YOUR_API_KEY'; const TIER_POLICIES = { base: new Set(['Malware', 'Phishing', 'Botnet']), family: new Set(['Malware', 'Phishing', 'Adult Content', 'Gambling', 'Violence', 'Drugs']), premium: new Set(['Malware', 'Phishing', 'Adult Content', 'Gambling', 'Violence', 'Social Media', 'Streaming']) }; async function filterSubscriberRequest(domain, tier) { const res = await fetch(API_URL, { method: 'POST', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify({ api_key: API_KEY, query: domain }) }); const data = await res.json(); const blocked = TIER_POLICIES[tier] || TIER_POLICIES.base; const violations = (data.categories || []) .filter(c => blocked.has(c)); if (violations.length) { return { action: 'BLOCK', reason: violations.join(', ') }; } return { action: 'ALLOW', category: data.primary_category }; } // Check subscriber requests ['streaming-service.com', 'adult-site.xxx', 'news.com', 'social-media.com'] .forEach(async d => { const r = await filterSubscriberRequest(d, 'family'); console.log(`${r.action}: ${d}`); });
# Check a streaming domain for subscriber filtering curl -X POST "https://webfilteringdatabase.com/api/moderate.php" \ -H "Content-Type: application/json" \ -d '{ "api_key": "YOUR_API_KEY", "query": "streaming-service.com" }' # Response: {"primary_category":"Streaming","risk_level":"low"} # Check an adult domain curl -X POST "https://webfilteringdatabase.com/api/moderate.php" \ -H "Content-Type: application/json" \ -d '{ "api_key": "YOUR_API_KEY", "query": "adult-site.xxx" }' # Response: {"primary_category":"Adult Content","risk_level":"high"} # Check a news domain curl -X POST "https://webfilteringdatabase.com/api/moderate.php" \ -H "Content-Type: application/json" \ -d '{ "api_key": "YOUR_API_KEY", "query": "news.com" }' # Response: {"primary_category":"News","risk_level":"low"} # Check a social media domain curl -X POST "https://webfilteringdatabase.com/api/moderate.php" \ -H "Content-Type: application/json" \ -d '{ "api_key": "YOUR_API_KEY", "query": "social-media.com" }' # Response: {"primary_category":"Social Media","risk_level":"low"}
<?php $apiUrl = 'https://webfilteringdatabase.com/api/moderate.php'; $apiKey = 'YOUR_API_KEY'; $tiers = [ 'base' => ['Malware', 'Phishing', 'Botnet'], 'family' => ['Malware', 'Phishing', 'Adult Content', 'Gambling', 'Violence', 'Drugs'], 'premium' => ['Malware', 'Phishing', 'Adult Content', 'Gambling', 'Violence', 'Social Media'] ]; function filterSubscriberRequest($domain, $tier, $apiUrl, $apiKey, $tiers) { $ch = curl_init($apiUrl); curl_setopt($ch, CURLOPT_POST, true); curl_setopt($ch, CURLOPT_POSTFIELDS, json_encode([ 'api_key' => $apiKey, 'query' => $domain ])); curl_setopt($ch, CURLOPT_HTTPHEADER, ['Content-Type: application/json']); curl_setopt($ch, CURLOPT_RETURNTRANSFER, true); $result = json_decode(curl_exec($ch), true); curl_close($ch); $blocked = $tiers[$tier] ?? $tiers['base']; $violations = array_intersect($result['categories'] ?? [], $blocked); if (!empty($violations)) { return ['BLOCK', implode(', ', $violations)]; } return ['ALLOW', $result['primary_category'] ?? 'Unknown']; } // ISP subscriber DNS request filtering $domains = ['streaming-service.com', 'adult-site.xxx', 'news.com', 'social-media.com']; foreach ($domains as $d) { [$action, $reason] = filterSubscriberRequest($d, 'family', $apiUrl, $apiKey, $tiers); echo "$action: $d — $reason\n"; } ?>
Partner with us to bring 100 million categorized domains to your subscriber base. Customizable clean-pipe services, white-label portals, and ISP-grade SLAs.