Submit up to 100 domains per API request and let our infrastructure classify them in parallel, backed by over 120 million categorized domains across 58 content categories.
Batch Processing
Classify up to 100 domains in a single API request with parallel processing for maximum throughput.
Async Webhooks
Fire-and-forget batch jobs with webhook callbacks that notify your endpoint when results are ready.
CSV Upload
Upload CSV files of domains directly and receive classified results in your preferred output format.
Classify a whole backlog
Turn a list of unknown hostnames into labeled rows
Log exports, customer lists and crawl results all end up as long columns of domains with no meaning attached. Our records add the categories, and you choose whether that happens through the API or from files you hold.
Send batches to the real-time API when the list changes often.
Load CSV or JSON files and run the whole join inside your own database.
Use the offline database when the list must never leave your network.
Submit domains in whatever format your workflow produces
1
JSON Array Submission
Submit a JSON array of domain strings directly in the request body for programmatic integrations. This is the fastest path for applications that already have domain lists in memory.
The API accepts arrays of up to 10,000 domains per request, with each domain automatically normalized by stripping protocols, paths, and query parameters to extract the registrable domain.
Duplicate domains are deduplicated server-side to avoid redundant classification work.
2
CSV File Upload
Upload CSV files containing domain lists extracted from spreadsheets, log analysis tools, or database exports.
The API auto-detects the column containing domains by scanning headers for common names like "domain," "url," "hostname," or "site."
For CSVs without headers, you can specify the column index.
Files up to 50 MB are processed synchronously; larger files are handled asynchronously with progress tracking and webhook notification upon completion.
3
Newline-Delimited Text
Pipe raw domain lists from command-line tools directly to the API using plain text with one domain per line.
This format integrates naturally with Unix pipelines, allowing commands like extracting domains from DNS logs and piping them through curl to the batch endpoint.
Comment lines starting with # are ignored, and blank lines are skipped.
Ideal for scripted automation and cron-based periodic classification jobs.
4
URL List Processing
Submit full URLs instead of bare domains when your source data includes complete web addresses.
The API extracts the registrable domain from each URL, handling subdomains, ports, paths, query strings, and fragments transparently.
This means you can feed raw proxy logs, browser history exports, or web crawler outputs directly without preprocessing.
Each result maps back to the original URL for easy correlation.
Processing Modes
Async Processing and Webhook Callbacks
1
Synchronous Mode
For batches of up to 10,000 domains, synchronous mode returns results directly in the HTTP response.
The connection stays open while processing completes, typically finishing in 5 to 30 seconds depending on batch size.
This mode is ideal for interactive applications, dashboards, and workflows where the calling process needs immediate results.
A progress header is sent periodically to prevent client-side timeouts during processing.
2
Asynchronous Mode
Batches exceeding 10,000 domains or explicit async requests return immediately with a job ID and status URL.
Your application can poll the status endpoint to track progress as a percentage, or register a webhook URL to receive a POST notification when processing completes.
The webhook payload includes a download URL for the results file, summary statistics, and any errors encountered during processing.
3
Webhook Configuration
Register webhook endpoints that receive signed POST notifications at key processing milestones: job accepted, 50% complete, processing finished, and results available for download.
Each webhook includes an HMAC-SHA256 signature header computed with your API secret, allowing your server to verify authenticity.
Retry logic delivers failed webhooks up to five times with exponential backoff.
You can configure multiple webhook URLs per job for redundancy or to notify different systems.
When volume is the problem
For millions of rows, bring the data to the job
Instead of sending a huge list out, run the database inside your own systems and join it locally. Lookups answer in milliseconds, and the set grows by about 300,000 categorized domains each day.
Scale predictably with transparent limits and intelligent throttling
Per-Request Limits
Synchronous batches support up to 10,000 domains per request. Async batches accept up to 1 million domains per job. These limits ensure predictable processing times and fair resource allocation across all API consumers. Requests exceeding limits receive a 413 status with guidance on splitting the batch.
Rate Throttling
Batch endpoints are rate-limited separately from single-lookup endpoints. Standard plans allow 60 batch requests per minute with up to 10 concurrent async jobs. Enterprise plans offer configurable concurrency up to 100 simultaneous jobs. Rate limit headers in every response show remaining quota and reset timestamps.
Monthly Domain Quotas
Each plan includes a monthly domain classification quota shared between single and batch lookups. Usage dashboards and threshold alerts help you monitor consumption.
Optimal Batch Sizing
While the API accepts up to 10,000 domains synchronously, optimal throughput is achieved with batches of 1,000 to 5,000 domains. Smaller batches reduce latency per request, allowing parallelism on the client side. For maximum throughput, submit multiple medium-sized batches concurrently rather than one maximum-sized batch.
Quota efficiency: Batch processing consumes one quota unit per unique domain classified, so submitting the same domain in multiple batches within a billing cycle only counts once.
Output Formats
Response Formats and Result Structure
Get classification results in the format that fits your data pipeline
JSON Response
The default response format returns a JSON object with each domain as a key mapping to its classification result.
Each result includes the primary category, classification detail, all matching categories with individual scores, category level indicator, and timestamp of last classification update.
Nested JSON preserves the full category taxonomy from our 58-category system including IAB tier mappings.
CSV Response
Request CSV output for direct import into spreadsheets, databases, or data warehouse ETL pipelines.
Each row contains the domain, primary category, classification detail, secondary categories, and category level.
Column headers match common BI tool conventions.
CSV responses for async jobs are delivered as downloadable files with gzip compression, reducing transfer size by up to 85 percent for large result sets.
NDJSON Streaming
Newline-delimited JSON format streams results one domain per line, enabling processing to begin before the full response arrives.
This format works naturally with streaming JSON parsers and command-line tools like jq.
Each line is a complete JSON object, so partial responses are still valid and parseable.
Ideal for piping results directly into downstream processing steps.
Enriched Output Mode
The enriched response mode adds metadata beyond basic classification: domain age, registrar information, DNS record summary, SSL certificate status, historical category changes, and related domains sharing the same infrastructure.
This comprehensive view supports content classification workflows where classification alone is insufficient and analysts need the full domain context to make informed decisions.
What a batch returns
Sample rows from a mixed list of hostnames
Real traffic logs mix shops, references, infrastructure and shorteners in a single column. Here is how our data labels six of them.
How organizations leverage batch classification across industries
content classification Enrichment
Security operations centers feed indicator lists, classification feeds, and suspicious domain extractions through the bulk API to categorize and prioritize content risks.
Domains classified as unsafe content, or Command-and-Control are escalated automatically while benign categories are deprioritized from analyst review queues.
The same bulk classification approach helps analysts find companies with owners nearing retirement by profiling web footprints at scale.
Ad Tech Brand Safety
Advertising platforms classify publisher domain inventories to ensure ads appear only alongside brand-safe content. Bulk lookup categorizes entire exchange inventories overnight, flagging domains in categories like Adult, Violence, or Hate Speech so demand-side platforms can apply advertiser exclusion lists before bidding begins.
Log Analysis and Auditing
Compliance teams export DNS query logs or proxy access logs, extract unique domains, and batch-classify them to identify policy violations. Monthly audit reports reveal category distributions, flag access to restricted categories, and track trends in web usage patterns across the organization over time.
Building Robust Batch Processing Pipelines
1
Idempotent Job Submission
Production batch workflows should implement idempotent job submission using client-generated request IDs to safely retry failed submissions without creating duplicate jobs. These same pipeline patterns apply when constructing an acquisition pipeline for aging business owners that processes thousands of business domains.
2
Partition Large Lists
Partition large domain lists into chunks of 5,000 before submission, allowing parallel processing across multiple batch requests.
3
Persist Job IDs
Store the job ID returned by the API to enable progress tracking and result retrieval even if the submitting process restarts.
4
Incremental Mode for Recurring Jobs
For recurring batch jobs, such as nightly log classification or weekly classification feed enrichment, use the incremental mode that only classifies domains not seen in previous batches.
The API maintains a per-account domain cache with configurable TTL, so resubmitting previously classified domains returns cached results instantly without consuming quota.
This makes daily incremental jobs extremely efficient, as only newly observed domains require fresh classification.
Integration
Bulk Classification API Integration
Classify thousands of domains in a single request with just a few lines of code
Batch Domain Classification
Submit a list of domains and receive categorization results for all of them in one response.
import requests
API_URL = "https://webfilteringdatabase.com/api/moderate.php"
API_KEY = "YOUR_API_KEY"# Batch of domains to classify
domains = [
"google.com",
"facebook.com",
"gambling-site.xyz",
"github.com",
"adult-content.xxx"
]
# Classify each domain and collect results
results = {}
for domain in domains:
response = requests.post(API_URL, json={
"api_key": API_KEY,
"query": domain
})
data = response.json()
results[domain] = {
"category": data.get("primary_category", "Unknown"),
"risk": data.get("risk_level", "low")
}
# Print classification reportfor domain, info in results.items():
print(f"{domain}: {info['category']} (risk: {info['risk']})")
You can begin with a database plan at $999 for the 1 million most popular domains. When you resell the classification inside a product, OEM licences start at $10,000 per year.
Ready to Classify Domains at Scale?
Process thousands to millions of domains with our batch classification API backed by 120 million categorized domains.
