Bulk Domain Classification - Batch Lookup API for Thousands of Domains

Batch Pipeline

Why Bulk Domain Classification?

Submit up to 100 domains per API request and let our infrastructure classify them in parallel, backed by over 100 million categorized domains across 59 content categories.

Batch Processing

Classify up to 100 domains in a single API request with parallel processing for maximum throughput.

Async Webhooks

Fire-and-forget batch jobs with webhook callbacks that notify your endpoint when results are ready.

CSV Upload

Upload CSV files of domains directly and receive classified results in your preferred output format.

Input Formats

Flexible Input Formats

Submit domains in whatever format your workflow produces

1

JSON Array Submission

Submit a JSON array of domain strings directly in the request body for programmatic integrations. This is the fastest path for applications that already have domain lists in memory.

The API accepts arrays of up to 10,000 domains per request, with each domain automatically normalized by stripping protocols, paths, and query parameters to extract the registrable domain.
Duplicate domains are deduplicated server-side to avoid redundant classification work.

2

CSV File Upload

Upload CSV files containing domain lists extracted from spreadsheets, log analysis tools, or database exports.

The API auto-detects the column containing domains by scanning headers for common names like "domain," "url," "hostname," or "site."
For CSVs without headers, you can specify the column index.
Files up to 50 MB are processed synchronously; larger files are handled asynchronously with progress tracking and webhook notification upon completion.

3

Newline-Delimited Text

Pipe raw domain lists from command-line tools directly to the API using plain text with one domain per line.

This format integrates naturally with Unix pipelines, allowing commands like extracting domains from DNS logs and piping them through curl to the batch endpoint.
Comment lines starting with # are ignored, and blank lines are skipped.
Ideal for scripted automation and cron-based periodic classification jobs.

4

URL List Processing

Submit full URLs instead of bare domains when your source data includes complete web addresses.

The API extracts the registrable domain from each URL, handling subdomains, ports, paths, query strings, and fragments transparently.
This means you can feed raw proxy logs, browser history exports, or web crawler outputs directly without preprocessing.
Each result maps back to the original URL for easy correlation.

Processing Modes

Async Processing and Webhook Callbacks

1

Synchronous Mode

For batches of up to 10,000 domains, synchronous mode returns results directly in the HTTP response.

The connection stays open while processing completes, typically finishing in 5 to 30 seconds depending on batch size.
This mode is ideal for interactive applications, dashboards, and workflows where the calling process needs immediate results.
A progress header is sent periodically to prevent client-side timeouts during processing.

2

Asynchronous Mode

Batches exceeding 10,000 domains or explicit async requests return immediately with a job ID and status URL.

Your application can poll the status endpoint to track progress as a percentage, or register a webhook URL to receive a POST notification when processing completes.
The webhook payload includes a download URL for the results file, summary statistics, and any errors encountered during processing.

3

Webhook Configuration

Register webhook endpoints that receive signed POST notifications at key processing milestones: job accepted, 50% complete, processing finished, and results available for download.

Each webhook includes an HMAC-SHA256 signature header computed with your API secret, allowing your server to verify authenticity.
Retry logic delivers failed webhooks up to five times with exponential backoff.
You can configure multiple webhook URLs per job for redundancy or to notify different systems.

Limits & Quotas

Rate Limits, Quotas, and Batch Sizing

Scale predictably with transparent limits and intelligent throttling

Per-Request Limits

Synchronous batches support up to 10,000 domains per request. Async batches accept up to 1 million domains per job. These limits ensure predictable processing times and fair resource allocation across all API consumers. Requests exceeding limits receive a 413 status with guidance on splitting the batch.

Rate Throttling

Batch endpoints are rate-limited separately from single-lookup endpoints. Standard plans allow 60 batch requests per minute with up to 10 concurrent async jobs. Enterprise plans offer configurable concurrency up to 100 simultaneous jobs. Rate limit headers in every response show remaining quota and reset timestamps.

Monthly Domain Quotas

Each plan includes a monthly domain classification quota shared between single and batch lookups. Usage dashboards and threshold alerts help you monitor consumption.

Optimal Batch Sizing

While the API accepts up to 10,000 domains synchronously, optimal throughput is achieved with batches of 1,000 to 5,000 domains. Smaller batches reduce latency per request, allowing parallelism on the client side. For maximum throughput, submit multiple medium-sized batches concurrently rather than one maximum-sized batch.

Quota efficiency: Batch processing consumes one quota unit per unique domain classified, so submitting the same domain in multiple batches within a billing cycle only counts once.

Output Formats

Response Formats and Result Structure

Get classification results in the format that fits your data pipeline

JSON Response

The default response format returns a JSON object with each domain as a key mapping to its classification result.

Each result includes the primary category, classification detail, all matching categories with individual scores, category level indicator, and timestamp of last classification update.
Nested JSON preserves the full category taxonomy from our 59-category system including IAB tier mappings.

CSV Response

Request CSV output for direct import into spreadsheets, databases, or data warehouse ETL pipelines.

Each row contains the domain, primary category, classification detail, secondary categories, and category level.
Column headers match common BI tool conventions.
CSV responses for async jobs are delivered as downloadable files with gzip compression, reducing transfer size by up to 85 percent for large result sets.

NDJSON Streaming

Newline-delimited JSON format streams results one domain per line, enabling processing to begin before the full response arrives.

This format works naturally with streaming JSON parsers and command-line tools like jq.
Each line is a complete JSON object, so partial responses are still valid and parseable.
Ideal for piping results directly into downstream processing steps.

Enriched Output Mode

The enriched response mode adds metadata beyond basic classification: domain age, registrar information, DNS record summary, SSL certificate status, historical category changes, and related domains sharing the same infrastructure.

This comprehensive view supports content classification workflows where classification alone is insufficient and analysts need the full domain context to make informed decisions.

Use Cases

Common Bulk Lookup Use Cases

How organizations leverage batch classification across industries

content classification Enrichment

Security operations centers feed indicator lists, classification feeds, and suspicious domain extractions through the bulk API to categorize and prioritize content risks.

Domains classified as unsafe content, or Command-and-Control are escalated automatically while benign categories are deprioritized from analyst review queues.
The same bulk classification approach helps analysts find companies with owners nearing retirement by profiling web footprints at scale.

Ad Tech Brand Safety

Advertising platforms classify publisher domain inventories to ensure ads appear only alongside brand-safe content. Bulk lookup categorizes entire exchange inventories overnight, flagging domains in categories like Adult, Violence, or Hate Speech so demand-side platforms can apply advertiser exclusion lists before bidding begins.

Log Analysis and Auditing

Compliance teams export DNS query logs or proxy access logs, extract unique domains, and batch-classify them to identify policy violations. Monthly audit reports reveal category distributions, flag access to restricted categories, and track trends in web usage patterns across the organization over time.

Building Robust Batch Processing Pipelines

1

Idempotent Job Submission

Production batch workflows should implement idempotent job submission using client-generated request IDs to safely retry failed submissions without creating duplicate jobs. These same pipeline patterns apply when constructing an acquisition pipeline for aging business owners that processes thousands of business domains.

2

Partition Large Lists

Partition large domain lists into chunks of 5,000 before submission, allowing parallel processing across multiple batch requests.

3

Persist Job IDs

Store the job ID returned by the API to enable progress tracking and result retrieval even if the submitting process restarts.

4

Incremental Mode for Recurring Jobs

For recurring batch jobs, such as nightly log classification or weekly classification feed enrichment, use the incremental mode that only classifies domains not seen in previous batches.

The API maintains a per-account domain cache with configurable TTL, so resubmitting previously classified domains returns cached results instantly without consuming quota.
This makes daily incremental jobs extremely efficient, as only newly observed domains require fresh classification.

Integration

Bulk Classification API Integration

Classify thousands of domains in a single request with just a few lines of code

Batch Domain Classification

Submit a list of domains and receive categorization results for all of them in one response.

import requests

API_URL = "https://webfilteringdatabase.com/api/moderate.php"
API_KEY = "YOUR_API_KEY"

# Batch of domains to classify
domains = [
    "google.com",
    "facebook.com",
    "gambling-site.xyz",
    "github.com",
    "adult-content.xxx"
]

# Classify each domain and collect results
results = {}
for domain in domains:
    response = requests.post(API_URL, json={
        "api_key": API_KEY,
        "query": domain
    })
    data = response.json()
    results[domain] = {
        "category": data.get("primary_category", "Unknown"),
        "risk": data.get("risk_level", "low")
    }

# Print classification report
for domain, info in results.items():
    print(f"{domain}: {info['category']} (risk: {info['risk']})")

const API_URL = 'https://webfilteringdatabase.com/api/moderate.php';
const API_KEY = 'YOUR_API_KEY';

const domains = [
    'google.com',
    'facebook.com',
    'gambling-site.xyz',
    'github.com',
    'adult-content.xxx'
];

async function bulkClassify(domainList) {
    const results = await Promise.all(
        domainList.map(async (domain) => {
            const res = await fetch(API_URL, {
                method: 'POST',
                headers: { 'Content-Type': 'application/json' },
                body: JSON.stringify({ api_key: API_KEY, query: domain })
            });
            const data = await res.json();
            return { domain, category: data.primary_category, risk: data.risk_level };
        })
    );
    return results;
}

bulkClassify(domains).then(results => {
    results.forEach(r =>
        console.log(`${r.domain}: ${r.category} (risk: ${r.risk})`)
    );
});

# Classify multiple domains in sequence
for domain in google.com facebook.com gambling-site.xyz github.com; do
  curl -s -X POST "https://webfilteringdatabase.com/api/moderate.php" \
    -H "Content-Type: application/json" \
    -d "{\"api_key\": \"YOUR_API_KEY\", \"query\": \"$domain\"}" \
    | jq "{domain: .domain, category: .primary_category, risk: .risk_level}"
done

# Example output for one domain:
# {
#   "domain": "gambling-site.xyz",
#   "category": "Gambling",
#   "risk": "high"
# }

<?php
$apiUrl = 'https://webfilteringdatabase.com/api/moderate.php';
$apiKey = 'YOUR_API_KEY';

$domains = ['google.com', 'facebook.com', 'gambling-site.xyz', 'github.com'];

foreach ($domains as $domain) {
    $ch = curl_init($apiUrl);
    curl_setopt($ch, CURLOPT_POST, true);
    curl_setopt($ch, CURLOPT_POSTFIELDS, json_encode([
        'api_key' => $apiKey,
        'query'   => $domain
    ]));
    curl_setopt($ch, CURLOPT_HTTPHEADER, ['Content-Type: application/json']);
    curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
    $result = json_decode(curl_exec($ch), true);
    curl_close($ch);

    echo "$domain: {$result['primary_category']} (risk: {$result['risk_level']})\n";
}
?>

Ready to Classify Domains at Scale?

Process thousands to millions of domains with our batch classification API backed by 100 million categorized domains.

Check Domain See Pricing Download Sample

Bulk Domain Classification API