Web Scraper

Extract comprehensive data from any website including metadata, contact information, social profiles, and page content.

[!NOTE] See the API Guide for the shared request lifecycle (sync, async polling, async webhook), error envelope, rate limits, custom_vars, and deduplication. This page documents only the fields and behaviour unique to this endpoint.

Try it

Pricing

Endpoint

Input Parameters

Examples

Output Fields

Error Responses

Validation Error (422)

{
    "message": "The url field is required.",
    "errors": {
        "url": ["The url field is required."]
    }
}

Common validation errors:

url field is required
url must be a valid URL
url may not be greater than 2048 characters

Failed Enrichment (422 Unprocessable Entity)

When the enrichment fails (e.g., target website unreachable, scraping blocked), the HTTP status code is 422 and the response body contains "status": "failed" with an error_message:

{
    "id": 123,
    "type": "web_scraper",
    "status": "failed",
    "input": {
        "url": "https://unreachable-site.com"
    },
    "output": null,
    "error_message": "Target website could not be found (DNS resolution failed)"
}

Common error messages include:

Target website could not be found (DNS resolution failed)
Target website has SSL/TLS configuration issues
Target website did not respond in time
Target website refused the connection
Web scraper service is temporarily unavailable (temporary service issue, retry after a short delay)

Rate Limit Exceeded (429)

{
    "message": "Web scraper rate limit exceeded"
}

The API enforces rate limits to ensure fair usage. Implement exponential backoff when retrying.

Integration Examples

JavaScript/Node.js

const response = await fetch(
    'https://api.skael.de/api/enrichments/web_scraper',
    {
        method: 'POST',
        headers: {
            Authorization: 'Bearer eak_your_api_key_here',
            'Content-Type': 'application/json',
        },
        body: JSON.stringify({
            url: 'https://example.com',
        }),
    },
);

const enrichment = await response.json();
console.log(enrichment.output.emails);
console.log(enrichment.output.social_links);

Python

import requests

response = requests.post(
    'https://api.skael.de/api/enrichments/web_scraper',
    headers={
        'Authorization': 'Bearer eak_your_api_key_here',
        'Content-Type': 'application/json'
    },
    json={'url': 'https://example.com'}
)

enrichment = response.json()
print(enrichment['output']['emails'])
print(enrichment['output']['social_links'])

PHP

$client = new GuzzleHttp\Client();

$response = $client->post('https://api.skael.de/api/enrichments/web_scraper', [
    'headers' => [
        'Authorization' => 'Bearer eak_your_api_key_here',
        'Content-Type' => 'application/json',
    ],
    'json' => [
        'url' => 'https://example.com',
    ],
]);

$enrichment = json_decode($response->getBody(), true);
print_r($enrichment['output']['emails']);