Robots.txt Parser

Fetch and parse a domain's robots.txt file into structured data

GET


                    api/v1/robots-txt

Fetch and parse a domain's robots.txt file into structured data

Rate Limiting

Standard rate limits apply to this endpoint.

Authentication

This endpoint requires authentication via API key. You can use the sandbox mode for testing without registering. See the authentication guide for details.

Query Parameters

Parameter	Type	Required	Description
`domain`	string	Required	The domain parameter

Response details for this API endpoint. See the response format guide for the standard response structure.

Example Response

{
    "status": "success",
    "http_code": 200,
    "code": "SUCCESS",
    "message": "Robots.txt parsed successfully",
    "data": {
        "domain": "example.com",
        "found": true,
        "user_agents": {
            "*": {
                "allow": [
                    "/"
                ],
                "disallow": [
                    "/admin/"
                ],
                "crawl_delay": 10
            }
        },
        "sitemaps": [
            "https://example.com/sitemap.xml"
        ]
    }
}

Example Usage

Here's how to use this endpoint in various languages. For more examples, see our code examples page.

cURL

curl -X GET \
    "https://apixies.io/api/v1/robots-txt" \
    -H "X-API-Key: YOUR_API_KEY"

JavaScript

fetch("https://apixies.io/api/v1/robots-txt", {
    method: "GET",
    headers: {
        "Content-Type": "application/json",
        "X-API-Key": "YOUR_API_KEY"
    }
})
.then(response => response.json())
.then(data => console.log(data))
.catch(error => console.error("Error:", error));

PHP

$ch = curl_init();

curl_setopt($ch, CURLOPT_URL, "https://apixies.io/api/v1/robots-txt");
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_HTTPHEADER, [
    "Content-Type: application/json",
    "X-API-Key: YOUR_API_KEY"
]);

$response = curl_exec($ch);
curl_close($ch);

echo $response;

Try It Yourself

Test this endpoint directly from your browser using our API sandbox. No authentication required for testing.

Try API Endpoint

This is a limited sandbox environment.

For full, unlimited API access, register for a free account during our beta period.

API Endpoint

/

Sandbox API Usage Get full API access

No token information available

Sandbox limit: 1 token per day with 25 requests per token

Your session has expired. A new token will be created automatically.

Your daily Sandbox API quota has been exhausted. Please try again tomorrow.

Need unlimited access? Register for free and get your own API key with higher limits.

Unable to validate your API token.

Response

Send a request to see the response

Common Use Cases

SEO crawlability auditing

Analyze robots.txt files across your domains to ensure important pages aren't accidentally blocked from search engine crawlers.

Competitor analysis

Examine competitors' robots.txt files to understand their crawl strategy, discover hidden sitemaps, and identify sections they prioritize or restrict.

Multi-site governance

Monitor robots.txt configurations across all your organization's domains. Alert when changes could affect SEO or expose restricted content.

Related API Endpoints

Explore related endpoints that work well with Robots.txt Parser.

GET Meta Tag Extractor

Extract OpenGraph, Twitter Card, and standard meta tags from any URL

GET Link Checker

Check if a URL is reachable and return its HTTP status code and response time

GET DNS Lookup

Resolve DNS records (A, AAAA, MX, TXT, CNAME, NS, SOA, SRV, CAA, PTR) for any domain

Share Your Feedback

Suggest a micro‑API