How to Extract All URLs from a Sitemap (and Why the One-Liner Breaks)

You want every URL a site lists in its sitemap. Maybe you're auditing links, feeding a crawler, or checking what Google can see. Here's the one-liner everyone posts, the four ways it breaks on real sites, and a version that doesn't.

The one-liner

curl -s https://www.cloudflare.com/sitemap.xml | grep -o "<loc>[^<]*" | sed 's/<loc>//'

On Cloudflare's sitemap this prints 897 URLs. It works because that file is the simple case: one plain XML file, sitting at /sitemap.xml, with every page listed inside it.

Most sites you'll actually care about aren't the simple case.

Where it breaks

I ran the same command against a handful of well-known sites. Four different failures.

1. You get sitemaps, not pages. Try it on laravel.com:

https://laravel.com/website-sitemap.xml
https://laravel.com/cloud/docs/sitemap.xml
https://laravel.com/forge/docs/sitemap.xml

Three lines, none of them a page. That file is a sitemap index: a sitemap that lists other sitemaps. Big sites all do this, because one file tops out at 50,000 URLs. gov.uk's index points to 35 children. IMDb's points to 4,101. To get pages you have to fetch each child and run the command again.

2. The sitemap isn't at /sitemap.xml. https://stripe.com/sitemap.xml returns a 404 page, so the command prints nothing, with no error. Stripe's real sitemap is at /sitemap/sitemap.xml. The only way to know is to read robots.txt:

curl -s https://stripe.com/robots.txt | grep -i "^sitemap"
# Sitemap: https://stripe.com/sitemap/sitemap.xml

3. It's gzipped. IMDb serves index.xml.gz as real gzip. grep sees binary and finds 0 URLs. Fine, pipe it through gunzip first.

4. It says .gz but isn't. Now run that gunzip version on the New York Times news sitemap, which also ends in .xml.gz:

gzip: stdin: not in gzip format

Their storage keeps the file gzipped and decompresses it on the way out, so you receive plain XML from a .gz URL. A script that trusts the file name breaks here. One that checks the first two bytes (1f 8b means gzip) doesn't.

None of this is hard to handle. It's just a lot of branches for a "one-liner", and you find them one site at a time.

One call that handles all four

The Sitemap Parser API takes a sitemap URL, or just the site address, and deals with the rest: it reads robots.txt, tries the usual paths, unzips by content, and follows index files.

curl -G "https://apixies.io/api/v1/parse-sitemap" \
     -H "X-API-Key: YOUR_API_KEY" \
     --data-urlencode "url=https://stripe.com" \
     --data-urlencode "limit=2"

{
  "status": "success",
  "data": {
    "url": "https://stripe.com/sitemap/sitemap.xml",
    "type": "sitemapindex",
    "total_urls": 9,
    "returned_urls": 2,
    "urls": [
      { "loc": "https://stripe.com/sitemap/partition-0.xml" },
      { "loc": "https://stripe.com/sitemap/partition-1.xml" }
    ],
    "requested_url": "https://stripe.com",
    "discovered_via": "robots.txt"
  }
}

I passed the bare site address. discovered_via says the sitemap came from robots.txt, and url is where it really lives. type tells you it's an index, so those nine entries are child sitemaps.

To get pages instead, add follow_index=true:

curl -G "https://apixies.io/api/v1/parse-sitemap" \
     -H "X-API-Key: YOUR_API_KEY" \
     --data-urlencode "url=https://www.gov.uk/sitemap.xml" \
     --data-urlencode "follow_index=true" \
     --data-urlencode "limit=1000"

That reads child sitemaps until it has your limit. On gov.uk the first child alone holds 25,000 URLs, so total_urls comes back as 25,000 and stats.child_sitemaps_followed as 1 of 35. limit caps what's returned (1,000 at most per call), not what's counted: total_urls and stats cover every URL that was read.

You can try all of this in the browser with the Sitemap Parser tool. No key needed there.

In code

JavaScript

async function sitemapUrls(site) {
  const params = new URLSearchParams({ url: site, follow_index: "true", limit: "1000" });
  const res = await fetch(`https://apixies.io/api/v1/parse-sitemap?${params}`, {
    headers: { "X-API-Key": process.env.APIXIES_API_KEY },
  });
  const body = await res.json();

  if (body.status !== "success") {
    throw new Error(`${body.code}: ${body.message}`);
  }
  return body.data.urls.map((u) => u.loc);
}

Python

import os
import requests

def sitemap_urls(site):
    res = requests.get(
        "https://apixies.io/api/v1/parse-sitemap",
        params={"url": site, "follow_index": "true", "limit": 1000},
        headers={"X-API-Key": os.environ["APIXIES_API_KEY"]},
        timeout=60,
    )
    body = res.json()
    if body["status"] != "success":
        raise RuntimeError(f"{body['code']}: {body['message']}")
    return [u["loc"] for u in body["data"]["urls"]]

PHP

function sitemapUrls(string $site): array
{
    $query = http_build_query(['url' => $site, 'follow_index' => 'true', 'limit' => 1000]);
    $context = stream_context_create(['http' => [
        'header' => 'X-API-Key: ' . getenv('APIXIES_API_KEY'),
        'ignore_errors' => true,
    ]]);

    $body = json_decode(file_get_contents("https://apixies.io/api/v1/parse-sitemap?$query", false, $context), true);

    if ($body['status'] !== 'success') {
        throw new RuntimeException("{$body['code']}: {$body['message']}");
    }

    return array_column($body['data']['urls'], 'loc');
}

Finding what's wrong with a sitemap

Getting the URLs out is half the job. The other half is noticing when the sitemap itself is the problem. These are the checks worth automating, and the field that answers each one.

• There's no sitemap at all. You get a 404 with code SITEMAP_NOT_FOUND, and errors.tried lists every location that was checked. github.com is a real example: nothing in robots.txt, nothing at the usual paths.
• The XML is broken. A 422 with code INVALID_SITEMAP and the parser's message. This one matters: a sitemap that doesn't parse is a sitemap Google ignores, and nothing on the site looks wrong.
• It's a feed, not a sitemap. Point it at an RSS URL and the message reads root element is <rss>. People mix these up more than you'd think.
• Missing lastmod. Compare stats.has_lastmod with total_urls. The first laravel.com sitemap has it on 481 of 647 URLs. gov.uk has it on 24,999 of 25,000, which makes you wonder about the one. Search engines use lastmod to decide what to recrawl, so every URL without it is a page you're not vouching for.
• Everything changed today. Look at stats.oldest_lastmod and stats.newest_lastmod. Cloudflare's 897 URLs all carry the same date, which is what a sitemap looks like when lastmod is just the build time. Google has said it ignores lastmod when it's not believable.
• Child sitemaps that can't be read. With follow_index=true, stats.child_sitemaps_failed lists them. A child can fail because it's gone, it isn't valid XML, or it's over 16 MB.
• Stale entries. IMDb's index still lists http:// URLs with a lastmod from 2018. The sitemap parses fine. It's just old.

A small monitoring script only needs three of those: status isn't success, total_urls dropped sharply since yesterday, or child_sitemaps_failed isn't empty.

Limits worth knowing

limit goes up to 1,000 URLs per call. With follow_index=true up to 10 child sitemaps are read per call, so a site like IMDb with 4,101 children needs you to walk the index yourself: call once without follow_index to get the children, then call each child.

The free tier is 75 requests a day. That's plenty for checking your own sites daily. It isn't meant for crawling the web.

Next steps

• Sitemap Parser API reference: every parameter and response field
• Sitemap Parser tool: try a site in the browser
• Broken Link Monitoring for SEO: feed these URLs into a link checker
• All guides