Why are only a few of my website’s pages being crawled?Question
If you’ve noticed that only 4-6 pages of your website are being crawled (your home page, sitemaps URLs and robots.txt), most likely this is because our bot couldn’t find outgoing internal links on your Homepage. Below you will find possible reasons for this issue.
In both cases there is a way to ensure that our bot will crawl your pages. To do this, you need to change the crawl source from “website” to “sitemap” or “URLs from file” in your campaign settings:
“Website” is a default source. It means we will crawl your website using a breadth-first search algorithm and navigate through the links we see on your page’s code - starting from the homepage.
If you choose one of the other options, we will crawl links that are found in the sitemap or in the file you upload.
Our crawler could have been blocked on some pages in the website’s robots.txt or by noindex/nofollow tags. You can check if this is the case in your Crawled pages report:
You can inspect your Robots.txt for any disallow commands that would prevent crawlers like ours from accessing your website.
If you see the following code on the main page of a website, it tells us that we’re not allowed to index/follow links on it and our access is blocked. Or, a page containing at least one of the two: "nofollow", "none", will lead to a crawling error.
<meta name="robots" content="noindex, nofollow">
You will find more information about these errors in our troubleshooting article.
- How is Site Health Score calculated in the Site Audit tool?
- Why does Semrush say I have duplicate content?
- How many pages can I crawl in a Site Audit?
- How do I audit a subdomain?
- What Issues Can Site Audit Identify?
- Why does Semrush say I have an incorrect certificate?
- What are unoptimized anchors and how does Site Audit identify them?
- What do the Structured Data Markup Items in Site Audit Mean?
- Pourquoi des pages actives de mon site web apparaissent-elles comme cassées ?
- Can I change automatic Audit re-run schedule or stop it completely?
- Why are only a few of my website’s pages being crawled?
- Why can’t I find URLs from the Audit report on my website?
- Combien de temps faut-il pour explorer un site web ? Il semble que mon audit soit bloqué.
- How Does Site Audit Select Pages to Analyze for Core Web Vitals?
- Why is there a difference between GSC and Semrush Core Web Vitals data?
- Can I set up a custom re-crawl schedule?
- Configurer Audit de site
- Site Audit Overview Report
- Site Audit Thematic Reports
- Reviewing Your Site Audit Issues
- Site Audit Crawled Pages Report
- Site Audit Statistics
- Compare Crawls and Progress
- Dépannage pour Audit de site
- Exporting Site Audit Results
- Comment optimiser la vitesse d'exploration dans Audit de site ?
- Comment intégrer Zapier à Audit de site
- Using Semrush to Find Areas for Improvement on Your Website
- Optimizing Page Title and Meta Description via Semrush