Issues with Crawling a Domain
To whitelist the bot, contact your webmaster or hosting provider and ask them to whitelist SplitSignalBot.
The bot’s IP subnetworks are: 126.96.36.199/28 and 188.8.131.52/28
The bot is using standard 80 HTTP and 443 HTTPS ports to connect.
If you use any plugins (Wordpress, for example) or CDNs (content delivery networks) to manage your site, you will have to whitelist the bot IP within those as well.
For whitelisting on Wordpress, contact Wordpress support.
Common CDNs that block our crawler include:
- Cloudflare - read how to whitelist here.
- Incapsula - read how to whitelist here (add Semrush as a “Good bot”).
- ModSecurity - read how to whitelist here.
- Sucuri - read how to whitelist here.
Please note: If you have shared hosting, it is possible that your hosting provider may not allow you to whitelist any bots or edit the Robots.txt file.
Below is a list of some of the most popular hosting providers on the web and how to whitelist a bot on each or reach their support team for assistance:
- Siteground - whitelisting instructions
- 1&1 IONOS - whitelisting instructions
- Bluehost* - whitelisting instructions
- Hostgator* - whitelisting instructions
- Hostinger - whitelisting instructions
- GoDaddy - whitelisting instructions
- GreenGeeks - whitelisting instructions
- Big Commerce - Must contact support
- Liquid Web - Must contact support
- iPage - Must contact support
- InMotion - Must contact support
- Glowhost - Must contact support
- A2 Hosting - Must contact support
- DreamHost - Must contact support
* Please note: these instructions work for HostGator and Bluehost if you have a website on VPS or Dedicated Hosting.
- SplitSignal FAQ
- Issues with Crawling a Domain
- What on-page changes are available for testing in SplitSignal?
- How to use the upload variables feature (import from file)?
- How to Read SplitSignal Test Results?
- SplitSignal Pre-SEO Test Launch Checklist
- SEO Split-Testing: Is My Website a Good Fit?