Fix for Crawling or Deployment Failure

WP Static Site Generators Static HTML Output
#1

Assuming your development site is https://dev.example.com which you’re trying to crawl and wanna publish at https://example.com

There are few tips which can help avoiding crawl failure.

  • If you’re using Cloudflare, I would recommend bypass cache for *dev.example.com/* for frontend.

  • Whitelist hosting server IP in Cloudflare, if you’ve static IP you can whitelist yourself as well.

  • For backend, you should point localhost IP to domain name using hosts file.

  • Make sure you have provided correct CDN/Storage login credential.

  • If you’re using HTTP Authentication, you should either provide correct details in plugin settings or consider whitelisting from HTTP web server configuration.

  • Sometime increasing fastcgi_read_timeout value can also help.

fastcgi_read_timeout 10800;
  • Last but not least, use faster Cloud Instance and keep server region same as your Storage region for faster deployment.
1 Like