Assuming your development site is
https://dev.example.com which you’re trying to crawl and wanna publish at
There are few tips which can help avoiding crawl failure.
If you’re using Cloudflare, I would recommend bypass cache for
*dev.example.com/* for frontend.
Whitelist hosting server IP in Cloudflare, if you’ve static IP you can whitelist yourself as well.
For backend, you should point localhost IP to domain name using hosts file.
Make sure you have provided correct CDN/Storage login credential.
If you’re using HTTP Authentication, you should either provide correct details in plugin settings or consider whitelisting from HTTP web server configuration.
Sometime increasing fastcgi_read_timeout value can also help.
- Last but not least, use faster Cloud Instance and keep server region same as your Storage region for faster deployment.