New Advanced Crawling Add-on

I’ve been testing some experimental changes in this new add-on. The add-on requires the latest version of WP2Static 7 from git. It comes with five settings:

  • Add URLs Detected While Crawling – This can pick up some pages, images, CSS, JS files, etc. that might not get detected normally.
  • Crawl Chunk Size – The add-on crawls in chunks, which changes the performance. You can tweak this setting if speed is an issue.
  • Crawl Only Changed URLs – Don’t use this. It’s a work-in-progress to avoid the need to crawl the entire site. Currently it will detect when a post has changed, but it won’t realize that the pagination pages have changed.
  • Crawl Sitemaps
  • Detect Redirects from the Redirection plugin – We can’t support regexes, but this will pick up any normal redirects. Not all of the deployment add-ons support redirects yet, but the S3 add-on definitely does.

Zip file | Release page | GitHub repo

3 Likes

Very exciting! Looking forward to try it out!

Updated with “Crawl sitemaps” option.

There’s some bug in the static analysis that is giving failures improperly.

Hey John! Thanks for the addon!
Crawl Only Changed URLs – looks very promising!
Mb it’s more stable now? Also I’m using a wp2static build from May27 so I assume the addon won’t work with it?

It’s pretty stable. I’ve been using it on all of our sites for a while now. But I don’t think it will work with the newer WP2Static builds. The addon’s latest build works with WP2Static 7.1.6, which came out in December. There were a lot of changes in WP2Static after that, and I never updated.

2 Likes

Thanks @john-shaffer!

Yes, I need to get that building with the latest WP2Static core codebase and publish them all once stable. No ETA right now, but “as soon as I can”

1 Like

Thanks to you both guys @leonstafford and @john-shaffer
You are doing amazing work!