Failed during "Crawling initial file list"

WP Static Site Generators Static HTML Output
#1

Hi,
for months, static html output worked perfectly on our site. but suddenly I always get the error Failed during “Crawling initial file list”.

This is the output of the export log:

2021-10-28 19:11:01 2021-10-28 07:11:01
PHP VERSION 7.4.10
OS VERSION Linux ip-10-0-25-91.eu-central-1.compute.internal 4.14.243-185.433.amzn2.x86_64 #1 SMP Mon Aug 9 05:55:52 UTC 2021 x86_64
WP VERSION 5.8.1
WP URL https://cms.compeon.de
WP SITEURL https://cms.compeon.de
WP HOME https://cms.compeon.de
WP ADDRESS https://cms.compeon.de
PLUGIN VERSION 6.6.21
VIA WP-CLI?
STATIC EXPORT URL https://www.compeon.de
PERMALINK STRUCTURE /%postname%/
SERVER SOFTWARE Apache/2.4.38 (Debian)
2021-10-28 19:11:01 Active plugins:
2021-10-28 19:11:01 404-to-301/404-to-301.php
2021-10-28 19:11:01 acf-extended/acf-extended.php
2021-10-28 19:11:01 acf-icon-picker/acf-icon-picker.php
2021-10-28 19:11:01 acf-image-select/acf-image-select.php
2021-10-28 19:11:01 advanced-custom-fields-nav-menu-field/fz-acf-nav-menu.php
2021-10-28 19:11:01 advanced-custom-fields-pro/acf.php
2021-10-28 19:11:01 advanced-custom-fields-row-field/acf-row.php
2021-10-28 19:11:01 ajax-thumbnail-rebuild/ajax-thumbnail-rebuild.php
2021-10-28 19:11:01 better-search-replace/better-search-replace.php
2021-10-28 19:11:01 classic-editor/classic-editor.php
2021-10-28 19:11:01 duplicate-post/duplicate-post.php
2021-10-28 19:11:01 login-logo/login-logo.php
2021-10-28 19:11:01 media-file-renamer/media-file-renamer.php
2021-10-28 19:11:01 plugin-organizer/plugin-organizer.php
2021-10-28 19:11:01 real-media-library/index.php
2021-10-28 19:11:01 real-physical-media/index.php
2021-10-28 19:11:01 redirection/redirection.php
2021-10-28 19:11:01 rename-wp-login/rename-wp-login.php
2021-10-28 19:11:01 shortpixel-adaptive-images/short-pixel-ai.php
2021-10-28 19:11:01 static-html-output-plugin/static-html-output-plugin.php
2021-10-28 19:11:01 updraftplus/updraftplus.php
2021-10-28 19:11:01 wordpress-importer/wordpress-importer.php
2021-10-28 19:11:01 wordpress-seo/wp-seo.php
2021-10-28 19:11:01 wp-410/wp-410.php
2021-10-28 19:11:01 wp-mail-smtp/wp_mail_smtp.php
2021-10-28 19:11:01 wp-migrate-db-pro-cli/wp-migrate-db-pro-cli.php
2021-10-28 19:11:01 wp-migrate-db-pro-media-files/wp-migrate-db-pro-media-files.php
2021-10-28 19:11:01 wp-migrate-db-pro/wp-migrate-db-pro.php
2021-10-28 19:11:01 wp-updates-notifier/class-sc-wp-updates-notifier.php
2021-10-28 19:11:01 Plugin options:
2021-10-28 19:11:01 additionalUrls: /news-kfw-schnellkredit/
/compeon-news/
2021-10-28 19:11:01 baseUrl: https://www.compeon.de
2021-10-28 19:11:01 baseUrl-bitbucket:
2021-10-28 19:11:01 baseUrl-bunnycdn:
2021-10-28 19:11:01 baseUrl-github:
2021-10-28 19:11:01 baseUrl-gitlab:
2021-10-28 19:11:01 baseUrl-netlify:
2021-10-28 19:11:01 baseUrl-s3:
2021-10-28 19:11:01 baseUrl-zip: https://www.compeon.de
2021-10-28 19:11:01 baseUrl-zip: https://www.compeon.de
2021-10-28 19:11:01 basicAuthPassword: *******************
2021-10-28 19:11:01 basicAuthUser: compeon
2021-10-28 19:11:01 bbBranch:
2021-10-28 19:11:01 bbRepo:
2021-10-28 19:11:01 bbToken: *******************
2021-10-28 19:11:01 bunnycdnStorageZoneAccessKey: *******************
2021-10-28 19:11:01 bunnycdnPullZoneAccessKey: *******************
2021-10-28 19:11:01 bunnycdnPullZoneID:
2021-10-28 19:11:01 bunnycdnStorageZoneName:
2021-10-28 19:11:01 bunnycdn_api_host: *******************
2021-10-28 19:11:01 cfDistributionId:
2021-10-28 19:11:01 completionEmail:
2021-10-28 19:11:01 crawl_delay: 0
2021-10-28 19:11:01 crawl_increment: 50
2021-10-28 19:11:01 crawlPort:
2021-10-28 19:11:01 delayBetweenAPICalls: 0
2021-10-28 19:11:01 deployBatchSize: 25
2021-10-28 19:11:01 excludeURLs:
2021-10-28 19:11:01 ghBranch:
2021-10-28 19:11:01 ghCommitMessage:
2021-10-28 19:11:01 ghRepo:
2021-10-28 19:11:01 ghToken: *******************
2021-10-28 19:11:01 glBranch:
2021-10-28 19:11:01 glProject:
2021-10-28 19:11:01 glToken: *******************
2021-10-28 19:11:01 netlifyHeaders:
2021-10-28 19:11:01 netlifyPersonalAccessToken: *******************
2021-10-28 19:11:01 netlifyRedirects:
2021-10-28 19:11:01 netlifySiteID:
2021-10-28 19:11:01 removeConditionalHeadComments: 1
2021-10-28 19:11:01 removeHTMLComments: 1
2021-10-28 19:11:01 removeWPLinks: 1
2021-10-28 19:11:01 removeWPMeta:
2021-10-28 19:11:01 rewrite_rules:
2021-10-28 19:11:01 rename_rules:
2021-10-28 19:11:01 s3Bucket:
2021-10-28 19:11:01 s3Key:
2021-10-28 19:11:01 s3Region:
2021-10-28 19:11:01 s3Secret: *******************
2021-10-28 19:11:01 selected_deployment_option: zip
2021-10-28 19:11:01 targetFolder:
2021-10-28 19:11:01 useBasicAuth: 1
2021-10-28 19:11:01 Installed extensions:
2021-10-28 19:11:01 Core,date,libxml,openssl,pcre,sqlite3,zlib,ctype,curl,dom,fileinfo,filter,ftp,hash,iconv,json,mbstring,SPL,PDO,session,posix,Reflection,standard,SimpleXML,pdo_sqlite,Phar,tokenizer,xml,xmlreader,xmlwriter,mysqlnd,apache2handler,bcmath,exif,gd,imagick,mysqli,sodium,zip,Zend OPcache

@leonstafford any idea what could be the problem?

Thanks,

Robert

#2

Hi Robert,

Hmm, working then stopped, with same version of Static HTML Output, my first thoughts:

  • environmental changes (operating system or components)
  • WordPress or plugin updates (WP shouldn’t matter, PHP versions, also, the newer the better)

That looks like an AWS box - is the AWS stuff setup yourself or using a quick setup thing like LightSail?

If you have any AWS WAF/loadbalancer stuff that could be falsely detecting the site crawl like a DDoS, that’s unlikely, but worth mentioning.

Do you have another log from within Static HTML Output? ie, one that shows which URLs were attempted to crawl?

PHP error log may be the next place to look for an error the plugin wasn’t able to capture/present to you…

#3

Thank you for your answer. We are using it on AWS ECS / Fargate. I am currently on holiday then I will check php and other Logs. Reg PHP Version i also have to check.