HR Wiki mirror — download status LIVE

Failures

3,825 failed attempts in concurrent log (~32% of completed attempts)

Recent error samples:

• <urlopen error [Errno 111] Connection refused>
• <urlopen error [Errno 111] Connection refused>
• <urlopen error [Errno 111] Connection refused>
• <urlopen error [Errno 111] Connection refused>
• <urlopen error [Errno 111] Connection refused>

Failed URLs can be retried later; resume skips files already saved.

After download

Convert → XML dump → MediaWiki import is chained to start when the download process exits. The wiki currently shows only pilot pages until that import finishes.

Processes

• 17889 python3 /root/hrwiki-mirror/scripts/download_from_manifest.py
• 44155 sh -c -- pgrep -af 'download_from_manifest.py' 2>/dev/null
• 44157 sh -c -- pgrep -af 'cdx_wayback_download.py' 2>/dev/null
• 17890 bash -c set -e DLPID=17889 echo Waiting for download PID $DLPID... while kill -0 $DLPID 2>/dev/null; do sleep 30; done echo Download finished at $(date) tail -12 /root/hrwiki-mirror/logs/full-download.log echo '=== Convert ===' rm -rf /root/hrwiki-mirror/wikitext mkdir -p /root/hrwiki-mirror/wikitext python3 /root/hrwiki-mirror/scripts/html_to_mediawiki.py 2>&1 | tee /root/hrwiki-mirror/logs/full-convert.log echo '=== Dump ===' python3 /root/hrwiki-mirror/scripts/build_xml_dump.py 2>&1 | tee /root/hrwiki-mirror/logs/full-dump.log echo '=== Import ===' cd /var/www/hrwikitwice.com/public_html php maintenance/run.php importDump /root/hrwiki-mirror/hrwiki-mirror-dump.xml 2>&1 | tee /root/hrwiki-mirror/logs/full-import.log php maintenance/run.php rebuildrecentchanges 2>&1 | tee -a /root/hrwiki-mirror/logs/full-import.log php maintenance/run.php initSiteStats --update 2>&1 | tee -a /root/hrwiki-mirror/logs/full-import.log php maintenance/run.php refreshLinks 2>&1 | tee -a /root/hrwiki-mirror/logs/full-import.log || true echo '=== DONE at' $(date) '===' mysql mediawiki -N -e 'SELECT COUNT(*) FROM page;'
• 44159 sh -c -- pgrep -af 'html_to_mediawiki.py' 2>/dev/null
• 44161 sh -c -- pgrep -af 'build_xml_dump.py' 2>/dev/null
• 44163 sh -c -- pgrep -af 'run_full_pipeline.sh' 2>/dev/null