pip install linkchecker

linkchecker http://localhost:8000 --check-extern

Or to account for some occasional latency do a slower crawl and output errors to a file use:

linkchecker --timeout=20 --threads=1 -F text/linkchecker_output.log http://localhost:8000

Note the --check-extern option tells linkchecker to check external links as well as internal links.

To output a file with the results of the linkchecker run, use the -F option followed by the path to the file to output to. For example:

linkchecker --timeout=30 --ignore-url='.*\.swf$' -F text/linkchecker_output.log http://localhost:8000  

If you want to check only HTML pages and ignore other resources, you can use the --no-warnings option. This will make linkchecker faster and reduce the number of URLs checked. However, it will also make linkchecker less thorough, as it won't check if your CSS, JavaScript, images, and other resources are loading correctly.

linkchecker will crawl all pages of your website and check all links on each page. It will print a report to the console, showing any broken links it found.

Also, please be aware that linkchecker can generate a lot of traffic and may be blocked by some websites. Always use it responsibly and respect the terms of service of the websites you're checking.

Increase the timeout that linkchecker uses when accessing URLs by using the --timeout option followed by the number of seconds to wait. For example, to wait up to 10 seconds for a response, you can use:

linkchecker --timeout=10 http://localhost:8000

Ignore URLs: If there are certain URLs you want linkchecker to ignore, you can use the -i or --ignore-url option followed by a regular expression that matches the URLs to ignore. For example, to ignore all URLs that contain example.com, you can use -i example.com.

Set the User-Agent: Some websites may block or limit requests from linkchecker because it identifies itself as a bot. You can change the User-Agent string that linkchecker sends with the -u or --user-agent option. For example, to identify as a regular Chrome browser, you can use -u "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.3".

Limit the Depth: By default, linkchecker follows all links it finds, no matter how deep. You can limit the depth of the crawl with the -r or --recursion-level option followed by a number. For example, to only check links on the homepage and one level deep, you can use -r 2.

Check Only Certain File Types: If you're only interested in certain types of files, you can use the --file-extension option followed by a comma-separated list of file extensions. For example, to check only HTML and CSS files, you can use --file-extension=html,css.

Use linkchecker --list-plugins to see a list of all available plugins.

linkchecker -h or linkchecker --help will show a list of all available options.

After testing Bakery and other Django static site generators, I settled on using wget to generate the static files for this project. Using wget I'm able to capture the paginated pages as needed.

Install wget if not already installed.

brew install wget

Prep Django for static file export:

• Set DJANGO_DEBUG=False in .env. (NOTE: I'm not sure if this is necessary anymore)
• Minify SASS using sass assets/scss/index.scss:static/portfolio/styles.css --style=compressed --no-source-map
• • In settings.py disable the django-debug-toolbar by commenting out the DEBUG_TOOLBAR_CONFIG setting.

DEBUG_TOOLBAR_CONFIG = {
    # "SHOW_TOOLBAR_CALLBACK": show_toolbar,
    'SHOW_TOOLBAR_CALLBACK': lambda r: False,  # Disables the debug toolbar
}

-N(or --timestamping): Tells wget to only download files that are newer than the local copies --recursive: download the entire website. --no-clobber: don't overwrite any existing files (useful for updating your local copy). --page-requisites: download all the files that are necessary to properly display a given HTML page (including images and stylesheets). --html-extension: save files with the .html extension. --convert-links: convert all links so that they work offline. --restrict-file-names=windows: modify filenames so that they will work in Windows as well (converts a colon to a plus sign for example). --domains localhost: don't follow links outside of the specified domain. --no-parent: don't follow links outside of the directory hierarchy that the starting URL specifies.

To download files to your current directory, use:

wget -N --recursive --no-clobber --page-requisites --html-extension --convert-links --restrict-file-names=windows --domains localhost --no-parent http://localhost:8000/

The URL to start downloading from should be the last argument in the command.

Note: The --domains option expects a domain name, not a URL. You should remove http:// from the --domains option.

The --mirror option in wget is a shortcut for enabling several options that are useful for mirroring a website. Specifically, it's equivalent to -r -N -l inf --no-remove-listing.

Here's what each of these options does:

• -r or --recursive: This option tells wget to follow links and download pages recursively.
• -N or --timestamping: This option tells wget to only download files that are newer than the local copies. It's useful for updating your local copy of the website without re-downloading everything.
• -l inf or --level=inf: This option sets the maximum recursion depth to infinite, meaning wget will follow links indefinitely. By default, wget only follows links up to 5 levels deep.
• --no-remove-listing: This option prevents wget from removing the temporary .listing files generated when downloading directories using FTP. This is generally not relevant when downloading websites over HTTP or HTTPS.

So, when you use --mirror, wget will download the entire website, including all linked pages, and only download files that have changed since the last download. It's a convenient option for creating a local mirror of a website.

The --mirror option also sets -l inf which means it will follow links indefinitely deep, while the default for --recursive is to follow links up to 5 levels deep.

Use the --mirror convenience option:

wget --mirror --no-clobber --page-requisites --html-extension --convert-links --restrict-file-names=windows --domains localhost --no-parent http://localhost:8000/

If you are not working on a Windows machine, you can remove the --restrict-file-names=windows option.

First, validate the URLs in the sitemap using linkchecker and output the results to a file:

wget --spider --recursive --no-verbose --force-html -i http://localhost:8000/sitemap.xml > linkchecker_output.txt

Download or copy the URLs in the output from http://localhost:8000/wget_sitemap/ to a plain text file named wget_urls.txt.

Make sure the URLs in wget_urls.txt are pointing to http://localhost:8000/ and NOT https://danpoynor.com/.

Use linkchecker to validate the URLs in wget_urls.txt and output the results to a file:

linkchecker --timeout=30 --threads=2 -F text/linkchecker_output.log http://localhost:8000

If no broken links are found, then cd to the parent directory outside of the project to avoid git tracking the initial downloaded files then run wget with the --config option followed by the path to the .wgetrc file:

wget --config=danpoynorcom-django/.wgetrc -i danpoynorcom-django/wget_urls.txt

wget will create a directory name localhost:8000 in the current directory and download the files there.

If wget pauses for a long time towards the end it's probably updating all the links in the project to be relative to the current directory. This is a good thing.

Move the files to a docs/ directory in the project root and remove the localhost:8000 directory, unless iterating.

Now you can add the docs/ directory to Git and push it to GitHub and GitHub Pages will serve the static files from there.

To run a server in the directory containing the output, cd into localhost+8000 directory and run:

python3 -m http.server 9876

Then visit http://localhost:9876 in a web browser to make sure the server is running and the site looks good.

You can then run linkchecker on the build directory to check for broken links and output the results to a file:

linkchecker --timeout=30 --threads=2 -F text/linkchecker_output_port9876.log http://localhost:9876

Check for unused CSS. If the PurgeCSS CLI is installed, you can run it from inside the localhost+8000 directory to create an output file with only the used CSS and compare it to the original CSS file.

purgecss --css static/portfolio/styles.css --content **/*.html --output static/portfolio/styles.purged.css

• Write more tests
• Make sure urls match up:
  • WGET Sitemap
  • sitemap.xml
  • SEO audit
• Evaluate automating the steps in the section 'Use wget To Generate Static Files' above.
• Evaluate automating adding the sitemap.xml, robots.txt, favicon.ico files to the docs/ directory.
• Add concept matrix project item to the portfolio.
• Update images to high res in the portfolio.
• Add some cool view-transition animations.
• Add info on how to view Flash projects in the portfolio.