GitHub - spire-rs/kit: 📚 Libraries for retrieving and parsing robots.txt & sitemaps.

Kit

countio: The wrapper struct to enable byte counting for std::io::Read and std::io::Write and its asynchronous variants from futures and tokio.
robotxt: The implementation of the Robots.txt (or URL exclusion) protocol with the support of crawl-delay, sitemap and universal match extensions.
sitemapo: The implementation of the Sitemap (or URL inclusion) protocol with the support of txt, xml formats and video, image, and news extensions.

Name		Name	Last commit message	Last commit date
Latest commit History 15 Commits
.github		.github
countio		countio
exclusion		exclusion
inclusion		inclusion
.gitignore		.gitignore
Cargo.lock		Cargo.lock
Cargo.toml		Cargo.toml
LICENSE.txt		LICENSE.txt
README.md		README.md