xojoc.pw/crawl/robots -

robots.txt

robots is a library written in golang for parsing the robots.txt file format.

First install the library with

go get xojoc.pw/crawl/robots

…then run…

go test

if something fails open an issue.

See godoc for the complete documentation.

robots was written by Alexandru Cojocaru (https://xojoc.pw).

Get the code