robots is a library written in golang for parsing the robots.txt file format.
First install the library with
go get xojoc.pw/crawl/robots
if something fails open an issue.
See godoc for the complete documentation.
robots was written by Alexandru Cojocaru (https://xojoc.pw).
See all the discussions around the web about this page.