#Skip to menu

robots.txt

robots is a library written in golang for parsing the robots.txt file format.

Usage

First install the library with

go get xojoc.pw/crawl/robots

…then run…

go test

if something fails open an issue.

See godoc for the complete documentation.

Who?

robots was written by Alexandru Cojocaru (https://xojoc.pw).

Get the code

See all the discussions around the web about this page.