Access true while robots.txt disallows all

[basvdijk]

I have a robots.txt file containing two lines:

User-agent: *
Disallow: /

And used the example shown on the documentation:

var robots = require('robots')
  , parser = new robots.RobotsParser(
                'http://localhost/robots.txt',
                'Mozilla/5.0 (compatible; RobotTxtBot/1.0)',
                after_parse
            );

function after_parse(parser, success) {
  if(success) {
    parser.canFetch('*', '/', function (access, url, reason) {
      if (access) {
        console.log(' url: '+url+', access: '+access);
        // parse url ...
      }
    });
  }
};

Still the code below gives the output:

url: /, access: true

While I expected access to be false

[slrendell]

I’m getting the same issue whilst running the sample code. Any fixes yet?

[basvdijk]

The last commit on this repo is from Sep 17, 2018. I guess this library is not maintained anymore...

[slrendell]

My mistake. I was checking against the full url, ‘http://facebook.com/‘, rather than just ‘/‘. Working properly now, although not thoroughly tested yet.