Skip to content

Promise support for node-crawler (Web Crawler/Spider for NodeJS + server-side jQuery)

License

Notifications You must be signed in to change notification settings

AppliedSoul/promise-crawler

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

15 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

promise-crawler

Promise support for node-crawler (Web Crawler/Spider for NodeJS + server-side jQuery)

npm package

Build Status Coverage Status Greenkeeper badge

Nodejs library for website crawling using node-crawler but on bluebird promises.

Install using npm:

npm i promise-crawler --save

Example:

const PromiseCrawler = require('promise-crawler');
//Initialize with node-crawler options
const crawler = new PromiseCrawler({
  maxConnections: 10,
  retries: 3
});

//perform setup and then use it
crawler.setup().then(() => {
  // makes request with node-crawler queue options
  crawler.request({
    url: 'http://example.com'
  }).then((res) => {
    //server side response parsing using cheerio
    let $ = res.$;
    console.log($("title").text());

    // destroy the instance
    process.nextTick(() => crawler.destroy())
  })
});

About

Promise support for node-crawler (Web Crawler/Spider for NodeJS + server-side jQuery)

Topics

Resources

License

Stars

Watchers

Forks

Packages

No packages published