Skip to content
/ dyer Public

Dyer is designed for reliable, flexible and fast web crawling, providing some high-level, comprehensive features without compromising speed.

Notifications You must be signed in to change notification settings

hominee/dyer

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

87 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

API Document crates.io Cookbook

Overview

dyer is designed for reliable, flexible and fast Request-Response based service, including data processing, web-crawling and so on, providing some friendly, interoperable, comprehensive features without compromising speed.

Features

  • asynchronous, concurrent streaming and I/O, make the best of thread pool, network, and system resource.
  • Event-driven, once you set the initials and recursive generator, dyer will handle the rest of it interoperably.
  • User-friendly and flexible, dyer offers high-level, flexible, easy to use wrappers and APIs what does a lot for you.

Feature Flag

To reduce code redundancy and speed up compilation, dyer use feature flag to mark the necessary modules/functions, Currently here are some supported Features:

  • xpath-stable: stably parse the response html with third-party C library libxml2
  • xpath-alpha: rust-native parse the html response, NOTE that it is experimental and unstale, remains to be improved
  • compression: Enable HTTP Compression: br, deflate, gzip
  • proxy: Enable use proxies
  • full: Enable all features

Guide

Get started by installing dyer-cli and looking over the examples and quick start.

Crates: Link 🔗
Documentation: Link 🔗
The Cookbook gives a detailed view of dyer.

About

Dyer is designed for reliable, flexible and fast web crawling, providing some high-level, comprehensive features without compromising speed.

Topics

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages