chore(release): v3.8.0

apify · Feb 21, 2024 · 36dc882 · 36dc882
1 parent 8e4218a
commit 36dc882
Show file tree

Hide file tree

Showing 35 changed files with 321 additions and 124 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -3,6 +3,32 @@
 All notable changes to this project will be documented in this file.
 See [Conventional Commits](https://conventionalcommits.org) for commit guidelines.
 
+# [3.8.0](https://github.com/apify/crawlee/compare/v3.7.3...v3.8.0) (2024-02-21)
+
+
+### Bug Fixes
+
+* `createRequests` works correctly with `exclude` (and nothing else) ([#2321](https://github.com/apify/crawlee/issues/2321)) ([048db09](https://github.com/apify/crawlee/commit/048db0964a57ac570320ad495425733128235491))
+* declare missing dependencies on `csv-stringify` and `fs-extra` ([#2326](https://github.com/apify/crawlee/issues/2326)) ([718959d](https://github.com/apify/crawlee/commit/718959dbbe1fa69f948d0b778d0f54d9c493ab25)), closes [/github.com/redabacha/crawlee/blob/2f05ed22b203f688095300400bb0e6d03a03283c/.eslintrc.json#L50](https://github.com//github.com/redabacha/crawlee/blob/2f05ed22b203f688095300400bb0e6d03a03283c/.eslintrc.json/issues/L50)
+* **puppeteer:** add 'process' to the browser bound methods ([#2329](https://github.com/apify/crawlee/issues/2329)) ([2750ba6](https://github.com/apify/crawlee/commit/2750ba646ef3c1d51eacdd8e7d67be0e14fb2a97))
+* **puppeteer:** replace `page.waitForTimeout()` with `sleep()` ([52d7219](https://github.com/apify/crawlee/commit/52d7219acdc19b34a727e5d26f7f9288d27ca57f)), closes [#2335](https://github.com/apify/crawlee/issues/2335)
+* **puppeteer:** support `puppeteer@v22` ([#2337](https://github.com/apify/crawlee/issues/2337)) ([3cc360a](https://github.com/apify/crawlee/commit/3cc360a1ea94147133f9785d65834f360f7b42a7))
+
+
+### Features
+
+* `KeyValueStore.recordExists()` ([#2339](https://github.com/apify/crawlee/issues/2339)) ([8507a65](https://github.com/apify/crawlee/commit/8507a65d1ad079f64c752a6ddb1d8fac9b494228))
+* accessing crawler state, key-value store and named datasets via crawling context ([#2283](https://github.com/apify/crawlee/issues/2283)) ([58dd5fc](https://github.com/apify/crawlee/commit/58dd5fcc25f31bb066402c46e48a9e5e91efd5c5))
+* adaptive playwright crawler ([#2316](https://github.com/apify/crawlee/issues/2316)) ([8e4218a](https://github.com/apify/crawlee/commit/8e4218ada03cf485751def46f8c465b2d2a825c7))
+* add Sitemap.tryCommonNames to check well known sitemap locations ([#2311](https://github.com/apify/crawlee/issues/2311)) ([85589f1](https://github.com/apify/crawlee/commit/85589f167196ac49c0cc10664ab3e9e5595208ed)), closes [#2307](https://github.com/apify/crawlee/issues/2307)
+* **ci:** snapshot docs automatically on minor/major publish ([#2344](https://github.com/apify/crawlee/issues/2344)) ([092f51e](https://github.com/apify/crawlee/commit/092f51e8e85d22953686897d7508e9c0c9af08ad))
+* **core:** add `userAgent` parameter to `RobotsFile.isAllowed()` + `RobotsFile.from()` helper ([#2338](https://github.com/apify/crawlee/issues/2338)) ([343c159](https://github.com/apify/crawlee/commit/343c159f20546a2006db33da4674e6ffd77db572))
+* Support plain-text sitemap files (sitemap.txt) ([#2315](https://github.com/apify/crawlee/issues/2315)) ([0bee7da](https://github.com/apify/crawlee/commit/0bee7daf9509fe61c8d83799e706f0bb030257ec))
+
+
+
+
+
 ## [3.7.3](https://github.com/apify/crawlee/compare/v3.7.2...v3.7.3) (2024-01-30)
 
 

diff --git a/lerna.json b/lerna.json
@@ -2,7 +2,7 @@
   "packages": [
     "packages/*"
   ],
-  "version": "3.7.3",
+  "version": "3.8.0",
   "command": {
     "version": {
       "conventionalCommits": true,

diff --git a/packages/basic-crawler/CHANGELOG.md b/packages/basic-crawler/CHANGELOG.md
@@ -3,6 +3,23 @@
 All notable changes to this project will be documented in this file.
 See [Conventional Commits](https://conventionalcommits.org) for commit guidelines.
 
+# [3.8.0](https://github.com/apify/crawlee/compare/v3.7.3...v3.8.0) (2024-02-21)
+
+
+### Bug Fixes
+
+* declare missing dependencies on `csv-stringify` and `fs-extra` ([#2326](https://github.com/apify/crawlee/issues/2326)) ([718959d](https://github.com/apify/crawlee/commit/718959dbbe1fa69f948d0b778d0f54d9c493ab25)), closes [/github.com/redabacha/crawlee/blob/2f05ed22b203f688095300400bb0e6d03a03283c/.eslintrc.json#L50](https://github.com//github.com/redabacha/crawlee/blob/2f05ed22b203f688095300400bb0e6d03a03283c/.eslintrc.json/issues/L50)
+
+
+### Features
+
+* accessing crawler state, key-value store and named datasets via crawling context ([#2283](https://github.com/apify/crawlee/issues/2283)) ([58dd5fc](https://github.com/apify/crawlee/commit/58dd5fcc25f31bb066402c46e48a9e5e91efd5c5))
+* adaptive playwright crawler ([#2316](https://github.com/apify/crawlee/issues/2316)) ([8e4218a](https://github.com/apify/crawlee/commit/8e4218ada03cf485751def46f8c465b2d2a825c7))
+
+
+
+
+
 ## [3.7.3](https://github.com/apify/crawlee/compare/v3.7.2...v3.7.3) (2024-01-30)
 
 **Note:** Version bump only for package @crawlee/basic

diff --git a/packages/basic-crawler/package.json b/packages/basic-crawler/package.json
@@ -1,6 +1,6 @@
 {
     "name": "@crawlee/basic",
-    "version": "3.7.3",
+    "version": "3.8.0",
     "description": "The scalable web crawling and scraping library for JavaScript/Node.js. Enables development of data extraction and web automation jobs (not only) with headless Chrome and Puppeteer.",
     "engines": {
         "node": ">=16.0.0"
@@ -48,9 +48,9 @@
         "@apify/log": "^2.4.0",
         "@apify/timeout": "^0.3.0",
         "@apify/utilities": "^2.7.10",
-        "@crawlee/core": "3.7.3",
-        "@crawlee/types": "3.7.3",
-        "@crawlee/utils": "3.7.3",
+        "@crawlee/core": "^3.8.0",
+        "@crawlee/types": "^3.8.0",
+        "@crawlee/utils": "^3.8.0",
         "csv-stringify": "^6.2.0",
         "fs-extra": "^11.0.0",
         "got-scraping": "^4.0.0",

diff --git a/packages/browser-crawler/CHANGELOG.md b/packages/browser-crawler/CHANGELOG.md
@@ -3,6 +3,17 @@
 All notable changes to this project will be documented in this file.
 See [Conventional Commits](https://conventionalcommits.org) for commit guidelines.
 
+# [3.8.0](https://github.com/apify/crawlee/compare/v3.7.3...v3.8.0) (2024-02-21)
+
+
+### Features
+
+* adaptive playwright crawler ([#2316](https://github.com/apify/crawlee/issues/2316)) ([8e4218a](https://github.com/apify/crawlee/commit/8e4218ada03cf485751def46f8c465b2d2a825c7))
+
+
+
+
+
 ## [3.7.3](https://github.com/apify/crawlee/compare/v3.7.2...v3.7.3) (2024-01-30)
 
 **Note:** Version bump only for package @crawlee/browser

diff --git a/packages/browser-crawler/package.json b/packages/browser-crawler/package.json
@@ -1,6 +1,6 @@
 {
     "name": "@crawlee/browser",
-    "version": "3.7.3",
+    "version": "3.8.0",
     "description": "The scalable web crawling and scraping library for JavaScript/Node.js. Enables development of data extraction and web automation jobs (not only) with headless Chrome and Puppeteer.",
     "engines": {
         "node": ">=16.0.0"
@@ -54,10 +54,10 @@
     },
     "dependencies": {
         "@apify/timeout": "^0.3.0",
-        "@crawlee/basic": "3.7.3",
-        "@crawlee/browser-pool": "3.7.3",
-        "@crawlee/types": "3.7.3",
-        "@crawlee/utils": "3.7.3",
+        "@crawlee/basic": "^3.8.0",
+        "@crawlee/browser-pool": "^3.8.0",
+        "@crawlee/types": "^3.8.0",
+        "@crawlee/utils": "^3.8.0",
         "ow": "^0.28.1",
         "tslib": "^2.4.0",
         "type-fest": "^4.0.0"

diff --git a/packages/browser-pool/CHANGELOG.md b/packages/browser-pool/CHANGELOG.md
@@ -3,6 +3,18 @@
 All notable changes to this project will be documented in this file.
 See [Conventional Commits](https://conventionalcommits.org) for commit guidelines.
 
+# [3.8.0](https://github.com/apify/crawlee/compare/v3.7.3...v3.8.0) (2024-02-21)
+
+
+### Bug Fixes
+
+* **puppeteer:** add 'process' to the browser bound methods ([#2329](https://github.com/apify/crawlee/issues/2329)) ([2750ba6](https://github.com/apify/crawlee/commit/2750ba646ef3c1d51eacdd8e7d67be0e14fb2a97))
+* **puppeteer:** support `puppeteer@v22` ([#2337](https://github.com/apify/crawlee/issues/2337)) ([3cc360a](https://github.com/apify/crawlee/commit/3cc360a1ea94147133f9785d65834f360f7b42a7))
+
+
+
+
+
 ## [3.7.3](https://github.com/apify/crawlee/compare/v3.7.2...v3.7.3) (2024-01-30)
 
 **Note:** Version bump only for package @crawlee/browser-pool

diff --git a/packages/browser-pool/package.json b/packages/browser-pool/package.json
@@ -1,6 +1,6 @@
 {
     "name": "@crawlee/browser-pool",
-    "version": "3.7.3",
+    "version": "3.8.0",
     "description": "Rotate multiple browsers using popular automation libraries such as Playwright or Puppeteer.",
     "engines": {
         "node": ">=16.0.0"
@@ -38,8 +38,8 @@
     "dependencies": {
         "@apify/log": "^2.4.0",
         "@apify/timeout": "^0.3.0",
-        "@crawlee/core": "3.7.3",
-        "@crawlee/types": "3.7.3",
+        "@crawlee/core": "^3.8.0",
+        "@crawlee/types": "^3.8.0",
         "fingerprint-generator": "^2.0.6",
         "fingerprint-injector": "^2.0.5",
         "lodash.merge": "^4.6.2",

diff --git a/packages/cheerio-crawler/CHANGELOG.md b/packages/cheerio-crawler/CHANGELOG.md
@@ -3,6 +3,14 @@
 All notable changes to this project will be documented in this file.
 See [Conventional Commits](https://conventionalcommits.org) for commit guidelines.
 
+# [3.8.0](https://github.com/apify/crawlee/compare/v3.7.3...v3.8.0) (2024-02-21)
+
+**Note:** Version bump only for package @crawlee/cheerio
+
+
+
+
+
 ## [3.7.3](https://github.com/apify/crawlee/compare/v3.7.2...v3.7.3) (2024-01-30)
 
 **Note:** Version bump only for package @crawlee/cheerio

diff --git a/packages/cheerio-crawler/package.json b/packages/cheerio-crawler/package.json
@@ -1,6 +1,6 @@
 {
     "name": "@crawlee/cheerio",
-    "version": "3.7.3",
+    "version": "3.8.0",
     "description": "The scalable web crawling and scraping library for JavaScript/Node.js. Enables development of data extraction and web automation jobs (not only) with headless Chrome and Puppeteer.",
     "engines": {
         "node": ">=16.0.0"
@@ -53,9 +53,9 @@
         "access": "public"
     },
     "dependencies": {
-        "@crawlee/http": "3.7.3",
-        "@crawlee/types": "3.7.3",
-        "@crawlee/utils": "3.7.3",
+        "@crawlee/http": "^3.8.0",
+        "@crawlee/types": "^3.8.0",
+        "@crawlee/utils": "^3.8.0",
         "cheerio": "^1.0.0-rc.12",
         "htmlparser2": "^9.0.0",
         "tslib": "^2.4.0"

diff --git a/packages/cli/CHANGELOG.md b/packages/cli/CHANGELOG.md
@@ -3,6 +3,14 @@
 All notable changes to this project will be documented in this file.
 See [Conventional Commits](https://conventionalcommits.org) for commit guidelines.
 
+# [3.8.0](https://github.com/apify/crawlee/compare/v3.7.3...v3.8.0) (2024-02-21)
+
+**Note:** Version bump only for package @crawlee/cli
+
+
+
+
+
 ## [3.7.3](https://github.com/apify/crawlee/compare/v3.7.2...v3.7.3) (2024-01-30)
 
 **Note:** Version bump only for package @crawlee/cli

diff --git a/packages/cli/package.json b/packages/cli/package.json
@@ -1,6 +1,6 @@
 {
     "name": "@crawlee/cli",
-    "version": "3.7.3",
+    "version": "3.8.0",
     "description": "The scalable web crawling and scraping library for JavaScript/Node.js. Enables development of data extraction and web automation jobs (not only) with headless Chrome and Puppeteer.",
     "engines": {
         "node": ">=16.0.0"
@@ -51,7 +51,7 @@
         "access": "public"
     },
     "dependencies": {
-        "@crawlee/templates": "3.7.3",
+        "@crawlee/templates": "^3.8.0",
         "ansi-colors": "^4.1.3",
         "fs-extra": "^11.0.0",
         "inquirer": "^8.2.4",

diff --git a/packages/core/CHANGELOG.md b/packages/core/CHANGELOG.md
@@ -3,6 +3,24 @@
 All notable changes to this project will be documented in this file.
 See [Conventional Commits](https://conventionalcommits.org) for commit guidelines.
 
+# [3.8.0](https://github.com/apify/crawlee/compare/v3.7.3...v3.8.0) (2024-02-21)
+
+
+### Bug Fixes
+
+* `createRequests` works correctly with `exclude` (and nothing else) ([#2321](https://github.com/apify/crawlee/issues/2321)) ([048db09](https://github.com/apify/crawlee/commit/048db0964a57ac570320ad495425733128235491))
+
+
+### Features
+
+* `KeyValueStore.recordExists()` ([#2339](https://github.com/apify/crawlee/issues/2339)) ([8507a65](https://github.com/apify/crawlee/commit/8507a65d1ad079f64c752a6ddb1d8fac9b494228))
+* accessing crawler state, key-value store and named datasets via crawling context ([#2283](https://github.com/apify/crawlee/issues/2283)) ([58dd5fc](https://github.com/apify/crawlee/commit/58dd5fcc25f31bb066402c46e48a9e5e91efd5c5))
+* adaptive playwright crawler ([#2316](https://github.com/apify/crawlee/issues/2316)) ([8e4218a](https://github.com/apify/crawlee/commit/8e4218ada03cf485751def46f8c465b2d2a825c7))
+
+
+
+
+
 ## [3.7.3](https://github.com/apify/crawlee/compare/v3.7.2...v3.7.3) (2024-01-30)
 
 

diff --git a/packages/core/package.json b/packages/core/package.json
@@ -1,6 +1,6 @@
 {
     "name": "@crawlee/core",
-    "version": "3.7.3",
+    "version": "3.8.0",
     "description": "The scalable web crawling and scraping library for JavaScript/Node.js. Enables development of data extraction and web automation jobs (not only) with headless Chrome and Puppeteer.",
     "engines": {
         "node": ">=16.0.0"
@@ -59,9 +59,9 @@
         "@apify/pseudo_url": "^2.0.30",
         "@apify/timeout": "^0.3.0",
         "@apify/utilities": "^2.7.10",
-        "@crawlee/memory-storage": "3.7.3",
-        "@crawlee/types": "3.7.3",
-        "@crawlee/utils": "3.7.3",
+        "@crawlee/memory-storage": "^3.8.0",
+        "@crawlee/types": "^3.8.0",
+        "@crawlee/utils": "^3.8.0",
         "@sapphire/async-queue": "^1.5.1",
         "@types/tough-cookie": "^4.0.2",
         "@vladfrangu/async_event_emitter": "^2.2.2",

diff --git a/packages/crawlee/CHANGELOG.md b/packages/crawlee/CHANGELOG.md
@@ -3,6 +3,14 @@
 All notable changes to this project will be documented in this file.
 See [Conventional Commits](https://conventionalcommits.org) for commit guidelines.
 
+# [3.8.0](https://github.com/apify/crawlee/compare/v3.7.3...v3.8.0) (2024-02-21)
+
+**Note:** Version bump only for package crawlee
+
+
+
+
+
 ## [3.7.3](https://github.com/apify/crawlee/compare/v3.7.2...v3.7.3) (2024-01-30)
 
 **Note:** Version bump only for package crawlee

diff --git a/packages/crawlee/package.json b/packages/crawlee/package.json
@@ -1,6 +1,6 @@
 {
     "name": "crawlee",
-    "version": "3.7.3",
+    "version": "3.8.0",
     "description": "The scalable web crawling and scraping library for JavaScript/Node.js. Enables development of data extraction and web automation jobs (not only) with headless Chrome and Puppeteer.",
     "engines": {
         "node": ">=16.0.0"
@@ -54,18 +54,18 @@
         "access": "public"
     },
     "dependencies": {
-        "@crawlee/basic": "3.7.3",
-        "@crawlee/browser": "3.7.3",
-        "@crawlee/browser-pool": "3.7.3",
-        "@crawlee/cheerio": "3.7.3",
-        "@crawlee/cli": "3.7.3",
-        "@crawlee/core": "3.7.3",
-        "@crawlee/http": "3.7.3",
-        "@crawlee/jsdom": "3.7.3",
-        "@crawlee/linkedom": "3.7.3",
-        "@crawlee/playwright": "3.7.3",
-        "@crawlee/puppeteer": "3.7.3",
-        "@crawlee/utils": "3.7.3",
+        "@crawlee/basic": "^3.8.0",
+        "@crawlee/browser": "^3.8.0",
+        "@crawlee/browser-pool": "^3.8.0",
+        "@crawlee/cheerio": "^3.8.0",
+        "@crawlee/cli": "^3.8.0",
+        "@crawlee/core": "^3.8.0",
+        "@crawlee/http": "^3.8.0",
+        "@crawlee/jsdom": "^3.8.0",
+        "@crawlee/linkedom": "^3.8.0",
+        "@crawlee/playwright": "^3.8.0",
+        "@crawlee/puppeteer": "^3.8.0",
+        "@crawlee/utils": "^3.8.0",
         "import-local": "^3.1.0",
         "tslib": "^2.4.0"
     },

diff --git a/packages/http-crawler/CHANGELOG.md b/packages/http-crawler/CHANGELOG.md
@@ -3,6 +3,14 @@
 All notable changes to this project will be documented in this file.
 See [Conventional Commits](https://conventionalcommits.org) for commit guidelines.
 
+# [3.8.0](https://github.com/apify/crawlee/compare/v3.7.3...v3.8.0) (2024-02-21)
+
+**Note:** Version bump only for package @crawlee/http
+
+
+
+
+
 ## [3.7.3](https://github.com/apify/crawlee/compare/v3.7.2...v3.7.3) (2024-01-30)
 
 **Note:** Version bump only for package @crawlee/http

diff --git a/packages/http-crawler/package.json b/packages/http-crawler/package.json
@@ -1,6 +1,6 @@
 {
     "name": "@crawlee/http",
-    "version": "3.7.3",
+    "version": "3.8.0",
     "description": "The scalable web crawling and scraping library for JavaScript/Node.js. Enables development of data extraction and web automation jobs (not only) with headless Chrome and Puppeteer.",
     "engines": {
         "node": ">=16.0.0"
@@ -55,9 +55,9 @@
     "dependencies": {
         "@apify/timeout": "^0.3.0",
         "@apify/utilities": "^2.7.10",
-        "@crawlee/basic": "3.7.3",
-        "@crawlee/types": "3.7.3",
-        "@crawlee/utils": "3.7.3",
+        "@crawlee/basic": "^3.8.0",
+        "@crawlee/types": "^3.8.0",
+        "@crawlee/utils": "^3.8.0",
         "@types/content-type": "^1.1.5",
         "cheerio": "^1.0.0-rc.12",
         "content-type": "^1.0.4",

diff --git a/packages/jsdom-crawler/CHANGELOG.md b/packages/jsdom-crawler/CHANGELOG.md
@@ -3,6 +3,14 @@
 All notable changes to this project will be documented in this file.
 See [Conventional Commits](https://conventionalcommits.org) for commit guidelines.
 
+# [3.8.0](https://github.com/apify/crawlee/compare/v3.7.3...v3.8.0) (2024-02-21)
+
+**Note:** Version bump only for package @crawlee/jsdom
+
+
+
+
+
 ## [3.7.3](https://github.com/apify/crawlee/compare/v3.7.2...v3.7.3) (2024-01-30)
 
 **Note:** Version bump only for package @crawlee/jsdom