chore(release): v3.7.0

apify · Dec 21, 2023 · 3641c87 · 3641c87
1 parent e19a773
commit 3641c87
Show file tree

Hide file tree

Showing 35 changed files with 318 additions and 120 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -3,6 +3,30 @@
 All notable changes to this project will be documented in this file.
 See [Conventional Commits](https://conventionalcommits.org) for commit guidelines.
 
+# [3.7.0](https://github.com/apify/crawlee/compare/v3.6.2...v3.7.0) (2023-12-21)
+
+
+### Bug Fixes
+
+* `retryOnBlocked` doesn't override the blocked HTTP codes ([#2243](https://github.com/apify/crawlee/issues/2243)) ([81672c3](https://github.com/apify/crawlee/commit/81672c3d1db1dcdcffb868de5740addff82cf112))
+* **browser-pool:** respect user options before assigning fingerpints ([#2190](https://github.com/apify/crawlee/issues/2190)) ([f050776](https://github.com/apify/crawlee/commit/f050776a916a0530aca6727a447a49252e643417)), closes [#2164](https://github.com/apify/crawlee/issues/2164)
+* filter out empty globs ([#2205](https://github.com/apify/crawlee/issues/2205)) ([41322ab](https://github.com/apify/crawlee/commit/41322ab32d7db7baf61638d00fd7eaec9e5330f1)), closes [#2200](https://github.com/apify/crawlee/issues/2200)
+* make CLI work on Windows too with `--no-purge` ([#2244](https://github.com/apify/crawlee/issues/2244)) ([83f3179](https://github.com/apify/crawlee/commit/83f3179d6963dbaca4232d866356cc766db11dd8))
+* make SessionPool queue up getSession calls to prevent overruns ([#2239](https://github.com/apify/crawlee/issues/2239)) ([0f5665c](https://github.com/apify/crawlee/commit/0f5665c473371bff5a5d3abee3c3a9d23f2aeb23)), closes [#1667](https://github.com/apify/crawlee/issues/1667)
+* **MemoryStorage:** lock request JSON file when reading to support multiple process crawling ([#2215](https://github.com/apify/crawlee/issues/2215)) ([eb84ce9](https://github.com/apify/crawlee/commit/eb84ce9ce5540b72d5799b1f66c80938d57bc1cc))
+
+
+### Features
+
+* allow configuring crawler statistics ([#2213](https://github.com/apify/crawlee/issues/2213)) ([9fd60e4](https://github.com/apify/crawlee/commit/9fd60e4036dce720c71f2d169a8eccbc4c813a96)), closes [#1789](https://github.com/apify/crawlee/issues/1789)
+* check enqueue link strategy post redirect ([#2238](https://github.com/apify/crawlee/issues/2238)) ([3c5f9d6](https://github.com/apify/crawlee/commit/3c5f9d6056158e042e12d75b2b1b21ef6c32e618)), closes [#2173](https://github.com/apify/crawlee/issues/2173)
+* log cause with `retryOnBlocked` ([#2252](https://github.com/apify/crawlee/issues/2252)) ([e19a773](https://github.com/apify/crawlee/commit/e19a773693cfc5e65c1e2321bfc8b73c9844ea8b)), closes [#2249](https://github.com/apify/crawlee/issues/2249)
+* robots.txt and sitemap.xml utils ([#2214](https://github.com/apify/crawlee/issues/2214)) ([fdfec4f](https://github.com/apify/crawlee/commit/fdfec4f4d0a0f925b49015d2d63932c4a82555ba)), closes [#2187](https://github.com/apify/crawlee/issues/2187)
+
+
+
+
+
 ## [3.6.2](https://github.com/apify/crawlee/compare/v3.6.1...v3.6.2) (2023-11-26)
 
 

diff --git a/lerna.json b/lerna.json
@@ -2,7 +2,7 @@
   "packages": [
     "packages/*"
   ],
-  "version": "3.6.2",
+  "version": "3.7.0",
   "command": {
     "version": {
       "conventionalCommits": true,

diff --git a/packages/basic-crawler/CHANGELOG.md b/packages/basic-crawler/CHANGELOG.md
@@ -3,6 +3,19 @@
 All notable changes to this project will be documented in this file.
 See [Conventional Commits](https://conventionalcommits.org) for commit guidelines.
 
+# [3.7.0](https://github.com/apify/crawlee/compare/v3.6.2...v3.7.0) (2023-12-21)
+
+
+### Features
+
+* allow configuring crawler statistics ([#2213](https://github.com/apify/crawlee/issues/2213)) ([9fd60e4](https://github.com/apify/crawlee/commit/9fd60e4036dce720c71f2d169a8eccbc4c813a96)), closes [#1789](https://github.com/apify/crawlee/issues/1789)
+* check enqueue link strategy post redirect ([#2238](https://github.com/apify/crawlee/issues/2238)) ([3c5f9d6](https://github.com/apify/crawlee/commit/3c5f9d6056158e042e12d75b2b1b21ef6c32e618)), closes [#2173](https://github.com/apify/crawlee/issues/2173)
+* log cause with `retryOnBlocked` ([#2252](https://github.com/apify/crawlee/issues/2252)) ([e19a773](https://github.com/apify/crawlee/commit/e19a773693cfc5e65c1e2321bfc8b73c9844ea8b)), closes [#2249](https://github.com/apify/crawlee/issues/2249)
+
+
+
+
+
 ## [3.6.2](https://github.com/apify/crawlee/compare/v3.6.1...v3.6.2) (2023-11-26)
 
 **Note:** Version bump only for package @crawlee/basic

diff --git a/packages/basic-crawler/package.json b/packages/basic-crawler/package.json
@@ -1,6 +1,6 @@
 {
     "name": "@crawlee/basic",
-    "version": "3.6.2",
+    "version": "3.7.0",
     "description": "The scalable web crawling and scraping library for JavaScript/Node.js. Enables development of data extraction and web automation jobs (not only) with headless Chrome and Puppeteer.",
     "engines": {
         "node": ">=16.0.0"
@@ -48,9 +48,9 @@
         "@apify/log": "^2.4.0",
         "@apify/timeout": "^0.3.0",
         "@apify/utilities": "^2.7.10",
-        "@crawlee/core": "3.6.2",
-        "@crawlee/types": "3.6.2",
-        "@crawlee/utils": "3.6.2",
+        "@crawlee/core": "^3.7.0",
+        "@crawlee/types": "^3.7.0",
+        "@crawlee/utils": "^3.7.0",
         "got-scraping": "^4.0.0",
         "ow": "^0.28.1",
         "tldts": "^6.0.0",

diff --git a/packages/browser-crawler/CHANGELOG.md b/packages/browser-crawler/CHANGELOG.md
@@ -3,6 +3,23 @@
 All notable changes to this project will be documented in this file.
 See [Conventional Commits](https://conventionalcommits.org) for commit guidelines.
 
+# [3.7.0](https://github.com/apify/crawlee/compare/v3.6.2...v3.7.0) (2023-12-21)
+
+
+### Bug Fixes
+
+* `retryOnBlocked` doesn't override the blocked HTTP codes ([#2243](https://github.com/apify/crawlee/issues/2243)) ([81672c3](https://github.com/apify/crawlee/commit/81672c3d1db1dcdcffb868de5740addff82cf112))
+
+
+### Features
+
+* check enqueue link strategy post redirect ([#2238](https://github.com/apify/crawlee/issues/2238)) ([3c5f9d6](https://github.com/apify/crawlee/commit/3c5f9d6056158e042e12d75b2b1b21ef6c32e618)), closes [#2173](https://github.com/apify/crawlee/issues/2173)
+* log cause with `retryOnBlocked` ([#2252](https://github.com/apify/crawlee/issues/2252)) ([e19a773](https://github.com/apify/crawlee/commit/e19a773693cfc5e65c1e2321bfc8b73c9844ea8b)), closes [#2249](https://github.com/apify/crawlee/issues/2249)
+
+
+
+
+
 ## [3.6.2](https://github.com/apify/crawlee/compare/v3.6.1...v3.6.2) (2023-11-26)
 
 **Note:** Version bump only for package @crawlee/browser

diff --git a/packages/browser-crawler/package.json b/packages/browser-crawler/package.json
@@ -1,6 +1,6 @@
 {
     "name": "@crawlee/browser",
-    "version": "3.6.2",
+    "version": "3.7.0",
     "description": "The scalable web crawling and scraping library for JavaScript/Node.js. Enables development of data extraction and web automation jobs (not only) with headless Chrome and Puppeteer.",
     "engines": {
         "node": ">=16.0.0"
@@ -54,10 +54,10 @@
     },
     "dependencies": {
         "@apify/timeout": "^0.3.0",
-        "@crawlee/basic": "3.6.2",
-        "@crawlee/browser-pool": "3.6.2",
-        "@crawlee/types": "3.6.2",
-        "@crawlee/utils": "3.6.2",
+        "@crawlee/basic": "^3.7.0",
+        "@crawlee/browser-pool": "^3.7.0",
+        "@crawlee/types": "^3.7.0",
+        "@crawlee/utils": "^3.7.0",
         "ow": "^0.28.1",
         "tslib": "^2.4.0"
     }

diff --git a/packages/browser-pool/CHANGELOG.md b/packages/browser-pool/CHANGELOG.md
@@ -3,6 +3,17 @@
 All notable changes to this project will be documented in this file.
 See [Conventional Commits](https://conventionalcommits.org) for commit guidelines.
 
+# [3.7.0](https://github.com/apify/crawlee/compare/v3.6.2...v3.7.0) (2023-12-21)
+
+
+### Bug Fixes
+
+* **browser-pool:** respect user options before assigning fingerpints ([#2190](https://github.com/apify/crawlee/issues/2190)) ([f050776](https://github.com/apify/crawlee/commit/f050776a916a0530aca6727a447a49252e643417)), closes [#2164](https://github.com/apify/crawlee/issues/2164)
+
+
+
+
+
 ## [3.6.2](https://github.com/apify/crawlee/compare/v3.6.1...v3.6.2) (2023-11-26)
 
 **Note:** Version bump only for package @crawlee/browser-pool

diff --git a/packages/browser-pool/package.json b/packages/browser-pool/package.json
@@ -1,6 +1,6 @@
 {
     "name": "@crawlee/browser-pool",
-    "version": "3.6.2",
+    "version": "3.7.0",
     "description": "Rotate multiple browsers using popular automation libraries such as Playwright or Puppeteer.",
     "engines": {
         "node": ">=16.0.0"
@@ -38,8 +38,8 @@
     "dependencies": {
         "@apify/log": "^2.4.0",
         "@apify/timeout": "^0.3.0",
-        "@crawlee/core": "3.6.2",
-        "@crawlee/types": "3.6.2",
+        "@crawlee/core": "^3.7.0",
+        "@crawlee/types": "^3.7.0",
         "fingerprint-generator": "^2.0.6",
         "fingerprint-injector": "^2.0.5",
         "lodash.merge": "^4.6.2",

diff --git a/packages/cheerio-crawler/CHANGELOG.md b/packages/cheerio-crawler/CHANGELOG.md
@@ -3,6 +3,14 @@
 All notable changes to this project will be documented in this file.
 See [Conventional Commits](https://conventionalcommits.org) for commit guidelines.
 
+# [3.7.0](https://github.com/apify/crawlee/compare/v3.6.2...v3.7.0) (2023-12-21)
+
+**Note:** Version bump only for package @crawlee/cheerio
+
+
+
+
+
 ## [3.6.2](https://github.com/apify/crawlee/compare/v3.6.1...v3.6.2) (2023-11-26)
 
 **Note:** Version bump only for package @crawlee/cheerio

diff --git a/packages/cheerio-crawler/package.json b/packages/cheerio-crawler/package.json
@@ -1,6 +1,6 @@
 {
     "name": "@crawlee/cheerio",
-    "version": "3.6.2",
+    "version": "3.7.0",
     "description": "The scalable web crawling and scraping library for JavaScript/Node.js. Enables development of data extraction and web automation jobs (not only) with headless Chrome and Puppeteer.",
     "engines": {
         "node": ">=16.0.0"
@@ -53,8 +53,8 @@
         "access": "public"
     },
     "dependencies": {
-        "@crawlee/http": "3.6.2",
-        "@crawlee/types": "3.6.2",
+        "@crawlee/http": "^3.7.0",
+        "@crawlee/types": "^3.7.0",
         "cheerio": "^1.0.0-rc.12",
         "htmlparser2": "^9.0.0",
         "tslib": "^2.4.0"

diff --git a/packages/cli/CHANGELOG.md b/packages/cli/CHANGELOG.md
@@ -3,6 +3,17 @@
 All notable changes to this project will be documented in this file.
 See [Conventional Commits](https://conventionalcommits.org) for commit guidelines.
 
+# [3.7.0](https://github.com/apify/crawlee/compare/v3.6.2...v3.7.0) (2023-12-21)
+
+
+### Bug Fixes
+
+* make CLI work on Windows too with `--no-purge` ([#2244](https://github.com/apify/crawlee/issues/2244)) ([83f3179](https://github.com/apify/crawlee/commit/83f3179d6963dbaca4232d866356cc766db11dd8))
+
+
+
+
+
 ## [3.6.2](https://github.com/apify/crawlee/compare/v3.6.1...v3.6.2) (2023-11-26)
 
 **Note:** Version bump only for package @crawlee/cli

diff --git a/packages/cli/package.json b/packages/cli/package.json
@@ -1,6 +1,6 @@
 {
     "name": "@crawlee/cli",
-    "version": "3.6.2",
+    "version": "3.7.0",
     "description": "The scalable web crawling and scraping library for JavaScript/Node.js. Enables development of data extraction and web automation jobs (not only) with headless Chrome and Puppeteer.",
     "engines": {
         "node": ">=16.0.0"
@@ -51,7 +51,7 @@
         "access": "public"
     },
     "dependencies": {
-        "@crawlee/templates": "3.6.2",
+        "@crawlee/templates": "^3.7.0",
         "ansi-colors": "^4.1.3",
         "fs-extra": "^11.0.0",
         "inquirer": "^8.2.4",

diff --git a/packages/core/CHANGELOG.md b/packages/core/CHANGELOG.md
@@ -3,6 +3,25 @@
 All notable changes to this project will be documented in this file.
 See [Conventional Commits](https://conventionalcommits.org) for commit guidelines.
 
+# [3.7.0](https://github.com/apify/crawlee/compare/v3.6.2...v3.7.0) (2023-12-21)
+
+
+### Bug Fixes
+
+* `retryOnBlocked` doesn't override the blocked HTTP codes ([#2243](https://github.com/apify/crawlee/issues/2243)) ([81672c3](https://github.com/apify/crawlee/commit/81672c3d1db1dcdcffb868de5740addff82cf112))
+* filter out empty globs ([#2205](https://github.com/apify/crawlee/issues/2205)) ([41322ab](https://github.com/apify/crawlee/commit/41322ab32d7db7baf61638d00fd7eaec9e5330f1)), closes [#2200](https://github.com/apify/crawlee/issues/2200)
+* make SessionPool queue up getSession calls to prevent overruns ([#2239](https://github.com/apify/crawlee/issues/2239)) ([0f5665c](https://github.com/apify/crawlee/commit/0f5665c473371bff5a5d3abee3c3a9d23f2aeb23)), closes [#1667](https://github.com/apify/crawlee/issues/1667)
+
+
+### Features
+
+* allow configuring crawler statistics ([#2213](https://github.com/apify/crawlee/issues/2213)) ([9fd60e4](https://github.com/apify/crawlee/commit/9fd60e4036dce720c71f2d169a8eccbc4c813a96)), closes [#1789](https://github.com/apify/crawlee/issues/1789)
+* check enqueue link strategy post redirect ([#2238](https://github.com/apify/crawlee/issues/2238)) ([3c5f9d6](https://github.com/apify/crawlee/commit/3c5f9d6056158e042e12d75b2b1b21ef6c32e618)), closes [#2173](https://github.com/apify/crawlee/issues/2173)
+
+
+
+
+
 ## [3.6.2](https://github.com/apify/crawlee/compare/v3.6.1...v3.6.2) (2023-11-26)
 
 

diff --git a/packages/core/package.json b/packages/core/package.json
@@ -1,6 +1,6 @@
 {
     "name": "@crawlee/core",
-    "version": "3.6.2",
+    "version": "3.7.0",
     "description": "The scalable web crawling and scraping library for JavaScript/Node.js. Enables development of data extraction and web automation jobs (not only) with headless Chrome and Puppeteer.",
     "engines": {
         "node": ">=16.0.0"
@@ -59,9 +59,9 @@
         "@apify/pseudo_url": "^2.0.30",
         "@apify/timeout": "^0.3.0",
         "@apify/utilities": "^2.7.10",
-        "@crawlee/memory-storage": "3.6.2",
-        "@crawlee/types": "3.6.2",
-        "@crawlee/utils": "3.6.2",
+        "@crawlee/memory-storage": "^3.7.0",
+        "@crawlee/types": "^3.7.0",
+        "@crawlee/utils": "^3.7.0",
         "@sapphire/async-queue": "^1.5.1",
         "@types/tough-cookie": "^4.0.2",
         "@vladfrangu/async_event_emitter": "^2.2.2",

diff --git a/packages/crawlee/CHANGELOG.md b/packages/crawlee/CHANGELOG.md
@@ -3,6 +3,14 @@
 All notable changes to this project will be documented in this file.
 See [Conventional Commits](https://conventionalcommits.org) for commit guidelines.
 
+# [3.7.0](https://github.com/apify/crawlee/compare/v3.6.2...v3.7.0) (2023-12-21)
+
+**Note:** Version bump only for package crawlee
+
+
+
+
+
 ## [3.6.2](https://github.com/apify/crawlee/compare/v3.6.1...v3.6.2) (2023-11-26)
 
 **Note:** Version bump only for package crawlee

diff --git a/packages/crawlee/package.json b/packages/crawlee/package.json
@@ -1,6 +1,6 @@
 {
     "name": "crawlee",
-    "version": "3.6.2",
+    "version": "3.7.0",
     "description": "The scalable web crawling and scraping library for JavaScript/Node.js. Enables development of data extraction and web automation jobs (not only) with headless Chrome and Puppeteer.",
     "engines": {
         "node": ">=16.0.0"
@@ -54,18 +54,18 @@
         "access": "public"
     },
     "dependencies": {
-        "@crawlee/basic": "3.6.2",
-        "@crawlee/browser": "3.6.2",
-        "@crawlee/browser-pool": "3.6.2",
-        "@crawlee/cheerio": "3.6.2",
-        "@crawlee/cli": "3.6.2",
-        "@crawlee/core": "3.6.2",
-        "@crawlee/http": "3.6.2",
-        "@crawlee/jsdom": "3.6.2",
-        "@crawlee/linkedom": "3.6.2",
-        "@crawlee/playwright": "3.6.2",
-        "@crawlee/puppeteer": "3.6.2",
-        "@crawlee/utils": "3.6.2",
+        "@crawlee/basic": "^3.7.0",
+        "@crawlee/browser": "^3.7.0",
+        "@crawlee/browser-pool": "^3.7.0",
+        "@crawlee/cheerio": "^3.7.0",
+        "@crawlee/cli": "^3.7.0",
+        "@crawlee/core": "^3.7.0",
+        "@crawlee/http": "^3.7.0",
+        "@crawlee/jsdom": "^3.7.0",
+        "@crawlee/linkedom": "^3.7.0",
+        "@crawlee/playwright": "^3.7.0",
+        "@crawlee/puppeteer": "^3.7.0",
+        "@crawlee/utils": "^3.7.0",
         "import-local": "^3.1.0",
         "tslib": "^2.4.0"
     },

diff --git a/packages/http-crawler/CHANGELOG.md b/packages/http-crawler/CHANGELOG.md
@@ -3,6 +3,18 @@
 All notable changes to this project will be documented in this file.
 See [Conventional Commits](https://conventionalcommits.org) for commit guidelines.
 
+# [3.7.0](https://github.com/apify/crawlee/compare/v3.6.2...v3.7.0) (2023-12-21)
+
+
+### Features
+
+* check enqueue link strategy post redirect ([#2238](https://github.com/apify/crawlee/issues/2238)) ([3c5f9d6](https://github.com/apify/crawlee/commit/3c5f9d6056158e042e12d75b2b1b21ef6c32e618)), closes [#2173](https://github.com/apify/crawlee/issues/2173)
+* log cause with `retryOnBlocked` ([#2252](https://github.com/apify/crawlee/issues/2252)) ([e19a773](https://github.com/apify/crawlee/commit/e19a773693cfc5e65c1e2321bfc8b73c9844ea8b)), closes [#2249](https://github.com/apify/crawlee/issues/2249)
+
+
+
+
+
 ## [3.6.2](https://github.com/apify/crawlee/compare/v3.6.1...v3.6.2) (2023-11-26)
 
 **Note:** Version bump only for package @crawlee/http

diff --git a/packages/http-crawler/package.json b/packages/http-crawler/package.json
@@ -1,6 +1,6 @@
 {
     "name": "@crawlee/http",
-    "version": "3.6.2",
+    "version": "3.7.0",
     "description": "The scalable web crawling and scraping library for JavaScript/Node.js. Enables development of data extraction and web automation jobs (not only) with headless Chrome and Puppeteer.",
     "engines": {
         "node": ">=16.0.0"
@@ -55,9 +55,9 @@
     "dependencies": {
         "@apify/timeout": "^0.3.0",
         "@apify/utilities": "^2.7.10",
-        "@crawlee/basic": "3.6.2",
-        "@crawlee/types": "3.6.2",
-        "@crawlee/utils": "3.6.2",
+        "@crawlee/basic": "^3.7.0",
+        "@crawlee/types": "^3.7.0",
+        "@crawlee/utils": "^3.7.0",
         "@types/content-type": "^1.1.5",
         "cheerio": "^1.0.0-rc.12",
         "content-type": "^1.0.4",

diff --git a/packages/jsdom-crawler/CHANGELOG.md b/packages/jsdom-crawler/CHANGELOG.md
@@ -3,6 +3,14 @@
 All notable changes to this project will be documented in this file.
 See [Conventional Commits](https://conventionalcommits.org) for commit guidelines.
 
+# [3.7.0](https://github.com/apify/crawlee/compare/v3.6.2...v3.7.0) (2023-12-21)
+
+**Note:** Version bump only for package @crawlee/jsdom
+
+
+
+
+
 ## [3.6.2](https://github.com/apify/crawlee/compare/v3.6.1...v3.6.2) (2023-11-26)
 
 **Note:** Version bump only for package @crawlee/jsdom