v3.3.0

3.3.0 (2023-03-09)

Bug Fixes

• add proxyUrl to DownloadListOfUrlsOptions (779be1e), closes #1780
• CheerioCrawler: pass ixXml down to response parser (#1807) (af7a5c4), closes #1794
• ignore invalid URLs in enqueueLinks in browser crawlers (#1803) (5ac336c)
• MemoryStorage: request queues race conditions causing crashes (#1806) (083a9db), closes #1792
• MemoryStorage: RequestQueue should respect forefront (#1816) (b68e86a), closes #1787
• MemoryStorage: RequestQueue#handledRequestCount should update (#1817) (a775e4a), closes #1764

Features

• add basic support for setStatusMessage (#1790), (#1808) (c318980)
• add exclude option to enqueueLinks (#1786) (2e833dc), closes #1785

v3.2.2

3.2.2 (2023-02-08)

Bug Fixes

• MemoryStorage: request queues saved in the wrong place (#1779) (19409db)

v3.2.1

3.2.1 (2023-02-07)

Bug Fixes

• add QueueOperationInfo export to the core package (5ec6c24)

v3.2.0

3.2.0 (2023-02-07)

Bug Fixes

• allow userData option in enqueueLinksByClickingElements (#1749) (736f85d), closes #1617
• clone request.userData when creating new request object (#1728) (222ef59), closes #1725
• Correctly compute pendingRequestCount in request queue (#1765) (946535f)
• declare missing dependency on tslib (27e96c8), closes #1747
• ensure CrawlingContext interface is inferred correctly in route handlers (aa84633)
• KeyValueStore: big buffers should not crash (#1734) (2f682f7), closes #1732 #1710
• memory-storage: dont fail when storage already purged (#1737) (8694027), closes #1736
• update playwright to 1.29.2 and make peer dep. less strict (#1735) (c654fcd), closes #1723
• utils: add missing dependency on ow (bf0e03c), closes #1716

Features

• add forefront option to all enqueueLinks variants (#1760) (a01459d), closes #1483
• enqueueLinks: add SameOrigin strategy and relax protocol matching for the other strategies (#1748) (4ba982a)
• MemoryStorage: read from fs if persistStorage is enabled, ram only otherwise (#1761) (e903980)

v3.1.4

3.1.4 (2022-12-14)

Bug Fixes

• session.markBad() on requestHandler error (#1709) (e87eb1f), closes #1635

v3.1.3

3.1.3 (2022-12-07)

Bug Fixes

• remove memory leaks from migration event handling (#1679) (49bba25), closes #1670

Features

• always show error origin if inside the userland (#1677) (bbe9045)
• hideInternalConsole in JSDOMCrawler (#1707) (8975f90)

v3.1.2

3.1.2 (2022-11-15)

Bug Fixes

• injectJQuery in context does not survive navs (#1661) (493a7cf)
• make router error message more helpful for undefined routes (#1678) (ab359d8)
• MemoryStorage: correctly respect the desc option (#1666) (b5f37f6)
• requestHandlerTimeout timing (#1660) (493ea0c)
• shallow clone browserPoolOptions before normalization (#1665) (22467ca)
• support headfull mode in playwright js project template (ea2e61b)
• support headfull mode in puppeteer js project template (e6aceb8)

Features

• jsdom-crawler: add runScripts option (#1668) (8ef90bc)

v3.1.1

3.1.1 (2022-11-07)

Bug Fixes

• utils.playwright.blockRequests warning message (#1632) (76549eb)
• concurrency option override order (#1649) (7bbad03)
• handle non-error objects thrown gracefully (#1652) (c3a4e1a)
• mark session as bad on failed requests (#1647) (445ae43)
• support reloading of sessions with lots of retries (ebc89d2)
• fix type errors when playwright is not installed (#1637) (de9db0c)
• upgrade to puppeteer@19.x (#1623) (ce36d6b)

Features

• add static set and useStorageClient shortcuts to Configuration (2e66fa2)
• enable migration testing (#1583) (ee3a68f)
• playwright: disable animations when taking screenshots (#1601) (4e63034)

v3.1.0

3.1.0 (2022-10-13)

Bug Fixes

• add overload for KeyValueStore.getValue with defaultValue (#1541) (e3cb509)
• add retry attempts to methods in CLI (#1588) (9142e59)
• allow label in enqueueLinksByClickingElements options (#1525) (18b7c25)
• basic-crawler: handle request.noRetry after errorHandler (#1542) (2a2040e)
• build storage classes by using this instead of the class (#1596) (2b14eb7)
• correct some typing exports (#1527) (4a136e5)
• do not hide stack trace of (retried) Type/Syntax/ReferenceErrors (469b4b5)
• enqueueLinks: ensure the enqueue strategy is respected alongside user patterns (#1509) (2b0eeed)
• enqueueLinks: prevent useless request creations when filtering by user patterns (#1510) (cb8fe36)
• export Cookie from crawlee metapackage (7b02ceb)
• handle redirect cookies (#1521) (2f7fc7c)
• http-crawler: do not hang on POST without payload (#1546) (8c87390)
• remove undeclared dependency on core package from puppeteer utils (827ae60)
• support TypeScript 4.8 (#1507) (4c3a504)
• wait for persist state listeners to run when event manager closes (#1481) (aa550ed)

Features

• add Dataset.exportToCSV and Dataset.exportToJSON
• add Dataset.getData() shortcut (522ed6e)
• add utils.downloadListOfUrls to crawlee metapackage (7b33b0a)
• add utils.parseOpenGraph() (#1555) (059f85e)
• add utils.playwright.compileScript (#1559) (2e14162)
• add utils.playwright.infiniteScroll (#1543) (60c8289), closes #1528
• add utils.playwright.saveSnapshot (#1544) (a4ceef0)
• add global useState helper (#1551) (2b03177)
• allow disabling storage persistence (#1539) (f65e3c6)
• bump puppeteer support to 17.x (#1519) (b97a852)
• core: add forefront option to enqueueLinks helper (f8755b6), closes #1595
• don't close page before calling errorHandler (#1548) (1c8cd82)
• enqueue links by clicking for Playwright (#1545) (3d25ade)
• error tracker (#1467) (6bfe1ce)
• make the CLI download directly from GitHub (#1540) (3ff398a)
• router: add userdata generic to addHandler (#1547) (19cdf13)
• use JSON5 for INPUT.json to support comments (#1538) (09133ff)

v3.0.4

3.0.4 (2022-08-22)

Features

• bump puppeteer support to 15.1

Bug Fixes

• key value stores emitting an error when multiple write promises ran in parallel (#1460) (f201cca)
• fix dockerfiles in project templates