Upgrading dependencies to resolve security vulnerabilities and usage of deprecated versions

[esakal]

Hi,

I opened this issue to consult about a sensitive area in the open source. It has been too long since we upgraded the dependencies of this project.

Due to the nature of this open-source, no one has access to all the scrappers; thus, doing such a change is risky and complicated:

1. We cannot write unit tests to cover it (most of the issues will appear during execution)
2. We cannot run end-to-end tests on all scrapers.
3. Only developers that have access to a company can develop it.

How do we proceed with it?

[sagiba]

How about having each scraper as a separate node module with its own set of dependencies? That way you'd be able to quickly update the ones that you can test without worrying about breaking the others, and you won't have to install modules that you don't need and might have stale dependencies.

[esakal]

@sagiba interesting idea, but by doing so we will enforce some complication on the consumer if they are using scrapers that have mixed overlapping dependencies with different versions. Also in many monorepo libraries like NRWL the dependencies list is shared to avoid complication.