refactor: add project management

[cwegener]

High-level overview of changes:

1. rename src to flaresolverr in order to namespace the package
   appropriately

2. Move package.json from root folder into flaresolverr so that it can
   be read by the utils.get_flaresolverr_version() method

3. Rewrite necessary import statements to use the full package name as
   per step 1

4. Move script entry point into separate method (cli_run)

5. Manually added poetry dependencies as per existing requirements.txt
   file

[HLFH]

Cool thanks.

For 5, why not using Hatch instead of Poetry?

[cwegener]

Cool thanks.

For 5, why not using Hatch instead of Poetry?

No particular reason. It's just what I'm used to using. Happy to have it be hatch instead of poetry.

I did give it a quick try to create pyproject.toml for hatch, but I'm not very comfortable in using hatch. @HLFH If you have a bit of experience with hatch, maybe you can paste what you think a solid pyproject.toml should look like for hatch ...

Here is what I have at the moment ...

[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"

[project]
name = "flaresolverr"
description = "Proxy server to bypass Cloudflare protection"
version = "3.0.0"
readme = "README.md"
requires-python = ">=3.7"
license = "MIT"
keywords = []
authors = [
  { name = "Diego Heras (ngosang)", email = "ngosang@hotmail.es" },
]
classifiers = [
  "Development Status :: 4 - Beta",
  "Programming Language :: Python",
  "Programming Language :: Python :: 3.7",
  "Programming Language :: Python :: 3.8",
  "Programming Language :: Python :: 3.9",
  "Programming Language :: Python :: 3.10",
  "Programming Language :: Python :: 3.11",
  "Programming Language :: Python :: Implementation :: CPython",
  "Programming Language :: Python :: Implementation :: PyPy",
]
dependencies = [
  "bottle==0.12.23",
  "waitress==2.1.2",
  "selenium==4.4.3",
  "func-timeout==4.3.5",
  "requests==2.28.1",
  "websockets==10.3",
  "xvfbwrapper==0.2.9",
]

[project.urls]
Documentation = "https://github.com/flaresolverr/flaresolverr#readme"
Issues = "https://github.com/flaresolverr/flaresolverr/issues"
Source = "https://github.com/flaresolverr/flaresolverr"

[project.scripts]
flaresolverr = "flaresolverr.flaresolverr:cli_run"

[tool.hatch.build]
include = [
  "flaresolverr",
]

[tool.hatch.envs.default]
dependencies = [
  "pytest",
  "pytest-cov",
]

[tool.hatch.envs.default.scripts]
cov = "pytest --cov-report=term-missing --cov-config=pyproject.toml --cov=flaresolverr --cov=tests {args}"
no-cov = "cov --no-cov {args}"

[[tool.hatch.envs.test.matrix]]
python = ["37", "38", "39", "310", "311"]

[tool.coverage.run]
branch = true
parallel = true

[tool.coverage.report]
exclude_lines = [
  "no cov",
  "if __name__ == .__main__.:",
  "if TYPE_CHECKING:",
]

[cwegener]

@HLFH I pushed the change from poetry to hatch.

Also, I just noticed that there is a Dockerfile in the project ... which would also need to be updated with the changes from the renaming of src to flaresolverr

[HLFH]

I pushed the change from poetry to hatch.

Thanks a lot! I have a few comments on your great pyproject.toml but I'll complete the review it later as it is 3:00am here.

Also, I just noticed that there is a Dockerfile in the project ... which would also need to be updated with the changes

Indeed, thanks!

[HLFH]

I updated the pyproject.toml after your PR here: https://github.com/HLFH/FlareSolverr/.
I have set Python >= 3.10 because of the updated TLS settings from the 3.10 version, and because the Dockerfile is using version 3.11.
The dynamic versioning needs to be adjusted to work.
The package.json file should be removed, and some code around versioning should be adjusted/removed as well.
I am looking into it.
I still need to update the *.txt files which I will do with pip-compile.

[ngosang]

@cwegener @HLFH @M4RC0Sx I don't know much about Python packaging so I have some questions.

• why hatch and no other package manager? there is a pr with pienv => V3beta - Python improvements #567
• what do you think about the custom logger? => V3beta - Python improvements #567
• what are the instructions to install the dependencies? we have to update the readme and the dockerfile

[HLFH]

https://github.com/HLFH/FlareSolverr/commits/master

I have turned off dynamic versioning until it is fixed.
requirements*.txt files are now autogenerated with pip-compile

@ngosang I'll reply to you ASAP.

[cwegener]

@cwegener @HLFH @M4RC0Sx I don't know much about Python packaging so I have some questions.

* why hatch and no other package manager? there is a pr with pienv => [V3beta - Python improvements #567](https://github.com/FlareSolverr/FlareSolverr/pull/567)

Here are my $.02

hatch is provided by the Python Packaging Authority (PyPA), which is a core working group that provides amongst other things:

• the pip software
• the pipx software
• the Python Packaging Guide

In terms of project management tools, there is no more authoritative and mature tool than hatch.
https://packaging.python.org/en/latest/key_projects/#pypa-projects

* what do you think about the custom logger? => [V3beta - Python improvements #567](https://github.com/FlareSolverr/FlareSolverr/pull/567)

Good idea. Not in scope for this change set though.

* what are the instructions to install the dependencies? we have to update the readme and the dockerfile

Dependencies are declared in the core metadata as specified in the PyPA Packaging Guide https://packaging.python.org/en/latest/specifications/core-metadata/#requires-dist-multiple-use

Any Python Source Distribution that follows the PyPA Packaging Guide should have these Requires-Dist fields declared in its metadata. Creating the correct metadata is handled by the project management tool, based on the configured input of installation time dependencies.

I have simply copied the contents of requirements.txt by hand into the respective pyproject.toml

As an enhancement to that, @HLFH has added the use of the pip-compile tool in his fork to enhance the management of those dependency declarations. pip-compile takes what is defined in pyproject.toml and builds an explicit tree of all dependencies including all transitive dependencies and stores them in one or more requirements.txt files.

The advantage of the requirements.txt file that is generated by pip-compile is that it has all transitive dependencies and their pinned versions, so therefore the requirements.txt files can then be used to perform reproducible builds, which may or may not be of importance depending on people's requirements.

The requirements.txt files are not used when performing a pip install of a source distribution though.

So, long story short:

For end-users, the FlareSolverr python releases should be installed from an index (PyPI is the authoritative index), using pip install flaresolverr

For Docker, I have no idea.

@ngosang Can you clarify what the original intention of the Dockerfile is? Is it so that potential contributors who want to use docker for local development can quickly launch a container? Or is the intention to use the Dockerfile as the basis for publishing a flaresolverr image to a Container registry like the Docker Hub or Quay?

[ngosang]

Thank you for the explanation. I will add all the changes from both PRs in the next release. Now I'm dealing with bug reports.

Can you clarify what the original intention of the Dockerfile is?

This porject is distributed as Docker image with Chrome and everithing configured inside. I have 12MM downloads as today => https://hub.docker.com/r/flaresolverr/flaresolverr/
There are few users installing from source code. Mostly Windows and MacOS users until I provide compilled binaries => #660