Drop Python3.7 support

[hramezani]

Closes #7187

[cloudflare-pages]

Deploying with    Cloudflare Pages

Latest commit:	7640c85
Status:	✅  Deploy successful!
Preview URL:	https://eafdae24.pydantic-docs2.pages.dev
Branch Preview URL:	https://drop-python37.pydantic-docs2.pages.dev

View logs

[hramezani]

Seem we need this check for other versions as well. by removing this check some tests fail. like

pydantic/tests/test_json_schema.py

Line 4891 in 8989f96

def test_json_schema_extras_on_ref() -> None:

[samuelcolvin]

please update the comment.

[samuelcolvin]

🐦 tweet here asking if anyone has any objections.

[samuelcolvin]

@davidhewitt suggested we breakdown data by whether the distro is "Amazon Linux".

Downloads by Python Version, major pydantic version and whether or not they're amazon linux.

In summary the percentage of downloads that are: Python 3.7, Pydantic V2, and not Amazon Linux is 2.19% - well bellow the 5% threshold that @graingert mentioned use as a threshold.

percentage	python_version	major	amazon
15.65	3.8	1	false
13.4	3.9	1	false
10.3	3.10	1	false
9.37	3.7	1	false
7.32	3.7	1	true
5.85		1	false
4.51	3.7	2	true
4.26	3.11	1	false
3.96	3.10	1	true
3.88	3.8	1	true
3.59	3.8	2	false
3.15	3.9	2	false
2.88	3.8	2	true
2.73	3.10	2	false
2.28	3.9	1	true
2.19	3.7	2	false
1.66	3.11	2	false
1.06	3.9	2	true

If you look at just V2 downloads, the picture is slightly different:

percentage	python_version	amazon
19.57	3.7	true
15.59	3.8	false
13.69	3.9	false
12.5	3.8	true
11.86	3.10	false
9.5	3.7	false
7.23	3.11	false
4.59	3.9	true
4.23		false
1.02	3.10	true
0.21	3.11	true
0.02	3.12	false

[lig]

FTR #6820 introduces sys.version_info >= (3, 8) check in pydantic/plugin/_loader.py

[samuelcolvin]

Our new rule for dropping a Python version are:

• Needs to be EOL
• <5% of downloads of the most recent minor release need to be using that version

These rules might be reviewed (e.g. we might increase it to 10%) if 3.7 drags on too long.

---

Still 21% of downloads of 2.3 are using python 3.7, so we can't drop it in 2.4.

num_downloads	percent_downloads	python_version
5481568	28.59	3.8
4069629	21.22	3.7
3566365	18.6	3.1
3193854	16.66	3.9
1967557	10.26	3.11
891239	4.65
5848	0.03	3.12
46	0.0	3.5
29	0.0	3.6
24	0.0	3.4
19	0.0	2.7
6	0.0	3.2
4	0.0	3.3
1	0.0	3.13

To get these numberse:

Run the following (altered) in BigQuery

SELECT
  num_downloads,
  ROUND(num_downloads/ total * 100, 2) as percent_downloads,
  python_version
from (
  SELECT
    COUNT(*) AS num_downloads,
    sum(count(*)) OVER() AS total,
    REGEXP_EXTRACT(details.python, r"[0-9]+\.[0-9]+") AS python_version
  FROM `bigquery-public-data.pypi.file_downloads`
  WHERE
    -- timestamp filter is just to avoid full table scan
    file.project = 'pydantic' and STARTS_WITH(file.version, '2.3') and DATE(timestamp) > DATE('2023-08-01')
  GROUP BY `python_version`
  ORDER BY `num_downloads` DESC
)

To print the Markdown table, download the csv and run

duckdb -markdown -c "select * from '<filename>.csv' order by num_downloads desc"

[jorenham]

@samuelcolvin I sure hope that the 3.1 is a typo 😅

[davidhewitt]

Pretty confident that's 3.10 with the 0 truncated by some part of the query pipeline treating it as a decimal number.

[samuelcolvin]

I checked again just now, and 3.7 has dropped to 7%, hopefully when we get to 2.5, the percentage will be below 5% 🤞

[codspeed-hq]

CodSpeed Performance Report

Merging #7188 will not alter performance

_{Comparing drop-python37 (7640c85) with main (7df25da)}

Summary

✅ 10 untouched benchmarks

[sydney-runkle]

@samuelcolvin, before we merge, could you post the updated download stats re Python versions?

[samuelcolvin]

2.4.* downloads in November

num_downloads	percent_downloads	python_version
5597586	36.28	3.9
3294617	21.35	3.10
3264764	21.16	3.8
1496884	9.7	3.11
786632	5.1	3.7
724466	4.7
263197	1.71	3.12
279	0.0	3.13
19	0.0	3.5
14	0.0	3.6
6	0.0	3.4
3	0.0	3.3
1	0.0	2.7
1	0.0	3.2

2.5.* downloads

Total downloads including beta since last week and 2.5.0 release yesterday: 881,249

num_downloads	percent_downloads	python_version
275327	31.24	3.9
220582	25.03	3.10
195593	22.19	3.8
95171	10.8	3.11
43150	4.9	3.7
31761	3.6
19618	2.23	3.12
35	0.0	3.13
4	0.0	3.5
4	0.0	3.3
2	0.0	2.7
2	0.0	3.6

---

Let's merge!