Python3.12

[potiuk]

---

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

[potiuk]

cc: @dirrao @Taragolis -> seems like apache-beam having numpy as dependency is the next problem to solve after pendulum is solved

 Downloading numpy-1.24.4.tar.gz (10.9 MB)
  #56 70.61            ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 10.9/10.9 MB 36.6 MB/s eta 0:00:00
  #56 70.61         Installing build dependencies: started
  #56 70.61         Installing build dependencies: finished with status 'done'
  #56 70.61         Getting requirements to build wheel: started
  #56 70.61         Getting requirements to build wheel: finished with status 'error'
  #56 70.61         error: subprocess-exited-with-error
  #56 70.61       
  #56 70.61         × Getting requirements to build wheel did not run successfully.
  #56 70.61         │ exit code: 1
  #56 70.61         ╰─> [33 lines of output]
  #56 70.61             Traceback (most recent call last):
  #56 70.61               File "/usr/local/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
  #56 70.61                 main()
  #56 70.61               File "/usr/local/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
  #56 70.61                 json_out['return_val'] = hook(**hook_input['kwargs'])
  #56 70.61                                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  #56 70.61               File "/usr/local/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 112, in get_requires_for_build_wheel
  #56 70.61                 backend = _build_backend()
  #56 70.61                           ^^^^^^^^^^^^^^^^
  #56 70.61               File "/usr/local/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 77, in _build_backend
  #56 70.61                 obj = import_module(mod_path)
  #56 70.61                       ^^^^^^^^^^^^^^^^^^^^^^^

[potiuk]

Looks like we need NumPy 1.26+ - from that long discussion here: numpy/numpy#23808 and
Apache Beam has <1.25 even in main: https://github.com/apache/beam/blob/master/sdks/python/setup.py#L304

So likely the next best thing to do is to exclude apache-beam provider for python 3.12

This is kinda expected, Beam is always dragging us behind

[potiuk]

Pushed a fixup marking it for exclusion - let's see.

[Taragolis]

Seems like 1.26.0 is the first release for numpy which officially support 3.12:

• https://github.com/numpy/numpy/releases/tag/v1.26.0rc1

[Taragolis]

Oh... I've post my comment without refresh page, and you've already found the same things

[Taragolis]

Seems like Google provider also not compatible with 3.12 yet

46.9 ERROR: Could not find a version that satisfies the requirement google-ads>=22.1.0; extra == "google" (from apache-airflow[google]) (from versions: 0.1.0, 0.2.0, 0.3.0, 0.4.0, 0.5.0, 0.5.1, 0.5.2, 0.6.0, 0.7.0, 1.0.0, 1.0.1, 1.1.0, 1.1.1, 1.2.0, 1.3.0, 1.3.1, 2.0.0, 2.1.0, 2.2.0, 2.3.0, 2.4.0, 2.4.1, 3.0.0, 3.1.0, 3.1.1, 3.1.2, 3.2.0, 3.3.0, 4.0.0, 4.1.0, 4.1.1, 5.0.0, 5.0.1, 5.0.2, 5.0.3, 5.0.4, 5.1.0, 6.0.0, 7.0.0, 8.0.0, 8.1.0, 8.2.0, 9.0.0, 10.0.0, 11.0.0, 11.0.1, 11.0.2, 12.0.0, 13.0.0, 14.0.0, 14.0.1, 14.1.0, 15.0.0, 15.1.0, 15.1.1, 16.0.0, 17.0.0, 18.0.0, 18.1.0, 18.2.0, 19.0.0, 20.0.0, 21.0.0, 21.1.0, 21.2.0, 21.3.0, 22.0.0)
  246.9 ERROR: No matching distribution found for google-ads>=22.1.0; extra == "google"

Latest google-ads package pinned to >=3.7, <3.12

Issue for add support of Python 3.12 already exists googleads/google-ads-python#813

[potiuk]

All right ... let me exclude google provider too then. At this stage I have a feeling that excluding few - even huge and important - providers and having an open -issue to bring the 3.12 support in would be a good thing.

And I know for a fact that google team wants to split the google provider and splitting of ads was the first thing to try anyway, so that might accelerate things a bit.

[potiuk]

Pushed.

[potiuk]

BTW. I really like how nicely and transpartently the new pyproject.toml exclusion works now simply pre-commit updating pyproject.toml and it's immediately visible what is excluded :D

[Taragolis]

I think one day we need to finally resolve Consider splitting Google Provider because google provider is really huge providers (28k+ lines which tracked by our test) and contains quite a few different components:

• Cloud (GCP)
• Google ADS
• Google Suite
• LevelDB
• Google Firebase (is it part of GCP?)

So if we found the way how it could be done it might prevent the situation that one of this component become a showstopper for others

[potiuk]

I think one day we need to finally resolve Consider splitting Google Provider because google provider is really huge providers (28k+ lines which tracked by our test) and contains quite a few different components:

This is precisely the plan I am discussing with Google team :)

[Taragolis]

So we have duckdb dependency issue now, it already has python 3.12 support which added by duckdb/duckdb#10144 (Linux and MacOS) but not released yet

[Taragolis]

Seems 0.9.3.dev2258 pre-release is a first which supports 3.12

❯ pip install duckdb==0.9.3.dev2258
Collecting duckdb==0.9.3.dev2258
  Obtaining dependency information for duckdb==0.9.3.dev2258 from https://files.pythonhosted.org/packages/3c/43/094637a1939e8ba6ae53a788bd46adfd0b71fe0a6e182c8e6179b2966e09/duckdb-0.9.3.dev2258-cp312-cp312-macosx_11_0_arm64.whl.metadata
  Downloading duckdb-0.9.3.dev2258-cp312-cp312-macosx_11_0_arm64.whl.metadata (768 bytes)
Downloading duckdb-0.9.3.dev2258-cp312-cp312-macosx_11_0_arm64.whl (13.7 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 13.7/13.7 MB 3.6 MB/s eta 0:00:00
Installing collected packages: duckdb
Successfully installed duckdb-0.9.3.dev2258

[potiuk]

Pushed a change for it :)

[Taragolis]

One more step and a new error, this time it is related to the LevelDB, which is also part of google provider, but I guess it has separate extra

I think this is the same issue: wbolster/plyvel#158

[potiuk]

Running

[dirrao]

Build Prod Images still picking the google ads provider.

#56 25.60 ERROR: Ignored the following versions that require a different python version: 2.7.3 Requires-Python <3.12,~=3.8; 2.7.3rc1 Requires-Python <3.12,~=3.8; 2.8.0 Requires-Python <3.12,~=3.8; 2.8.0b1 Requires-Python <3.12,~=3.8; 2.8.0rc1 Requires-Python <3.12,~=3.8; 2.8.0rc2 Requires-Python <3.12,~=3.8; 2.8.0rc3 Requires-Python <3.12,~=3.8; 2.8.0rc4 Requires-Python <3.12,~=3.8; 22.1.0 Requires-Python >=3.7, <3.12
[17469](https://github.com/apache/airflow/actions/runs/7509965786/job/20447997500?pr=36755#step:5:17544)
  #56 25.60 ERROR: Could not find a version that satisfies the requirement google-ads>=22.1.0 (from apache-airflow-providers-google) (from versions: 0.1.0, 0.2.0, 0.3.0, 0.4.0, 0.5.0, 0.5.1, 0.5.2, 0.6.0, 0.7.0, 1.0.0, 1.0.1, 1.1.0, 1.1.1, 1.2.0, 1.3.0, 1.3.1, 2.0.0, 2.1.0, 2.2.0, 2.3.0, 2.4.0, 2.4.1, 3.0.0, 3.1.0, 3.1.1, 3.1.2, 3.2.0, 3.3.0, 4.0.0, 4.1.0, 4.1.1, 5.0.0, 5.0.1, 5.0.2, 5.0.3, 5.0.4, 5.1.0, 6.0.0, 7.0.0, 8.0.0, 8.1.0, 8.2.0, 9.0.0, 10.0.0, 11.0.0, 11.0.1, 11.0.2, 12.0.0, 13.0.0, 14.0.0, 14.0.1, 14.1.0, 15.0.0, 15.1.0, 15.1.1, 16.0.0, 17.0.0, 18.0.0, 18.1.0, 18.2.0, 19.0.0, 20.0.0, 21.0.0, 21.1.0, 21.2.0, 21.3.0, 22.0.0)
[17470](https://github.com/apache/airflow/actions/runs/7509965786/job/20447997500?pr=36755#step:5:17545)
  #56 25.60 ERROR: No matching distribution found for google-ads>=22.1.0
[17471](https://github.com/apache/airflow/actions/runs/7509965786/job/20447997500?pr=36755#step:5:17546)

[potiuk]

Yes. Because google provider is installed and epxected to be installed when PROD image is built. So if we do not build it locally for Pythin 3.12, it will install the one from PyPI. Luckily it looks like google ads maintainers are going to release 3.12-compatible version this week googleads/google-ads-python#813 (comment) so we should - I think - wait for it. Releasing 3.12 version without google provider, when we know we will likely be able to install it in two days, is likely just not worth the effort (we would have to add a code to exclude certain providers from PROD image installation.

In the meatime - we could take a close look at the failing tests for Python 3.13 https://github.com/apache/airflow/actions/runs/7509965786/job/20448000561?pr=36755)

I think they mostly fail because google and beam providers are missing - but if there are any other tests we should look at them. I have not looked in detail yet but there are at least a few with "real" 3.12 incompatiblities (in test code at least) that could be fixed in the meantime:

           AttributeError: 'called_once' is not a valid assertion. Use a spec for the mock if 'called_once' is meant to be an attribute.

[potiuk]

cc: @dirrao @Taragolis -> Ads released with 3.12 support https://pypi.org/project/google-ads/ - removed the limit from Google provider, let's see.

[potiuk]

Duckdb removed devel version we had pinned for 3.12 -> replaced it with >= for the new devel