THRIFT-5915: Fix Python 3.12 build issues in thrift Python #3276

greggdonovan · 2025-12-22T10:49:56Z

JIRA link for THRIFT-5915

Summary:

Switch Python setup scripts to use setuptools instead of stdlib distutils (Python 3.12 removed distutils).
Use setuptools._distutils errors for build_ext error handling when available.
Guard ntohll/htonll definitions in lib/py/src/ext/endian.h to avoid macOS macro collisions that break C++ builds.
Update tornado, twisted, and zope to the minimum versions that support Python 3.12.

Why:

Python 3.12 no longer ships distutils, causing thrift builds from sdist to fail during wheel build.
On macOS, system headers define ntohll/htonll macros; the local inline definitions collided, causing compilation failures.

Validation:

Built a wheel from lib/py with Python 3.12 successfully after the changes.

Related issues:

THRIFT-5899 - "Python tests fail for the Appveyor MSVC builds". My PR updates MSVC to 2022 and tests with Python 3.12. This is green/passing, so maybe we'll be able to close that JIRA too?

N.B.: there are multiple commits at the moment but I will squash them to a single commit before merge per guidelines.

fishy · 2025-12-22T17:22:27Z

contrib/fb303/py/setup.py

-try:
-    from setuptools import setup, Extension
-except:
-    from distutils.core import setup, Extension, Command


should we keep this (but remove Command, looks like that's unused) as fallback for earlier versions of python? (although I'm not familiar of what scenario will have no setuptools but distutils)

Thanks for taking a look, @fishy! Do you know what the Thrift support guidelines are for older python versions? e.g. We have a followup PR to add better type support, ty type checking, etc. but that's dependent on more modern python versions. The guidelines in LANGUAGES.md appear pretty outdated (i.e. python 3.6.8 is the latest supported).

Would we be open to specifying that EOL python versions are not supported (currently 3.9 and older are EOL, with 3.10 scheduled to go EOL in October 2026)? That would make it much easier to evolve type support and modernize the codebase.

In this case, I think relying on setuptools >= 61.0 and dropping distutils limits us to python 3.7+, which seems very reasonable.

there's no single support guideline for older language versions (each language has their own), but don't support EOL versions is a very reasonable and common approach.

fishy · 2025-12-22T17:22:46Z

lib/py/setup.py

-try:
-    from setuptools import setup, Extension
-except Exception:
-    from distutils.core import setup, Extension


same question here regarding fallback to earlier versions of python

lib/py/setup.py

switching to just building as a source allows us to patch in apache/thrift#3276 once that has landed/published we can switch back to pypi Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: phlax <454682+phlax@users.noreply.github.com>

fishy · 2026-01-14T15:57:25Z

.github/workflows/build.yml

        run: make -C test/py check

+  lib-python-macos:
+    needs: compiler-macos


can this also be a matrix instead? (I'm not familiar with github actions enough so it's totally fine if this is not supported)

@fishy Thanks - let me give it a try.

fishy · 2026-01-14T15:58:39Z

.github/workflows/build.yml

+    runs-on: macos-14
+    strategy:
+      matrix:
+        python-version: ["3.10", "3.11", "3.12", "3.13", "3.14"]


if needs cannot be a matrix, I wonder whether we can define this as a "variable" somewhere in this yaml, so when we change the supported python versions in the future we only need to change one place.

fishy · 2026-01-14T15:59:02Z

.github/workflows/build.yml

+
+      - name: Python setup
+        run: |
+          python -m pip install --upgrade pip setuptools wheel flake8 tornado twisted zope.interface


similar here on possibility to define the list as a variable

- Add pyproject.toml with setuptools build requirement for PEP 517 compliance - Replace distutils imports with setuptools equivalents - Use setuptools error names directly (CompileError, ExecError, PlatformError) - Fix macOS header collision with ntohll/htonll macros in endian.h This fixes the ModuleNotFoundError: No module named 'distutils' error when building thrift with Python 3.12+. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>

The system bison (2.3) doesn't support the %code directive. Add homebrew's bison to GITHUB_PATH so it persists across all steps. Co-Authored-By: Claude <noreply@anthropic.com>

…nds.

…ck a matrix of python versions.

… be committed).

make -C test/cpp TestServer src/TestServer.cpp:599:9: error: no viable conversion from '__bind<int &, sockaddr *, unsigned long>' to 'int' 599 | int rv = bind(socket_fd, (struct sockaddr*)&sa, sizeof(sa)); | ^ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

greggdonovan · 2026-01-15T17:20:17Z

@fishy - I think this is ready for another look. I have another PR I'm working on for THRIFT-5537 - "drop eol python versions" - but it probably makes sense to keep that separate. Thanks!

greggdonovan force-pushed the etsy-python312-fix branch from b70cdb9 to ea60e94 Compare December 22, 2025 10:59

fishy reviewed Dec 22, 2025

View reviewed changes

lib/py/setup.py Outdated Show resolved Hide resolved

greggdonovan force-pushed the etsy-python312-fix branch from ea60e94 to ed73c20 Compare January 4, 2026 19:33

Jens-G added the python label Jan 7, 2026

phlax mentioned this pull request Jan 8, 2026

thrift: Dont use the pypi package envoyproxy/envoy#42908

Merged

mergeable bot added the github_actions Pull requests that update GitHub Actions code label Jan 14, 2026

fishy reviewed Jan 14, 2026

View reviewed changes

greggdonovan force-pushed the etsy-python312-fix branch from c6f9e1f to fc04dad Compare January 14, 2026 17:42

mergeable bot added the c++ label Jan 14, 2026

greggdonovan changed the title ~~Fix Python 3.12 build issues in thrift Python~~ THRIFT-5915: Fix Python 3.12 build issues in thrift Python Jan 15, 2026

greggdonovan and others added 17 commits January 15, 2026 11:32

Fix macOS CI: add homebrew bison to GITHUB_PATH

d314acb

The system bison (2.3) doesn't support the %code directive. Add homebrew's bison to GITHUB_PATH so it persists across all steps. Co-Authored-By: Claude <noreply@anthropic.com>

Windows: remove support for EOL MSVC2015.

8b1c3b8

Windows: remove deprecated cup/cinst commands in favor of choco comma…

4dd0fa9

…nds.

AppVeyor: bump zlib to 1.3.1

d62af4d

Fix workflow anchors in build.yml

754f581

CMake: honor BOOST_ROOT with CMP0144

83438bc

python: ensure we specify that we are building a wheel.

ff8e8b2

windows: focus on one VS2022 build till it works, then maybe bring ba…

bdeb3e8

…ck a matrix of python versions.

Windows: disable MINGW (which works) temporarily while fixing MSVC2022.

8869c50

windows: libevent UINT32_MAX errors

caa7e5c

AppVeyor: add Python/choco diagnostics

c10ab90

CMake: link iphlpapi for libevent on Windows

d6bc235

windows: restore the full grid now that 3.12 is working with MSVC2022

87e3d6e

windows: remove debug logging

99b0832

node: dependency updates from local testing (not just if these should…

a554689

… be committed).

win: drop extra logging

9dc0b35

greggdonovan force-pushed the etsy-python312-fix branch from 4daefe5 to 9dc0b35 Compare January 15, 2026 16:47

mergeable bot added the nodejs label Jan 15, 2026

greggdonovan marked this pull request as ready for review January 15, 2026 17:18

greggdonovan requested a review from fishy January 15, 2026 17:18

greggdonovan added 7 commits January 15, 2026 15:09

macos: install py to python prefix in CI

ecdca82

macos: pass PY_PREFIX to make install

e51ab46

py: harden socket test and improve RunClientServer/TestServer

89b665e

py: require tornado>=6.3.0 in ci and tests

ff29fa0

py: require twisted>=24.3.0

9b1ec07

py: require zope.interface>=6.1

8cedc0e

py: isolate RunClientServer ports and cleanup

7ec3fd9

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

THRIFT-5915: Fix Python 3.12 build issues in thrift Python #3276

THRIFT-5915: Fix Python 3.12 build issues in thrift Python #3276

greggdonovan commented Dec 22, 2025 •

edited

Loading

Uh oh!

fishy Dec 22, 2025

Uh oh!

greggdonovan Jan 14, 2026

Uh oh!

fishy Jan 14, 2026

Uh oh!

fishy Dec 22, 2025

Uh oh!

Uh oh!

fishy Jan 14, 2026

Uh oh!

greggdonovan Jan 14, 2026

Uh oh!

fishy Jan 14, 2026

Uh oh!

fishy Jan 14, 2026

Uh oh!

greggdonovan commented Jan 15, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

THRIFT-5915: Fix Python 3.12 build issues in thrift Python #3276

Are you sure you want to change the base?

THRIFT-5915: Fix Python 3.12 build issues in thrift Python #3276

Conversation

greggdonovan commented Dec 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

greggdonovan commented Jan 15, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

greggdonovan commented Dec 22, 2025 •

edited

Loading