mirror of
https://github.com/pybind/pybind11.git
synced 2026-03-14 20:27:47 +00:00
0057e4945d3a77b7027c5970148e5addaf65ef1b
gil_safe_call_once_and_store (#5933)
* Add new argument to `gil_safe_call_once_and_store::call_once_and_store_result` * Add per-interpreter storage for `gil_safe_call_once_and_store` * Make `~gil_safe_call_once_and_store` a no-op * Fix C++11 compatibility * Improve thread-safety and add default finalizer * Try fix thread-safety * Try fix thread-safety * Add a warning comment * Simplify `PYBIND11_INTERNALS_VERSION >= 12` * Try fix thread-safety * Try fix thread-safety * Revert get_pp() * Update comments * Move call-once storage out of internals * Revert internal version bump * Cleanup outdated comments * Move atomic_bool alias into pybind11::detail namespace The `using atomic_bool = ...` declaration was at global scope, polluting the global namespace. Move it into pybind11::detail to avoid potential conflicts with user code. * Add explicit #include <unordered_map> for subinterpreter support The subinterpreter branch uses std::unordered_map but relied on transitive includes. Add an explicit include for robustness. * Remove extraneous semicolon after destructor definition Style fix: remove trailing semicolon after ~call_once_storage() destructor body. * Add comment explaining unused finalize parameter Clarify why the finalize callback parameter is intentionally ignored when subinterpreter support is disabled: the storage is process-global and leaked to avoid destructor calls after interpreter finalization. * Add comment explaining error_scope usage Clarify why error_scope is used: to preserve any existing Python error state that might be cleared or modified by dict_getitemstringref. * Improve exception safety in get_or_create_call_once_storage_map() Use std::unique_ptr to hold the newly allocated storage map until the capsule is successfully created. This prevents a memory leak if capsule creation throws an exception. * Add timeout-minutes: 3 to cpptest workflow steps Add a 3-minute timeout to all C++ test (cpptest) steps across all platforms to detect hangs early. This uses GitHub Actions' built-in timeout-minutes property which works on Linux, macOS, and Windows. * Add progress reporter for test_with_catch Catch2 runner Add a custom Catch2 streaming reporter that prints one line per test case as it starts and ends, with immediate flushing to keep CI logs current. This makes it easy to see where the embedded/interpreter tests are spending time and to pinpoint which test case is stuck when builds hang (e.g., free-threading issues). The reporter: - Prints "[ RUN ]" when each test starts - Prints "[ OK ]" or "[ FAILED ]" when each test ends - Prints the Python version once at the start via Py_GetVersion() - Uses StreamingReporterBase for immediate output (not buffered) - Is set as the default reporter via CATCH_CONFIG_DEFAULT_REPORTER This approach gives visibility into all tests without changing their behavior, turning otherwise opaque 90-minute CI timeouts into locatable issues in the Catch output. * clang-format auto-fix (overlooked before) * Disable "Move Subinterpreter" test on free-threaded Python 3.14+ This test hangs in Py_EndInterpreter() when the subinterpreter is destroyed from a different thread than it was created on. The hang was observed: - Intermittently on macOS with Python 3.14.0t - Predictably on macOS, Ubuntu, and Windows with Python 3.14.1t and 3.14.2t Root cause analysis points to an interaction between pybind11's subinterpreter creation code and CPython's free-threaded runtime, specifically around PyThreadState_Swap() after PyThreadState_DeleteCurrent(). See detailed analysis: https://github.com/pybind/pybind11/pull/5933 * style: pre-commit fixes * Add test for gil_safe_call_once_and_store per-interpreter isolation This test verifies that gil_safe_call_once_and_store provides separate storage for each interpreter when subinterpreter support is enabled. The test caches the interpreter ID in the main interpreter, then creates a subinterpreter and verifies it gets its own cached value (not the main interpreter's). Without per-interpreter storage, the subinterpreter would incorrectly see the main interpreter's cached object. * Add STARTING/DONE timestamps to test_with_catch output Print UTC timestamps at the beginning and end of the test run to make it immediately clear when tests started and whether they ran to completion. The DONE message includes the Catch session result value. Example output: [ STARTING ] 2025-12-21 03:23:20.497Z [ PYTHON ] 3.14.2 ... [ RUN ] Threads [ OK ] Threads [ DONE ] 2025-12-21 03:23:20.512Z (result 0) * Disable stdout buffering in test_with_catch Ensure test output appears immediately in CI logs by disabling stdout buffering. Without this, output may be lost if the process is killed by a timeout, making it difficult to diagnose which test was hanging. * EXPERIMENT: Re-enable hanging test to verify CI log buffering fix This is a temporary commit to verify that the unbuffered stdout fix makes the hanging test visible in CI logs. REVERT THIS COMMIT after confirming the output appears. * Revert "Disable stdout buffering in test_with_catch" This reverts commit0f8f32a92a. * Use USES_TERMINAL for cpptest to show output immediately Ninja buffers subprocess output until completion. When a test hangs, the output is never shown, making it impossible to diagnose which test is hanging. USES_TERMINAL gives the command direct terminal access, bypassing ninja's buffering. This explains why Windows CI showed test progress but Linux/macOS did not - Windows uses MSBuild which doesn't buffer the same way. * Fix clang-tidy performance-avoid-endl warning Use '\n' instead of std::endl since USES_TERMINAL now handles output buffering at the CMake level. * Add SIGTERM handler to show when test is killed by timeout When a test hangs and is killed by `timeout`, Catch2 marks it as failed but the process exits before printing [ DONE ]. This made it unclear whether the test failed normally or was terminated. The signal handler prints a clear message when SIGTERM is received, making timeout-related failures obvious in CI logs. * Fix typo: atleast -> at_least * Fix GCC warn_unused_result error for write() in signal handler Assign the return value to a variable to satisfy GCC's warn_unused_result attribute, then cast to void to suppress unused variable warning. * Add USES_TERMINAL to other C++ test targets Apply the same ninja output buffering fix to test_cross_module_rtti and test_pure_cpp targets. Also add explanatory comments to all USES_TERMINAL usages. * Revert "EXPERIMENT: Re-enable hanging test to verify CI log buffering fix" This reverts commita3abdeea89. * Update comment to reference PR #5940 for Move Subinterpreter fix * Add alias `interpid_t = std::int64_t` * Add isolation and gc test for `gil_safe_call_once_and_store` * Add thread local cache for gil_safe_call_once_and_store * Revert "Add thread local cache for gil_safe_call_once_and_store" This reverts commit 5d6681956d2d326fe74c7bf80e845c8e8ddb2a7c. * Revert changes according to code review * Relocate multiple-interpreters tests * Add more tests for multiple interpreters * Remove copy constructor * Apply suggestions from code review * Refactor to use per-storage capsule instead * Update comments * Update singleton tests * Use interpreter id type for `get_num_interpreters_seen()` * Suppress unused variable warning * HACKING * Revert "HACKING" This reverts commit534235ea55. * Try fix concurrency * Test even harder * Reorg code to avoid duplicates * Fix unique_ptr::reset -> unique_ptr::release * Extract reusable functions * Fix indentation * Appease warnings for MSVC * Appease warnings for MSVC * Appease warnings for MSVC * Try fix concurrency by not using `get_num_interpreters_seen() > 1` * Try fix tests * Make Python path handling more robust * Update comments and assertion messages * Revert changes according to code review * Disable flaky tests * Use `@pytest.mark.xfail` rather than `pytest.skip` * Retrigger CI * Retrigger CI * Revert file moves * Refactor atomic_get_or_create_in_state_dict: improve API and fix on_fetch_ bug Three improvements to atomic_get_or_create_in_state_dict: 1. Return std::pair<Payload*, bool> instead of just Payload* - The bool indicates whether storage was newly created (true) or already existed (false), following std::map::insert convention. - This fixes a bug where on_fetch_ was called even for newly created internals, when it should only run for fetched (existing) ones. (Identified by @b-pass in code review) 2. Change LeakOnInterpreterShutdown from template param to runtime arg - Renamed to `clear_destructor` to describe what it does locally, rather than embedding assumptions about why it's used. - Reduces template instantiations (header-only library benefits). - The check is in the slow path (create) anyway, so negligible cost. 3. Remove unnecessary braces around the fast-path lookup - The braces created a nested scope but declared no local variables that would benefit from scoping. * Remove unused PYBIND11_MULTIPLE_INTERPRETERS_TEST_FILES variable This variable was defined but never used. --------- Co-authored-by: Ralf W. Grosse-Kunstleve <rgrossekunst@nvidia.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Improve C++ test infrastructure: progress reporter, timeouts, and skip hanging Move Subinterpreter test (#5942)
This file contains invisible Unicode characters
This file contains invisible Unicode characters that are indistinguishable to humans but may be processed differently by a computer. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
.. figure:: https://github.com/pybind/pybind11/raw/master/docs/pybind11-logo.png
:alt: pybind11 logo
**pybind11 (v3) — Seamless interoperability between C++ and Python**
|Latest Documentation Status| |Stable Documentation Status| |Gitter chat| |GitHub Discussions|
|CI| |Build status| |SPEC 4 — Using and Creating Nightly Wheels|
|Repology| |PyPI package| |Conda-forge| |Python Versions|
`Setuptools example <https://github.com/pybind/python_example>`_
• `Scikit-build example <https://github.com/pybind/scikit_build_example>`_
• `CMake example <https://github.com/pybind/cmake_example>`_
.. start
**pybind11** is a lightweight header-only library that exposes C++ types
in Python and vice versa, mainly to create Python bindings of existing
C++ code. Its goals and syntax are similar to the excellent
`Boost.Python <http://www.boost.org/doc/libs/1_58_0/libs/python/doc/>`_
library by David Abrahams: to minimize boilerplate code in traditional
extension modules by inferring type information using compile-time
introspection.
The main issue with Boost.Python—and the reason for creating such a
similar project—is Boost. Boost is an enormously large and complex suite
of utility libraries that works with almost every C++ compiler in
existence. This compatibility has its cost: arcane template tricks and
workarounds are necessary to support the oldest and buggiest of compiler
specimens. Now that C++11-compatible compilers are widely available,
this heavy machinery has become an excessively large and unnecessary
dependency.
Think of this library as a tiny self-contained version of Boost.Python
with everything stripped away that isn't relevant for binding
generation. Without comments, the core header files only require ~4K
lines of code and depend on Python (CPython 3.8+, PyPy, or GraalPy) and the C++
standard library. This compact implementation was possible thanks to some C++11
language features (specifically: tuples, lambda functions and variadic
templates). Since its creation, this library has grown beyond Boost.Python in
many ways, leading to dramatically simpler binding code in many common
situations.
Tutorial and reference documentation is provided at
`pybind11.readthedocs.io <https://pybind11.readthedocs.io/en/latest>`_.
A PDF version of the manual is available
`here <https://pybind11.readthedocs.io/_/downloads/en/latest/pdf/>`_.
And the source code is always available at
`github.com/pybind/pybind11 <https://github.com/pybind/pybind11>`_.
Core features
-------------
pybind11 can map the following core C++ features to Python:
- Functions accepting and returning custom data structures per value,
reference, or pointer
- Instance methods and static methods
- Overloaded functions
- Instance attributes and static attributes
- Arbitrary exception types
- Enumerations
- Callbacks
- Iterators and ranges
- Custom operators
- Single and multiple inheritance
- STL data structures
- Smart pointers with reference counting like ``std::shared_ptr``
- Internal references with correct reference counting
- C++ classes with virtual (and pure virtual) methods can be extended
in Python
- Integrated NumPy support (NumPy 2 requires pybind11 2.12+)
Goodies
-------
In addition to the core functionality, pybind11 provides some extra
goodies:
- CPython 3.8+, PyPy3 7.3.17+, and GraalPy 24.1+ are supported with an
implementation-agnostic interface (see older versions for older CPython
and PyPy versions).
- It is possible to bind C++11 lambda functions with captured
variables. The lambda capture data is stored inside the resulting
Python function object.
- pybind11 uses C++11 move constructors and move assignment operators
whenever possible to efficiently transfer custom data types.
- It's easy to expose the internal storage of custom data types through
Pythons' buffer protocols. This is handy e.g. for fast conversion
between C++ matrix classes like Eigen and NumPy without expensive
copy operations.
- pybind11 can automatically vectorize functions so that they are
transparently applied to all entries of one or more NumPy array
arguments.
- Python's slice-based access and assignment operations can be
supported with just a few lines of code.
- Everything is contained in just a few header files; there is no need
to link against any additional libraries.
- Binaries are generally smaller by a factor of at least 2 compared to
equivalent bindings generated by Boost.Python. A recent pybind11
conversion of PyRosetta, an enormous Boost.Python binding project,
`reported <https://graylab.jhu.edu/Sergey/2016.RosettaCon/PyRosetta-4.pdf>`_
a binary size reduction of **5.4x** and compile time reduction by
**5.8x**.
- Function signatures are precomputed at compile time (using
``constexpr``), leading to smaller binaries.
- With little extra effort, C++ types can be pickled and unpickled
similar to regular Python objects.
Supported platforms & compilers
-------------------------------
pybind11 is exercised in continuous integration across a range of operating
systems, Python versions, C++ standards, and toolchains. For an up-to-date
view of the combinations we currently test, please see the
`pybind11 GitHub Actions <https://github.com/pybind/pybind11/actions?query=branch%3Amaster>`_
logs.
The test matrix naturally evolves over time as older platforms and compilers
fall out of use and new ones are added by the community. Closely related
versions of a tested compiler or platform will often work as well in practice,
but we cannot promise to validate every possible combination. If a
configuration you rely on is missing from the matrix or regresses, issues and
pull requests to extend coverage are very welcome. At the same time, we need
to balance the size of the test matrix with the available CI resources,
such as GitHub's limits on concurrent jobs under the free tier.
About
-----
This project was created by `Wenzel
Jakob <http://rgl.epfl.ch/people/wjakob>`_. Significant features and/or
improvements to the code were contributed by
Jonas Adler,
Lori A. Burns,
Sylvain Corlay,
Eric Cousineau,
Aaron Gokaslan,
Ralf Grosse-Kunstleve,
Trent Houliston,
Axel Huebl,
@hulucc,
Yannick Jadoul,
Sergey Lyskov,
Johan Mabille,
Tomasz Miąsko,
Dean Moldovan,
Ben Pritchard,
Jason Rhinelander,
Boris Schäling,
Pim Schellart,
Henry Schreiner,
Ivan Smirnov,
Dustin Spicuzza,
Boris Staletic,
Ethan Steinberg,
Patrick Stewart,
Ivor Wanders,
and
Xiaofei Wang.
We thank Google for a generous financial contribution to the continuous
integration infrastructure used by this project.
Contributing
~~~~~~~~~~~~
See the `contributing
guide <https://github.com/pybind/pybind11/blob/master/.github/CONTRIBUTING.md>`_
for information on building and contributing to pybind11.
License
~~~~~~~
pybind11 is provided under a BSD-style license that can be found in the
`LICENSE <https://github.com/pybind/pybind11/blob/master/LICENSE>`_
file. By using, distributing, or contributing to this project, you agree
to the terms and conditions of this license.
.. |Latest Documentation Status| image:: https://readthedocs.org/projects/pybind11/badge?version=latest
:target: http://pybind11.readthedocs.org/en/latest
.. |Stable Documentation Status| image:: https://img.shields.io/badge/docs-stable-blue.svg
:target: http://pybind11.readthedocs.org/en/stable
.. |Gitter chat| image:: https://img.shields.io/gitter/room/gitterHQ/gitter.svg
:target: https://gitter.im/pybind/Lobby
.. |CI| image:: https://github.com/pybind/pybind11/workflows/CI/badge.svg
:target: https://github.com/pybind/pybind11/actions
.. |Build status| image:: https://ci.appveyor.com/api/projects/status/riaj54pn4h08xy40?svg=true
:target: https://ci.appveyor.com/project/wjakob/pybind11
.. |PyPI package| image:: https://img.shields.io/pypi/v/pybind11.svg
:target: https://pypi.org/project/pybind11/
.. |Conda-forge| image:: https://img.shields.io/conda/vn/conda-forge/pybind11.svg
:target: https://github.com/conda-forge/pybind11-feedstock
.. |Repology| image:: https://repology.org/badge/latest-versions/python:pybind11.svg
:target: https://repology.org/project/python:pybind11/versions
.. |Python Versions| image:: https://img.shields.io/pypi/pyversions/pybind11.svg
:target: https://pypi.org/project/pybind11/
.. |GitHub Discussions| image:: https://img.shields.io/static/v1?label=Discussions&message=Ask&color=blue&logo=github
:target: https://github.com/pybind/pybind11/discussions
.. |SPEC 4 — Using and Creating Nightly Wheels| image:: https://img.shields.io/badge/SPEC-4-green?labelColor=%23004811&color=%235CA038
:target: https://scientific-python.org/specs/spec-0004/
Description
Languages
C++
70.2%
Python
24%
CMake
5.3%
C
0.4%