mirror of
https://github.com/pybind/pybind11.git
synced 2026-03-14 20:27:47 +00:00
4a77b97725e310903a45e98177ac660e1ad9b5de
7 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Sam Gross
|
4f81a12507 |
Fix deadlock in test with free threading (#5973)
Importing "widget_module" re-enables the GIL. In current versions of CPython, this requires pausing all threads attached to all interpreters. The spinning on sync/num without a py::gil_scoped_release causes occasional deadlocks. |
||
|
Xuehai Pan
|
fee2527dfa |
Fix concurrency consistency for internals_pp_manager under multiple-interpreters (#5947)
* Add per-interpreter storage for `gil_safe_call_once_and_store`
* Disable thread local cache for `internals_pp_manager`
* Disable thread local cache for `internals_pp_manager` for multi-interpreter only
* Use anonymous namespace to separate these type_ids from other tests with the same class names.
* style: pre-commit fixes
* Revert internals_pp_manager changes
* This is the crux of fix for the subinterpreter_before_main failure.
The pre_init needs to check if it is in a subinterpreter or not. But in 3.13+ this static initializer runs in the main interpreter. So we need to check this later, during the exec phase.
* Continue to do the ensure in both places, there might be a reason it was where it was...
Should not hurt anything to do it extra times here.
* Change get_num_interpreters_seen to a boolean flag instead.
The count was not used, it was just checked for > 1, we now accomplish this by setting the flag.
* Spelling typo
* Work around older python versions, only need this check for newish versions
* Add more comments for test case
* Add more comments for test case
* Stop traceback propagation
* Re-enable subinterpreter support on ubuntu 3.14 builds
Was disabled in
|
||
|
Xuehai Pan
|
0057e4945d |
Add per-interpreter storage for gil_safe_call_once_and_store (#5933)
* Add new argument to `gil_safe_call_once_and_store::call_once_and_store_result` * Add per-interpreter storage for `gil_safe_call_once_and_store` * Make `~gil_safe_call_once_and_store` a no-op * Fix C++11 compatibility * Improve thread-safety and add default finalizer * Try fix thread-safety * Try fix thread-safety * Add a warning comment * Simplify `PYBIND11_INTERNALS_VERSION >= 12` * Try fix thread-safety * Try fix thread-safety * Revert get_pp() * Update comments * Move call-once storage out of internals * Revert internal version bump * Cleanup outdated comments * Move atomic_bool alias into pybind11::detail namespace The `using atomic_bool = ...` declaration was at global scope, polluting the global namespace. Move it into pybind11::detail to avoid potential conflicts with user code. * Add explicit #include <unordered_map> for subinterpreter support The subinterpreter branch uses std::unordered_map but relied on transitive includes. Add an explicit include for robustness. * Remove extraneous semicolon after destructor definition Style fix: remove trailing semicolon after ~call_once_storage() destructor body. * Add comment explaining unused finalize parameter Clarify why the finalize callback parameter is intentionally ignored when subinterpreter support is disabled: the storage is process-global and leaked to avoid destructor calls after interpreter finalization. * Add comment explaining error_scope usage Clarify why error_scope is used: to preserve any existing Python error state that might be cleared or modified by dict_getitemstringref. * Improve exception safety in get_or_create_call_once_storage_map() Use std::unique_ptr to hold the newly allocated storage map until the capsule is successfully created. This prevents a memory leak if capsule creation throws an exception. * Add timeout-minutes: 3 to cpptest workflow steps Add a 3-minute timeout to all C++ test (cpptest) steps across all platforms to detect hangs early. This uses GitHub Actions' built-in timeout-minutes property which works on Linux, macOS, and Windows. * Add progress reporter for test_with_catch Catch2 runner Add a custom Catch2 streaming reporter that prints one line per test case as it starts and ends, with immediate flushing to keep CI logs current. This makes it easy to see where the embedded/interpreter tests are spending time and to pinpoint which test case is stuck when builds hang (e.g., free-threading issues). The reporter: - Prints "[ RUN ]" when each test starts - Prints "[ OK ]" or "[ FAILED ]" when each test ends - Prints the Python version once at the start via Py_GetVersion() - Uses StreamingReporterBase for immediate output (not buffered) - Is set as the default reporter via CATCH_CONFIG_DEFAULT_REPORTER This approach gives visibility into all tests without changing their behavior, turning otherwise opaque 90-minute CI timeouts into locatable issues in the Catch output. * clang-format auto-fix (overlooked before) * Disable "Move Subinterpreter" test on free-threaded Python 3.14+ This test hangs in Py_EndInterpreter() when the subinterpreter is destroyed from a different thread than it was created on. The hang was observed: - Intermittently on macOS with Python 3.14.0t - Predictably on macOS, Ubuntu, and Windows with Python 3.14.1t and 3.14.2t Root cause analysis points to an interaction between pybind11's subinterpreter creation code and CPython's free-threaded runtime, specifically around PyThreadState_Swap() after PyThreadState_DeleteCurrent(). See detailed analysis: https://github.com/pybind/pybind11/pull/5933 * style: pre-commit fixes * Add test for gil_safe_call_once_and_store per-interpreter isolation This test verifies that gil_safe_call_once_and_store provides separate storage for each interpreter when subinterpreter support is enabled. The test caches the interpreter ID in the main interpreter, then creates a subinterpreter and verifies it gets its own cached value (not the main interpreter's). Without per-interpreter storage, the subinterpreter would incorrectly see the main interpreter's cached object. * Add STARTING/DONE timestamps to test_with_catch output Print UTC timestamps at the beginning and end of the test run to make it immediately clear when tests started and whether they ran to completion. The DONE message includes the Catch session result value. Example output: [ STARTING ] 2025-12-21 03:23:20.497Z [ PYTHON ] 3.14.2 ... [ RUN ] Threads [ OK ] Threads [ DONE ] 2025-12-21 03:23:20.512Z (result 0) * Disable stdout buffering in test_with_catch Ensure test output appears immediately in CI logs by disabling stdout buffering. Without this, output may be lost if the process is killed by a timeout, making it difficult to diagnose which test was hanging. * EXPERIMENT: Re-enable hanging test to verify CI log buffering fix This is a temporary commit to verify that the unbuffered stdout fix makes the hanging test visible in CI logs. REVERT THIS COMMIT after confirming the output appears. * Revert "Disable stdout buffering in test_with_catch" This reverts commit |
||
|
Ralf W. Grosse-Kunstleve
|
799f591ec3 |
Re-enable Move Subinterpreter test for free-threaded Python 3.14 (#5940)
* Remove skip for Move Subinterpreter test on free-threaded Python 3.14+ * Fix deadlock by detaching from the main interpreter before joining the thread. * style: pre-commit fixes --------- Co-authored-by: b-pass <b-pass@users.noreply.github.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> |
||
|
Ralf W. Grosse-Kunstleve
|
78381e5e28 |
Improve C++ test infrastructure: progress reporter, timeouts, and skip hanging Move Subinterpreter test (#5942)
* Improve C++ test infrastructure and disable hanging test This commit improves the C++ test infrastructure to ensure test output is visible in CI logs, and disables a test that hangs on free-threaded Python 3.14+. Changes: ## CI/test infrastructure improvements - .github/workflows: Added `timeout-minutes: 3` to all C++ test steps to prevent indefinite hangs. - tests/**/CMakeLists.txt: Added `USES_TERMINAL` to C++ test targets (cpptest, test_cross_module_rtti, test_pure_cpp) to ensure output is shown immediately rather than buffered and possibly lost on crash/timeout. - tests/test_with_catch/catch.cpp: Added a custom Catch2 progress reporter with timestamps, Python version info, and a SIGTERM handler to make test execution and failures clearly visible in CI logs. ## Disabled hanging test - The "Move Subinterpreter" test is disabled on free-threaded Python 3.14+ due to a hang in Py_EndInterpreter() when the subinterpreter is destroyed from a different thread than it was created on. Work on fixing the underlying issue will continue under PR #5940. Context: We were in the dark for months (since we started testing with Python 3.14t) because CI logs gave no clue about the root cause of hangs. This led to ignoring intermittent hangs (mostly on macOS). Our hand was forced only with the Python 3.14.1 release, when hangs became predictable on all platforms. For the full development history of these changes, see PR #5933. * Add test summary to progress reporter Print the total number of test cases and assertions at the end of the test run, making it easy to spot if tests are disabled or added. Example output: [ PASSED ] 20 test cases, 1589 assertions. * Add PYBIND11_CATCH2_SKIP_IF macro to skip tests at runtime Catch2 v2 doesn't have native skip support (v3 does with SKIP()). This macro allows tests to be skipped with a visible message while still appearing in the test list. Use this for the Move Subinterpreter test on free-threaded Python 3.14+ so it shows as skipped rather than being conditionally compiled out. Example output: [ RUN ] Move Subinterpreter [ SKIPPED ] Skipped on free-threaded Python 3.14+ (see PR #5940) [ OK ] Move Subinterpreter * Fix clang-tidy bugprone-macro-parentheses warning in PYBIND11_CATCH2_SKIP_IF |
||
|
Scott Wolchok
|
3262000195 |
Add fast_type_map, use it authoritatively for local types and as a hint for global types (ABI breaking) (#5842)
* Add fast_type_map, use it authoritatively for local types and as a hint for global types
nanobind has a similar two-level lookup strategy, added and explained
by
|
||
|
Scott Wolchok
|
30748f863f |
Avoid heap allocation for function calls with a small number of args (#5824)
* Avoid heap allocation for function calls with a small number of arguments
We don't have access to llvm::SmallVector or similar, but given the
limited subset of the `std::vector` API that
`function_call::args{,_convert}` need and the "reserve-then-fill"
usage pattern, it is relatively straightforward to implement custom
containers that get the job done.
Seems to improves time to call the collatz function in
pybind/pybind11_benchmark significantly; numbers are a little noisy
but there's a clear improvement from "about 60 ns per call" to "about
45 ns per call" on my machine (M4 Max Mac), as measured with
`timeit.repeat('collatz(4)', 'from pybind11_benchmark import
collatz')`.
* clang-tidy
* more clang-tidy
* clang-tidy NOLINTBEGIN/END instead of NOLINTNEXTLINE
* forgot to increase inline size after removing std::variant
* constexpr arg_vector_small_size, use move instead of swap to hopefully clarify second_pass_convert
* rename test_embed to test_low_level
* rename test_low_level to test_with_catch
* Be careful to NOINLINE slow paths
* rename array/vector members to iarray/hvector. Move comment per request. Add static_asserts for our untagged union implementation per request.
* drop is_standard_layout assertions; see https://github.com/pybind/pybind11/pull/5824#issuecomment-3308616072
|