pybind/pybind11
1
0
Fork 0
mirror of https://github.com/pybind/pybind11.git synced 2026-05-24 06:45:03 +00:00
Code Issues Packages Projects Releases Wiki Activity

Add PYBIND11_SIMPLE_GIL_MANAGEMENT option (cmake, C++ define) (#4216)

Browse Source 
* Add option to force the use of the PYPY GIL scoped acquire/release logic to support nested gil access, see https://github.com/pybind/pybind11/issues/1276 and https://github.com/pytorch/pytorch/issues/83101

* Apply suggestions from code review

* Update CMakeLists.txt

* docs: update upgrade guide

* Update docs/upgrade.rst

* All bells & whistles.

* Add Reminder to common.h, so that we will not forget to purge `!WITH_THREAD` branches when dropping Python 3.6

* New sentence instead of semicolon.

* Temporarily pull in snapshot of PR #4246

* Add `test_release_acquire`

* Add more unit tests for nested gil locking

* Add test_report_builtins_internals_keys

* Very minor enhancement: sort list only after filtering.

* Revert change in docs/upgrade.rst

* Add test_multi_acquire_release_cross_module, while also forcing unique PYBIND11_INTERNALS_VERSION for cross_module_gil_utils.cpp

* Hopefully fix apparently new ICC error.

```
2022-10-28T07:57:54.5187728Z -- The CXX compiler identification is Intel 2021.7.0.20220726
...
2022-10-28T07:58:53.6758994Z icpc: remark #10441: The Intel(R) C++ Compiler Classic (ICC) is deprecated and will be removed from product release in the second half of 2023. The Intel(R) oneAPI DPC++/C++ Compiler (ICX) is the recommended compiler moving forward. Please transition to use this compiler. Use '-diag-disable=10441' to disable this message.
2022-10-28T07:58:54.5801597Z In file included from /home/runner/work/pybind11/pybind11/include/pybind11/detail/../detail/type_caster_base.h(15),
2022-10-28T07:58:54.5803794Z                  from /home/runner/work/pybind11/pybind11/include/pybind11/detail/../cast.h(15),
2022-10-28T07:58:54.5805740Z                  from /home/runner/work/pybind11/pybind11/include/pybind11/detail/../attr.h(14),
2022-10-28T07:58:54.5809556Z                  from /home/runner/work/pybind11/pybind11/include/pybind11/detail/class.h(12),
2022-10-28T07:58:54.5812154Z                  from /home/runner/work/pybind11/pybind11/include/pybind11/pybind11.h(13),
2022-10-28T07:58:54.5948523Z                  from /home/runner/work/pybind11/pybind11/tests/cross_module_gil_utils.cpp(13):
2022-10-28T07:58:54.5949009Z /home/runner/work/pybind11/pybind11/include/pybind11/detail/../detail/internals.h(177): error #2282: unrecognized GCC pragma
2022-10-28T07:58:54.5949374Z       PYBIND11_TLS_KEY_INIT(tstate)
2022-10-28T07:58:54.5949579Z       ^
2022-10-28T07:58:54.5949695Z
```

* clang-tidy fixes

* Workaround for PYPY WIN exitcode None

* Revert "Temporarily pull in snapshot of PR #4246"

This reverts commit 23ac16e859150f27fda25ca865cabcb4444e0770.

* Another workaround for PYPY WIN exitcode None

* Clean up how the tests are run "run in process" Part 1: uniformity

* Clean up how the tests are run "run in process" Part 2: use `@pytest.mark.parametrize` and clean up the naming.

* Skip some tests `#if defined(THREAD_SANITIZER)` (tested with TSAN using the Google-internal toolchain).

* Run all tests again but ignore ThreadSanitizer exitcode 66 (this is less likely to mask unrelated ThreadSanitizer issues in the future).

* bug fix: missing common.h include before using `PYBIND11_SIMPLE_GIL_MANAGEMENT`

For the tests in the github CI this does not matter, because
`PYBIND11_SIMPLE_GIL_MANAGEMENT` is always defined from the command line,
but when monkey-patching common.h locally, it matters.

* if process.exitcode is None: assert t_delta > 9.9

* More sophisiticated `_run_in_process()` implementation, clearly reporting `DEADLOCK`, additionally exercised via added `intentional_deadlock()`

* Wrap m.intentional_deadlock in a Python function, for `ForkingPickler` compatibility.

```
>       ForkingPickler(file, protocol).dump(obj)
E       TypeError: cannot pickle 'PyCapsule' object
```

Observed with all Windows builds including mingw but not PyPy, and macos-latest with Python 3.9, 3.10, 3.11 but not 3.6.

* Add link to potential solution for WOULD-BE-NICE-TO-HAVE feature.

* Add `SKIP_IF_DEADLOCK = True` option, to not pollute the CI results with expected `DEADLOCK` failures while we figure out what to do about them.

* Add COPY-PASTE-THIS: gdb ... command (to be used for debugging the detected deadlock)

* style: pre-commit fixes

* Do better than automatic pre-commit fixes.

* Add `PYBIND11_SIMPLE_GIL_MANAGEMENT` to `pytest_report_header()` (so that we can easily know when harvesting deadlock information from the CI logs).

Co-authored-by: Arnim Balzer <arnim@seechange.ai>
Co-authored-by: Henry Schreiner <HenrySchreinerIII@gmail.com>
Co-authored-by: Ralf W. Grosse-Kunstleve <rwgk@google.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
This commit is contained in:
Chekov2k
2022-10-30 15:57:23 +00:00
committed by Henry Schreiner
parent 5b395c9b44
commit 15fde1def2
11 changed files with 433 additions and 60 deletions
Download Patch File Download Diff File Expand all files Collapse all files

67
tests/cross_module_gil_utils.cpp
View File

@@ -6,9 +6,15 @@
All rights reserved. Use of this source code is governed by a
BSD-style license that can be found in the LICENSE file.
*/
#if defined(PYBIND11_INTERNALS_VERSION)
# undef PYBIND11_INTERNALS_VERSION
#endif
#define PYBIND11_INTERNALS_VERSION 21814642 // Ensure this module has its own `internals` instance.
#include <pybind11/pybind11.h>
#include <cstdint>
#include <string>
#include <thread>
// This file mimics a DSO that makes pybind11 calls but does not define a
// PYBIND11_MODULE. The purpose is to test that such a DSO can create a
@@ -21,8 +27,54 @@
namespace {
namespace py = pybind11;
void gil_acquire() { py::gil_scoped_acquire gil; }
std::string gil_multi_acquire_release(unsigned bits) {
if ((bits & 0x1u) != 0u) {
py::gil_scoped_acquire gil;
}
if ((bits & 0x2u) != 0u) {
py::gil_scoped_release gil;
}
if ((bits & 0x4u) != 0u) {
py::gil_scoped_acquire gil;
}
if ((bits & 0x8u) != 0u) {
py::gil_scoped_release gil;
}
return PYBIND11_INTERNALS_ID;
}
struct CustomAutoGIL {
CustomAutoGIL() : gstate(PyGILState_Ensure()) {}
~CustomAutoGIL() { PyGILState_Release(gstate); }
PyGILState_STATE gstate;
};
struct CustomAutoNoGIL {
CustomAutoNoGIL() : save(PyEval_SaveThread()) {}
~CustomAutoNoGIL() { PyEval_RestoreThread(save); }
PyThreadState *save;
};
template <typename Acquire, typename Release>
void gil_acquire_inner() {
Acquire acquire_outer;
Acquire acquire_inner;
Release release;
}
template <typename Acquire, typename Release>
void gil_acquire_nested() {
Acquire acquire_outer;
Acquire acquire_inner;
Release release;
auto thread = std::thread(&gil_acquire_inner<Acquire, Release>);
thread.join();
}
constexpr char kModuleName[] = "cross_module_gil_utils";
struct PyModuleDef moduledef = {
@@ -30,6 +82,9 @@ struct PyModuleDef moduledef = {
} // namespace
#define ADD_FUNCTION(Name, ...) \
PyModule_AddObject(m, Name, PyLong_FromVoidPtr(reinterpret_cast<void *>(&__VA_ARGS__)));
extern "C" PYBIND11_EXPORT PyObject *PyInit_cross_module_gil_utils() {
PyObject *m = PyModule_Create(&moduledef);
@@ -37,8 +92,16 @@ extern "C" PYBIND11_EXPORT PyObject *PyInit_cross_module_gil_utils() {
if (m != nullptr) {
static_assert(sizeof(&gil_acquire) == sizeof(void *),
"Function pointer must have the same size as void*");
PyModule_AddObject(
m, "gil_acquire_funcaddr", PyLong_FromVoidPtr(reinterpret_cast<void *>(&gil_acquire)));
ADD_FUNCTION("gil_acquire_funcaddr", gil_acquire)
ADD_FUNCTION("gil_multi_acquire_release_funcaddr", gil_multi_acquire_release)
ADD_FUNCTION("gil_acquire_inner_custom_funcaddr",
gil_acquire_inner<CustomAutoGIL, CustomAutoNoGIL>)
ADD_FUNCTION("gil_acquire_nested_custom_funcaddr",
gil_acquire_nested<CustomAutoGIL, CustomAutoNoGIL>)
ADD_FUNCTION("gil_acquire_inner_pybind11_funcaddr",
gil_acquire_inner<py::gil_scoped_acquire, py::gil_scoped_release>)
ADD_FUNCTION("gil_acquire_nested_pybind11_funcaddr",
gil_acquire_nested<py::gil_scoped_acquire, py::gil_scoped_release>)
}
return m;