squash-merge smart_holder branch into master (#5542)

* Pure `git merge --squash smart_holder` (no manual interventions).

* Remove ubench/ directory.

* Remove include/pybind11/smart_holder.h

* [ci skip] smart_ptrs.rst updates [WIP/unfinished]

* [ci skip] smart_ptrs.rst updates continued; also updating classes.rst, advanced/classes.rst

* Remove README_smart_holder.rst

* Restore original README.rst from master

* [ci skip] Minimal change to README.rst, to leave a hint that this is pybind11v3

* [ci skip] Work in ChatGPT suggestions.

* Change macro name to PYBIND11_RUN_TESTING_WITH_SMART_HOLDER_AS_DEFAULT_BUT_NEVER_USE_IN_PRODUCTION_PLEASE

* Add a note pointing to the holder reinterpret_cast.

* Incorporate suggestion by @virtuald: https://github.com/pybind/pybind11/pull/5542#discussion_r1967000989

* Systematically change most py::class_ to py::classh under docs/

* Remove references to README_smart_holder.rst

This should have been part of commit eb550d03d3.

* [ci skip] Fix minor oversight (``class_`` -> ``py::class_``) noticed by chance.

* [ci skip] Resolve suggestion by @virtuald

https://github.com/pybind/pybind11/pull/5542#discussion_r1969940605

* [ci skip] Apply suggestions by @timohl (thanks!)

* https://github.com/pybind/pybind11/pull/5542#discussion_r1970714551
* https://github.com/pybind/pybind11/pull/5542#discussion_r1971315329
* https://github.com/pybind/pybind11/pull/5542#discussion_r1971322821

* Replace `classh : class_` inhertance with `using`, as suggested by @henryiii

https://github.com/pybind/pybind11/pull/5542#issuecomment-2689034104

* Revert "Systematically change most py::class_ to py::classh under docs/"

This reverts commit ac9d31e13f.

* docs: focus on py::smart_holder instead of py::classh

Signed-off-by: Henry Schreiner <henryschreineriii@gmail.com>

* Restore minor general fixes that got lost when ac9d31e13f was reverted.

* Remove `- smart_holder` from list of branches in all .github/workflows

* Extend classh note to explain whitespace noise motivation.

* Suggest `py::smart_holder` for "most situations for safety"

* Add back PYBIND11_HAS_INTERNALS_WITH_SMART_HOLDER_SUPPORT

This define was
* introduced with https://github.com/pybind/pybind11/pull/5286
* removed with https://github.com/pybind/pybind11/pull/5531

It is has been in use here:
* f02a2b7653/pybind11_protobuf/native_proto_caster.h (L89-L101)

Currently pybind11 unit tests for the two holder caster backwards compatibility traits

* `copyable_holder_caster_shared_ptr_with_smart_holder_support_enabled`
* `move_only_holder_caster_unique_ptr_with_smart_holder_support_enabled`

are missing.

* Add py::trampoline_self_life_support to all trampoline examples under docs/.

Address suggestion by @timohl:

* https://github.com/pybind/pybind11/pull/5542#issuecomment-2686452062

Add to the "please think twice" note: the overhead for safety is likely in the noise.

Also fix a two-fold inconsistency introduced by revert-commit 1e646c91b4:

1.

py::trampoline_self_life_support is mentioned in a note, but is missing in the example right before.

2.

The section starting with

    To enable safely passing a ``std::unique_ptr`` to a trampoline object between

is obsolete.

* Fix whitespace accident (indentation) introduced with 1e646c91b4

Apparently the mis-indentation was introduced when resolving merge conflicts for what became 1e646c91b4

* WHITESPACE CHANGES ONLY in README.rst (list of people that made significant contributions)

* Add Ethan Steinberg to list of people that made significant contributions (for completeness, unrelated to smart_holder work).

* [ci skip] Add to list of people that made significant contributions: major and/or influential contributors to smart_holder branch

* #2904 by @rhaschke was merged on Mar 16, 2021
* #3012 by @rhaschke was merged on May 28, 2021
* #3039 by @jakobandersen was merged on Jun 29, 2021
* #3048 by @Skylion007 was merged on Jun 18, 2021
* #3588 by @virtuald was merged on Jan 3, 2022
* #3633 by @wangxf123456 was merged on Jan 25, 2022
* #3635 by @virtuald was merged on Jan 26, 2022
* #3645 by @wangxf123456 was merged on Jan 25, 2022
* #3796 by @wangxf123456 was merged on Mar 10, 2022
* #3807 by @wangxf123456 was merged on Mar 18, 2022
* #3838 by @wangxf123456 was merged on Apr 15, 2022
* #3929 by @tomba was merged on May 7, 2022
* #4031 by @wangxf123456 was merged on Jun 27, 2022
* #4343 by @wangxf123456 was merged on Nov 18, 2022
* #4381 by @wangxf123456 was merged on Dec 5, 2022
* #4539 by @wangxf123456 was merged on Feb 28, 2023
* #4609 by @wangxf123456 was merged on Apr 6, 2023
* #4775 by @wangxf123456 was merged on Aug 3, 2023
* #4921 by @iwanders was merged on Nov 7, 2023
* #4924 by @iwanders was merged on Nov 6, 2023
* #5401 by @msimacek was merged on Oct 8, 2024

Co-authored-by: Aaron Gokaslan <aaronGokaslan@gmail.com>
Co-authored-by: Dustin Spicuzza <dustin@virtualroadside.com>
Co-authored-by: Ivor Wanders <iwanders@users.noreply.github.com>
Co-authored-by: Jakob Lykke Andersen <Jakob@caput.dk>
Co-authored-by: Michael Šimáček <michael.simacek@oracle.com>
Co-authored-by: Robert Haschke <rhaschke@users.noreply.github.com>
Co-authored-by: Tomi Valkeinen <tomi.valkeinen@iki.fi>
Co-authored-by: Xiaofei Wang <6218006+wangxf123456@users.noreply.github.com>

---------

Signed-off-by: Henry Schreiner <henryschreineriii@gmail.com>
Co-authored-by: Henry Schreiner <henryschreineriii@gmail.com>
Co-authored-by: Aaron Gokaslan <aaronGokaslan@gmail.com>
Co-authored-by: Dustin Spicuzza <dustin@virtualroadside.com>
Co-authored-by: Ivor Wanders <iwanders@users.noreply.github.com>
Co-authored-by: Jakob Lykke Andersen <Jakob@caput.dk>
Co-authored-by: Michael Šimáček <michael.simacek@oracle.com>
Co-authored-by: Robert Haschke <rhaschke@users.noreply.github.com>
Co-authored-by: Tomi Valkeinen <tomi.valkeinen@iki.fi>
Co-authored-by: Xiaofei Wang <6218006+wangxf123456@users.noreply.github.com>
This commit is contained in:
Ralf W. Grosse-Kunstleve
2025-03-05 12:40:53 -08:00
committed by GitHub
parent d8565ac731
commit 2943a27a14
66 changed files with 5533 additions and 169 deletions

View File

@@ -10,8 +10,10 @@
#pragma once
#include "detail/class.h"
#include "detail/dynamic_raw_ptr_cast_if_possible.h"
#include "detail/exception_translation.h"
#include "detail/init.h"
#include "detail/using_smart_holder.h"
#include "attr.h"
#include "gil.h"
#include "gil_safe_call_once.h"
@@ -1444,8 +1446,8 @@ protected:
tinfo->dealloc = rec.dealloc;
tinfo->simple_type = true;
tinfo->simple_ancestors = true;
tinfo->default_holder = rec.default_holder;
tinfo->module_local = rec.module_local;
tinfo->holder_enum_v = rec.holder_enum_v;
with_internals([&](internals &internals) {
auto tindex = std::type_index(*rec.type);
@@ -1618,6 +1620,239 @@ auto method_adaptor(Return (Class::*pmf)(Args...) const) -> Return (Derived::*)(
return pmf;
}
PYBIND11_NAMESPACE_BEGIN(detail)
// Helper for the property_cpp_function static member functions below.
// The only purpose of these functions is to support .def_readonly & .def_readwrite.
// In this context, the PM template parameter is certain to be a Pointer to a Member.
// The main purpose of must_be_member_function_pointer is to make this obvious, and to guard
// against accidents. As a side-effect, it also explains why the syntactical overhead for
// perfect forwarding is not needed.
template <typename PM>
using must_be_member_function_pointer = enable_if_t<std::is_member_pointer<PM>::value, int>;
// Note that property_cpp_function is intentionally in the main pybind11 namespace,
// because user-defined specializations could be useful.
// Classic (non-smart_holder) implementations for .def_readonly and .def_readwrite
// getter and setter functions.
// WARNING: This classic implementation can lead to dangling pointers for raw pointer members.
// See test_ptr() in tests/test_class_sh_property.py
// However, this implementation works as-is (and safely) for smart_holder std::shared_ptr members.
template <typename T, typename D>
struct property_cpp_function_classic {
template <typename PM, must_be_member_function_pointer<PM> = 0>
static cpp_function readonly(PM pm, const handle &hdl) {
return cpp_function([pm](const T &c) -> const D & { return c.*pm; }, is_method(hdl));
}
template <typename PM, must_be_member_function_pointer<PM> = 0>
static cpp_function read(PM pm, const handle &hdl) {
return readonly(pm, hdl);
}
template <typename PM, must_be_member_function_pointer<PM> = 0>
static cpp_function write(PM pm, const handle &hdl) {
return cpp_function([pm](T &c, const D &value) { c.*pm = value; }, is_method(hdl));
}
};
PYBIND11_NAMESPACE_END(detail)
template <typename T, typename D, typename SFINAE = void>
struct property_cpp_function : detail::property_cpp_function_classic<T, D> {};
PYBIND11_NAMESPACE_BEGIN(detail)
template <typename T, typename D, typename SFINAE = void>
struct both_t_and_d_use_type_caster_base : std::false_type {};
// `T` is assumed to be equivalent to `intrinsic_t<T>`.
// `D` is may or may not be equivalent to `intrinsic_t<D>`.
template <typename T, typename D>
struct both_t_and_d_use_type_caster_base<
T,
D,
enable_if_t<all_of<std::is_base_of<type_caster_base<T>, type_caster<T>>,
std::is_base_of<type_caster_base<intrinsic_t<D>>, make_caster<D>>>::value>>
: std::true_type {};
// Specialization for raw pointer members, using smart_holder if that is the class_ holder,
// or falling back to the classic implementation if not.
// WARNING: Like the classic implementation, this implementation can lead to dangling pointers.
// See test_ptr() in tests/test_class_sh_property.py
// However, the read functions return a shared_ptr to the member, emulating the PyCLIF approach:
// https://github.com/google/clif/blob/c371a6d4b28d25d53a16e6d2a6d97305fb1be25a/clif/python/instance.h#L233
// This prevents disowning of the Python object owning the raw pointer member.
template <typename T, typename D>
struct property_cpp_function_sh_raw_ptr_member {
using drp = typename std::remove_pointer<D>::type;
template <typename PM, must_be_member_function_pointer<PM> = 0>
static cpp_function readonly(PM pm, const handle &hdl) {
type_info *tinfo = get_type_info(typeid(T), /*throw_if_missing=*/true);
if (tinfo->holder_enum_v == holder_enum_t::smart_holder) {
return cpp_function(
[pm](handle c_hdl) -> std::shared_ptr<drp> {
std::shared_ptr<T> c_sp
= type_caster<std::shared_ptr<T>>::shared_ptr_with_responsible_parent(
c_hdl);
D ptr = (*c_sp).*pm;
return std::shared_ptr<drp>(c_sp, ptr);
},
is_method(hdl));
}
return property_cpp_function_classic<T, D>::readonly(pm, hdl);
}
template <typename PM, must_be_member_function_pointer<PM> = 0>
static cpp_function read(PM pm, const handle &hdl) {
return readonly(pm, hdl);
}
template <typename PM, must_be_member_function_pointer<PM> = 0>
static cpp_function write(PM pm, const handle &hdl) {
type_info *tinfo = get_type_info(typeid(T), /*throw_if_missing=*/true);
if (tinfo->holder_enum_v == holder_enum_t::smart_holder) {
return cpp_function([pm](T &c, D value) { c.*pm = std::forward<D>(std::move(value)); },
is_method(hdl));
}
return property_cpp_function_classic<T, D>::write(pm, hdl);
}
};
// Specialization for members held by-value, using smart_holder if that is the class_ holder,
// or falling back to the classic implementation if not.
// The read functions return a shared_ptr to the member, emulating the PyCLIF approach:
// https://github.com/google/clif/blob/c371a6d4b28d25d53a16e6d2a6d97305fb1be25a/clif/python/instance.h#L233
// This prevents disowning of the Python object owning the member.
template <typename T, typename D>
struct property_cpp_function_sh_member_held_by_value {
template <typename PM, must_be_member_function_pointer<PM> = 0>
static cpp_function readonly(PM pm, const handle &hdl) {
type_info *tinfo = get_type_info(typeid(T), /*throw_if_missing=*/true);
if (tinfo->holder_enum_v == holder_enum_t::smart_holder) {
return cpp_function(
[pm](handle c_hdl) -> std::shared_ptr<typename std::add_const<D>::type> {
std::shared_ptr<T> c_sp
= type_caster<std::shared_ptr<T>>::shared_ptr_with_responsible_parent(
c_hdl);
return std::shared_ptr<typename std::add_const<D>::type>(c_sp,
&(c_sp.get()->*pm));
},
is_method(hdl));
}
return property_cpp_function_classic<T, D>::readonly(pm, hdl);
}
template <typename PM, must_be_member_function_pointer<PM> = 0>
static cpp_function read(PM pm, const handle &hdl) {
type_info *tinfo = get_type_info(typeid(T), /*throw_if_missing=*/true);
if (tinfo->holder_enum_v == holder_enum_t::smart_holder) {
return cpp_function(
[pm](handle c_hdl) -> std::shared_ptr<D> {
std::shared_ptr<T> c_sp
= type_caster<std::shared_ptr<T>>::shared_ptr_with_responsible_parent(
c_hdl);
return std::shared_ptr<D>(c_sp, &(c_sp.get()->*pm));
},
is_method(hdl));
}
return property_cpp_function_classic<T, D>::read(pm, hdl);
}
template <typename PM, must_be_member_function_pointer<PM> = 0>
static cpp_function write(PM pm, const handle &hdl) {
type_info *tinfo = get_type_info(typeid(T), /*throw_if_missing=*/true);
if (tinfo->holder_enum_v == holder_enum_t::smart_holder) {
return cpp_function([pm](T &c, const D &value) { c.*pm = value; }, is_method(hdl));
}
return property_cpp_function_classic<T, D>::write(pm, hdl);
}
};
// Specialization for std::unique_ptr members, using smart_holder if that is the class_ holder,
// or falling back to the classic implementation if not.
// read disowns the member unique_ptr.
// write disowns the passed Python object.
// readonly is disabled (static_assert) because there is no safe & intuitive way to make the member
// accessible as a Python object without disowning the member unique_ptr. A .def_readonly disowning
// the unique_ptr member is deemed highly prone to misunderstandings.
template <typename T, typename D>
struct property_cpp_function_sh_unique_ptr_member {
template <typename PM, must_be_member_function_pointer<PM> = 0>
static cpp_function readonly(PM, const handle &) {
static_assert(!is_instantiation<std::unique_ptr, D>::value,
"def_readonly cannot be used for std::unique_ptr members.");
return cpp_function{}; // Unreachable.
}
template <typename PM, must_be_member_function_pointer<PM> = 0>
static cpp_function read(PM pm, const handle &hdl) {
type_info *tinfo = get_type_info(typeid(T), /*throw_if_missing=*/true);
if (tinfo->holder_enum_v == holder_enum_t::smart_holder) {
return cpp_function(
[pm](handle c_hdl) -> D {
std::shared_ptr<T> c_sp
= type_caster<std::shared_ptr<T>>::shared_ptr_with_responsible_parent(
c_hdl);
return D{std::move(c_sp.get()->*pm)};
},
is_method(hdl));
}
return property_cpp_function_classic<T, D>::read(pm, hdl);
}
template <typename PM, must_be_member_function_pointer<PM> = 0>
static cpp_function write(PM pm, const handle &hdl) {
return cpp_function([pm](T &c, D &&value) { c.*pm = std::move(value); }, is_method(hdl));
}
};
PYBIND11_NAMESPACE_END(detail)
template <typename T, typename D>
struct property_cpp_function<
T,
D,
detail::enable_if_t<detail::all_of<std::is_pointer<D>,
detail::both_t_and_d_use_type_caster_base<T, D>>::value>>
: detail::property_cpp_function_sh_raw_ptr_member<T, D> {};
template <typename T, typename D>
struct property_cpp_function<T,
D,
detail::enable_if_t<detail::all_of<
detail::none_of<std::is_pointer<D>,
std::is_array<D>,
detail::is_instantiation<std::unique_ptr, D>,
detail::is_instantiation<std::shared_ptr, D>>,
detail::both_t_and_d_use_type_caster_base<T, D>>::value>>
: detail::property_cpp_function_sh_member_held_by_value<T, D> {};
template <typename T, typename D>
struct property_cpp_function<
T,
D,
detail::enable_if_t<detail::all_of<
detail::is_instantiation<std::unique_ptr, D>,
detail::both_t_and_d_use_type_caster_base<T, typename D::element_type>>::value>>
: detail::property_cpp_function_sh_unique_ptr_member<T, D> {};
#ifdef PYBIND11_RUN_TESTING_WITH_SMART_HOLDER_AS_DEFAULT_BUT_NEVER_USE_IN_PRODUCTION_PLEASE
// NOTE: THIS IS MEANT FOR STRESS-TESTING OR TRIAGING ONLY!
// Running the pybind11 unit tests with smart_holder as the default holder is to ensure
// that `py::smart_holder` / `py::classh` is backward-compatible with all pre-existing
// functionality.
// Be careful not to link translation units compiled with different default holders, because
// this will cause ODR violations (https://en.wikipedia.org/wiki/One_Definition_Rule).
template <typename>
using default_holder_type = smart_holder;
#else
template <typename T>
using default_holder_type = std::unique_ptr<T>;
#endif
template <typename type_, typename... options>
class class_ : public detail::generic_type {
template <typename T>
@@ -1634,7 +1869,7 @@ public:
using type = type_;
using type_alias = detail::exactly_one_t<is_subtype, void, options...>;
constexpr static bool has_alias = !std::is_void<type_alias>::value;
using holder_type = detail::exactly_one_t<is_holder, std::unique_ptr<type>, options...>;
using holder_type = detail::exactly_one_t<is_holder, default_holder_type<type>, options...>;
static_assert(detail::all_of<is_valid_class_option<options>...>::value,
"Unknown/invalid class_ template parameters provided");
@@ -1665,7 +1900,16 @@ public:
record.type_align = alignof(conditional_t<has_alias, type_alias, type> &);
record.holder_size = sizeof(holder_type);
record.init_instance = init_instance;
record.default_holder = detail::is_instantiation<std::unique_ptr, holder_type>::value;
if (detail::is_instantiation<std::unique_ptr, holder_type>::value) {
record.holder_enum_v = detail::holder_enum_t::std_unique_ptr;
} else if (detail::is_instantiation<std::shared_ptr, holder_type>::value) {
record.holder_enum_v = detail::holder_enum_t::std_shared_ptr;
} else if (std::is_same<holder_type, smart_holder>::value) {
record.holder_enum_v = detail::holder_enum_t::smart_holder;
} else {
record.holder_enum_v = detail::holder_enum_t::custom_holder;
}
set_operator_new<type>(&record);
@@ -1804,9 +2048,11 @@ public:
class_ &def_readwrite(const char *name, D C::*pm, const Extra &...extra) {
static_assert(std::is_same<C, type>::value || std::is_base_of<C, type>::value,
"def_readwrite() requires a class member (or base class member)");
cpp_function fget([pm](const type &c) -> const D & { return c.*pm; }, is_method(*this)),
fset([pm](type &c, const D &value) { c.*pm = value; }, is_method(*this));
def_property(name, fget, fset, return_value_policy::reference_internal, extra...);
def_property(name,
property_cpp_function<type, D>::read(pm, *this),
property_cpp_function<type, D>::write(pm, *this),
return_value_policy::reference_internal,
extra...);
return *this;
}
@@ -1814,8 +2060,10 @@ public:
class_ &def_readonly(const char *name, const D C::*pm, const Extra &...extra) {
static_assert(std::is_same<C, type>::value || std::is_base_of<C, type>::value,
"def_readonly() requires a class member (or base class member)");
cpp_function fget([pm](const type &c) -> const D & { return c.*pm; }, is_method(*this));
def_property_readonly(name, fget, return_value_policy::reference_internal, extra...);
def_property_readonly(name,
property_cpp_function<type, D>::readonly(pm, *this),
return_value_policy::reference_internal,
extra...);
return *this;
}
@@ -1992,6 +2240,8 @@ private:
/// instance. Should be called as soon as the `type` value_ptr is set for an instance. Takes
/// an optional pointer to an existing holder to use; if not specified and the instance is
/// `.owned`, a new holder will be constructed to manage the value pointer.
template <typename H = holder_type,
detail::enable_if_t<!detail::is_smart_holder<H>::value, int> = 0>
static void init_instance(detail::instance *inst, const void *holder_ptr) {
auto v_h = inst->get_value_and_holder(detail::get_type_info(typeid(type)));
if (!v_h.instance_registered()) {
@@ -2001,6 +2251,65 @@ private:
init_holder(inst, v_h, (const holder_type *) holder_ptr, v_h.value_ptr<type>());
}
template <typename WrappedType>
static bool try_initialization_using_shared_from_this(holder_type *, WrappedType *, ...) {
return false;
}
// Adopting existing approach used by type_caster_base, although it leads to somewhat fuzzy
// ownership semantics: if we detected via shared_from_this that a shared_ptr exists already,
// it is reused, irrespective of the return_value_policy in effect.
// "SomeBaseOfWrappedType" is needed because std::enable_shared_from_this is not necessarily a
// direct base of WrappedType.
template <typename WrappedType, typename SomeBaseOfWrappedType>
static bool try_initialization_using_shared_from_this(
holder_type *uninitialized_location,
WrappedType *value_ptr_w_t,
const std::enable_shared_from_this<SomeBaseOfWrappedType> *) {
auto shd_ptr = std::dynamic_pointer_cast<WrappedType>(
detail::try_get_shared_from_this(value_ptr_w_t));
if (!shd_ptr) {
return false;
}
// Note: inst->owned ignored.
new (uninitialized_location) holder_type(holder_type::from_shared_ptr(shd_ptr));
return true;
}
template <typename H = holder_type,
detail::enable_if_t<detail::is_smart_holder<H>::value, int> = 0>
static void init_instance(detail::instance *inst, const void *holder_const_void_ptr) {
// Need for const_cast is a consequence of the type_info::init_instance type:
// void (*init_instance)(instance *, const void *);
auto *holder_void_ptr = const_cast<void *>(holder_const_void_ptr);
auto v_h = inst->get_value_and_holder(detail::get_type_info(typeid(type)));
if (!v_h.instance_registered()) {
register_instance(inst, v_h.value_ptr(), v_h.type);
v_h.set_instance_registered();
}
auto *uninitialized_location = std::addressof(v_h.holder<holder_type>());
auto *value_ptr_w_t = v_h.value_ptr<type>();
// Try downcast from `type` to `type_alias`:
inst->is_alias
= detail::dynamic_raw_ptr_cast_if_possible<type_alias>(value_ptr_w_t) != nullptr;
if (holder_void_ptr) {
// Note: inst->owned ignored.
auto *holder_ptr = static_cast<holder_type *>(holder_void_ptr);
new (uninitialized_location) holder_type(std::move(*holder_ptr));
} else if (!try_initialization_using_shared_from_this(
uninitialized_location, value_ptr_w_t, value_ptr_w_t)) {
if (inst->owned) {
new (uninitialized_location) holder_type(holder_type::from_raw_ptr_take_ownership(
value_ptr_w_t, /*void_cast_raw_ptr*/ inst->is_alias));
} else {
new (uninitialized_location)
holder_type(holder_type::from_raw_ptr_unowned(value_ptr_w_t));
}
}
v_h.set_holder_constructed();
}
// Deallocates an instance; via holder, if constructed; otherwise via operator delete.
// NOTE: The Python error indicator needs to cleared BEFORE this function is called.
// This is because we could be deallocating while cleaning up after a Python exception.
@@ -2066,6 +2375,11 @@ private:
}
};
// Supports easier switching between py::class_<T> and py::class_<T, py::smart_holder>:
// users can simply replace the `_` in `class_` with `h` or vice versa.
template <typename type_, typename... options>
using classh = class_<type_, smart_holder, options...>;
/// Binds an existing constructor taking arguments Args...
template <typename... Args>
detail::initimpl::constructor<Args...> init() {