Two mgr fixes

- Work around mgr `restful` module PyO3 failures
- Fix NOTIFY_TYPES exceptions during module import (rhbz#2361850)
This commit is contained in:
Hector Martin
2025-08-20 20:28:03 +09:00
parent 675842896e
commit 10c2ce06df
3 changed files with 218 additions and 1 deletions
@@ -0,0 +1,152 @@
From 0000000000000000000000000000000000000000 Mon Sep 17 00:00:00 2001
From: "Max R. Carrara" <m.carrara@proxmox.com>
Date: Wed, 16 Jul 2025 13:14:39 +0200
Subject: [PATCH 58/59] pybind/mgr/restful: provide workaround for PyO3
ImportError
Move the self-signed cert generation into a separate module
inside python-common/ceph and run the module in a separate Python
process.
This provides a workaround for the ImportError thrown by PyO3 when
the `restful` module is loaded in the context of multiple Python
sub-interpreters being present. In particular, the ImportError is
thrown by the `crypto` module of the `OpenSSL` package.
Inspired by an upstream PR [0].
[0]: https://github.com/ceph/ceph/pull/62951
Signed-off-by: Max R. Carrara <m.carrara@proxmox.com>
---
src/pybind/mgr/restful/module.py | 24 +++------
src/python-common/ceph/_crypto_wrap.py | 69 ++++++++++++++++++++++++++
2 files changed, 76 insertions(+), 17 deletions(-)
create mode 100644 src/python-common/ceph/_crypto_wrap.py
diff --git a/src/pybind/mgr/restful/module.py b/src/pybind/mgr/restful/module.py
index 0f8c78e0bd8..7f93c41f1e6 100644
--- a/src/pybind/mgr/restful/module.py
+++ b/src/pybind/mgr/restful/module.py
@@ -7,6 +7,7 @@ import json
import time
import errno
import inspect
+import subprocess
import tempfile
import threading
import traceback
@@ -19,7 +20,6 @@ from . import context
from uuid import uuid4
from pecan import jsonify, make_app
-from OpenSSL import crypto
from pecan.rest import RestController
from werkzeug.serving import make_server, make_ssl_devcert
@@ -401,24 +401,14 @@ class Module(MgrModule):
def create_self_signed_cert(self):
- # create a key pair
- pkey = crypto.PKey()
- pkey.generate_key(crypto.TYPE_RSA, 2048)
-
- # create a self-signed cert
- cert = crypto.X509()
- cert.get_subject().O = "IT"
- cert.get_subject().CN = "ceph-restful"
- cert.set_serial_number(int(uuid4()))
- cert.gmtime_adj_notBefore(0)
- cert.gmtime_adj_notAfter(10*365*24*60*60)
- cert.set_issuer(cert.get_subject())
- cert.set_pubkey(pkey)
- cert.sign(pkey, 'sha512')
+ cmd = ["python3", "-m", "ceph._crypto_wrap", "create_self_signed_cert"]
+
+ response = subprocess.run(cmd, capture_output=True, check=True)
+ response_obj = json.loads(response.stdout)
return (
- crypto.dump_certificate(crypto.FILETYPE_PEM, cert),
- crypto.dump_privatekey(crypto.FILETYPE_PEM, pkey)
+ response_obj["cert"].encode("utf-8"),
+ response_obj["key"].encode("utf-8"),
)
diff --git a/src/python-common/ceph/_crypto_wrap.py b/src/python-common/ceph/_crypto_wrap.py
new file mode 100644
index 00000000000..16a19a5345e
--- /dev/null
+++ b/src/python-common/ceph/_crypto_wrap.py
@@ -0,0 +1,69 @@
+"""CLI wrapper for cryptographic functions of the :mod:`restful` module.
+
+To be called via :func:`subprocess.run()` as a workaround for
+:class:`ImportError`s related to PyO3's current lack of sub-interpreter
+support.
+
+Note:
+ Since this module is installed as part of the ``ceph`` package,
+ it should be called like so::
+
+ python3 -m ceph._crypto_wrap create_self_signed_cert
+"""
+
+import argparse
+import sys
+import json
+
+from argparse import Namespace
+from typing import Any
+from uuid import uuid4
+
+from OpenSSL import crypto
+
+
+def _respond(data: dict[str, Any]) -> None:
+ json.dump(data, sys.stdout)
+ sys.stdout.flush()
+
+
+def create_self_signed_cert(args: Namespace) -> None:
+ cert_key_pair = _create_self_signed_cert()
+ _respond(cert_key_pair)
+
+
+def _create_self_signed_cert() -> dict[str, str]:
+ # create a key pair
+ pubkey = crypto.PKey()
+ pubkey.generate_key(crypto.TYPE_RSA, 2048)
+
+ # create a self-signed cert
+ cert = crypto.X509()
+ cert.get_subject().O = "IT"
+ cert.get_subject().CN = "ceph-restful"
+ cert.set_serial_number(int(uuid4()))
+ cert.gmtime_adj_notBefore(0)
+ cert.gmtime_adj_notAfter(10 * 365 * 24 * 60 * 60)
+ cert.set_issuer(cert.get_subject())
+ cert.set_pubkey(pubkey)
+ cert.sign(pubkey, "sha512")
+
+ return {
+ "cert": crypto.dump_certificate(crypto.FILETYPE_PEM, cert).decode(),
+ "key": crypto.dump_privatekey(crypto.FILETYPE_PEM, pubkey).decode(),
+ }
+
+
+def main() -> None:
+ parser = argparse.ArgumentParser(prog="_crypto_wrap.py")
+ subparsers = parser.add_subparsers(required=True)
+
+ parser_cssc = subparsers.add_parser("create_self_signed_cert")
+ parser_cssc.set_defaults(func=create_self_signed_cert)
+
+ args = parser.parse_args()
+ args.func(args)
+
+
+if __name__ == "__main__":
+ main()
@@ -0,0 +1,56 @@
From 0000000000000000000000000000000000000000 Mon Sep 17 00:00:00 2001
From: "Max R. Carrara" <m.carrara@proxmox.com>
Date: Wed, 16 Jul 2025 16:31:43 +0200
Subject: [PATCH 59/59] mgr: fix module import by making NOTIFY_TYPES in py
modules optional
If NOTIFY_TYPES isn't an attribute of the passed class, the Python
(sub-)interpreter raises an AttributeError that must be handled or cleared
explicitly via the Python C-API. Unfortunately, this isn't done here,
which means that the exception sticks around until handled.
This caused a call to PyModule::load_subclass_of() to fail and
incorrectly report the AttributeError as cause.
Checking whether the class has NOTIFY_TYPES as attribute in the first
place fixes this.
Note that there's an upstream PR [0] that wasn't backported that aimed
to fix this, but does so incorrectly, as the exception is still not
cleared there. The warnings regarding NOTIFY_TYPES missing also occurs
on Reef but doesn't cause any module imports to fail there. As the
affected Ceph code has stayed mostly the same between bookworm and
trixie releases, this suggests that some behavior between Python 3.11
and 3.13 likely changed.
Either way, avoiding the AttributeError altogether fixes this.
[0]: https://github.com/ceph/ceph/pull/57106
Signed-off-by: Max R. Carrara <m.carrara@proxmox.com>
---
src/mgr/PyModule.cc | 10 ++++++----
1 file changed, 6 insertions(+), 4 deletions(-)
diff --git a/src/mgr/PyModule.cc b/src/mgr/PyModule.cc
index 084cf3ffc1e..e6fd269dca5 100644
--- a/src/mgr/PyModule.cc
+++ b/src/mgr/PyModule.cc
@@ -513,11 +513,13 @@ int PyModule::register_options(PyObject *cls)
int PyModule::load_notify_types()
{
- PyObject *ls = PyObject_GetAttrString(pClass, "NOTIFY_TYPES");
- if (ls == nullptr) {
- derr << "Module " << get_name() << " has missing NOTIFY_TYPES member" << dendl;
- return -EINVAL;
+ if (!PyObject_HasAttrString(pClass, "NOTIFY_TYPES")) {
+ dout(10) << "Module " << get_name() << " has no NOTIFY_TYPES member" << dendl;
+ return 0;
}
+
+ PyObject *ls = PyObject_GetAttrString(pClass, "NOTIFY_TYPES");
+
if (!PyObject_TypeCheck(ls, &PyList_Type)) {
// Relatively easy mistake for human to make, e.g. defining COMMANDS
// as a {} instead of a []
+10 -1
View File
@@ -185,7 +185,7 @@
#################################################################################
Name: ceph
Version: 19.2.3
Release: 6%{?dist}
Release: 7%{?dist}
%if 0%{?fedora} || 0%{?rhel}
Epoch: 2
%endif
@@ -233,6 +233,11 @@ Patch: 0058-mgr-dashboard-Make-saml2-robust-against-module-load-.patch
# Fixed in upcoming 19.2.4
Patch: 0059-mgr-dashboard-catch-protobuf-error-due-to-mismatch-i.patch
# Squid fixes from https://git.proxmox.com/?p=ceph.git;a=tree;f=patches
# Should be fixed upstream in Ceph 20.
Patch: 0158-pybind-mgr-restful-provide-workaround-for-PyO3-Impor.patch
Patch: 0159-mgr-fix-module-import-by-making-NOTIFY_TYPES-in-py-m.patch
# ceph 14.0.1 does not support 32-bit architectures, bugs #1727788, #1727787
ExcludeArch: i686 armv7hl
%if 0%{?suse_version}
@@ -2748,6 +2753,10 @@ exit 0
%{python3_sitelib}/ceph_node_proxy-*
%changelog
* Wed Aug 20 2025 Hector Martin <marcan@marcan.st> - 2:19.2.3-7
- Work around mgr `restful` module PyO3 failures
- Fix NOTIFY_TYPES exceptions during module import (rhbz#2361850)
* Wed Aug 20 2025 Hector Martin <marcan@marcan.st> - 2:19.2.3-6
- Fix F41 build