Add initial draft of Rust API design document - #45
Conversation
|
|
||
| - **Subinterpreters.** Future support requires an interpreter-identity model | ||
| for contexts, owning handles, module state, and process-global Rust values. | ||
| - **Buffer protocol.** No safe `PyBuffer` API is included until the design |
There was a problem hiding this comment.
For a safe buffer protocol, I'm thinking for now we could ensure the buffer is uniquely held, and copy it if it isn't? There are usability and performance implications of that however.
| Rust slices from racing with Python mutation or escaping across detach and | ||
| reentrant calls. Copying from an exact immutable bytes object is the safe | ||
| interim input path. | ||
| - **Interpreter-aware locking.** The design must either provide a lock which |
There was a problem hiding this comment.
For this I am thinking it would make the most sense to either use PyMutex and wrap that, or write a lock which cooperates with the interpreter like the PyOnceLock in PyO3.
b334292 to
4a6e800
Compare
ngoldbaum
left a comment
ngoldbaum
left a comment
There was a problem hiding this comment.
Did a read-through and had a few comments questions and suggestions
| pub type PyResult<T> = Result<T, PyErrRaised>; | ||
| ``` | ||
|
|
||
| `PyErrRaised` is `!Send` and `#[must_use]`. Built-in and custom exception types |
There was a problem hiding this comment.
Maybe worth an external link for must_use? I’d never seen it before this and had to look up the docs: https://doc.rust-lang.org/std/attribute.must_use.html
|
|
||
| Generic implementations compose. `Option<T>` maps `None`, borrowed handles can | ||
| be promoted with an incref, and an existing `Bound<T>` can be returned without | ||
| refcount traffic: |
| ``` | ||
|
|
||
| Only generated `ModuleDef<S>` glue constructs `ModuleRef<S>` and proves the | ||
| state type. |
There was a problem hiding this comment.
not quite sure what “proves the static type” means here
There was a problem hiding this comment.
Probably a mis-edit on my part, I'll fix that!
| Generated modules declare `Py_mod_gil = Py_MOD_GIL_NOT_USED`, the internal | ||
| exact-version ABI, the GIL/free-threaded build mode, and | ||
| `Py_mod_multiple_interpreters = Py_MOD_MULTIPLE_INTERPRETERS_NOT_SUPPORTED`. | ||
| Stable-ABI and subinterpreter support are out of scope. |
There was a problem hiding this comment.
Is lack of subinterpreter support a problem for the PoC? Or would we degrade to a pure-python library instead of using a Rust accelerator under subinterpreters?
There was a problem hiding this comment.
I expect it will be. There's been a lot of effort to have the stdlib support subinterpreters. I'm not sure if we would be able to degrade gracefully.
It would be useful to have some solution, but I'm okay punting until the core sprints.
| pub struct ThreadContext<'py> { | ||
| ptr: NonNull<ffi::PyThreadState>, | ||
| detach: Option<DetachPermit<'py>>, | ||
| #[allow(clippy::type_complexity)] |
There was a problem hiding this comment.
It's better to use expect where possible and to provide a reason as well:
| #[allow(clippy::type_complexity)] | |
| #[expect(clippy::type_complexity, reason = "...")] |
maybe it's a bit much for a documentation 🤷
| ```rust | ||
| #[derive(PyClassGc)] | ||
| struct Node { | ||
| parent: Option<Py<PyInstance<Node>>>, |
There was a problem hiding this comment.
| parent: Option<Py<PyInstance<Node>>>, | |
| parent: Option<Py<PyInstance<Self>>>, |
3 participants
Okay, it took a while to play with these API design ideas and get them to a state where I was happy with them. I have a rough prototype similar to #41 that is based on this, but I figured it would make more sense to discuss the API design and make sure I haven't missed things or there isn't unsoundness in this design. It turns out it is just darn difficult to get thread-state passing right when you consider re-entrancy and multi-threading.
As mentioned in the document, my overall priorities in this design were:
PyModExport.I think 3 is probably the furthest from achievable with this current design, we query the TLS quite frequently with this design, but I believe a lot of those queries could be cleaned up. 4 is really a matter of taste, but I tried to be somewhat similar while also trying to have as small an API surface as possible.
Things left undone in this design that we will need to figure out:
Dropproperly, including if one occurs while detached.I also wasn't totally sure where to put this, perhaps it would go better in InternalDocs?
AI disclosure: I worked with GPT 5.6 Sol when iterating on this API and the content is a mix of writing me and the LLM. I've reviewed every word, but please let me know if there is anything unclear!