Add minimal C-api implementation that builds with Pyo3#7562
Add minimal C-api implementation that builds with Pyo3#7562bschoenmaeckers wants to merge 74 commits intoRustPython:mainfrom
Conversation
|
Important Review skippedDraft detected. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Path: .coderabbit.yml Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
Could you try with hpy if you dont mind? We are looking for sustainable way |
That would require significant changes in codebase that are using the CPython api spec. I would like to explore the possibility to use the ab3/abi3t api. Would you be willing to accept that? |
youknowone
left a comment
There was a problem hiding this comment.
This way will not work.
Just in case, the code can be generated, but I hope we carefully review the decisions. unlikely-to-happen decisions must be reviewed and justified by document(comment) the decisions.
If you are not familiar enough to the project to decide good way, please start from a smaller and simpler issue.
We only need abi3t, because we are compatible to free-threading.
We have to minimize the surface of C API. if HPy helps it, we need HPy.
Ideally c api must be very thin wrappers to RustPython features. "code" must not be included in this crate.
| use std::cell::RefCell; | ||
|
|
||
| thread_local! { | ||
| static VM: RefCell<Option<ThreadedVirtualMachine>> = const { RefCell::new(None) }; |
There was a problem hiding this comment.
C API must be simple API set. It must not contains any state ideally. And never the VM.
There was a problem hiding this comment.
The VM_CURRENT static in vm/thread.rs is scoped. So I cannot set and unset it in PyGILState_Ensure/ PyGILState_Release. (These methods have nothing to do with the GIL in free-threaded world despite GIL in the name)
There was a problem hiding this comment.
I could use VM_STACK instead... 🤔
There was a problem hiding this comment.
Indeed. Let's try to expose it
There was a problem hiding this comment.
VM_STACK only stores pointers (NonNull<VirtualMachine>) to a vm, so I need some other place to store it. What do you recommend?
There was a problem hiding this comment.
Could you tell me what's exactly the expectation? Please tell me what do you more need than accessing NonNull<VirtualMachine>. I will try to find a way to expose what you need.
There was a problem hiding this comment.
Could you tell me what's exactly the expectation? Please tell me what do you more need than accessing
NonNull<VirtualMachine>. I will try to find a way to expose what you need.
Thank's for helping me solve this!
Let's assume the Interpreter is already initialized in the current process and Py_IsInitialized will return true (Let's ignore the implementation of Py_IsInitialized for now).
What a CPython extension in general will do first is making sure the current thread (could be the main thread or some other thread that got spawned using python or native code like rust's std::thread::spawn) is attached to a interpreter by calling PyGILState_Ensure. After calling PyGILState_Ensure the current thread should be allowed to call into python code. What this means in RustPython is that we have a VirtualMachine instance for the current thread.
In short what I need is:
- a random thread can call
PyGILState_Ensurewithout having a reference to the mainInterpreterinstance - after
PyGILState_Ensurereturn there should be a thread localVirtualMachineinstance available for usage in the current thread.
There was a problem hiding this comment.
a random thread can call PyGILState_Ensure without having a reference to the main Interpreter instance
This is actually breaking our Interpreter assumptions. RustPython VMs share genesis but not about interpreter. Probably we need to pin an interpreter when RustPython is initialized by C API.
There was a problem hiding this comment.
HPy currently does not have any API's to do this, so that is also not a way out. See hpyproject/hpy#268 and hpyproject/hpy#439
I wanted to create as little of modificaties to the existing code base without significant motivation. But I definitely agree with you. |
| let (response_tx, response_rx) = mpsc::channel(); | ||
| tx.send(response_tx).expect("Failed to send VM request"); | ||
| response_rx.recv().expect("Failed to receive VM response") | ||
| } |
There was a problem hiding this comment.
This is also somewhat cursed. I want to be able to create ThreadedVirtualMachine scoped to the thread where PyGILState_Ensure is called. This may be from a different thread than where the main interpreter is running. So I need a way to share the Interpreter between different threads.
There was a problem hiding this comment.
I'd like to understand why rustpython_vm::vm::thread::with_vm is not enough but this is required
There was a problem hiding this comment.
I'd like to understand why
rustpython_vm::vm::thread::with_vmis not enough but this is required
This is so I can call PyGILState_Ensure from any thread, without holding a reference to a Interpreter. This may happen when creating a new thread using std::thread::spawn, and trying to run python code in there. See my test_new_thread testcase.
RustPython/crates/capi/src/pystate.rs
Lines 61 to 75 in 56f3823
This is my shot at implementing a minimal Cpython compatible C-api. I've implemented the bare minimum to get the included Pyo3 example running where most of the api is stubbed. I'm not familiar with the rest of the RustPython code base so let me know what you think and where I did stupid things.
Please take extra care reviewing the pylifecycle.rs & pystate.rs files where I try to setup the RustPython interpreter.
xref #5604