Fix libevwrapper atexit cleanup bug causing Python shutdown crashes

[Copilot]

Description

The atexit callback in libevreactor.py was registered with _global_loop=None at import time, causing it to receive None during shutdown instead of the actual loop instance. This prevented cleanup from running, leaving active libev callbacks that could execute during Python shutdown and crash.

This PR implements two complementary solutions to fix the issue and prevent shutdown crashes.

Changes

Analysis and Documentation

• LIBEV_SHUTDOWN_ANALYSIS.md: Comprehensive analysis document identifying the root cause and 6 different crash scenarios:

  1. Race condition during thread join timeout
  2. GIL state issues during interpreter finalization
  3. Object lifecycle issues (callbacks accessing deallocated Python objects)
  4. Connection cleanup not triggered (active connections with pending I/O)
  5. Module deallocation order problems
  6. Fork handling issues

• IMPLEMENTATION_SUMMARY.md: Complete documentation of implemented solutions, code changes, and testing approach

Solution 1: Fix atexit Registration (Minimal Change)

File: cassandra/io/libevreactor.py

Replaced the problematic atexit.register(partial(_cleanup, _global_loop)) with a wrapper function that looks up _global_loop at shutdown time:

def _atexit_cleanup():
    """Cleanup function that looks up _global_loop at shutdown time."""
    global _global_loop
    if _global_loop is not None:
        _cleanup(_global_loop)

_global_loop = None
atexit.register(_atexit_cleanup)

• Removed unused partial import from functools
• Ensures cleanup receives the actual LibevLoop instance, not None
• Minimal change (14 lines modified)

Solution 2: Add loop.stop() Method (Enhanced Robustness)

Files: cassandra/io/libevwrapper.c, cassandra/io/libevreactor.py

Added explicit loop stopping mechanism to the C extension:

C Extension Changes:

• Added ev_async async_watcher field to libevwrapper_Loop struct
• Implemented async_stop_cb() callback that calls ev_break(EVBREAK_ALL)
• Added Loop_stop() method callable from Python
• Initialize and start async_watcher in Loop_init()
• Clean up async_watcher in Loop_dealloc()

Python Changes:

• Updated _atexit_cleanup() to call loop.stop() before cleanup
• Thread-safe loop stopping via libev's async mechanism
• Wrapped in try/except to continue cleanup if stop fails

Tests

File: tests/unit/io/test_libevreactor_shutdown.py

• test_atexit_callback_uses_current_global_loop() - Verifies atexit handler is the wrapper function, not a partial
• test_shutdown_cleanup_works_with_fix() - Subprocess test verifying proper cleanup
• test_cleanup_with_fix_properly_shuts_down() - Verifies cleanup actually shuts down the loop

Root Cause

Line 231 in libevreactor.py:

_global_loop = None
atexit.register(partial(_cleanup, _global_loop))  # Captures None!

The partial function captures None at import time. Later when _global_loop is set to a LibevLoop instance, the atexit callback still has None captured, so cleanup never runs.

How the Fixes Work Together

1. Solution 1 ensures the atexit cleanup callback receives the actual loop instance
2. Solution 2 explicitly stops the event loop before cleanup starts
3. Together, they provide defense in depth:
   • The loop is stopped cleanly via thread-safe async mechanism
   • Watchers are properly shut down
   • The event loop thread is joined with proper cleanup
   • Callbacks won't fire during Python interpreter shutdown

Pre-review checklist

• I have split my patch into logically separate commits.
• All commit messages clearly explain what they change and why.
• I added relevant tests for new features and bug fixes.
• All commits compile, pass static checks and pass test.
• PR description sums up the changes and reasons why they should be introduced.
• I have provided docstrings for the public items that I want to introduce.
• I have adjusted the documentation in ./docs/source/.

Original prompt

---

This section details on the original issue you should resolve

<issue_title>Refactor libevwrapper cleanup situation</issue_title>
<issue_description>libevwrapper.py is using atexit module to do it's cleanup of global loop

from time to time we see the following crashes during python shutdown:

• Artifact test crashes on python shutdown (libevwrapper) scylla-cluster-tests#11713
• Sporadic SIGSEGV (exit code -11) in Python when exiting the Alternator test scylladb#17564 (suspect, since we don't collect there the dumps, to see it)

for years we had a workaround of some sort in SCT that was clearing the atexit hooks, blindly.
we removed it since it breaking other part that SCT or other part are using atexit.

and now we are getting this crash from time to time.

there two things that I think needs to improve:

1. ability to stop the libev loop, so we won't get into case it wakes up during python shutdown
2. refactor code to no use global loop and do cleanup directly without atexit

complete vibe code example of part one, it got compiled.
but didn't have a clear path (nor time) to test it

diff --git a/cassandra/io/libevwrapper.c b/cassandra/io/libevwrapper.c
index f32504fa3..0689bf3b9 100644
--- a/cassandra/io/libevwrapper.c
+++ b/cassandra/io/libevwrapper.c
@@ -6,6 +6,7 @@
 typedef struct libevwrapper_Loop {
     PyObject_HEAD
     struct ev_loop *loop;
+    ev_async async_watcher;
 } libevwrapper_Loop;

 static void
@@ -30,12 +31,24 @@ Loop_new(PyTypeObject *type, PyObject *args, PyObject *kwds) {
     return (PyObject *)self;
 };

+static void async_stop_cb(EV_P_ ev_async *w, int revents) {
+    ev_break(EV_A_ EVBREAK_ALL);
+}
+
+static PyObject *
+Loop_stop(libevwrapper_Loop *self, PyObject *args) {
+    ev_async_send(self->loop, &self->async_watcher);
+    Py_RETURN_NONE;
+}
+
 static int
 Loop_init(libevwrapper_Loop *self, PyObject *args, PyObject *kwds) {
     if (!PyArg_ParseTuple(args, "")) {
         PyErr_SetString(PyExc_TypeError, "Loop.__init__() takes no arguments");
         return -1;
     }
+    ev_async_init(&self->async_watcher, async_stop_cb);
+    ev_async_start(self->loop, &self->async_watcher);
     return 0;
 };

@@ -56,6 +69,7 @@ Loop_unref(libevwrapper_Loop *self, PyObject *args) {
 static PyMethodDef Loop_methods[] = {
     {"start", (PyCFunction)Loop_start, METH_NOARGS, "Start the event loop"},
     {"unref", (PyCFunction)Loop_unref, METH_NOARGS, "Unrefrence the event loop"},
+    {"stop", (PyCFunction)Loop_stop, METH_NOARGS, "Stop the event loop from any thread"},
     {NULL} /* Sentinel */
 };

</issue_description>

<agent_instructions>Please suggest a fix to clearing out the callback during shutdown, that doesn't include changing any 3rd party library, and wait to approval before implantation </agent_instructions>

Comments on the Issue (you are @copilot in this section)

@fruch NOTE: we are using libev backend across the board in testing, it's the most recommended and tested one from POV. this might be the cause of miss-using or leaving opening connection behind, but even if that happens we should prevent crashing python shutdown @fruch @absurdfarce are you familiar with such issues during the python shutdown ?

I did a quick search, and found any references </comment_new>
<comment_new>@fruch
@absurdfarce thanks, all fit with my suspicion that we have in flight request (in our tests that it happens, the Cassandra/scylla server is still up, when we shutdown the interpreter), and since we don't hold reference to the libev and stop the loop, it's callback can be called after/during shutdown.

• Fixes Refactor libevwrapper cleanup situation #524

---

💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.

[fruch]

@copilot suggest more directions that might cause the issue.

Also write a test that show case the problem, before implantation of any fix

[Copilot]

@copilot suggest more directions that might cause the issue.

Also write a test that show case the problem, before implantation of any fix

I've created a comprehensive analysis in commit f56442c:

Analysis Document: LIBEV_SHUTDOWN_ANALYSIS.md identifies 6 crash scenarios:

1. Race condition during thread join - Event loop thread keeps running after 1s timeout
2. GIL state issues - PyGILState_Ensure() may fail during interpreter finalization
3. Object lifecycle issues - Callbacks access deallocated Python objects (the main crash cause)
4. Connection cleanup not triggered - Active connections with pending I/O fire callbacks during shutdown
5. Module deallocation order - libev accesses modules (logging, os, threading) that may be torn down
6. Fork handling issues - Child process atexit may try to clean parent's loop state

Test File: tests/unit/io/test_libevreactor_shutdown.py demonstrates:

• The atexit callback captures None instead of the actual loop instance
• Cleanup receives None at shutdown and does nothing
• A subprocess test showing the exact bug scenario

Root Cause: Line 231 in libevreactor.py:

atexit.register(partial(_cleanup, _global_loop))  # _global_loop is None here!

The partial captures None, so cleanup never runs, leaving callbacks active during Python shutdown.

Recommended Fix (minimal change):

def _atexit_cleanup():
    global _global_loop
    if _global_loop is not None:
        _cleanup(_global_loop)

atexit.register(_atexit_cleanup)

This ensures cleanup looks up the current _global_loop value at shutdown time.

[fruch]

@copilot continue with implementation of the suggested fixes, one at a time

[Copilot]

@copilot continue with implementation of the suggested fixes, one at a time

Implemented Solution 1: Fix atexit Registration in commit 8c90f05.

Changes Made:

• Replaced atexit.register(partial(_cleanup, _global_loop)) with a wrapper function _atexit_cleanup()
• The wrapper looks up the current _global_loop value at shutdown time instead of capturing None at import time
• Removed unused partial import
• Updated tests to verify the fix works

How it fixes the bug:

• Old code: partial(_cleanup, _global_loop) captured None → cleanup received None at shutdown → did nothing
• New code: _atexit_cleanup() looks up _global_loop when called → cleanup receives actual loop instance → properly shuts down

This is the minimal change approach (14 lines modified). The next step would be Solution 2 (add loop.stop() method in C extension) if you'd like additional robustness.