Skip to content

Python 3.13 breaks circular imports through single-phase extension modules using non-isolated subinterpreters #138045

@marcan

Description

@marcan

Bug report

Bug description:

I believe this regression was caused by the changes made to fix gh-117953.

Repro:

import _interpreters
interp = _interpreters.create()
_interpreters.run_string(interp, "import numpy")

Tested on Fedora 41 with Python 3.13.5 and numpy 1.26.4, and Fedora 42 with 3.13.7 and 2.2.6, respectively.

What happens is that:

  • The subinterpreter tries to load numpy
  • Numpy loads a bunch of Python modules
  • numpy eventually tries to load the multiarray extension
  • import_run_extension switches to the main interpreter, as explained in the comment
  • The module init is called in the main interpreter
  • PyInit__multiarray_umath actually initializes in the main interpreter
  • initialize_static_globals() attempts to load some numpy Python modules from inside the extension
  • Since these modules aren't loaded in the main interpreter yet, the whole process starts over again loading numpy, now in the main interpreter
  • This then leads to loading the multiarray extension again, this time already in the main interpreter, so no switching happens
  • Python reentrantly attempts to initialize the (single-phase!) extension module in the main interpreter again
  • Kaboom

The easiest way to see the carnage in action is to run it under gdb and break on PyInit__multiarray_umath. You'll see it gets called twice, the second time reentrantly (with a 276-frame backtrace going through numpy initialization twice). If you don't set any breakpoints, you'll get crashes trying to process exceptions and other craziness when the original reentrant execution backtrace is long gone.

numpy does not support subinterpreters, but I would expect it to work when it is used in a single subinterpreter without isolation.

This is the root cause for:

CPython versions tested on:

3.13

Operating systems tested on:

Linux

Metadata

Metadata

Assignees

Labels

3.13bugs and security fixes3.14bugs and security fixes3.15new features, bugs and security fixesextension-modulesC modules in the Modules dirtopic-subinterpreterstype-bugAn unexpected behavior, bug, or error

Projects

Status

Todo

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions