-
-
Notifications
You must be signed in to change notification settings - Fork 32.7k
gh-79805: Indicate that Shelve.sync()
is not thread-safe
#127507
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is it thread-safe if writeback is not set to True (by default)?
Shelve.sync()
is not thread-safe.Shelve.sync()
is not thread-safe
I don't know :( friendly ping @rhettinger |
I have tested four scenarios regarding thread safety issues with the shelve module. When threading with a shared object with Scenario 1: Shared Object, writeback=False (FAILS) import shelve
import threading
db = shelve.open("example.shelve") # writeback=False by default
for i in range(100):
db[str(i)] = i**2
def read_shelve(thread_id):
for i in range(50):
value = db[str(i)]
print(f"Thread {thread_id} - {i}: {value}")
t1 = threading.Thread(target=read_shelve, args=(1,))
t2 = threading.Thread(target=read_shelve, args=(2,))
t1.start()
t2.start()
t1.join()
t2.join() Output: Exception in thread Thread-1 (read_shelve):
Exception in thread Thread-2 (read_shelve):
Traceback (most recent call last):
File "/usr/lib/python3.13/shelve.py", line 111, in __getitem__
value = self.cache[key]
~~~~~~~~~~^^^^^
KeyError: '0'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3.13/dbm/sqlite3.py", line 79, in _execute
return closing(self._cx.execute(*args, **kwargs))
~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
sqlite3.ProgrammingError: SQLite objects created in a thread can only be used in that same thread. The object was created in thread id 134486782028736 and this is thread id 134486754248384.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
Traceback (most recent call last):
File "/usr/lib/python3.13/threading.py", line 1043, in _bootstrap_inner
self.run()
~~~~~~~~^^
File "/usr/lib/python3.13/shelve.py", line 111, in __getitem__
value = self.cache[key]
~~~~~~~~~~^^^^^
File "/usr/lib/python3.13/threading.py", line 994, in run
self._target(*self._args, **self._kwargs)
~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
KeyError: '0'
File "/home/arf/test_shelve_multi/shelve_thread_shared_object.py", line 11, in read_shelve
value = db[str(i)]
~~^^^^^^^^
During handling of the above exception, another exception occurred:
File "/usr/lib/python3.13/shelve.py", line 113, in __getitem__
f = BytesIO(self.dict[key.encode(self.keyencoding)])
~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Traceback (most recent call last):
File "/usr/lib/python3.13/dbm/sqlite3.py", line 89, in __getitem__
with self._execute(LOOKUP_KEY, (key,)) as cu:
~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.13/dbm/sqlite3.py", line 79, in _execute
return closing(self._cx.execute(*args, **kwargs))
~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.13/dbm/sqlite3.py", line 81, in _execute
raise error(str(exc))
sqlite3.ProgrammingError: SQLite objects created in a thread can only be used in that same thread. The object was created in thread id 134486782028736 and this is thread id 134486762641088.
dbm.sqlite3.error: SQLite objects created in a thread can only be used in that same thread. The object was created in thread id 134486782028736 and this is thread id 134486754248384.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3.13/threading.py", line 1043, in _bootstrap_inner
self.run()
~~~~~~~~^^
File "/usr/lib/python3.13/threading.py", line 994, in run
self._target(*self._args, **self._kwargs)
~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/arf/test_shelve_multi/shelve_thread_shared_object.py", line 11, in read_shelve
value = db[str(i)]
~~^^^^^^^^
File "/usr/lib/python3.13/shelve.py", line 113, in __getitem__
f = BytesIO(self.dict[key.encode(self.keyencoding)])
~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.13/dbm/sqlite3.py", line 89, in __getitem__
with self._execute(LOOKUP_KEY, (key,)) as cu:
~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.13/dbm/sqlite3.py", line 81, in _execute
raise error(str(exc))
dbm.sqlite3.error: SQLite objects created in a thread can only be used in that same thread. The object was created in thread id 134486782028736 and this is thread id 134486762641088. Scenario 2: Shared Object, writeback=True (WORKS) import shelve
import threading
db = shelve.open("example.shelve", writeback=True)
for i in range(100):
db[str(i)] = i**2
def read_shelve(thread_id):
for i in range(50):
value = db[str(i)]
print(f"Thread {thread_id} - {i}: {value}")
t1 = threading.Thread(target=read_shelve, args=(1,))
t2 = threading.Thread(target=read_shelve, args=(2,))
t1.start()
t2.start()
t1.join()
t2.join() Scenario 3: Separate Objects Per Thread writeback=True (WORKS) import shelve
import threading
db = shelve.open("example.shelve", writeback=True)
for i in range(100):
db[str(i)] = i**2
def read_shelve(thread_id):
for i in range(50):
value = db[str(i)]
print(f"Thread {thread_id} - {i}: {value}")
t1 = threading.Thread(target=read_shelve, args=(1,))
t2 = threading.Thread(target=read_shelve, args=(2,))
t1.start()
t2.start()
t1.join()
t2.join() Scenario 4: Separate Objects Per Thread writeback=False (WORKS) import shelve
import threading
db = shelve.open("example.shelve") # writeback=False by default
for i in range(100):
db[str(i)] = i**2
def read_shelve(thread_id):
for i in range(50):
value = db[str(i)]
print(f"Thread {thread_id} - {i}: {value}")
t1 = threading.Thread(target=read_shelve, args=(1,))
t2 = threading.Thread(target=read_shelve, args=(2,))
t1.start()
t2.start()
t1.join()
t2.join() Output of successfully executed scenarios:
|
📚 Documentation preview 📚: https://cpython-previews--127507.org.readthedocs.build/