Skip to content

Conversation

picnixz
Copy link
Member

@picnixz picnixz commented Dec 2, 2024

@bedevere-app bedevere-app bot added docs Documentation in the Doc dir skip news labels Dec 2, 2024
Copy link
Member

@serhiy-storchaka serhiy-storchaka left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it thread-safe if writeback is not set to True (by default)?

@picnixz picnixz changed the title gh-79805: Indicate that Shelve.sync() is not thread-safe. gh-79805: Indicate that Shelve.sync() is not thread-safe May 11, 2025
@picnixz
Copy link
Member Author

picnixz commented May 11, 2025

Is it thread-safe if writeback is not set to True (by default)?

I don't know :( friendly ping @rhettinger

@furkanonder
Copy link
Contributor

furkanonder commented Aug 8, 2025

Is it thread-safe if writeback is not set to True (by default)?

I have tested four scenarios regarding thread safety issues with the shelve module.

When threading with a shared object with writeback=false, we can consider it not thread-safe.

Scenario 1: Shared Object, writeback=False (FAILS)

import shelve
import threading

db = shelve.open("example.shelve")  # writeback=False by default
for i in range(100):
    db[str(i)] = i**2

def read_shelve(thread_id):
    for i in range(50):
        value = db[str(i)]
        print(f"Thread {thread_id} - {i}: {value}")

t1 = threading.Thread(target=read_shelve, args=(1,))
t2 = threading.Thread(target=read_shelve, args=(2,))
t1.start()
t2.start()
t1.join()
t2.join()

Output:

Exception in thread Thread-1 (read_shelve):
Exception in thread Thread-2 (read_shelve):
Traceback (most recent call last):
  File "/usr/lib/python3.13/shelve.py", line 111, in __getitem__
    value = self.cache[key]
            ~~~~~~~~~~^^^^^
KeyError: '0'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.13/dbm/sqlite3.py", line 79, in _execute
    return closing(self._cx.execute(*args, **kwargs))
                   ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
sqlite3.ProgrammingError: SQLite objects created in a thread can only be used in that same thread. The object was created in thread id 134486782028736 and this is thread id 134486754248384.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
Traceback (most recent call last):
  File "/usr/lib/python3.13/threading.py", line 1043, in _bootstrap_inner
    self.run()
    ~~~~~~~~^^
  File "/usr/lib/python3.13/shelve.py", line 111, in __getitem__
    value = self.cache[key]
            ~~~~~~~~~~^^^^^
  File "/usr/lib/python3.13/threading.py", line 994, in run
    self._target(*self._args, **self._kwargs)
    ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
KeyError: '0'
  File "/home/arf/test_shelve_multi/shelve_thread_shared_object.py", line 11, in read_shelve
    value = db[str(i)]
            ~~^^^^^^^^

During handling of the above exception, another exception occurred:

  File "/usr/lib/python3.13/shelve.py", line 113, in __getitem__
    f = BytesIO(self.dict[key.encode(self.keyencoding)])
                ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Traceback (most recent call last):
  File "/usr/lib/python3.13/dbm/sqlite3.py", line 89, in __getitem__
    with self._execute(LOOKUP_KEY, (key,)) as cu:
         ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.13/dbm/sqlite3.py", line 79, in _execute
    return closing(self._cx.execute(*args, **kwargs))
                   ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.13/dbm/sqlite3.py", line 81, in _execute
    raise error(str(exc))
sqlite3.ProgrammingError: SQLite objects created in a thread can only be used in that same thread. The object was created in thread id 134486782028736 and this is thread id 134486762641088.
dbm.sqlite3.error: SQLite objects created in a thread can only be used in that same thread. The object was created in thread id 134486782028736 and this is thread id 134486754248384.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.13/threading.py", line 1043, in _bootstrap_inner
    self.run()
    ~~~~~~~~^^
  File "/usr/lib/python3.13/threading.py", line 994, in run
    self._target(*self._args, **self._kwargs)
    ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/arf/test_shelve_multi/shelve_thread_shared_object.py", line 11, in read_shelve
    value = db[str(i)]
            ~~^^^^^^^^
  File "/usr/lib/python3.13/shelve.py", line 113, in __getitem__
    f = BytesIO(self.dict[key.encode(self.keyencoding)])
                ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.13/dbm/sqlite3.py", line 89, in __getitem__
    with self._execute(LOOKUP_KEY, (key,)) as cu:
         ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.13/dbm/sqlite3.py", line 81, in _execute
    raise error(str(exc))
dbm.sqlite3.error: SQLite objects created in a thread can only be used in that same thread. The object was created in thread id 134486782028736 and this is thread id 134486762641088.

Scenario 2: Shared Object, writeback=True (WORKS)

import shelve
import threading

db = shelve.open("example.shelve", writeback=True)
for i in range(100):
    db[str(i)] = i**2


def read_shelve(thread_id):
    for i in range(50):
        value = db[str(i)]
        print(f"Thread {thread_id} - {i}: {value}")


t1 = threading.Thread(target=read_shelve, args=(1,))
t2 = threading.Thread(target=read_shelve, args=(2,))

t1.start()
t2.start()
t1.join()
t2.join()

Scenario 3: Separate Objects Per Thread writeback=True (WORKS)

import shelve
import threading

db = shelve.open("example.shelve", writeback=True)
for i in range(100):
    db[str(i)] = i**2


def read_shelve(thread_id):
    for i in range(50):
        value = db[str(i)]
        print(f"Thread {thread_id} - {i}: {value}")


t1 = threading.Thread(target=read_shelve, args=(1,))
t2 = threading.Thread(target=read_shelve, args=(2,))

t1.start()
t2.start()
t1.join()
t2.join()

Scenario 4: Separate Objects Per Thread writeback=False (WORKS)

import shelve
import threading

db = shelve.open("example.shelve")   # writeback=False by default
for i in range(100):
    db[str(i)] = i**2


def read_shelve(thread_id):
    for i in range(50):
        value = db[str(i)]
        print(f"Thread {thread_id} - {i}: {value}")


t1 = threading.Thread(target=read_shelve, args=(1,))
t2 = threading.Thread(target=read_shelve, args=(2,))

t1.start()
t2.start()
t1.join()
t2.join()

Output of successfully executed scenarios:

Thread 1 - 0: 0
Thread 1 - 1: 1
Thread 1 - 2: 4
Thread 1 - 3: 9
Thread 1 - 4: 16
Thread 1 - 5: 25
Thread 1 - 6: 36
Thread 2 - 0: 0
Thread 2 - 1: 1
Thread 1 - 7: 49
Thread 2 - 2: 4
Thread 2 - 3: 9
Thread 2 - 4: 16
Thread 2 - 5: 25
Thread 2 - 6: 36
Thread 2 - 7: 49
Thread 1 - 8: 64
Thread 2 - 8: 64
Thread 1 - 9: 81
Thread 1 - 10: 100
Thread 2 - 9: 81
Thread 2 - 10: 100
Thread 2 - 11: 121
Thread 2 - 12: 144
Thread 2 - 13: 169
Thread 1 - 11: 121
Thread 2 - 14: 196
Thread 1 - 12: 144
Thread 2 - 15: 225
Thread 1 - 13: 169
Thread 1 - 14: 196
Thread 1 - 15: 225
Thread 1 - 16: 256
Thread 1 - 17: 289
Thread 1 - 18: 324
Thread 1 - 19: 361
Thread 2 - 16: 256
Thread 1 - 20: 400
Thread 1 - 21: 441
Thread 1 - 22: 484
Thread 2 - 17: 289
Thread 1 - 23: 529
Thread 1 - 24: 576
Thread 1 - 25: 625
Thread 1 - 26: 676
Thread 1 - 27: 729
Thread 2 - 18: 324
Thread 2 - 19: 361
Thread 2 - 20: 400
Thread 1 - 28: 784
Thread 2 - 21: 441
Thread 1 - 29: 841
Thread 1 - 30: 900
Thread 2 - 22: 484
Thread 1 - 31: 961
Thread 1 - 32: 1024
Thread 2 - 23: 529
Thread 1 - 33: 1089
Thread 1 - 34: 1156
Thread 1 - 35: 1225
Thread 1 - 36: 1296
Thread 2 - 24: 576
Thread 1 - 37: 1369
Thread 2 - 25: 625
Thread 2 - 26: 676
Thread 2 - 27: 729
Thread 1 - 38: 1444
Thread 2 - 28: 784
Thread 1 - 39: 1521
Thread 1 - 40: 1600
Thread 1 - 41: 1681
Thread 2 - 29: 841
Thread 2 - 30: 900
Thread 2 - 31: 961
Thread 2 - 32: 1024
Thread 2 - 33: 1089
Thread 2 - 34: 1156
Thread 2 - 35: 1225
Thread 2 - 36: 1296
Thread 2 - 37: 1369
Thread 2 - 38: 1444
Thread 2 - 39: 1521
Thread 2 - 40: 1600
Thread 2 - 41: 1681
Thread 2 - 42: 1764
Thread 2 - 43: 1849
Thread 2 - 44: 1936
Thread 2 - 45: 2025
Thread 2 - 46: 2116
Thread 2 - 47: 2209
Thread 2 - 48: 2304
Thread 2 - 49: 2401
Thread 1 - 42: 1764
Thread 1 - 43: 1849
Thread 1 - 44: 1936
Thread 1 - 45: 2025
Thread 1 - 46: 2116
Thread 1 - 47: 2209
Thread 1 - 48: 2304
Thread 1 - 49: 2401

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
awaiting review docs Documentation in the Doc dir skip news
Projects
Status: Todo
Development

Successfully merging this pull request may close these issues.

3 participants