Handling optional attributes in SQLAlchemy Declarative Mapping abstract classes with dataclass #10027
-
DescriptionI'm currently trying to implement SQLAlchemy's Declarative Mapping with Pydantic's dataclass with SQLAlchemy 2.0 in an Async context. However, I encountered a use case that I'm unable to solve with my existing ObjectiveThe workflow that I'm trying to achieve is as follows:
The challenge lies in handling the optional attribute ProblemThe current implementation does not handle
However, what I expect to see is: QuestionIs there an alternative approach to Environment and SetupThe issue can be reproduced in the following environment:
The relevant model and API are as follows: My model: class Base(
MappedAsDataclass,
DeclarativeBase,
dataclass_callable=pydantic.dataclasses.dataclass,
):
class Config:
orm_mode = True
class BInput(Base):
__abstract__ = True
name: Mapped[str] = mapped_column(String(50))
attribute1: Mapped[bool] = mapped_column(Boolean(), default=False)
class BOutput(BInput):
__tablename__ = "b"
attribute2: Mapped[int] = mapped_column(Integer(), default=100)
computed_info_1: int = 0
computed_info_2: str = ""My API: @router.post("", response_model=BOutput, status_code=201)
async def add(*, db: AsyncSession = Depends(get_db), b_input: BInput):
b_output = BOutput(**asdict(b_input))
b_output.computed_info_2 = "Some custom logic too add here"
db.add(b_output)
await db.commit()
return b_outputTest CaseI've also provided a test case to help reproduce the issue: @pytest.mark.asyncio
async def test_create_b(db_session: AsyncSession, client: AsyncClient):
response = await client.post(
"/v1/b",
json={
"name": "Test B 1",
},
)
assert response.status_code == 201
pack = response.json()
print(pack)
assert pack["name"] == "Test B 1"
assert pack["attribute1"] == False
assert pack["attribute2"] == 100ErrorThe error that occurs during execution is as follows: Python stack tracebackIf required, I can provide a minimal reproducible example which will be quite similar to the one I've provided here. Looking forward to any insights and guidance that might help me resolve this issue. Update: I've clarified and updated the original issue to ease troubleshooting |
Beta Was this translation helpful? Give feedback.
Replies: 4 comments 19 replies
-
|
Hi, I'm not sure what's at fault. Trying to create a runnable example from your snipped does not seem to result in any error. Try to modify it to make it return an error EDIT: updated script to reproduce. This happens also with plain dataclasses from sqlalchemy.orm import (
DeclarativeBase,
MappedAsDataclass,
Mapped,
mapped_column,
)
from sqlalchemy import Boolean, Integer, String
import dataclasses as DC
import pydantic
class Base(
MappedAsDataclass,
DeclarativeBase,
# dataclass_callable=pydantic.dataclasses.dataclass,
):
class Config:
orm_mode = True
class BInput(Base):
__abstract__ = True
id: Mapped[int] = mapped_column(Integer, primary_key=True, init=False)
name: Mapped[str] = mapped_column(String(50))
attribute1: Mapped[bool] = mapped_column(Boolean(), default=False)
class BOutput(BInput):
__tablename__ = "b"
attribute2: Mapped[int] = mapped_column(Integer(), default=100)
computed_info_1: int = 0
computed_info_2: str = ""
b_input = BInput(name="foo")
print(DC.asdict(b_input))
b_output = BOutput(**DC.asdict(b_input))
b_output.attribute2 = 42
print(DC.asdict(b_output)) |
Beta Was this translation helpful? Give feedback.
-
|
oh this may be caused by how dataclasses sets default values here, since they are placed on the class dict. But when a class is mapped by sqlalchemy those defaults are removed entirely, and you have a descriptor there. I'll try using |
Beta Was this translation helpful? Give feedback.
-
|
This seems indeed a bug. b_input = BInput(name="foo")
print(b_input.id, b_input.attribute1)
print(b_input.__dict__)The issue here seems duet to |
Beta Was this translation helpful? Give feedback.
-
|
it's not valid to instantiate BInput, so that would not be a "bug". I get a stack trace here for some reason also $ python test3.py
Traceback (most recent call last):
File "/mnt/photon_home/classic/dev/sqlalchemy/test3.py", line 35, in <module>
print(DC.asdict(b_input))
^^^^^^^^^^^^^^^^^^
File "/usr/lib64/python3.11/dataclasses.py", line 1277, in asdict
return _asdict_inner(obj, dict_factory)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib64/python3.11/dataclasses.py", line 1284, in _asdict_inner
value = _asdict_inner(getattr(obj, f.name), dict_factory)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib64/python3.11/dataclasses.py", line 1318, in _asdict_inner
return copy.deepcopy(obj)
^^^^^^^^^^^^^^^^^^
File "/usr/lib64/python3.11/copy.py", line 172, in deepcopy
y = _reconstruct(x, memo, *rv)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib64/python3.11/copy.py", line 271, in _reconstruct
state = deepcopy(state, memo)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib64/python3.11/copy.py", line 146, in deepcopy
y = copier(x, memo)
^^^^^^^^^^^^^^^
File "/usr/lib64/python3.11/copy.py", line 211, in _deepcopy_tuple
y = [deepcopy(a, memo) for a in x]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib64/python3.11/copy.py", line 211, in <listcomp>
y = [deepcopy(a, memo) for a in x]
^^^^^^^^^^^^^^^^^
File "/usr/lib64/python3.11/copy.py", line 146, in deepcopy
y = copier(x, memo)
^^^^^^^^^^^^^^^
File "/usr/lib64/python3.11/copy.py", line 231, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib64/python3.11/copy.py", line 172, in deepcopy
y = _reconstruct(x, memo, *rv)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib64/python3.11/copy.py", line 271, in _reconstruct
state = deepcopy(state, memo)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib64/python3.11/copy.py", line 146, in deepcopy
y = copier(x, memo)
^^^^^^^^^^^^^^^
File "/usr/lib64/python3.11/copy.py", line 231, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib64/python3.11/copy.py", line 172, in deepcopy
y = _reconstruct(x, memo, *rv)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib64/python3.11/copy.py", line 271, in _reconstruct
state = deepcopy(state, memo)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib64/python3.11/copy.py", line 146, in deepcopy
y = copier(x, memo)
^^^^^^^^^^^^^^^
File "/usr/lib64/python3.11/copy.py", line 231, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib64/python3.11/copy.py", line 172, in deepcopy
y = _reconstruct(x, memo, *rv)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib64/python3.11/copy.py", line 271, in _reconstruct
state = deepcopy(state, memo)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib64/python3.11/copy.py", line 146, in deepcopy
y = copier(x, memo)
^^^^^^^^^^^^^^^
File "/usr/lib64/python3.11/copy.py", line 231, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib64/python3.11/copy.py", line 172, in deepcopy
y = _reconstruct(x, memo, *rv)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib64/python3.11/copy.py", line 271, in _reconstruct
state = deepcopy(state, memo)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib64/python3.11/copy.py", line 146, in deepcopy
y = copier(x, memo)
^^^^^^^^^^^^^^^
File "/usr/lib64/python3.11/copy.py", line 231, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib64/python3.11/copy.py", line 172, in deepcopy
y = _reconstruct(x, memo, *rv)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib64/python3.11/copy.py", line 271, in _reconstruct
state = deepcopy(state, memo)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib64/python3.11/copy.py", line 146, in deepcopy
y = copier(x, memo)
^^^^^^^^^^^^^^^
File "/usr/lib64/python3.11/copy.py", line 211, in _deepcopy_tuple
y = [deepcopy(a, memo) for a in x]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib64/python3.11/copy.py", line 211, in <listcomp>
y = [deepcopy(a, memo) for a in x]
^^^^^^^^^^^^^^^^^
File "/usr/lib64/python3.11/copy.py", line 146, in deepcopy
y = copier(x, memo)
^^^^^^^^^^^^^^^
File "/usr/lib64/python3.11/copy.py", line 231, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib64/python3.11/copy.py", line 161, in deepcopy
rv = reductor(4)
^^^^^^^^^^^
File "/mnt/photon_home/classic/dev/sqlalchemy/lib/sqlalchemy/util/langhelpers.py", line 1311, in __getattr__
value = getattr(self, f"_memoized_attr_{key}")()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/mnt/photon_home/classic/dev/sqlalchemy/lib/sqlalchemy/event/attr.py", line 81, in _memoized_attr_ref
return weakref.ref(self, registry._collection_gced)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: cannot create weak reference to '_JoinedListener' objectthis is weird? |
Beta Was this translation helpful? Give feedback.
-
|
asdict is trying to copy a mapped columns and failing |
Beta Was this translation helpful? Give feedback.
-
|
Got the same error |
Beta Was this translation helpful? Give feedback.
-
There's no ORM constructor or ORM-handling of these attributes on the abstract class because it's not valid to instantiate an abstract class: https://en.wikipedia.org/wiki/Abstract_type
SQLAlchemy isn't adding special steps here to prevent the instantiation but I guess we should. |
Beta Was this translation helpful? Give feedback.
-
|
Sorry if that was not clear enough. The issue is that I can't write a SQLAlchemy class without specifying tablename. So the only (bad) solution is to specify an abstract class, which is not sutaible either. What I wanted to do is to have What would have been so cool is to be able to write Declerative Mapping style class without specifying tablename (so with no table generation). Maybe I'm over-engineering too much for nothing and need to work to improve SQLModel compability to do such a thing. |
Beta Was this translation helpful? Give feedback.
-
That's possible. The issue here is that you are trying to use an abstract class from sqlalchemy point of view as your types. If you were to use
You may be able to fiddle around with the dataclass constructor function, by wrapping the pydantic one in your own to present a different class state to the real dataclass function. Using default_factory instead of default may also improve things |
Beta Was this translation helpful? Give feedback.
-
|
On second thoughts, abstract class is a false debate. The issue is still the same if I write this code: from sqlalchemy.orm import Mapped, MappedAsDataclass, mapped_column
class Base(
MappedAsDataclass,
DeclarativeBase,
# dataclass_callable=pydantic.dataclasses.dataclass,
):
class Config:
orm_mode = True
class BInput(MappedAsDataclass):
id: Mapped[int] = mapped_column(Integer, primary_key=True, init=False)
name: Mapped[str] = mapped_column(String(50))
attribute1: Mapped[bool] = mapped_column(Boolean(), default=False)
class BOutput(Base, BInput):
__tablename__ = "b"
attribute2: Mapped[int] = mapped_column(Integer(), default=100)
computed_info_1: int = 0
computed_info_2: str = ""
b_input = BInput(name="foo", attribute1=False)
print(DC.asdict(b_input))
b_output = BOutput(**DC.asdict(b_input))
b_output.attribute2 = 42
print(DC.asdict(b_output))In this case, I have a bug on IMO, at least there is something to change in exception sent. Anyway, thanks for your support. Really appreciated for your patience and SQLAlchemy is still a gem in the python echosystem ;) PS @CaselIT: Still trying to apply your advices but seems taht |
Beta Was this translation helpful? Give feedback.
-
|
sure, that's a mixin, you can "instantiate" that but SQLAlchemy does not map it, so the mapped_column() attributes are not removed and there is no solution to that problem here other than setting up init=False on the class. @CaselIT didn't we have some issue where we were going to force init=False for some subset of dataclasses ? |
Beta Was this translation helpful? Give feedback.
-
I remember that point, but maybe it was this issue here. IIRC the idea was that if a class is abstract we were to wrap init to warn on instantiation and/or force init=False.
class Id(MappedAsDataclass):
id: Mapped[int] = mapped_column(Integer, primary_key=True, init=False)
class BInput(Base):
__abstract__ = True
name: Mapped[str] = mapped_column(String(50))
attribute1: Mapped[bool] = mapped_column(Boolean(), default_factory=lambda: False)
class BOutput(BInput, Id):
__tablename__ = "b"
attribute2: Mapped[int] = mapped_column(Integer(), default=100)
computed_info_1: int = 0
computed_info_2: str = ""works in the example below. that said the fact that it's working is incidental, and probably not sorething sqlalchemy should commit to support |
Beta Was this translation helpful? Give feedback.
-
A SQLAlchemy class is mapped to a database table, always, either directly or via inheritance. that's their purpose. if you want to instantiate classes that are not mapped to database tables, then you would not make that class a SQLAlchemy mapped class. So make a separate class that does the business case you need and don't map it (that is, does not extend from the declarative base at all). A mixin class satisfies this need as well.
right but you are putting mapped_column() directives on that class, which doesnt make sense if you want to instantiate that class and have it not be associated with any database table. pydantic has some kind of "two class approach" to this sort of thing, apparently.
a mixin or abstract class can have this, but they are not mapped, so instantiation makes no sense. instantiation means the class is managed from a persistence perspective and requires a Mapper. The Mapper does not have a "exist, but we are not really mapped to any table" mode - that would be extremely complicated without there being a real use case. the mapped_column() directives are not actual instance-level constructs, when the class is mapped, these get replaced with Python descriptors that do the real work for instances.
what seems awkward here is that you are going for "classes that aren't mapped", which is a standard "high degree of separation of concerns" approach, that's OK, but then you want that same class to be used in a mapped hierarchy, which seems to defeat the whole purpose of separation of concerns. you might want to look into imperative mappings, either declarative with imperative table, or full imperative, if you are going for "the datamodel classes and how they are persisted in a database should be separate concerns" model. |
Beta Was this translation helpful? Give feedback.
-
|
added #10064. Instantiation of any MappedAsDataclass class that is not mapped is not supported by SQLAlchemy, and behavior is currently undefined. |
Beta Was this translation helpful? Give feedback.
added #10064. Instantiation of any MappedAsDataclass class that is not mapped is not supported by SQLAlchemy, and behavior is currently undefined.