Skip to content

gh-107001: Add a stdlib decorator that copies/applies the ParameterSpec from one function to another #121693

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 12 commits into
base: main
Choose a base branch
from

Conversation

CarliJoy
Copy link

@CarliJoy CarliJoy commented Jul 13, 2024

Add a stdlib decorator that copies/applies the ParameterSpec from one function to another.

All information can be found in the related issue

Note for review for JelleZijlstra and AlexWaygood (and maybe the typing council)


📚 Documentation preview 📚: https://cpython-previews--121693.org.readthedocs.build/en/121693/library/typing.html#typing.copy_func_params

@ghost
Copy link

ghost commented Jul 13, 2024

All commit authors signed the Contributor License Agreement.
CLA signed

@bedevere-app
Copy link

bedevere-app bot commented Jul 13, 2024

Most changes to Python require a NEWS entry. Add one using the blurb_it web app or the blurb command-line tool.

If this change has little impact on Python users, wait for a maintainer to apply the skip news label instead.

@JelleZijlstra
Copy link
Member

Since this decorator would need to be handled specially by type checkers, let's not add it to CPython until there is a corresponding change to the typing spec.

@CarliJoy
Copy link
Author

Since this decorator would need to be handled specially by type checkers, let's not add it to CPython until there is a corresponding change to the typing spec.

What are you referring to as "handled specially"?
Do you mean that checking that the signature of the decorated and source function shall be checked "by specification" by the type checkers?
Otherwise it already works with the existing type system without any changes to the type checkers (see linked SO/MyPy Play in related Issue)

I can create a PR for the Spec as well.

Independent of this I still would be interested if the name is okay and if I should include skip_first (see PR description).

@hugovk
Copy link
Member

hugovk commented Jul 13, 2024

Let's convert to draft for now.

@hugovk hugovk marked this pull request as draft July 13, 2024 15:02
@JelleZijlstra
Copy link
Member

Otherwise it already works with the existing type system without any changes to the type checkers (see linked SO/MyPy Play in related Issue)

You're right, I missed this, sorry. If this signature can be expressed in the existing type system, it doesn't need to be specified explicitly.

Copy link
Member

@JelleZijlstra JelleZijlstra left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If this function doesn't need special handling from type checkers, what is the motivation for adding it to typing?

We have a few existing helpers in typing that are not special forms (e.g., assert_never, AnyStr), but I don't know if this proposed addition is quite as pervasively useful as the existing ones.

As an alternative, have you considered contributing this function to https://github.com/hauntsaninja/useful_types/?

Lib/typing.py Outdated
def upstream_func(a: int, b: float, *, double: bool = False) -> float:
...

@copy_kwargs(upstream_func)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The name copy_kwargs implies that it does something to only the kwargs and not the args, but that's not the case; it copies both.

I would use @copy_signature.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi,

sorry haven't had time to work on this in the meanwhile.
I agree that copy_kwargs is not correct but neither is copy_signature. As in my understanding a signature includes also the return value. And we explicitly do not copy these.

After some thinking I came up with:
copy_func_params.

This also opens up the possibility to create a copy_meth_params.
The copy_meth_params is the same as copy_func_params but keeps the self (or cls) argument.

@CarliJoy
Copy link
Author

If this function doesn't need special handling from type checkers, what is the motivation for adding it to typing?

We have a few existing helpers in typing that are not special forms (e.g., assert_never, AnyStr), but I don't know if this proposed addition is quite as pervasively useful as the existing ones.

As an alternative, have you considered contributing this function to https://github.com/hauntsaninja/useful_types/?

The function doesn't currently require type checkers to add any special support. However, introducing it in the typing module could pave the way for future enhancements, where type checkers or static analysis tools might validate that the parameters of two functions actually match. This would allow Python projects to ensure that signature changes—whether in their own code or due to third-party library updates—are caught early, preventing mismatches that could otherwise lead to subtle bugs.

While contributing this function to a separate project, like useful_types, is an option, it has limitations. Projects that suggest vendorization can complicate maintenance, especially in larger codebases. Moreover, this decorator addresses a common enough use case that it warrants inclusion in the standard library, ensuring both accessibility and support across Python versions.

@CarliJoy CarliJoy marked this pull request as ready for review October 25, 2024 16:24
@JelleZijlstra
Copy link
Member

While contributing this function to a separate project, like useful_types, is an option, it has limitations. Projects that suggest vendorization can complicate maintenance, especially in larger codebases. Moreover, this decorator addresses a common enough use case that it warrants inclusion in the standard library, ensuring both accessibility and support across Python versions.

Then I'd suggest you contribute it to useful_types first, and only add it to the standard library after it has actually proven to be widely useful.

Note that anything we add now will only be released with Python 3.14, in October 2025 (though we'd add it to typing-extensions earlier).

@CarliJoy
Copy link
Author

Then I'd suggest you contribute it to useful_types first, and only add it to the standard library after it has actually proven to be widely useful.

Actually, I believe this decorator needs the visibility only the standard library can provide.

The purpose of this PR is to make typing in Python both easier and more robust. Extending functions and methods is common across all experience levels, but creating this kind of decorator is challenging—I found it difficult to arrive at the solution myself. The number of threads on discuss.python.org, along with the Stack Overflow issue referenced in #107001, demonstrates that there’s a real demand and that others are also struggling.

Note that anything we add now will only be released with Python 3.14, in October 2025 (though we'd add it to typing-extensions earlier).

Adding it now means it will already be visible in the Python 3.14 documentation, making it easy to find and use as an example.

I’m not in favor of placing this in a third-party library, as it would limit additional ideas I have for this decorator. For example, as noted in the documentation, using it incorrectly can actually reduce type safety. I’m already considering ways to support tools like ruff in checking that decorated functions correctly include *args and **kwargs.

In the longer term, I’d even like to explore extending the Python type system itself to allow checking if a source function’s call signature is compatible with a decorated function.

Adding this to a third-party library would impede this progress and keep the solution obscure.

I understand every addition has maintenance costs, but the actual code is just six lines returning the function. So what’s the reason for being so reluctant to add it?

In the end, my goal is to contribute to a Python ecosystem with fewer untyped *args and **kwargs that neither I nor my IDE or type checker can fully interpret. I hope that including this decorator is a small but meaningful step in that direction.

@ghost
Copy link

ghost commented Oct 28, 2024

All commit authors signed the Contributor License Agreement.
CLA signed

@CarliJoy
Copy link
Author

CarliJoy commented Jan 7, 2025

@JelleZijlstra Happy New year.
Did you have some time to look at my arguments?

@JelleZijlstra
Copy link
Member

Your statements about "additional ideas" and "extending the Python type system" make me even more reluctant to add this function now. Putting something in the standard library effectively means freezing it, and any future additions or tweaks have to overcome a large compatibility barrier.

And if nobody is willing to contribute this feature to a third-party library like useful-types, that again makes me question how commonly useful the feature is. The standard library isn't the place to incubate new features.

@cyberw
Copy link

cyberw commented Jan 11, 2025

that again makes me question how commonly useful the feature is.

Let me just chime in here. I found this PR while looking for a feature such as this. My project, Locust, subclasses requests.Session and overrides a few methods to add logging/tracking to each HTTP request, sometimes with identical signature, sometimes adding a few additional keyword arguments. Requests has a lot methods (get, post, put, etc) that support the same parameters as a more generic request()-method. Being able to copy the method signatures from that class to further wrappers in subclasses is very convenient (and I've been able to successfully use a copy of the code in this PR).

Most importantly, I think there is overwhelming evidence out there that a feature such as this is frequently sought, with no really good answers:
https://stackoverflow.com/questions/59717828/copy-type-signature-from-another-function
https://stackoverflow.com/questions/65954587/copying-function-signature-in-a-method
https://stackoverflow.com/questions/42420810/copy-signature-forward-all-arguments-from-wrapper-function
https://stackoverflow.com/questions/35512112/how-can-i-copy-a-python-functions-signature-and-define-a-new-function
https://www.reddit.com/r/learnpython/comments/sksy85/copy_type_signature_of_a_parameter_of_another/
https://discuss.python.org/t/copying-signature-of-init-to-a-mixin-class-method/58907/4

(that being said, personally I would have no problem using this feature just because it wasn't in the stdlib, I just want it somewhere :)

@CarliJoy
Copy link
Author

CarliJoy commented Jan 13, 2025

Thanks for chiming in, cyberw. I didn't find all the results you were looking for even so I looked quite intensively in the past.

The sole reason for this MR is to make the already existing way to copy function parameters from one function to another known and easily usable.

This PR achieves both goals.

As documented, there is a possible issue with this decorator: if the copied call signature is incompatible with the decorated function's call signature, it can lead to a Runtime TypeError that type checkers won’t catch 💣. Essentially, the decorated function’s signature is overwritten. [Details]

My "additional ideas" about extending the Python type system aim to assist user to prevent this.

"Putting something in the standard library effectively means freezing it, and any future additions or tweaks have to overcome a large compatibility barrier."

I did not propose changing the stdlib implementation in future here. Instead, I have two ideas to address the issue above,:

  1. Short-term: Implement a Ruff rule to ensure all decorated functions use *args and **kwargs, reducing the risk of runtime type errors (as noted in the documentation). The stdlib "frozen" state is desired here.
  2. Long-term/Maybe: Advocate for a change to the typing specification, enabling type checkers to validate the compatibility of the copied call signature with the decorated function.

Including this in useful-types wouldn’t make it easier to find or use¹. I’d rather keep it on StackOverflow, where users can also learn about the potential pitfalls..

Never the less: Both options make it equally hard to create a Ruff rule to help users avoid the issue.

¹ useful-types does not have a documentation and is rather unknown.

@last-partizan
Copy link

I haven't heard about useful-types, but I deeply care about typing my python code, and that's not the first time I'm looking for such function.

I copied it into my app/utils/typing.py, and it works fine.

Would it be too stupid to put this into typeshed as functools.wraps? It is copying signature...

In [1]: from functools import wraps

In [2]: def f1(a: str, b: str):
   ...:     """Hello from f1"""
   ...:

In [3]: @wraps(f1)
   ...: def f2(*args, **kwargs):
   ...:     return f1(*args, **kwargs)
   ...:

In [4]: f2?
Signature: f2(a: str, b: str)
Docstring: Hello from f1
File:      ~/<ipython-input-2-8ca42cdf805a>
Type:      function

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants