Version dependence in the test suite #2069
Description
While working on #2068 I ran the conformance test suite under Python 3.14 and got a number of changes:
diff --git a/conformance/results/mypy/classes_classvar.toml b/conformance/results/mypy/classes_classvar.toml
index 0072d63..ce30b65 100644
--- a/conformance/results/mypy/classes_classvar.toml
+++ b/conformance/results/mypy/classes_classvar.toml
@@ -12,7 +12,6 @@ classes_classvar.py:40: error: Name "var" is not defined [name-defined]
classes_classvar.py:45: error: ClassVar cannot contain type variables [misc]
classes_classvar.py:46: error: ClassVar cannot contain type variables [misc]
classes_classvar.py:52: error: Incompatible types in assignment (expression has type "dict[Never, Never]", variable has type "list[str]") [assignment]
-classes_classvar.py:54: error: Variable should not be annotated with both ClassVar and Final [misc]
classes_classvar.py:55: error: Invalid type: ClassVar nested inside other type [valid-type]
classes_classvar.py:67: error: Invalid type: ClassVar nested inside other type [valid-type]
classes_classvar.py:69: error: ClassVar can only be used for assignments in class body [misc]
@@ -31,5 +30,6 @@ classes_classvar.py:140: note: Protocol member ProtoA.y expected class variable,
conformance_automated = "Fail"
errors_diff = """
Line 47: Expected 1 errors
+Line 54: Expected 1 errors
Line 67: Unexpected errors ['classes_classvar.py:67: error: Invalid type: ClassVar nested inside other type [valid-type]']
"""
diff --git a/conformance/results/mypy/dataclasses_final.toml b/conformance/results/mypy/dataclasses_final.toml
index fdf8691..eb91705 100644
--- a/conformance/results/mypy/dataclasses_final.toml
+++ b/conformance/results/mypy/dataclasses_final.toml
@@ -7,12 +7,10 @@ conformance_automated = "Fail"
errors_diff = """
Line 27: Expected 1 errors
Line 16: Unexpected errors ['dataclasses_final.py:16: error: Final name must be initialized with a value [misc]']
-Line 18: Unexpected errors ['dataclasses_final.py:18: error: Final can be only used as an outermost qualifier in a variable annotation [valid-type]']
Line 24: Unexpected errors ['dataclasses_final.py:24: error: Expression is of type "Any", not "int" [assert-type]']
"""
output = """
dataclasses_final.py:16: error: Final name must be initialized with a value [misc]
-dataclasses_final.py:18: error: Final can be only used as an outermost qualifier in a variable annotation [valid-type]
dataclasses_final.py:24: error: Expression is of type "Any", not "int" [assert-type]
dataclasses_final.py:35: error: Cannot assign to final attribute "final_no_default" [misc]
dataclasses_final.py:36: error: Cannot assign to final attribute "final_with_default" [misc]
diff --git a/conformance/results/mypy/generics_variance_inference.toml b/conformance/results/mypy/generics_variance_inference.toml
index b1158c7..ef877e1 100644
--- a/conformance/results/mypy/generics_variance_inference.toml
+++ b/conformance/results/mypy/generics_variance_inference.toml
@@ -6,6 +6,7 @@ generics_variance_inference.py:28: error: Incompatible types in assignment (expr
generics_variance_inference.py:41: error: Incompatible types in assignment (expression has type "ShouldBeCovariant1[float]", variable has type "ShouldBeCovariant1[int]") [assignment]
generics_variance_inference.py:49: error: Incompatible types in assignment (expression has type "ShouldBeCovariant2[float]", variable has type "ShouldBeCovariant2[int]") [assignment]
generics_variance_inference.py:58: error: Incompatible types in assignment (expression has type "ShouldBeCovariant3[float]", variable has type "ShouldBeCovariant3[int]") [assignment]
+generics_variance_inference.py:66: error: Incompatible types in assignment (expression has type "ShouldBeCovariant4[int]", variable has type "ShouldBeCovariant4[float]") [assignment]
generics_variance_inference.py:67: error: Incompatible types in assignment (expression has type "ShouldBeCovariant4[float]", variable has type "ShouldBeCovariant4[int]") [assignment]
generics_variance_inference.py:80: error: Incompatible types in assignment (expression has type "ShouldBeCovariant5[float]", variable has type "ShouldBeCovariant5[int]") [assignment]
generics_variance_inference.py:96: error: Incompatible types in assignment (expression has type "ShouldBeInvariant1[int]", variable has type "ShouldBeInvariant1[float]") [assignment]
@@ -24,6 +25,7 @@ generics_variance_inference.py:170: error: Incompatible types in assignment (exp
generics_variance_inference.py:181: error: Incompatible types in assignment (expression has type "ShouldBeCovariant6[float]", variable has type "ShouldBeCovariant6[int]") [assignment]
generics_variance_inference.py:194: error: Incompatible types in assignment (expression has type "ShouldBeContravariant2[int]", variable has type "ShouldBeContravariant2[float]") [assignment]
"""
-conformance_automated = "Pass"
+conformance_automated = "Fail"
errors_diff = """
+Line 66: Unexpected errors ['generics_variance_inference.py:66: error: Incompatible types in assignment (expression has type "ShouldBeCovariant4[int]", variable has type "ShouldBeCovariant4[float]") [assignment]']
"""
diff --git a/conformance/results/mypy/qualifiers_final_annotation.toml b/conformance/results/mypy/qualifiers_final_annotation.toml
index 2d0cea3..8c0830f 100644
--- a/conformance/results/mypy/qualifiers_final_annotation.toml
+++ b/conformance/results/mypy/qualifiers_final_annotation.toml
@@ -20,8 +20,6 @@ qualifiers_final_annotation.py:71: error: Cannot assign to final name "RATE" [m
qualifiers_final_annotation.py:81: error: Cannot assign to final attribute "DEFAULT_ID" [misc]
qualifiers_final_annotation.py:94: error: Cannot assign to final name "BORDER_WIDTH" [misc]
qualifiers_final_annotation.py:96: error: Cannot assign to final name "__private" [misc]
-qualifiers_final_annotation.py:107: error: Final can be only used as an outermost qualifier in a variable annotation [valid-type]
-qualifiers_final_annotation.py:108: error: Variable should not be annotated with both ClassVar and Final [misc]
qualifiers_final_annotation.py:118: error: Final can be only used as an outermost qualifier in a variable annotation [valid-type]
qualifiers_final_annotation.py:121: error: Final can be only used as an outermost qualifier in a variable annotation [valid-type]
qualifiers_final_annotation.py:131: error: Invalid "NamedTuple()" field name [misc]
@@ -40,6 +38,8 @@ qualifiers_final_annotation.py:170: error: Cannot assign to final name "PI" [mi
"""
conformance_automated = "Fail"
errors_diff = """
+Line 107: Expected 1 errors
+Line 108: Expected 1 errors
Line 149: Expected 1 errors
Line 59: Unexpected errors ['qualifiers_final_annotation.py:59: error: Cannot assign to final attribute "ID6" [misc]']
Line 96: Unexpected errors ['qualifiers_final_annotation.py:96: error: Cannot assign to final name "__private" [misc]']
diff --git a/conformance/results/pyright/annotations_forward_refs.toml b/conformance/results/pyright/annotations_forward_refs.toml
index ca35b5b..b91c665 100644
--- a/conformance/results/pyright/annotations_forward_refs.toml
+++ b/conformance/results/pyright/annotations_forward_refs.toml
@@ -1,7 +1,5 @@
conformant = "Pass"
output = """
-annotations_forward_refs.py:22:7 - error: "ClassA" is not defined (reportUndefinedVariable)
-annotations_forward_refs.py:23:12 - error: "ClassA" is not defined (reportUndefinedVariable)
annotations_forward_refs.py:24:7 - error: Union syntax cannot be used with string operand; use quotes around entire expression (reportGeneralTypeIssues)
annotations_forward_refs.py:25:13 - error: Union syntax cannot be used with string operand; use quotes around entire expression (reportGeneralTypeIssues)
annotations_forward_refs.py:41:10 - error: Call expression not allowed in type expression (reportInvalidTypeForm)
@@ -29,11 +27,13 @@ annotations_forward_refs.py:52:11 - error: Unary operator not allowed in type ex
annotations_forward_refs.py:53:11 - error: Binary operator not allowed in type expression (reportInvalidTypeForm)
annotations_forward_refs.py:54:11 - error: Type expressions cannot use format string literals (f-strings) (reportGeneralTypeIssues)
annotations_forward_refs.py:55:10 - error: Module cannot be used as a type (reportGeneralTypeIssues)
-annotations_forward_refs.py:66:26 - error: "ClassB" is not defined (reportUndefinedVariable)
annotations_forward_refs.py:80:14 - error: Type of "ClassF" could not be determined because it refers to itself (reportGeneralTypeIssues)
annotations_forward_refs.py:80:14 - error: Variable not allowed in type expression (reportInvalidTypeForm)
-annotations_forward_refs.py:89:8 - error: Expected class but received "(self: Self@ClassD) -> None" (reportGeneralTypeIssues)
"""
-conformance_automated = "Pass"
+conformance_automated = "Fail"
errors_diff = """
+Line 22: Expected 1 errors
+Line 23: Expected 1 errors
+Line 66: Expected 1 errors
+Line 89: Expected 1 errors
"""
diff --git a/conformance/results/pyright/generics_variance_inference.toml b/conformance/results/pyright/generics_variance_inference.toml
index 9a378ad..5c0f32e 100644
--- a/conformance/results/pyright/generics_variance_inference.toml
+++ b/conformance/results/pyright/generics_variance_inference.toml
@@ -23,10 +23,12 @@ generics_variance_inference.py:58:35 - error: Type "ShouldBeCovariant3[float]" i
"ShouldBeCovariant3[float]" is not assignable to "ShouldBeCovariant3[int]"
Type parameter "T@ShouldBeCovariant3" is covariant, but "float" is not a subtype of "int"
"float" is not assignable to "int" (reportAssignmentType)
+generics_variance_inference.py:66:36 - error: Type "ShouldBeCovariant4[int]" is not assignable to declared type "ShouldBeCovariant4[float]"
+ "ShouldBeCovariant4[int]" is not assignable to "ShouldBeCovariant4[float]"
+ Type parameter "T@ShouldBeCovariant4" is invariant, but "int" is not the same as "float" (reportAssignmentType)
generics_variance_inference.py:67:34 - error: Type "ShouldBeCovariant4[float]" is not assignable to declared type "ShouldBeCovariant4[int]"
"ShouldBeCovariant4[float]" is not assignable to "ShouldBeCovariant4[int]"
- Type parameter "T@ShouldBeCovariant4" is covariant, but "float" is not a subtype of "int"
- "float" is not assignable to "int" (reportAssignmentType)
+ Type parameter "T@ShouldBeCovariant4" is invariant, but "float" is not the same as "int" (reportAssignmentType)
generics_variance_inference.py:80:34 - error: Type "ShouldBeCovariant5[float]" is not assignable to declared type "ShouldBeCovariant5[int]"
"ShouldBeCovariant5[float]" is not assignable to "ShouldBeCovariant5[int]"
Type parameter "T@ShouldBeCovariant5" is covariant, but "float" is not a subtype of "int"
@@ -80,6 +82,7 @@ generics_variance_inference.py:194:37 - error: Type "ShouldBeContravariant2[int]
Type parameter "T@ShouldBeContravariant2" is contravariant, but "int" is not a supertype of "float"
"float" is not assignable to "int" (reportAssignmentType)
"""
-conformance_automated = "Pass"
+conformance_automated = "Fail"
errors_diff = """
+Line 66: Unexpected errors ['generics_variance_inference.py:66:36 - error: Type "ShouldBeCovariant4[int]" is not assignable to declared type "ShouldBeCovariant4[float]"']
"""Analysis:
- In
classes_classvar.py, mypy allowsFinal[ClassVar[int]]only in 3.13+. This is in line with the runtime behavior, as this construct throws a runtime error in 3.12 and earlier. Interestingly, mypy also disallows this whenfrom __future__ import annotationsis used (so there wouldn't be a runtime error). Mypy seems to be in the wrong here according to the spec: we still explicitly disallow Final + ClassVar except in dataclasses, even if it works at runtime. qualifiers_final_annotation.pyalso has mypy allowing a Final/ClassVar it shouldn't.dataclasses_final.pyis a similar issue with mypy, but here it's a positive; it correctly allows Final + ClassVar in a dataclass.- In
generics_variance_inference.py, both mypy and pyright now infer a generic frozen dataclass as invariant. This is due to https://discuss.python.org/t/make-replace-stop-interfering-with-variance-inference/96092 . - In
annotations_forward_refs.py, pyright changes its behavior regarding names in annotations. This is mostly correct and in line with PEP 649, but one of the new errors (the last line) seems wrong to me.
Regardless of the specifics of these changes, the general lesson is that the behavior of the test suite depends on the Python version, and we need to make sure we use a consistent Python version to run the test suite (presumably, the most recent stable version, so currently 3.13). Optionally, we could also extend the framework to support checking some tests under a more specific Python version, so we can test version-dependent behavior.