Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Test failure with Python 3.13.0b1 #533

Closed
frenzymadness opened this issue May 13, 2024 · 1 comment · Fixed by #534
Closed

Test failure with Python 3.13.0b1 #533

frenzymadness opened this issue May 13, 2024 · 1 comment · Fixed by #534

Comments

@frenzymadness
Copy link
Contributor

This assertion fails:

assert list(clsdict.keys()) == ["C_CONSTANT", "__doc__", "method_c"]

with the following error:

=================================== FAILURES ===================================
___________________________ test_extract_class_dict ____________________________

    def test_extract_class_dict():
        class A(int):
            """A docstring"""
    
            def method(self):
                return "a"
    
        class B:
            """B docstring"""
    
            B_CONSTANT = 42
    
            def method(self):
                return "b"
    
        class C(A, B):
            C_CONSTANT = 43
    
            def method_c(self):
                return "c"
    
        clsdict = _extract_class_dict(C)
>       assert sorted(clsdict.keys()) == ["C_CONSTANT", "__doc__", "method_c"]
E       AssertionError: assert ['C_CONSTANT'...', 'method_c'] == ['C_CONSTANT'...', 'method_c']
E         At index 2 diff: '__firstlineno__' != 'method_c'
E         Left contains one more item: 'method_c'
E         Full diff:
E         - ['C_CONSTANT', '__doc__', 'method_c']
E         + ['C_CONSTANT', '__doc__', '__firstlineno__', 'method_c']
E         ?                          +++++++++++++++++++

tests/cloudpickle_test.py:111: AssertionError
=========================== short test summary info ============================
FAILED tests/cloudpickle_test.py::test_extract_class_dict - AssertionError: a...
============ 1 failed, 232 passed, 25 skipped, 2 warnings in 11.75s ============

This change in Python seems to be related: python/cpython#118475

@frenzymadness
Copy link
Contributor Author

Hmm, we have the test failing with Python 3.13 beta 1 and cloudpickle 3.0.0 and it's true that this problem is fixed by the proposed PR but when I run all the tests with cloudpickle from the main branch, I'm getting more errors:

========================================= FAILURES ==========================================
____ CloudPickleTest.test_deterministic_dynamic_class_attr_ordering_for_chained_pickling ____

self = <tests.cloudpickle_test.CloudPickleTest testMethod=test_deterministic_dynamic_class_attr_ordering_for_chained_pickling>

    def test_deterministic_dynamic_class_attr_ordering_for_chained_pickling(self):
        # Check that the pickle produced by pickling a reconstructed class definition
        # in a remote process matches the pickle produced by pickling the original
        # class definition.
        # In particular, this test checks that the order of the class attributes is
        # deterministic.
    
        with subprocess_worker(protocol=self.protocol) as w:
    
            class A:
                """Simple class definition"""
    
                pass
    
            A_dump = w.run(cloudpickle.dumps, A)
            check_deterministic_pickle(A_dump, cloudpickle.dumps(A))
    
            # If the `__doc__` attribute is defined after some other class
            # attribute, this can cause class attribute ordering changes due to
            # the way we reconstruct the class definition in
            # `_make_class_skeleton`, which creates the class and thus its
            # `__doc__` attribute before populating the class attributes.
            class A:
                name = "A"
                __doc__ = "Updated class definition"
    
            A_dump = w.run(cloudpickle.dumps, A)
            check_deterministic_pickle(A_dump, cloudpickle.dumps(A))
    
            # If a `__doc__` is defined on the `__init__` method, this can
            # cause ordering changes due to the way we reconstruct the class
            # with `_make_class_skeleton`.
            class A:
                def __init__(self):
                    """Class definition with explicit __init__"""
                    pass
    
            A_dump = w.run(cloudpickle.dumps, A)
>           check_deterministic_pickle(A_dump, cloudpickle.dumps(A))

A          = <class 'tests.cloudpickle_test.CloudPickleTest.test_deterministic_dynamic_class_attr_ordering_for_chained_pickling.<locals>.A'>
A_dump     = b"\x80\x05\x95\x14\x04\x00\x00\x00\x00\x00\x00\x8c\x17cloudpickle.cloudpickle\x94\x8c\x14_make_skeleton_class\x94\x93\...x0b__globals__\x94}\x94u\x86\x94\x86R0\x8c\n__module__\x94h\x0c\x8c\x15__static_attributes__\x94)u}\x94\x86\x94\x86R0."
self       = <tests.cloudpickle_test.CloudPickleTest testMethod=test_deterministic_dynamic_class_attr_ordering_for_chained_pickling>
w          = <tests.testutils._Worker object at 0x7f46db632510>

tests/cloudpickle_test.py:2090: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

a = b"\x80\x05\x95\x14\x04\x00\x00\x00\x00\x00\x00\x8c\x17cloudpickle.cloudpickle\x94\x8c\x14_make_skeleton_class\x94\x93\...x0b__globals__\x94}\x94u\x86\x94\x86R0\x8c\n__module__\x94h\x0c\x8c\x15__static_attributes__\x94)u}\x94\x86\x94\x86R0."
b = b"\x80\x05\x95\xac\x03\x00\x00\x00\x00\x00\x00\x8c\x17cloudpickle.cloudpickle\x94\x8c\x14_make_skeleton_class\x94\x93\...x0b__globals__\x94}\x94u\x86\x94\x86R0\x8c\n__module__\x94h\x0c\x8c\x15__static_attributes__\x94)u}\x94\x86\x94\x86R0."

    def check_deterministic_pickle(a, b):
        """Check that two pickle output are bitwise equal.
    
        If it is not the case, print the diff between the disassembled pickle
        payloads.
    
        This helper is useful to investigate non-deterministic pickling.
        """
        if a != b:
            with io.StringIO() as out:
                pickletools.dis(pickletools.optimize(a), out)
                a_out = out.getvalue()
                # Remove the 11 first characters of each line to remove the bytecode offset
                # of each object, which is different on each line for very small differences,
                # making the diff very hard to read.
                a_out = "\n".join(ll[11:] for ll in a_out.splitlines())
            with io.StringIO() as out:
                pickletools.dis(pickletools.optimize(b), out)
                b_out = out.getvalue()
                b_out = "\n".join(ll[11:] for ll in b_out.splitlines())
>           assert a_out == b_out
E           AssertionError: assert ' PROTO      ...g opcodes = 4' == ' PROTO      ...g opcodes = 4'
E             
E                PROTO      5
E             -  FRAME      879
E             ?              ^^
E             +  FRAME      981
E             ?             + ^
E                SHORT_BINUNICODE 'cloudpickle.cloudpickle'...
E             
E             ...Full output truncated (150 lines hidden), use '-vv' to show

a          = b"\x80\x05\x95\x14\x04\x00\x00\x00\x00\x00\x00\x8c\x17cloudpickle.cloudpickle\x94\x8c\x14_make_skeleton_class\x94\x93\...x0b__globals__\x94}\x94u\x86\x94\x86R0\x8c\n__module__\x94h\x0c\x8c\x15__static_attributes__\x94)u}\x94\x86\x94\x86R0."
a_out      = " PROTO      5\n FRAME      981\n SHORT_BINUNICODE 'cloudpickle.cloudpickle'\n MEMOIZE    (as 0)\n SHORT_BINUNICODE '_...PTY_TUPLE\n     SETITEMS   (MARK at 195)\n EMPTY_DICT\n TUPLE2\n TUPLE2\n REDUCE\n POP\n STOP\ntocol among opcodes = 4"
b          = b"\x80\x05\x95\xac\x03\x00\x00\x00\x00\x00\x00\x8c\x17cloudpickle.cloudpickle\x94\x8c\x14_make_skeleton_class\x94\x93\...x0b__globals__\x94}\x94u\x86\x94\x86R0\x8c\n__module__\x94h\x0c\x8c\x15__static_attributes__\x94)u}\x94\x86\x94\x86R0."
b_out      = " PROTO      5\n FRAME      879\n SHORT_BINUNICODE 'cloudpickle.cloudpickle'\n MEMOIZE    (as 0)\n SHORT_BINUNICODE '_...PTY_TUPLE\n     SETITEMS   (MARK at 195)\n EMPTY_DICT\n TUPLE2\n TUPLE2\n REDUCE\n POP\n STOP\ntocol among opcodes = 4"
out        = <_io.StringIO object at 0x7f46db6ad6c0>

tests/testutils.py:239: AssertionError
____ CloudPickleTest.test_deterministic_str_interning_for_chained_dynamic_class_pickling ____

self = <tests.cloudpickle_test.CloudPickleTest testMethod=test_deterministic_str_interning_for_chained_dynamic_class_pickling>

    def test_deterministic_str_interning_for_chained_dynamic_class_pickling(self):
        # Check that the pickle produced by the unpickled instance is the same.
        # This checks that there is no issue related to the string interning of
        # the names of attributes of class definitions and names of attributes
        # of the `__code__` objects of the methods.
    
        with subprocess_worker(protocol=self.protocol) as w:
            # Due to interning of class attributes, check that this does not
            # create issues with dynamic function definition.
            class A:
                """Class with potential string interning issues."""
    
                arg_1 = "class_value"
    
                def join(self):
                    pass
    
                def test_method(self, arg_1, join):
                    pass
    
            A_dump = w.run(cloudpickle.dumps, A)
>           check_deterministic_pickle(A_dump, cloudpickle.dumps(A))

A          = <class 'tests.cloudpickle_test.CloudPickleTest.test_deterministic_str_interning_for_chained_dynamic_class_pickling.<locals>.A'>
A_dump     = b'\x80\x05\x95\x94\x05\x00\x00\x00\x00\x00\x00\x8c\x17cloudpickle.cloudpickle\x94\x8c\x14_make_skeleton_class\x94\x93\...ass_pickling.<locals>.A.test_method\x94h<}\x94h>Nh?Nh\x0bh\x0ch@NhANhB]\x94hD}\x94u\x86\x94\x86R0u}\x94\x86\x94\x86R0.'
self       = <tests.cloudpickle_test.CloudPickleTest testMethod=test_deterministic_str_interning_for_chained_dynamic_class_pickling>
w          = <tests.testutils._Worker object at 0x7f46db410690>

tests/cloudpickle_test.py:2113: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

a = b'\x80\x05\x95\x94\x05\x00\x00\x00\x00\x00\x00\x8c\x17cloudpickle.cloudpickle\x94\x8c\x14_make_skeleton_class\x94\x93\...ass_pickling.<locals>.A.test_method\x94h<}\x94h>Nh?Nh\x0bh\x0ch@NhANhB]\x94hD}\x94u\x86\x94\x86R0u}\x94\x86\x94\x86R0.'
b = b'\x80\x05\x95\xc5\x04\x00\x00\x00\x00\x00\x00\x8c\x17cloudpickle.cloudpickle\x94\x8c\x14_make_skeleton_class\x94\x93\...\x94}\x94(h0\x8c\x0btest_method\x94h:hLh;}\x94h=Nh>Nh\x0bh\x0ch?Nh@NhA]\x94hC}\x94u\x86\x94\x86R0u}\x94\x86\x94\x86R0.'

    def check_deterministic_pickle(a, b):
        """Check that two pickle output are bitwise equal.
    
        If it is not the case, print the diff between the disassembled pickle
        payloads.
    
        This helper is useful to investigate non-deterministic pickling.
        """
        if a != b:
            with io.StringIO() as out:
                pickletools.dis(pickletools.optimize(a), out)
                a_out = out.getvalue()
                # Remove the 11 first characters of each line to remove the bytecode offset
                # of each object, which is different on each line for very small differences,
                # making the diff very hard to read.
                a_out = "\n".join(ll[11:] for ll in a_out.splitlines())
            with io.StringIO() as out:
                pickletools.dis(pickletools.optimize(b), out)
                b_out = out.getvalue()
                b_out = "\n".join(ll[11:] for ll in b_out.splitlines())
>           assert a_out == b_out
E           AssertionError: assert ' PROTO      ...g opcodes = 4' == ' PROTO      ...g opcodes = 4'
E             
E                PROTO      5
E             -  FRAME      1158
E             ?               --
E             +  FRAME      1361
E             ?              ++
E                SHORT_BINUNICODE 'cloudpickle.cloudpickle'...
E             
E             ...Full output truncated (288 lines hidden), use '-vv' to show

a          = b'\x80\x05\x95\x94\x05\x00\x00\x00\x00\x00\x00\x8c\x17cloudpickle.cloudpickle\x94\x8c\x14_make_skeleton_class\x94\x93\...ass_pickling.<locals>.A.test_method\x94h<}\x94h>Nh?Nh\x0bh\x0ch@NhANhB]\x94hD}\x94u\x86\x94\x86R0u}\x94\x86\x94\x86R0.'
a_out      = " PROTO      5\n FRAME      1361\n SHORT_BINUNICODE 'cloudpickle.cloudpickle'\n MEMOIZE    (as 0)\n SHORT_BINUNICODE '...n     POP\n     SETITEMS   (MARK at 195)\n EMPTY_DICT\n TUPLE2\n TUPLE2\n REDUCE\n POP\n STOP\ntocol among opcodes = 4"
b          = b'\x80\x05\x95\xc5\x04\x00\x00\x00\x00\x00\x00\x8c\x17cloudpickle.cloudpickle\x94\x8c\x14_make_skeleton_class\x94\x93\...\x94}\x94(h0\x8c\x0btest_method\x94h:hLh;}\x94h=Nh>Nh\x0bh\x0ch?Nh@NhA]\x94hC}\x94u\x86\x94\x86R0u}\x94\x86\x94\x86R0.'
b_out      = " PROTO      5\n FRAME      1158\n SHORT_BINUNICODE 'cloudpickle.cloudpickle'\n MEMOIZE    (as 0)\n SHORT_BINUNICODE '...n     POP\n     SETITEMS   (MARK at 195)\n EMPTY_DICT\n TUPLE2\n TUPLE2\n REDUCE\n POP\n STOP\ntocol among opcodes = 4"
out        = <_io.StringIO object at 0x7f46db6add80>

tests/testutils.py:239: AssertionError
________ CloudPickleTest.test_dynamic_class_determinist_subworker_tuple_memoization _________

self = <tests.cloudpickle_test.CloudPickleTest testMethod=test_dynamic_class_determinist_subworker_tuple_memoization>

    def test_dynamic_class_determinist_subworker_tuple_memoization(self):
        # Check that the pickle produced by the unpickled instance is the same.
        # This highlights some issues with tuple memoization.
    
        with subprocess_worker(protocol=self.protocol) as w:
            # Arguments' tuple is memoized in the main process but not in the
            # subprocess as the tuples do not share the same id in the loaded
            # class.
    
            # XXX - this does not seem to work, and I am not sure there is an easy fix.
            class A:
                """Class with potential tuple memoization issues."""
    
                def func1(self):
                    pass
    
                def func2(self):
                    pass
    
            A_dump = w.run(cloudpickle.dumps, A)
>           check_deterministic_pickle(A_dump, cloudpickle.dumps(A))

A          = <class 'tests.cloudpickle_test.CloudPickleTest.test_dynamic_class_determinist_subworker_tuple_memoization.<locals>.A'>
A_dump     = b'\x80\x05\x953\x05\x00\x00\x00\x00\x00\x00\x8c\x17cloudpickle.cloudpickle\x94\x8c\x14_make_skeleton_class\x94\x93\x94..._tuple_memoization.<locals>.A.func2\x94h:}\x94h<Nh=Nh\x0bh\x0ch>Nh?Nh@]\x94hB}\x94u\x86\x94\x86R0u}\x94\x86\x94\x86R0.'
self       = <tests.cloudpickle_test.CloudPickleTest testMethod=test_dynamic_class_determinist_subworker_tuple_memoization>
w          = <tests.testutils._Worker object at 0x7f46db6f6c10>

tests/cloudpickle_test.py:2153: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

a = b'\x80\x05\x953\x05\x00\x00\x00\x00\x00\x00\x8c\x17cloudpickle.cloudpickle\x94\x8c\x14_make_skeleton_class\x94\x93\x94..._tuple_memoization.<locals>.A.func2\x94h:}\x94h<Nh=Nh\x0bh\x0ch>Nh?Nh@]\x94hB}\x94u\x86\x94\x86R0u}\x94\x86\x94\x86R0.'
b = b'\x80\x05\x95{\x04\x00\x00\x00\x00\x00\x00\x8c\x17cloudpickle.cloudpickle\x94\x8c\x14_make_skeleton_class\x94\x93\x94...4h4hL}\x94}\x94(h.\x8c\x05func2\x94h8hHh9}\x94h;Nh<Nh\x0bh\x0ch=Nh>Nh?]\x94hA}\x94u\x86\x94\x86R0u}\x94\x86\x94\x86R0.'

    def check_deterministic_pickle(a, b):
        """Check that two pickle output are bitwise equal.
    
        If it is not the case, print the diff between the disassembled pickle
        payloads.
    
        This helper is useful to investigate non-deterministic pickling.
        """
        if a != b:
            with io.StringIO() as out:
                pickletools.dis(pickletools.optimize(a), out)
                a_out = out.getvalue()
                # Remove the 11 first characters of each line to remove the bytecode offset
                # of each object, which is different on each line for very small differences,
                # making the diff very hard to read.
                a_out = "\n".join(ll[11:] for ll in a_out.splitlines())
            with io.StringIO() as out:
                pickletools.dis(pickletools.optimize(b), out)
                b_out = out.getvalue()
                b_out = "\n".join(ll[11:] for ll in b_out.splitlines())
>           assert a_out == b_out
E           AssertionError: assert ' PROTO      ...g opcodes = 4' == ' PROTO      ...g opcodes = 4'
E             
E                PROTO      5
E             -  FRAME      1088
E             ?              ^^
E             +  FRAME      1268
E             ?              ^^
E                SHORT_BINUNICODE 'cloudpickle.cloudpickle'...
E             
E             ...Full output truncated (284 lines hidden), use '-vv' to show

a          = b'\x80\x05\x953\x05\x00\x00\x00\x00\x00\x00\x8c\x17cloudpickle.cloudpickle\x94\x8c\x14_make_skeleton_class\x94\x93\x94..._tuple_memoization.<locals>.A.func2\x94h:}\x94h<Nh=Nh\x0bh\x0ch>Nh?Nh@]\x94hB}\x94u\x86\x94\x86R0u}\x94\x86\x94\x86R0.'
a_out      = " PROTO      5\n FRAME      1268\n SHORT_BINUNICODE 'cloudpickle.cloudpickle'\n MEMOIZE    (as 0)\n SHORT_BINUNICODE '...n     POP\n     SETITEMS   (MARK at 195)\n EMPTY_DICT\n TUPLE2\n TUPLE2\n REDUCE\n POP\n STOP\ntocol among opcodes = 4"
b          = b'\x80\x05\x95{\x04\x00\x00\x00\x00\x00\x00\x8c\x17cloudpickle.cloudpickle\x94\x8c\x14_make_skeleton_class\x94\x93\x94...4h4hL}\x94}\x94(h.\x8c\x05func2\x94h8hHh9}\x94h;Nh<Nh\x0bh\x0ch=Nh>Nh?]\x94hA}\x94u\x86\x94\x86R0u}\x94\x86\x94\x86R0.'
b_out      = " PROTO      5\n FRAME      1088\n SHORT_BINUNICODE 'cloudpickle.cloudpickle'\n MEMOIZE    (as 0)\n SHORT_BINUNICODE '...n     POP\n     SETITEMS   (MARK at 195)\n EMPTY_DICT\n TUPLE2\n TUPLE2\n REDUCE\n POP\n STOP\ntocol among opcodes = 4"
out        = <_io.StringIO object at 0x7f46db6ae080>

tests/testutils.py:239: AssertionError
_ Protocol2CloudPickleTest.test_deterministic_dynamic_class_attr_ordering_for_chained_pickling _

self = <tests.cloudpickle_test.Protocol2CloudPickleTest testMethod=test_deterministic_dynamic_class_attr_ordering_for_chained_pickling>

    def test_deterministic_dynamic_class_attr_ordering_for_chained_pickling(self):
        # Check that the pickle produced by pickling a reconstructed class definition
        # in a remote process matches the pickle produced by pickling the original
        # class definition.
        # In particular, this test checks that the order of the class attributes is
        # deterministic.
    
        with subprocess_worker(protocol=self.protocol) as w:
    
            class A:
                """Simple class definition"""
    
                pass
    
            A_dump = w.run(cloudpickle.dumps, A)
            check_deterministic_pickle(A_dump, cloudpickle.dumps(A))
    
            # If the `__doc__` attribute is defined after some other class
            # attribute, this can cause class attribute ordering changes due to
            # the way we reconstruct the class definition in
            # `_make_class_skeleton`, which creates the class and thus its
            # `__doc__` attribute before populating the class attributes.
            class A:
                name = "A"
                __doc__ = "Updated class definition"
    
            A_dump = w.run(cloudpickle.dumps, A)
            check_deterministic_pickle(A_dump, cloudpickle.dumps(A))
    
            # If a `__doc__` is defined on the `__init__` method, this can
            # cause ordering changes due to the way we reconstruct the class
            # with `_make_class_skeleton`.
            class A:
                def __init__(self):
                    """Class definition with explicit __init__"""
                    pass
    
            A_dump = w.run(cloudpickle.dumps, A)
>           check_deterministic_pickle(A_dump, cloudpickle.dumps(A))

A          = <class 'tests.cloudpickle_test.CloudPickleTest.test_deterministic_dynamic_class_attr_ordering_for_chained_pickling.<locals>.A'>
A_dump     = b"\x80\x05\x95\x14\x04\x00\x00\x00\x00\x00\x00\x8c\x17cloudpickle.cloudpickle\x94\x8c\x14_make_skeleton_class\x94\x93\...x0b__globals__\x94}\x94u\x86\x94\x86R0\x8c\n__module__\x94h\x0c\x8c\x15__static_attributes__\x94)u}\x94\x86\x94\x86R0."
self       = <tests.cloudpickle_test.Protocol2CloudPickleTest testMethod=test_deterministic_dynamic_class_attr_ordering_for_chained_pickling>
w          = <tests.testutils._Worker object at 0x7f46d9f6ff20>

tests/cloudpickle_test.py:2090: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

a = b"\x80\x05\x95\x14\x04\x00\x00\x00\x00\x00\x00\x8c\x17cloudpickle.cloudpickle\x94\x8c\x14_make_skeleton_class\x94\x93\...x0b__globals__\x94}\x94u\x86\x94\x86R0\x8c\n__module__\x94h\x0c\x8c\x15__static_attributes__\x94)u}\x94\x86\x94\x86R0."
b = b"\x80\x05\x95\xac\x03\x00\x00\x00\x00\x00\x00\x8c\x17cloudpickle.cloudpickle\x94\x8c\x14_make_skeleton_class\x94\x93\...x0b__globals__\x94}\x94u\x86\x94\x86R0\x8c\n__module__\x94h\x0c\x8c\x15__static_attributes__\x94)u}\x94\x86\x94\x86R0."

    def check_deterministic_pickle(a, b):
        """Check that two pickle output are bitwise equal.
    
        If it is not the case, print the diff between the disassembled pickle
        payloads.
    
        This helper is useful to investigate non-deterministic pickling.
        """
        if a != b:
            with io.StringIO() as out:
                pickletools.dis(pickletools.optimize(a), out)
                a_out = out.getvalue()
                # Remove the 11 first characters of each line to remove the bytecode offset
                # of each object, which is different on each line for very small differences,
                # making the diff very hard to read.
                a_out = "\n".join(ll[11:] for ll in a_out.splitlines())
            with io.StringIO() as out:
                pickletools.dis(pickletools.optimize(b), out)
                b_out = out.getvalue()
                b_out = "\n".join(ll[11:] for ll in b_out.splitlines())
>           assert a_out == b_out
E           AssertionError: assert ' PROTO      ...g opcodes = 4' == ' PROTO      ...g opcodes = 4'
E             
E                PROTO      5
E             -  FRAME      879
E             ?              ^^
E             +  FRAME      981
E             ?             + ^
E                SHORT_BINUNICODE 'cloudpickle.cloudpickle'...
E             
E             ...Full output truncated (150 lines hidden), use '-vv' to show

a          = b"\x80\x05\x95\x14\x04\x00\x00\x00\x00\x00\x00\x8c\x17cloudpickle.cloudpickle\x94\x8c\x14_make_skeleton_class\x94\x93\...x0b__globals__\x94}\x94u\x86\x94\x86R0\x8c\n__module__\x94h\x0c\x8c\x15__static_attributes__\x94)u}\x94\x86\x94\x86R0."
a_out      = " PROTO      5\n FRAME      981\n SHORT_BINUNICODE 'cloudpickle.cloudpickle'\n MEMOIZE    (as 0)\n SHORT_BINUNICODE '_...PTY_TUPLE\n     SETITEMS   (MARK at 195)\n EMPTY_DICT\n TUPLE2\n TUPLE2\n REDUCE\n POP\n STOP\ntocol among opcodes = 4"
b          = b"\x80\x05\x95\xac\x03\x00\x00\x00\x00\x00\x00\x8c\x17cloudpickle.cloudpickle\x94\x8c\x14_make_skeleton_class\x94\x93\...x0b__globals__\x94}\x94u\x86\x94\x86R0\x8c\n__module__\x94h\x0c\x8c\x15__static_attributes__\x94)u}\x94\x86\x94\x86R0."
b_out      = " PROTO      5\n FRAME      879\n SHORT_BINUNICODE 'cloudpickle.cloudpickle'\n MEMOIZE    (as 0)\n SHORT_BINUNICODE '_...PTY_TUPLE\n     SETITEMS   (MARK at 195)\n EMPTY_DICT\n TUPLE2\n TUPLE2\n REDUCE\n POP\n STOP\ntocol among opcodes = 4"
out        = <_io.StringIO object at 0x7f46d88cee00>

tests/testutils.py:239: AssertionError
_ Protocol2CloudPickleTest.test_deterministic_str_interning_for_chained_dynamic_class_pickling _

self = <tests.cloudpickle_test.Protocol2CloudPickleTest testMethod=test_deterministic_str_interning_for_chained_dynamic_class_pickling>

    def test_deterministic_str_interning_for_chained_dynamic_class_pickling(self):
        # Check that the pickle produced by the unpickled instance is the same.
        # This checks that there is no issue related to the string interning of
        # the names of attributes of class definitions and names of attributes
        # of the `__code__` objects of the methods.
    
        with subprocess_worker(protocol=self.protocol) as w:
            # Due to interning of class attributes, check that this does not
            # create issues with dynamic function definition.
            class A:
                """Class with potential string interning issues."""
    
                arg_1 = "class_value"
    
                def join(self):
                    pass
    
                def test_method(self, arg_1, join):
                    pass
    
            A_dump = w.run(cloudpickle.dumps, A)
>           check_deterministic_pickle(A_dump, cloudpickle.dumps(A))

A          = <class 'tests.cloudpickle_test.CloudPickleTest.test_deterministic_str_interning_for_chained_dynamic_class_pickling.<locals>.A'>
A_dump     = b'\x80\x05\x95\x94\x05\x00\x00\x00\x00\x00\x00\x8c\x17cloudpickle.cloudpickle\x94\x8c\x14_make_skeleton_class\x94\x93\...ass_pickling.<locals>.A.test_method\x94h<}\x94h>Nh?Nh\x0bh\x0ch@NhANhB]\x94hD}\x94u\x86\x94\x86R0u}\x94\x86\x94\x86R0.'
self       = <tests.cloudpickle_test.Protocol2CloudPickleTest testMethod=test_deterministic_str_interning_for_chained_dynamic_class_pickling>
w          = <tests.testutils._Worker object at 0x7f46d845faf0>

tests/cloudpickle_test.py:2113: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

a = b'\x80\x05\x95\x94\x05\x00\x00\x00\x00\x00\x00\x8c\x17cloudpickle.cloudpickle\x94\x8c\x14_make_skeleton_class\x94\x93\...ass_pickling.<locals>.A.test_method\x94h<}\x94h>Nh?Nh\x0bh\x0ch@NhANhB]\x94hD}\x94u\x86\x94\x86R0u}\x94\x86\x94\x86R0.'
b = b'\x80\x05\x95\xc5\x04\x00\x00\x00\x00\x00\x00\x8c\x17cloudpickle.cloudpickle\x94\x8c\x14_make_skeleton_class\x94\x93\...\x94}\x94(h0\x8c\x0btest_method\x94h:hLh;}\x94h=Nh>Nh\x0bh\x0ch?Nh@NhA]\x94hC}\x94u\x86\x94\x86R0u}\x94\x86\x94\x86R0.'

    def check_deterministic_pickle(a, b):
        """Check that two pickle output are bitwise equal.
    
        If it is not the case, print the diff between the disassembled pickle
        payloads.
    
        This helper is useful to investigate non-deterministic pickling.
        """
        if a != b:
            with io.StringIO() as out:
                pickletools.dis(pickletools.optimize(a), out)
                a_out = out.getvalue()
                # Remove the 11 first characters of each line to remove the bytecode offset
                # of each object, which is different on each line for very small differences,
                # making the diff very hard to read.
                a_out = "\n".join(ll[11:] for ll in a_out.splitlines())
            with io.StringIO() as out:
                pickletools.dis(pickletools.optimize(b), out)
                b_out = out.getvalue()
                b_out = "\n".join(ll[11:] for ll in b_out.splitlines())
>           assert a_out == b_out
E           AssertionError: assert ' PROTO      ...g opcodes = 4' == ' PROTO      ...g opcodes = 4'
E             
E                PROTO      5
E             -  FRAME      1158
E             ?               --
E             +  FRAME      1361
E             ?              ++
E                SHORT_BINUNICODE 'cloudpickle.cloudpickle'...
E             
E             ...Full output truncated (288 lines hidden), use '-vv' to show

a          = b'\x80\x05\x95\x94\x05\x00\x00\x00\x00\x00\x00\x8c\x17cloudpickle.cloudpickle\x94\x8c\x14_make_skeleton_class\x94\x93\...ass_pickling.<locals>.A.test_method\x94h<}\x94h>Nh?Nh\x0bh\x0ch@NhANhB]\x94hD}\x94u\x86\x94\x86R0u}\x94\x86\x94\x86R0.'
a_out      = " PROTO      5\n FRAME      1361\n SHORT_BINUNICODE 'cloudpickle.cloudpickle'\n MEMOIZE    (as 0)\n SHORT_BINUNICODE '...n     POP\n     SETITEMS   (MARK at 195)\n EMPTY_DICT\n TUPLE2\n TUPLE2\n REDUCE\n POP\n STOP\ntocol among opcodes = 4"
b          = b'\x80\x05\x95\xc5\x04\x00\x00\x00\x00\x00\x00\x8c\x17cloudpickle.cloudpickle\x94\x8c\x14_make_skeleton_class\x94\x93\...\x94}\x94(h0\x8c\x0btest_method\x94h:hLh;}\x94h=Nh>Nh\x0bh\x0ch?Nh@NhA]\x94hC}\x94u\x86\x94\x86R0u}\x94\x86\x94\x86R0.'
b_out      = " PROTO      5\n FRAME      1158\n SHORT_BINUNICODE 'cloudpickle.cloudpickle'\n MEMOIZE    (as 0)\n SHORT_BINUNICODE '...n     POP\n     SETITEMS   (MARK at 195)\n EMPTY_DICT\n TUPLE2\n TUPLE2\n REDUCE\n POP\n STOP\ntocol among opcodes = 4"
out        = <_io.StringIO object at 0x7f46d88cf580>

tests/testutils.py:239: AssertionError
===================================== warnings summary ======================================
tests/cloudpickle_test.py: 12 warnings
  /usr/lib64/python3.13/multiprocessing/popen_fork.py:67: DeprecationWarning: This process (pid=62090) is multi-threaded, use of fork() may lead to deadlocks in the child.
    self.pid = os.fork()

tests/cloudpickle_test.py::CloudPickleTest::test_itertools_count
  /home/lbalhar/Software/cloudpickle/cloudpickle/cloudpickle.py:1284: DeprecationWarning: Pickle, copy, and deepcopy support will be removed from itertools in Python 3.14.
    return super().dump(obj)

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================== short test summary info ==================================
FAILED tests/cloudpickle_test.py::CloudPickleTest::test_deterministic_dynamic_class_attr_ordering_for_chained_pickling - AssertionError: assert ' PROTO      ...g opcodes = 4' == ' PROTO      ...g opcodes = 4'
FAILED tests/cloudpickle_test.py::CloudPickleTest::test_deterministic_str_interning_for_chained_dynamic_class_pickling - AssertionError: assert ' PROTO      ...g opcodes = 4' == ' PROTO      ...g opcodes = 4'
FAILED tests/cloudpickle_test.py::CloudPickleTest::test_dynamic_class_determinist_subworker_tuple_memoization - AssertionError: assert ' PROTO      ...g opcodes = 4' == ' PROTO      ...g opcodes = 4'
FAILED tests/cloudpickle_test.py::Protocol2CloudPickleTest::test_deterministic_dynamic_class_attr_ordering_for_chained_pickling - AssertionError: assert ' PROTO      ...g opcodes = 4' == ' PROTO      ...g opcodes = 4'
FAILED tests/cloudpickle_test.py::Protocol2CloudPickleTest::test_deterministic_str_interning_for_chained_dynamic_class_pickling - AssertionError: assert ' PROTO      ...g opcodes = 4' == ' PROTO      ...g opcodes = 4'
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 5 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
============= 5 failed, 156 passed, 4 skipped, 1 xfailed, 13 warnings in 4.43s ==============
py313: exit 1 (4.62 seconds) /home/lbalhar/Software/cloudpickle> pytest -lv --maxfail=5 pid=62090

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant