0.4.4 tests fail in PyPy

[isuruf]

test_fmpz...Error in test_fmpz
Traceback (most recent call last):
  File "/home/conda/feedstock_root/build_artifacts/python-flint_1694094201440/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pla/lib/pypy3.9/site-packages/flint/test/__main__.py", line 36, in run_tests
    test()
  File "/home/conda/feedstock_root/build_artifacts/python-flint_1694094201440/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pla/lib/pypy3.9/site-packages/flint/test/test.py", line 141, in test_fmpz
    assert pow(a, flint.fmpz(b), c) == ab_mod_c
TypeError: operands do not support pow()

This used to work in 0.4.2

[oscarbenjamin]

I haven't tested with PyPy at all. What version of PyPy are you using for this?

[isuruf]

PyPy3.9 7.3.12

[oscarbenjamin]

PyPy3.9 7.3.12

The tests all passed for me locally with PyPy3.9 7.3.11.

[oscarbenjamin]

I tried reproducing this in CI in #76 but the wheel build fails under PyPy.

[oscarbenjamin]

the wheel build fails under PyPy.

  + python -m pip wheel /Users/runner/work/python-flint/python-flint --wheel-dir=/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/cibw-run-ck51u289/pp39-macosx_x86_64/built_wheel --no-deps
  Processing /Users/runner/work/python-flint/python-flint
    Preparing metadata (setup.py): started
    Preparing metadata (setup.py): finished with status 'error'
    error: subprocess-exited-with-error
    
    × python setup.py egg_info did not run successfully.
    │ exit code: 1
    ╰─> [16 lines of output]
        Traceback (most recent call last):
          File "<string>", line 2, in <module>
          File "<pip-setuptools-caller>", line 34, in <module>
          File "/Users/runner/work/python-flint/python-flint/setup.py", line 9, in <module>
            from numpy.distutils.system_info import default_include_dirs, default_lib_dirs
          File "/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/cibw-run-ck51u289/pp39-macosx_x86_64/build/venv/lib/pypy3.9/site-packages/numpy/__init__.py", line 139, in <module>
            from . import core
          File "/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/cibw-run-ck51u289/pp39-macosx_x86_64/build/venv/lib/pypy3.9/site-packages/numpy/core/__init__.py", line 23, in <module>
            from . import multiarray
          File "/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/cibw-run-ck51u289/pp39-macosx_x86_64/build/venv/lib/pypy3.9/site-packages/numpy/core/multiarray.py", line 86, in <module>
            def empty_like(prototype, dtype=None, order=None, subok=None, shape=None):
          File "/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/cibw-run-ck51u289/pp39-macosx_x86_64/build/venv/lib/pypy3.9/site-packages/numpy/core/overrides.py", line 178, in decorator
            return array_function_dispatch(
          File "/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/cibw-run-ck51u289/pp39-macosx_x86_64/build/venv/lib/pypy3.9/site-packages/numpy/core/overrides.py", line 160, in decorator
            public_api = _ArrayFunctionDispatcher(dispatcher, implementation)
        AttributeError: 'builtin_function_or_method' object has no attribute '__qualname__'

[oscarbenjamin]

PyPy3.9 7.3.12

The tests all passed for me locally with PyPy3.9 7.3.11.

I'm a bit stuck here without being able to reproduce this.

[isuruf]

Do you have a local conda installation? If so, I can give instructions on how to reproduce, else I can try to set it up on CI.

Note that the tests fail on all three OSes at conda-forge/python-flint-feedstock#31

[oscarbenjamin]

I don't even want to try to fix this until we have CI that can reproduce the problem. If we don't test PyPy in CI then I would rather just say that it is not supported.

[oscarbenjamin]

I have just tested locally that I can build a wheel and install from it and all tests are passing.

This is with pypy-3.9-7.3.11 which is the newest version available from pyenv.

If the difference is pypy-3.9-7.3.12 rather than 7.3.11 then that might indicate a PyPy bug rather than a python-flint bug.

The PRs to python-flint in between 0.4.2 and 0.4.4 are:

• Introduce submodules into python-flint #61
• Finish the flint_base submodule, depreciate old roots and factor out more utils #63
• Mod experiment #65
• Bump version to 0.4.3 #69
• Include files in sdist and bump version to 0.4.4 #71

They are all just refactoring or unrelated changes.

The bug reported is with 3-arg pow like:

pow(1, flint.fmpz(2), 3)

This is the code for fmpz.pow in 0.4.2:

python-flint/src/flint/fmpz.pyx

Lines 365 to 412 in b658fd3

	def __pow__(s, t, m):
	cdef fmpz_struct tval[1]
	cdef fmpz_struct mval[1]
	cdef int ttype = FMPZ_UNKNOWN
	cdef int mtype = FMPZ_UNKNOWN
	cdef int success
	u = NotImplemented
	ttype = fmpz_set_any_ref(tval, t)
	if ttype == FMPZ_UNKNOWN:
	return NotImplemented
	if m is None:
	# fmpz_pow_fmpz throws if x is negative
	if fmpz_sgn(tval) == -1:
	if ttype == FMPZ_TMP: fmpz_clear(tval)
	raise ValueError("negative exponent")
	u = fmpz.__new__(fmpz)
	success = fmpz_pow_fmpz((<fmpz>u).val, (<fmpz>s).val, tval)
	if not success:
	if ttype == FMPZ_TMP: fmpz_clear(tval)
	raise OverflowError("fmpz_pow_fmpz: exponent too large")
	else:
	# Modular exponentiation
	mtype = fmpz_set_any_ref(mval, m)
	if mtype != FMPZ_UNKNOWN:
	if fmpz_is_zero(mval):
	if ttype == FMPZ_TMP: fmpz_clear(tval)
	if mtype == FMPZ_TMP: fmpz_clear(mval)
	raise ValueError("pow(): modulus cannot be zero")
	# The Flint docs say that fmpz_powm will throw if m is zero
	# but it also throws if m is negative. Python generally allows
	# e.g. pow(2, 2, -3) == (2^2) % (-3) == -2. We could implement
	# that here as well but it is not clear how useful it is.
	if fmpz_sgn(mval) == -1:
	if ttype == FMPZ_TMP: fmpz_clear(tval)
	if mtype == FMPZ_TMP: fmpz_clear(mval)
	raise ValueError("pow(): negative modulua not supported")
	u = fmpz.__new__(fmpz)
	fmpz_powm((<fmpz>u).val, (<fmpz>s).val, tval, mval)
	if ttype == FMPZ_TMP: fmpz_clear(tval)
	if mtype == FMPZ_TMP: fmpz_clear(mval)
	return u

This is the code in 0.4.4:

python-flint/src/flint/types/fmpz.pyx

Lines 358 to 405 in 24267a5

	def __pow__(s, t, m):
	cdef fmpz_struct tval[1]
	cdef fmpz_struct mval[1]
	cdef int ttype = FMPZ_UNKNOWN
	cdef int mtype = FMPZ_UNKNOWN
	cdef int success
	u = NotImplemented
	ttype = fmpz_set_any_ref(tval, t)
	if ttype == FMPZ_UNKNOWN:
	return NotImplemented
	if m is None:
	# fmpz_pow_fmpz throws if x is negative
	if fmpz_sgn(tval) == -1:
	if ttype == FMPZ_TMP: fmpz_clear(tval)
	raise ValueError("negative exponent")
	u = fmpz.__new__(fmpz)
	success = fmpz_pow_fmpz((<fmpz>u).val, (<fmpz>s).val, tval)
	if not success:
	if ttype == FMPZ_TMP: fmpz_clear(tval)
	raise OverflowError("fmpz_pow_fmpz: exponent too large")
	else:
	# Modular exponentiation
	mtype = fmpz_set_any_ref(mval, m)
	if mtype != FMPZ_UNKNOWN:
	if fmpz_is_zero(mval):
	if ttype == FMPZ_TMP: fmpz_clear(tval)
	if mtype == FMPZ_TMP: fmpz_clear(mval)
	raise ValueError("pow(): modulus cannot be zero")
	# The Flint docs say that fmpz_powm will throw if m is zero
	# but it also throws if m is negative. Python generally allows
	# e.g. pow(2, 2, -3) == (2^2) % (-3) == -2. We could implement
	# that here as well but it is not clear how useful it is.
	if fmpz_sgn(mval) == -1:
	if ttype == FMPZ_TMP: fmpz_clear(tval)
	if mtype == FMPZ_TMP: fmpz_clear(mval)
	raise ValueError("pow(): negative modulua not supported")
	u = fmpz.__new__(fmpz)
	fmpz_powm((<fmpz>u).val, (<fmpz>s).val, tval, mval)
	if ttype == FMPZ_TMP: fmpz_clear(tval)
	if mtype == FMPZ_TMP: fmpz_clear(mval)
	return u

So unless I've missed something the code for fmpz.__pow__ is unchanged although there was a refactor that moved it to a different file.

The code still passes under CPython in any case.

[isuruf]

If the difference is pypy-3.9-7.3.12 rather than 7.3.11 then that might indicate a PyPy bug rather than a python-flint bug.

I checked with 7.3.11 in conda and the bug is still there

[isuruf]

It is a bug in 7.3.12. Disregard my previous comment.

[oscarbenjamin]

Is there an associated PyPy bug report for this?

[oscarbenjamin]

gh-93 fixes a bug to do with 3-arg pow but I am not sure if it is related to the bug here.

[oscarbenjamin]

I'm not sure how easy it is to support the 3-arg pow under PyPy. The mechanism that makes it work in CPython is quite obscure and only works from Cython (not pure Python):
https://discuss.python.org/t/awkward-dunders-for-pow-x-y-z/35185

I can reproduce this locally now and also there is a new test failure under PyPy:

test_polys...Error in test_polys
Traceback (most recent call last):
  File "/home/oscar/current/active/python-flint/src/flint/test/__main__.py", line 36, in run_tests
    test()
  File "/home/oscar/current/active/python-flint/src/flint/test/test.py", line 2362, in test_polys
    assert raises(lambda: P([1, 2, 1]) // 0, ZeroDivisionError)
  File "/home/oscar/current/active/python-flint/src/flint/test/test.py", line 18, in raises
    f()
  File "/home/oscar/current/active/python-flint/src/flint/test/test.py", line 2362, in <lambda>
    assert raises(lambda: P([1, 2, 1]) // 0, ZeroDivisionError)
  File "src/flint/types/fmpz_mod_poly.pyx", line 522, in flint.types.fmpz_mod_poly.fmpz_mod_poly.__floordiv__
    return fmpz_mod_poly._floordiv_(self, other)
  File "src/flint/types/fmpz_mod_poly.pyx", line 512, in flint.types.fmpz_mod_poly.fmpz_mod_poly._floordiv_
    if not right.leading_coefficient().is_unit():
  File "src/flint/types/fmpz_mod_poly.pyx", line 892, in flint.types.fmpz_mod_poly.fmpz_mod_poly.leading_coefficient
    return self[self.degree()]
IndexError: %T index out of range

[oscarbenjamin]

I haven't been able to set this up but does gmpy2 manage to do the 3-arg pow under PyPy?

Under CPython it works just like python-flint does:

>>> import gmpy2
>>> pow(2, gmpy2.mpz(3), 4)
mpz(0)

This is what fails for python-flint under PyPy:

>>>> import flint
>>>> pow(2, flint.fmpz(3), 4)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: operands do not support pow()

[oscarbenjamin]

I'm not sure how easy it is to support the 3-arg pow under PyPy.

I think that we just have to add an exception for this in the test suite. Either PyPy and CPython need to find a compatible way of supporting the overload of 3-arg pow or it just is not possible to make this work under PyPy.

Another possibility would be to just make it always a TypeError to do e.g. pow(int, fmpz, int) so that behaviour is at least consistent across PyPy and CPython.

[oscarbenjamin]

there is a new test failure under PyPy:

This needs investigating before releasing 0.5.0.

[oscarbenjamin]

The new test failure is basically this:

In [14]: p = flint.fmpz_mod_poly_ctx(3)(0)

In [15]: p
Out[15]: 0

In [16]: p.leading_coefficient()
---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
Cell In[16], line 1
----> 1 p.leading_coefficient()

File ~/current/active/python-flint/src/flint/types/fmpz_mod_poly.pyx:893, in flint.types.fmpz_mod_poly.fmpz_mod_poly.leading_coefficient()
    891         fmpz_mod(3, 163)
    892     """
--> 893     return self[self.degree()]
    894 
    895 def reverse(self, degree=None):

IndexError: %T index out of range

Running the python code shown in the traceback gives:

In [18]: p.degree()
Out[18]: -1

In [19]: p[-1]
Out[19]: fmpz_mod(0, 3)

In [17]: p[p.degree()]
Out[17]: fmpz_mod(0, 3)

So p[p.degree()] works but the leading_coefficient method that does self[self.degree()] fails.

This is either a Cython bug or a PyPy bug. Minimal reproducer without python-flint:

# polymod.pyx

cdef class poly:

    def __len__(self):
        return 0

    cpdef long degree(self):
        return -1

    def __getitem__(self, key):
        if key < 0:
            return 0

    def leading_coefficient(self):
        return self[self.degree()]

Then under PyPy:

>>>> import polymod
>>>> p = polymod.poly()
>>>> print(p.degree())
-1
>>>> print(p[-1])
0
>>>> print(p[p.degree()])
0
>>>> print(p.leading_coefficient())
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "polymod.pyx", line 14, in polymod.poly.leading_coefficient
    return self[self.degree()]
IndexError: %T index out of range