Cannot use path names containing dollar signs

[zooba]

(Migrated from https://bugs.python.org/issue33193 - the below text is from a reply with more useful context than the OP)

An exotic case, but it also affects Linux:

python3.7 -m venv 'at$test'
Error: Command '['/home/maier/at$test/bin/python3.7', '-Im', 'ensurepip', '--upgrade', '--default-pip']' returned non-zero exit status 2.
[maier@nb19 ~]$ mkdir 'at$test'
mkdir: cannot create directory ‘at$test’: File exists
[maier@nb19 ~]$ cd 'at$test'
[maier@nb19 at$test]$ bin/python -m ensurepip
Collecting setuptools
Collecting pip
Installing collected packages: setuptools, pip
Exception:
Traceback (most recent call last):
  File "/usr/lib64/python3.7/distutils/util.py", line 187, in subst_vars
    return re.sub(r'\$([a-zA-Z_][a-zA-Z_0-9]*)', _subst, s)
  File "/usr/lib64/python3.7/re.py", line 198, in sub
    return _compile(pattern, flags).sub(repl, string, count)
  File "/usr/lib64/python3.7/distutils/util.py", line 184, in _subst
    return os.environ[var_name]
  File "/usr/lib64/python3.7/os.py", line 678, in __getitem__
    raise KeyError(key) from None
KeyError: 'test'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/tmp/tmpppaapmyk/pip-9.0.1-py2.py3-none-any.whl/pip/basecommand.py", line 215, in main
    status = self.run(options, args)
  File "/tmp/tmpppaapmyk/pip-9.0.1-py2.py3-none-any.whl/pip/commands/install.py", line 342, in run
    prefix=options.prefix_path,
  File "/tmp/tmpppaapmyk/pip-9.0.1-py2.py3-none-any.whl/pip/req/req_set.py", line 784, in install
    **kwargs
  File "/tmp/tmpppaapmyk/pip-9.0.1-py2.py3-none-any.whl/pip/req/req_install.py", line 851, in install
    self.move_wheel_files(self.source_dir, root=root, prefix=prefix)
  File "/tmp/tmpppaapmyk/pip-9.0.1-py2.py3-none-any.whl/pip/req/req_install.py", line 1064, in move_wheel_files
    isolated=self.isolated,
  File "/tmp/tmpppaapmyk/pip-9.0.1-py2.py3-none-any.whl/pip/wheel.py", line 247, in move_wheel_files
    prefix=prefix,
  File "/tmp/tmpppaapmyk/pip-9.0.1-py2.py3-none-any.whl/pip/locations.py", line 153, in distutils_scheme
    i.finalize_options()
  File "/usr/lib64/python3.7/distutils/command/install.py", line 307, in finalize_options
    self.expand_basedirs()
  File "/usr/lib64/python3.7/distutils/command/install.py", line 486, in expand_basedirs
    self._expand_attrs(['install_base', 'install_platbase', 'root'])
  File "/usr/lib64/python3.7/distutils/command/install.py", line 480, in _expand_attrs
    val = subst_vars(val, self.config_vars)
  File "/usr/lib64/python3.7/distutils/util.py", line 189, in subst_vars
    raise ValueError("invalid variable '$%s'" % var)
ValueError: invalid variable '$'test''

So the venv actually gets created, but it's ensurepip which chokes on the $.

[zooba]

Presumably the "real" fix here for CPython will be in pip and will eventually be adopted into ensurepip, but the underlying issue belongs to distutils.

[jaraco]

This issue might have been fixed incidentally in #23, except that backward compatibility for $ substitutions was retained.

[jaraco]

But more important is that literal characters in the path should never trigger substitutions.

[jaraco]

I started to explore how it might be possible to allow these characters to pass without triggering substitutions, but then I realized that it's currently possible for someone to pass values that the intend to be substituted, like:

setup.py install --install-purelib '$userbase/foo' ...
# or
setup.py install --install-purelib '{userbase}/foo' ...

And in that case, distutils would substitute the userbase value just as it does for sysconfig-supplied parameters.

I'm not sure if that's a supported use-case or an incidental behavior that should be disallowed.

If use-cases demand that substitutions be allowed for supplied parameters (and not just those from sysconfig or defaults), then it will be impossible to distinguish $userbase (literal) from $userbase (intended substitution) when supplied.

I think it's going to be difficult to solve this until we can have this codebase widely adopted, so I'm going to defer this until after Setuptools is using it by default.

[smartYSC]

We are seeing this issue as well. But we made the (questionable) decision to use a $ sign in the user name of the executing user (on Windows).

To work around the issue decribed above we would set the environment variable test to $test. This works, because the substitution is only executed once and not recursively. Then it finds the variable and substitutes it with the original string, effectively doing nothing.