Extra borg arguments with white space raise an error

[goebbe]

A Vorta user had a hard time when pattern matching a folder-name with white space: #2156 (reply in thread)
Since matching names that contain a single space is a common use case, I decided to open this issue.

I am able to reproduce the issues, using Vorta 0.10.1 with borg 1.4 on LinuxMint22 (based on Ubuntu 24.4)

The issue appears e.g. when trying to pattern match an item name (Folder/ File) that contains white space,
via Vortas "Extra borg arguments for borg create" found at Vorta>Schedule>Shell Commands>Extra arguments for borg create.

To reproduce, I created a folder named test folder, with a single space in the middle of the folder-name.
I used the following --pattern option matcher and tried to exclude the folder from my backup by passing:
--pattern=-home/User/test\ folder

This led to an error message: borg create:error: argument ARCHIVE: "folder": No archive specified.

This is the borg command from the Vorta log files:
/usr/bin/borg create --remote-path=/usr/local/bin/borg --list --progress --info --log-json --json --filter=AM -C lz4 --pattern=-home/User/test\ folder ssh://backupUser@192.168.0.5:22/~/backup/vorta-repo-EB845::User-EliteBook-845-2024-12-06-110323 /home/User

Since this borg command in the log-files seems to be correct, I presumed, that his must be a borg issue and opened an issue on borg: borgbackup/borg#8578

However, it appears that the borg command (from the Vorta logs) actually work as expected, when copied and executed directly on the command line.

To summarize the issue:

The following syntax works for matching test folder with a white space, when using borg directly on the command line:

$ borg create  --pattern=-subdir/test\ folder  repo::space subdir
$ borg create  '--pattern=-subdir/test folder'  repo::space subdir
$ borg create  --pattern='-subdir/test folder'  repo::space subdir

However, when passing these valid options using Vorta (Extra borg arguments) an error is raised:

--pattern=-subdir/test\ folder
'--pattern=-subdir/test folder'
--pattern='-subdir/test folder'

The error messages differ slightly, depending on which of the three options are used with Vorta.
But while each of these option works on the command line, none of these options could be used successfully when passed/ applied via Vorta.

Extract from the Vorta log files when escaping the white space by \:

2024-12-06 11:03:23,454 - vorta.borg.jobs_manager - DEBUG - Start job on site: 5
2024-12-06 11:03:23,473 - vorta.borg.borg_job - INFO - Running command /usr/bin/borg create --remote-path=/usr/local/bin/borg --list --progress --info --log-json --json --filter=AM -C lz4 --pattern=-home/User/test\ folder  ssh://backupUser@192.168.0.5:22/~/backup/vorta-repo-EB845::User-EliteBook-845-2024-12-06-110323 /home/User
2024-12-06 11:03:23,560 - vorta.borg.borg_job - WARNING - usage: borg create [-h] [--critical] [--error] [--warning] [--info] [--debug]
2024-12-06 11:03:23,560 - vorta.borg.borg_job - WARNING - [--debug-topic TOPIC] [-p] [--iec] [--log-json]
2024-12-06 11:03:23,560 - vorta.borg.borg_job - WARNING - [--lock-wait SECONDS] [--bypass-lock] [--show-version]
2024-12-06 11:03:23,560 - vorta.borg.borg_job - WARNING - [--show-rc] [--umask M] [--remote-path PATH]
2024-12-06 11:03:23,562 - vorta.borg.borg_job - WARNING - [--remote-ratelimit RATE] [--upload-ratelimit RATE]
2024-12-06 11:03:23,562 - vorta.borg.borg_job - WARNING - [--remote-buffer UPLOAD_BUFFER]
2024-12-06 11:03:23,562 - vorta.borg.borg_job - WARNING - [--upload-buffer UPLOAD_BUFFER] [--consider-part-files]
2024-12-06 11:03:23,562 - vorta.borg.borg_job - WARNING - [--debug-profile FILE] [--rsh RSH] [-n] [-s] [--list]
2024-12-06 11:03:23,562 - vorta.borg.borg_job - WARNING - [--filter STATUSCHARS] [--json] [--no-cache-sync]
2024-12-06 11:03:23,562 - vorta.borg.borg_job - WARNING - [--stdin-name NAME] [--stdin-user USER]
2024-12-06 11:03:23,562 - vorta.borg.borg_job - WARNING - [--stdin-group GROUP] [--stdin-mode M]
2024-12-06 11:03:23,562 - vorta.borg.borg_job - WARNING - [--content-from-command] [--paths-from-stdin]
2024-12-06 11:03:23,562 - vorta.borg.borg_job - WARNING - [--paths-from-command] [--paths-delimiter DELIM]
2024-12-06 11:03:23,562 - vorta.borg.borg_job - WARNING - [-e PATTERN] [--exclude-from EXCLUDEFILE]
2024-12-06 11:03:23,562 - vorta.borg.borg_job - WARNING - [--pattern PATTERN] [--patterns-from PATTERNFILE]
2024-12-06 11:03:23,562 - vorta.borg.borg_job - WARNING - [--exclude-caches] [--exclude-if-present NAME]
2024-12-06 11:03:23,562 - vorta.borg.borg_job - WARNING - [--keep-exclude-tags] [--exclude-nodump] [-x]
2024-12-06 11:03:23,562 - vorta.borg.borg_job - WARNING - [--numeric-owner] [--numeric-ids] [--noatime] [--atime]
2024-12-06 11:03:23,562 - vorta.borg.borg_job - WARNING - [--noctime] [--nobirthtime] [--nobsdflags] [--noflags]
2024-12-06 11:03:23,562 - vorta.borg.borg_job - WARNING - [--noacls] [--noxattrs] [--sparse] [--files-cache MODE]
2024-12-06 11:03:23,562 - vorta.borg.borg_job - WARNING - [--read-special] [--comment COMMENT]
2024-12-06 11:03:23,562 - vorta.borg.borg_job - WARNING - [--timestamp TIMESTAMP] [-c SECONDS]
2024-12-06 11:03:23,562 - vorta.borg.borg_job - WARNING - [--chunker-params PARAMS] [-C COMPRESSION]
2024-12-06 11:03:23,562 - vorta.borg.borg_job - WARNING - ARCHIVE [PATH ...]
2024-12-06 11:03:23,562 - vorta.borg.borg_job - WARNING - borg create: error: argument ARCHIVE: "folder": No archive specified
2024-12-06 11:03:23,690 - vorta.borg.jobs_manager - DEBUG - Finish job for site: 5

[goebbe]

Here is the extract from the log file when using:
--pattern='-subdir/test folder'

Extract from the log file:

2024-12-06 11:31:12,710 - vorta.borg.jobs_manager - DEBUG - Start job on site: 5
2024-12-06 11:31:12,730 - vorta.borg.borg_job - INFO - Running command /usr/bin/borg create --remote-path=/usr/local/bin/borg --list --progress --info --log-json --json --filter=AM -C lz4 --pattern='-subdir/test folder' ssh://backpUser@192.168.0.5:22/~/backup/vorta-repo-EB845::User-EliteBook-845-2024-12-06-113112 /home/User
2024-12-06 11:31:12,827 - vorta.borg.borg_job - WARNING - A pattern/command must start with any of: -, !, +, R, r, P, p
2024-12-06 11:31:12,955 - vorta.borg.jobs_manager - DEBUG - Finish job for site: 5

Again, when copying the "Running command" to the terminal, the execution went fine.

[goebbe]

I try to understand the source of the issue and collect some insights, here.
Please add comments to confirm or explain, if my interpretation is wrong.

First, in vora/borg/create.py line 102-106

            s1, sep, s2 = profile.repo.create_backup_cmd.partition('-- ')
            extra_cmd_options = s1.split()
            suffix_command = (sep + s2).split()

Here, the .split() function splits item-names with white-space into two parts:

s1 = "--pattern=-subdir/test\ folder"
x = s1.split()
print(x)
$ ['--pattern=-subdir/test\\', 'folder']

Note that s1 will be part of the cmd list that contains all command line options in the form of a list.

Now with respect to the (correct) borg command, that is printed in the log-file:
In the file vorta/borg/borg_job.py, line 231:
logger.info('Running command %s', ' '.join(self.cmd))
This line will print the command e.g. borg create and then append all parameters from the cmd list, separated by a white space, to the log files.
In our example, this would lead to the following command in the log files (which is correctly escaping the white space):
borg create --pattern=-subdir/test\ folder

Next, in line 233, the cmd is passed to Pipe open (Popen). cmd is just a list of arguments (i.e.strings):

        p = Popen(
            self.cmd,
            stdout=PIPE,
            stderr=PIPE,
            bufsize=1,
            universal_newlines=True,
            env=self.env,
            cwd=self.cwd,
            start_new_session=True,
        )

Here, Vorta passes the list of options to the Pipe. (to get the command executed(?)

Current working hypothesis:
The cmd options are passed to borg in form of a list of arguments - and borg is checking each item of the list separately, which leads to an error, since in our example Vorta passes: ['--pattern=-subdir/test\\', 'folder'] and 'folder' is not a valid borg option/ parameter.

[goebbe]

The likely solution is to use of shlex.split() instead of split(). This will split the commands correctly (and preserve the whitespace in the item name):

import shlex
s1 = "--pattern=-subdir/test\ folder"
x = shlex.split(s1, posix=True)
print(x)
$ ['--pattern=-subdir/test folder']

The resulting cmd can likely be passed directly to Popen (using the default Shell=False option).

However, the resulting cmd cannot be used directly to construct the corresponding borg shell-command (for the log files), since the white space would not be escaped.

Perhaps one way to construct the borg-shell-command would be to simply replace the whitespace by an escaped whitespace, before joining the resulting arguments, as follows?

    cmd1 = [s.replace(" ", "\\ ") for s in cmd]
    logger.info('Running command %s', ' '.join(self.cmd1))

[m3nu]

Should be fixed via #2171 . Thanks to @goebbe for stepping up!