Download Quoted Tweet to Appropriate Users Folder?

[Xylemon]

Hi, as an archivist, I try to crawl every single tweet from a specified user on Twitter, this includes anytime they quote a tweet and the quoted tweet itself. The issue I'm having is that both tweets will save in the same folder. Is there a way I can place the quoted tweet in that author's folder automatically with gallery-dl? Currently, I run gallery-dl through a script, and this is the command my script generally will spit out:

gallery-dl -D /path/to/folder/GameSack -o archive=/path/to/folder/GameSack/urls.db -o skip=abort:10 --write-info-json https://twitter.com/GameSack/status/1532221645376475137

My config file looks like this:

{
    "extractor":
    {
        "base-directory": "./",
        "parent-directory": false,
        "#": "set global archive file for all extractors",
        "archive": null,
        "cookies": null,
        "cookies-update": true,
        "proxy": null,
        "skip": true,

        "user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:91.0) Gecko/20100101 Firefox/91.0",
        "retries": 4,
        "timeout": 30.0,
        "verify": true,
        "fallback": true,

        "sleep": 0,
        "sleep-request": 0,
        "sleep-extractor": 0,

        "#": "replace invalid path characters with unicode alternatives",
        "path-restrict": "auto",
        "path-replace": "_",
        "path-remove": "\\u0000-\\u001f\\u007f",
        "path-strip": "auto",
        "extension-map": {
            "jpeg": "jpg",
            "jpe" : "jpg",
            "jfif": "jpg",
            "jif" : "jpg",
            "jfi" : "jpg"
        },

        "#": "write plaintext descriptions, and json metadata",
        "postprocessors": [
            {
                "name": "mtime",
                "event": "post"
            },
            {
                "name": "metadata",
                "mode": "json",
                "filename": "{post_id|tweet_id|id}.json",
                "mtime": true
            },
            {
                "name" : "metadata",
                "event": "post",
                "filename": "{post_id|tweet_id|id}.txt",

                "#": "write only the values for 'content' or 'description'",
                "mode" : "custom",
                "format": "{content|description}\n",
                "mtime": true
            }
        ],

        "twitter":
        {
            "username": null,
            "password": null,
            "cards": true,
            "conversations": false,
            "pinned": true,
            "quoted": true,
            "replies": "self",
            "retweets": false,
            "syndication": false,
            "text-tweets": true,
            "twitpic": true,
            "users": "timeline",
            "videos": true,
            "postprocessors": [
            {
                "name": "mtime",
                "event": "post"
            },
            {
                "name": "metadata",
                "event": "post",
                "filename": "{tweet_id}.json",
                "mtime": true
            }
        ]
        },
        "ytdl":
        {
            "enabled": false,
            "format": null,
            "generic": true,
            "logging": true,
            "module": "yt_dlp",
            "raw-options": null
        }
    },

    "downloader":
    {
        "#": "restrict download speed to 1 MB/s",
        "rate": "1M",

        "#": "show download progress indicator after 2 seconds",
        "progress": 2.0,

        "#": "retry failed downloads up to 3 times",
        "retries": 3,

        "#": "consider a download 'failed' after 8 seconds of inactivity",
        "timeout": 8.0,

        "#": "write '.part' files into a special directory",
        "part-directory": "/tmp/.download/"
    },

        "output":
    {
        "mode": "auto",
        "progress": true,
        "shorten": true,
        "colors": {
            "success": "1;32",
            "skip"   : "2"
        },
        "skip": true,
        "log": "[{name}][{levelname}] {message}",
        "logfile": null,
        "unsupportedfile": null
    },

    "netrc": false
}

I'm assuming I'll have to change the config and my script a little bit, which I'm open to, but I'm just unsure of what to change. I would like to have different download directories for different sites.

[mikf]

Do not use -D if you want to download into different directories. -D forces all files to be downloaded into that specific directory.

Use -d (lower case) or base-directory, together with appropriate directory format strings.

This will split Tweet downloads into separate directories depending on retweet/quote status:

            "directory": {
                "retweet_id"              : ["{category}", "{user[name]}", "Retweets", "{author[name]}"],
                "locals().get('quote_by')": ["{category}", "{user[name]}", "Quoted"  , "{author[name]}"],
                ""                        : ["{category}", "{user[name]}"]
            }

[Xylemon]

Thank you for your response, this is almost what I want, but not quite:

Do not use -D if you want to download into different directories. -D forces all files to be downloaded into that specific directory.

Use -d (lower case) or base-directory, together with appropriate directory format strings.

Ah ok, I used '-d' but gallery-dl always wants to append the site name, so if the base dir was "/home/user/Downloads" the tweets would always go in a sub-folder called "twitter". Am I able to change the site folder name behavior?

This will split Tweet downloads into separate directories depending on retweet/quote status:

            "directory": {
                "retweet_id"              : ["{category}", "{user[name]}", "Retweets", "{author[name]}"],
                "locals().get('quote_by')": ["{category}", "{user[name]}", "Quoted"  , "{author[name]}"],
                ""                        : ["{category}", "{user[name]}"]
            }

This is good, but it puts the quoted user (let's just call them user2) in the same folder as user1, so I get "twitter/user1/Quoted/user2/tweetid.json" but what I would really want is the tweets in separate user folders like so:

twitter/user1/tweetid.json
twitter/user2/tweetid.json

Basically, the quoted tweet would just be stored in the appropriate user's folder like any other tweet that was downloaded.

[mikf]

Am I able to change the site folder name behavior?

This is what "{category}" in the directory format strings I posted results in. Just remove it if you don't want it.

This is good, but it puts the quoted user (let's just call them user2) in the same folder as user1, so I get "twitter/user1/Quoted/user2/tweetid.json" but what I would really want is the tweets in separate user folders like so:

twitter/user1/tweetid.json
twitter/user2/tweetid.json

Basically, the quoted tweet would just be stored in the appropriate user's folder like any other tweet that was downloaded.

Then change it accordingly. Take my example and play with it until it produces what you want.
In this case just "directory": ["{author[name]}"] should be enough.

[Hrxn]

Couldn't test it here yet, but maybe try this?

            "directory": {
                "retweet_id"              : ["{category}", "{user[name]}", "Retweets", "{author[name]}"],
                "user != author"          : ["{category}", "{user[name]}", "Quoted"  , "{author[name]}"],
                ""                        : ["{category}", "{user[name]}"]
            }

I.e. with "user != author" instead of "locals().get('quote_by')"

[Xylemon]

Thank you both for your response, I tried @mikf's suggestion and was able to get gallery-dl doing exactly what I wanted. I actually tried this a couple days ago, but I was very busy with life so apologies for not responding sooner. My twitter snippet from my config:

            "directory": {
                "retweet_id"              : ["{author[name]}"],
                "locals().get('quote_by')": ["{author[name]}"],
                ""                        : ["{author[name]}"]
            },

Thanks again, gonna mark as solved.