From 0d2ebd8800b09128d86de63f88ef1d8658b1aa44 Mon Sep 17 00:00:00 2001 From: HelpSeeker Date: Thu, 30 Jan 2020 20:48:54 +0100 Subject: [PATCH 01/40] Add basic GUI This is supposed to make CoubDownloader more accessible for people, who are not comfortable using the command line. I can't stress enough though that it is only very basic. CoubDownloader is first and foremost a CLI tool. Any graphical frontend will always be an addition and not the main focus. I also make it a rule not to change the main script to accommodate to the GUI. If an option doesn't make sense with a GUI, it's coub-gui.py's job to either find a way around it or make sure it doesn't interfere with the execution. The README.md outlines the small differences regarding functionality, so I refrain from repeating them here. I'm also aware that this is only a small step in the right direction. People still have to go through the trouble of installing Python and some 3rd party packages to fully utilize CoubDownloader. --- .gitignore | 4 +- README.md | 62 ++++--- coub-gui.py | 253 ++++++++++++++++++++++++++++ images/coub-gui_execution_Linux.png | Bin 0 -> 12998 bytes images/coub-gui_input_Windows.png | Bin 0 -> 10421 bytes 5 files changed, 296 insertions(+), 23 deletions(-) create mode 100755 coub-gui.py create mode 100644 images/coub-gui_execution_Linux.png create mode 100644 images/coub-gui_input_Windows.png diff --git a/.gitignore b/.gitignore index 38d1301..258b66e 100644 --- a/.gitignore +++ b/.gitignore @@ -1,6 +1,8 @@ -* +/* !.gitignore !LICENSE !README.md !coub.py +!coub-gui.py +!/images/ diff --git a/README.md b/README.md index f52fea1..b7618d0 100644 --- a/README.md +++ b/README.md @@ -4,25 +4,26 @@ CoubDownloader is a simple script to download videos (called coubs) from [Coub]( ## Contents -1. [Usage](https://github.com/HelpSeeker/CoubDownloader#usage) -2. [Requirements](https://github.com/HelpSeeker/CoubDownloader#requirements) -2.1 [Optional](https://github.com/HelpSeeker/CoubDownloader#optional) -3. [Input](https://github.com/HelpSeeker/CoubDownloader#input) -3.1. [Overview](https://github.com/HelpSeeker/CoubDownloader#overview) -3.2. [Direct coub links](https://github.com/HelpSeeker/CoubDownloader#direct-coub-links) -3.3. [Lists](https://github.com/HelpSeeker/CoubDownloader#lists) -3.4. [Channels](https://github.com/HelpSeeker/CoubDownloader#channels) -3.5. [Searches](https://github.com/HelpSeeker/CoubDownloader#searches) -3.6. [Random](https://github.com/HelpSeeker/CoubDownloader#random) -3.7. [Tags](https://github.com/HelpSeeker/CoubDownloader#tags) -3.8. [Communities](https://github.com/HelpSeeker/CoubDownloader#communities) -3.9. [Hot section](https://github.com/HelpSeeker/CoubDownloader#hot-section) -4. [Misc. information](https://github.com/HelpSeeker/CoubDownloader#misc-information) -4.1. [Video resolution vs. quality](https://github.com/HelpSeeker/CoubDownloader#video-resolution-vs-quality) -4.2. [AAC audio](https://github.com/HelpSeeker/CoubDownloader#aac-audio) -4.3. ['share' videos](https://github.com/HelpSeeker/CoubDownloader#share-videos) -5. [Changes since Coub's database upgrade (watermark & co)](https://github.com/HelpSeeker/CoubDownloader#changes-since-coubs-database-upgrade-watermark--co) -6. [Changes since switching to Coub's API (previously used youtube-dl)](https://github.com/HelpSeeker/CoubDownloader#changes-since-switching-to-coubs-api-previously-used-youtube-dl) +1. [Usage](#usage) +2. [Requirements](#requirements) + 2.1 [Optional](#optional) +3. [Input](#input) + 3.1. [Overview](#overview) + 3.2. [Direct coub links](#direct-coub-links) + 3.3. [Lists](#lists) + 3.4. [Channels](#channels) + 3.5. [Searches](#searches) + 3.6. [Random](#random) + 3.7. [Tags](#tags) + 3.8. [Communities](#communities) + 3.9. [Hot section](#hot-section) +4. [GUI](#gui) +5. [Misc. information](#misc-information) + 5.1. [Video resolution vs. quality](#video-resolution-vs-quality) + 5.2. [AAC audio](#aac-audio) + 5.3. ['share' videos](#share-videos) +6. [Changes since Coub's database upgrade (watermark & co)](#changes-since-coubs-database-upgrade-watermark--co) +7. [Changes since switching to Coub's API (previously used youtube-dl)](#changes-since-switching-to-coubs-api-previously-used-youtube-dl) ## Usage @@ -118,6 +119,7 @@ Output: * [aiohttp](https://aiohttp.readthedocs.io/en/stable/) for asynchronous execution **(recommended)** * [colorama](https://github.com/tartley/colorama) for colorized terminal output on Windows +* [Gooey](https://github.com/chriskiehl/Gooey) to run `coub-gui.py` ## Input @@ -364,6 +366,21 @@ refer to the hot section and can be used as input. The default sort order (most popular coubs of the month) may provide less results than other sort orders. +## GUI + +A basic GUI, powered by [Gooey](https://github.com/chriskiehl/Gooey), is provided via `coub-gui.py`. + +![Settings window on Windows](/images/coub-gui_input_Windows.png) ![Progress window on Linux](/images/coub-gui_execution_Linux.png) + +It provides the same functionality as the main CLI tool, with a few notable exceptions: + +* No quiet mode +* No overwrite prompt (default prompt answer is set to "no") +* No option equivalent to `--random#top` (direct URL input must be used) +* The output path defaults to "coubs" in the user's home directory instead of the current one + +Another important difference is that `coub-gui.py` is **NOT** a standalone script. It depends on `coub.py` being in the same location. + ## Misc. information ### Video resolution vs. quality @@ -453,7 +470,7 @@ There's no fallback for *share* videos. If the *share* version is not yet availa Coub started to massively overhaul their database and API. Of course those changes aren't documented (why would you document API changes anyway?). - [x] Only repair video streams that are actually broken -- [x] Remove mobile option (they now come with a watermark and are the exact same as html5 med) +- [x] Remove mobile option (they now come with a watermark and are the exact same as html5 med) - [x] Add AAC mobile audio as another possible audio version (ranked between low and high quality MP3 audio) - [x] Add options to prefer AAC or only download AAC audio - [x] Add shared option (video+audio already combined) @@ -462,7 +479,7 @@ Coub started to massively overhaul their database and API. Of course those chang - [x] Asynchronous coub processing - [x] Asynchronous timeline parsing - [x] Detect stream corruption (incl. old Coub storage method) -- [x] Workspace cleanup (incomplete coubs) after user interrupt +- [x] Workspace cleanup (incomplete coubs) after user interrupt - [x] Colorized terminal output - [x] Download retries - [x] URL input without input options @@ -471,6 +488,7 @@ Coub started to massively overhaul their database and API. Of course those chang - [x] Support for sort order related URLs - [x] Download random coubs - [x] Option to change the container format for stream remuxing +- [x] Basic GUI frontend ## Changes since switching to Coub's API (previously used youtube-dl) @@ -481,7 +499,7 @@ Coub started to massively overhaul their database and API. Of course those chang - [x] ~~Limit download speed~~ (was only possible in the Bash version) - [x] Download all coubs with a certain tag - [x] Check for the existence of a coub before downloading -- [x] Specify max. coub duration (FFmpeg syntax) +- [x] Specify max. coub duration (FFmpeg syntax) - [x] Keep track of already downloaded coubs - [x] Export parsed coub links (from channels or tags) to a file for later usage - [x] Different verbosity levels diff --git a/coub-gui.py b/coub-gui.py new file mode 100755 index 0000000..4fff28c --- /dev/null +++ b/coub-gui.py @@ -0,0 +1,253 @@ +#!/usr/bin/env python3 + +import os +from textwrap import dedent + +from gooey import Gooey, GooeyParser + +import coub + + +class GuiDefaultOptions(coub.DefaultOptions): + """Custom default option class to reflect the differences between CLI and GUI.""" + # There's no way for the user to enter input if a prompt occurs + # So only "yes" or "no" make sense + PROMPT = "no" + + # Outputting to the current dir is a viable strategy for a CLI tool + # Not so much for a GUI + if coub.DefaultOptions.PATH == ".": + PATH = os.path.join(os.path.expanduser("~"), "coubs") + else: + PATH = os.path.abspath(coub.DefaultOptions.PATH) + + # "%id%" by itself gets replaced by None anyway and it's less confusing + # than just showing None as default value + # This might be worth changing in the main script as well + if not coub.DefaultOptions.OUT_FORMAT: + OUT_FORMAT = "%id%" + + # Create special labels for dropdown menus + # Some internally used values would cause confusion + # Some menus also combine options + QUALITY_LABEL = ["Worst quality", "Best quality"] + AAC_LABEL = ["Only MP3", "No Bias", "Prefer AAC", "Only AAC"] + RECOUB_LABEL = { + (True, False): "With Recoubs", + (False, False): "No Recoubs", + (True, True): "Only Recoubs", + } + SPECIAL_LABEL = { + (True, False, False): "Share", + (False, True, False): "Video only", + (False, False, True): "Audio only", + (False, False, False): None, + } + + +def translate_to_cli(options): + """Make GUI-specific options object compatible with the main script.""" + # Special dropdown menu labels and what they translate to + QUALITY_LABEL = {"Worst quality": 0, "Best quality": -1} + AAC_LABEL = {"Only MP3": 0, "No Bias": 1, "Prefer AAC": 2, "Only AAC": 3} + RECOUB_LABEL = { + "With Recoubs": (True, False), + "No Recoubs": (False, False), + "Only Recoubs": (True, True), + } + SPECIAL_LABEL = { + "Share": (True, False, False), + "Video only": (False, True, False), + "Audio only": (False, False, True), + None: (False, False, False), + } + + # Convert GUI labels to valid options for the main script + options.v_quality = QUALITY_LABEL[options.v_quality] + options.a_quality = QUALITY_LABEL[options.a_quality] + options.aac = AAC_LABEL[options.aac] + options.recoubs, options.only_recoubs = RECOUB_LABEL[options.recoubs] + options.share, options.v_only, options.a_only = SPECIAL_LABEL[options.special] + + return options + + +@Gooey( + program_name="CoubDownloader", + default_size=(800, 600), + progress_regex=r"^\[\s*(?P\d+)\/(?P\d+)\](.*)$", + progress_expr="current / total * 100", + tabbed_groups=True, + show_success_modal=False, + show_failure_modal=False, + hide_progress_msg=False, + terminal_font_family="monospace", # didn't work when I tested it on Windows +) +def parse_cli(): + """Create Gooey GUI.""" + defs = GuiDefaultOptions() + parser = GooeyParser( + description="Download videos from coub.com", + usage="%(prog)s [OPTIONS] INPUT [INPUT]..." + ) + + # Input + input_ = parser.add_argument_group( + "Input", + description="Specify various input sources\n\n" + "All input fields support several items (i.e. names, IDs, " + "tags, etc.). Items must be comma-separated.", + gooey_options={'columns': 1} + ) + input_.add_argument("--urls", default="", metavar="Direct URLs", + help="Provide direct URL input") + input_.add_argument("--ids", default="", metavar="Coub IDs", + help="Download coubs with the given IDs") + input_.add_argument("--channels", default="", metavar="Channels", + help="Download channels with the given names") + input_.add_argument("--recoubs", metavar="Recoubs", + default=defs.RECOUB_LABEL[(defs.RECOUBS, defs.ONLY_RECOUBS)], + choices=["With Recoubs", "No Recoubs", "Only Recoubs"], + help="How to treat recoubs during channel downloads") + input_.add_argument("--tags", default="", metavar="Tags", + help="Download coubs with at least one of the given tags") + input_.add_argument("--searches", default="", metavar="Search Terms", + help="Download search results for the given terms") + input_.add_argument("--communities", default="", metavar="Communities", + help="Download coubs from the given communities") + input_.add_argument("--lists", default="", widget="MultiFileChooser", + metavar="Link Lists", help="Read coub links from input lists", + gooey_options={'message': "Choose link lists"}) + input_.add_argument("--random", action="count", metavar="Random", + help="Download N*1000 randomly generated coubs") + input_.add_argument("--hot", action="store_true", widget="BlockCheckbox", + metavar="Hot Section", help="Download coubs from the hot section") + + # Common Options + common = parser.add_argument_group("General", gooey_options={'columns': 1}) + common.add_argument("--prompt", choices=["yes", "no"], default=defs.PROMPT, + metavar="Prompt Behavior", help="How to answer user prompts") + common.add_argument("--repeat", type=coub.positive_int, default=defs.REPEAT, + metavar="Loop Count", help="How often to loop the video stream") + common.add_argument("--dur", type=coub.valid_time, default=defs.DUR, + metavar="Limit duration", + help="Max. duration of the output (FFmpeg syntax)") + common.add_argument("--preview", default=defs.PREVIEW, metavar="Preview Command", + help="Command to invoke to preview each finished coub") + common.add_argument("--archive-path", type=coub.valid_archive, + default=defs.ARCHIVE_PATH, widget="FileSaver", + metavar="Archive", gooey_options={'message': "Choose archive file"}, + help="Use an archive file to keep track of already downloaded coubs") + common.add_argument("--keep", action=f"store_{'false' if defs.KEEP else 'true'}", + widget="BlockCheckbox", metavar="Keep streams", + help="Whether to keep the individual streams after merging") + + # Download Options + download = parser.add_argument_group("Download", gooey_options={'columns': 1}) + download.add_argument("--connect", type=coub.positive_int, + default=defs.CONNECT, metavar="Number of connections", + help="How many connections to use (>100 not recommended)") + download.add_argument("--retries", type=int, default=defs.RETRIES, + metavar="Retry Attempts", + help="How often to reconnect to Coub after connection loss " + "(<0 for infinite retries)") + download.add_argument("--max-coubs", type=coub.positive_int, + default=defs.MAX_COUBS, metavar="Limit Quantity", + help="How many coub links to parse") + + # Format Selection + formats = parser.add_argument_group("Format", gooey_options={'columns': 1}) + formats.add_argument("--v-quality", choices=["Best quality", "Worst quality"], + default=defs.QUALITY_LABEL[defs.V_QUALITY], + metavar="Video Quality", help="Which video quality to download") + formats.add_argument("--a-quality", choices=["Best quality", "Worst quality"], + default=defs.QUALITY_LABEL[defs.A_QUALITY], + metavar="Audio Quality", help="Which audio quality to download") + formats.add_argument("--v-max", choices=["med", "high", "higher"], + default=defs.V_MAX, metavar="Max. Video Quality", + help="Cap the max. video quality considered for download") + formats.add_argument("--v-min", choices=["med", "high", "higher"], + default=defs.V_MIN, metavar="Min. Video Quality", + help="Cap the min. video quality considered for download") + formats.add_argument("--aac", default=defs.AAC_LABEL[defs.AAC], + choices=["Only MP3", "No Bias", "Prefer AAC", "Only AAC"], + metavar="Audio Format", help="How much to prefer AAC over MP3") + formats.add_argument("--special", choices=["Share", "Video only", "Audio only"], + default=defs.SPECIAL_LABEL[(defs.SHARE, defs.V_ONLY, defs.A_ONLY)], + metavar="Special Formats", help="Use a special format selection") + + # Output + output = parser.add_argument_group("Output", gooey_options={'columns': 1}) + output.add_argument("--out-file", type=os.path.abspath, widget="FileSaver", + default=defs.OUT_FILE, metavar="Output to List", + gooey_options={'message': "Save link list"}, + help="Save all parsed links in a list (no download)") + output.add_argument("--path", type=os.path.abspath, default=defs.PATH, + widget="DirChooser", metavar="Output Directory", + help="Where to save downloaded coubs", + gooey_options={ + 'message': "Pick output destination", + 'default_path': defs.PATH, + }) + output.add_argument("--merge-ext", default=defs.MERGE_EXT, + metavar="Output Container", + choices=["mkv", "mp4", "asf", "avi", "flv", "f4v", "mov"], + help="What extension to use for merged output files " + "(has no effect if no merge is required)") + output.add_argument("--out-format", default=defs.OUT_FORMAT, + metavar="Name Template", + help=dedent(f"""\ + Change the naming convention of output files + + Special strings: + %id% - coub ID (identifier in the URL) + %title% - coub title + %creation% - creation date/time + %community% - coub community + %channel% - channel title + %tags% - all tags (separated by _) + + Other strings will be interpreted literally + This option has no influence on the file extension + """)) + + # Advanced Options + parser.set_defaults( + verbosity=1, + coubs_per_page=25, # allowed: 1-25 + tag_sep="_", + write_method="w", # w -> overwrite, a -> append + ) + + args = parser.parse_args() + args.input = [] + args.input.extend([coub.mapped_input(u) for u in args.urls.split(",") if u]) + args.input.extend([f"https://coub.com/view/{i}" for i in args.ids.split(",") if i]) + args.input.extend([coub.LinkList(l) for l in args.lists.split(",") if l]) + args.input.extend([coub.Channel(c) for c in args.channels.split(",") if c]) + args.input.extend([coub.Tag(t) for t in args.tags.split(",") if t]) + args.input.extend([coub.Search(s) for s in args.searches.split(",") if s]) + args.input.extend([coub.Community(c) for c in args.communities.split(",") if c]) + if args.hot: + args.input.append(coub.HotSection()) + if args.random: + for _ in range(args.random): + args.input.append(coub.RandomCategory()) + + # Read archive content + if args.archive_path and os.path.exists(args.archive_path): + with open(args.archive_path, "r") as f: + args.archive = [l.strip() for l in f] + else: + args.archive = None + # The default naming scheme is the same as using %id% + # but internally the default value is None + if args.out_format == "%id%": + args.out_format = None + + return translate_to_cli(args) + + +if __name__ == '__main__': + coub.opts = parse_cli() + coub.main() diff --git a/images/coub-gui_execution_Linux.png b/images/coub-gui_execution_Linux.png new file mode 100644 index 0000000000000000000000000000000000000000..d697040c5d8f425b3e39f9eee7239a9c4f2fc18c GIT binary patch literal 12998 zcmZ{K1yoyIvvz<0!Gl|Y;82QFibEh2FJ356+>5&vEfgs36iRW5YjJleT8c|?clST^ zmGArSx;JYP&Y3f__w3ovJTqtQgebj|!a^rQ2LJ$AGSV;=000RB03d3Dkl>QyZw}&c z0YFJ!O_GL{i;{_lOV=YRtEOXOerR;KJU_`p|J75~*zcn=Lk;nYd>kxd?pfoDgHzLe z)n6I8_PU#zgUwKU4Uwym3z^HosoxUf_CcwN}| zaFvaaa7O18JJg&_Sj{oH%S=J6r>!EzMo!7Ez9Y?%h6}3lsr8$ckXLj;w5N$?*mrM5 zem1>K*RqAi5H;(((b&}Jo|51XEo(i|YEPv?q+h(Vvw!dA_D1@ptguePXs)e{>o*ze zBsskRZEfw>+S*Bmd?|s64-XHAhlitczb$-{^(@2X4T8=uu2wepZyz4o`UaO)RvsR% z?3zG&8;P-xm*ALh2#?@z?0Z_xT)!W~>BR3Ck zH5GOr9$GKq1N6J}-OS9)%)?dhw!1g4SK$}uWnt5|8{YuhEn1WlLEM*(gSjG*nuB@)is%34^eECgNdP`G$Nt1s39&`aPbZJh6O1fq7o8 zgM-2Oh7N-ogAuMW@6fWiuwjWCLM^ zDbXS$p`dX1qX`FtM4~mZQC1Vgb6Wl7SxK-_k!K@?D|K;+jB7*rFrr03HTWF7zl?9PfZ+?-%*kP#Owij`FSf{;p^W0R4{MI3(%XAi>K}b zp3g7?xlwU+%13?KQHWIW^sqCB+&?kD>qTfBbmuxk50BeFpW4E_$qLD* z3a)%(XVL@4$&z|nqqI({Ls>Vgbz+ALp^}v!aMq>oCx^YaqJ4uQWpza&{17G1&0Iq; z!62T9+V^f`N#}+o;U`h1pSY}BPTwQ?``RhV6CGwenLZ5`c-^11?@N#mb!abJt^3_e zJwQnofh0~VB%4TKkbjRa13@E^=`$e|j4%6*Ct^6>qM+R+p^c|yq3nSvu1FtUUSb9OjADQ3G z)DwZ5n%+Aj%LoVz>_?P+-kDZj>#co*+8Ot$!EbQRN^-XT^+vB352R6B%r4;UC`RgN zXf^Y~O|08pEg86);@7~4#5VFCVLC=)t7-SlxAq401~o;L9a7unZ>fNZ|7#FS%`1~i zyB1U0;nVjSB6?YYZ$m;e#pR-ov=EuPAr_}QqIVM$NNN|qhA}{tW6;5ui%?jfP`2xg zN8dVilW}uE682b*#21Rm2HdD@p?5c*tZk}GgtI@1U2}tao1S*NdJSScy^66}7_<2< zX$isX9rG3DDOzUVBHz}MGJBI}JVB}c0#W=+txz>>m-67(j0G(ouzKE&JCg#aR}i>* zB1+)zVZL&74jcZ$)rgt<`5#;X0Ii$=ArJzB+l4Ox0s;8``DV-`e0RKiYiN70e7h^^ z>C!gWjMZ)5_#@p~ByoDr&z6!~5fxp?r4KnVnN&*6YqoL-DZZV5WARobkjXCfK)BjC z-W6sILO1S4C~)k`2jE^JA>U2 zVdc?F%a}m@@#W4~h)p<=5d`3o>gE8mZ`y=g4}`eM2KVnRqXS#ZgY&|kF>d&Uy>FYk zECVpXUWK`lF|E^~b$-#0;pOL}SK$g5OVqSD3Hq-9>`7+^)mH+sU;4TqUwvcj@?$JQ>Sq6xhF&1L>;gk+PJEM3{c+gfHr9B}|OaRqY#B{ggtuqBMv$^7e z&B(-hPy*9WEI8FI*RS(SCq9hJXNp)6@#;{lsJ||~BtUFj1N$b)p03e|wMq;yXYt7` zs8>Ly`IEZKH62(KXg|?IHuSaRmj;_LJx#DA;DgZzw{CN~KH*-I!p7IvFQPE$qa^6T zCdJKo-VlJxvlKBym*uz5uJ(9+9PY$c6CcbHUPF%Yc8GNhp;NA#Zy>^bY!jsiy|`R= zDMADmrHbrPzZsg)Uu-FHAbn6dH2y&M8FE}v;HX`6&iOSXhcNhwXfuL57^Hw~-<2Kh zFH@2p3SUDn=iyS@snvI04!)EHVGuStwN;v=jzLlmeGYwyk3s-TuLm`F2|a!TeVF|9 zqRZEkukR%n0=(QHfOZk244I#|!puKYvlEBJWgWB+)+VieZr{A<{L!+GEH+iM$rV+& zlk%aYG)|0e?1U(|al}yNq67d6IHQZ2ZTMjG6==A|3chH5|Mn;5_6!(cP_{CSidS<2 z4)$PNM)S4f!5tO=hZgt&8Yyk0&i(AR-E#Z@aT_#I4nV|*o0KnwBN5~%{w7XW{oi;@ znz;g@Ca0eKAD@_-4s~#FhIM8Eh#mR>INOf zW0e}NeE!^W6*ckneS^xJG3$cRs(SmR6JCv$l#W7C`1SyT){bkK)%Wu2!-m6UPfWx{ z%`Qk@#R8RxUG<(Irb<(u=7m~)rG^$yZ{cz0gH`~D`uQtyOGE28i&fcpVfqWPJ_5ksL>1mAnxRKdu^Gs45vcXyJ!3Ptl_+RN{h;~M*H zY@Dsr=gIG)jjcpLD*oi3GLd#A3*vu{dPtJlXwE@(vRI87Bamx#33nJ8Y(8m;aSCuu zkGNk@yi=Wi3#<;Fs)|iS(n;VzVf`J33Dv3Rk~EGwK1m#FJz3JAIc>m1EV2Y2U2s?% z>;rA6>NbbSDKJXN`NzKS&Uh8Dt>5<0bV@=$PHg8_Eqvo^U54W4l(Ni^+Qv8b@w6O$ ziexCOrYFTYf<~QD3$!9SG9H*0n^VU)>ZkmE86YL@V%Sk~gfx@hM@hIb=gvdjsT77S zU)^?~pq%kq3&<;pk|gMQ+`1sA+%%Tn2O%N#Uy#$t>*bO@C{~@u8~*6i1L23>CmXWT zT@c#op4S!&u!MSARx4nZW%DV9EfOrB>(W#nE#tc`RJ}(3%+%E^F_iKQy74p8tgPCE zbU6URnA)t(_tgZ=oFdxoXus6(P#T1`Lq)}^Y!fG(hHJ?55|yS0tGp{_n%8<}dr`HH zl`J3#`VEO+8o@bK-sK`#P;&X|u+*yjyk5ui?%-9(_mL$$FnK6B0^4L($Tt6EVR&6C z?JJDSnV!|2n`_gj4L`+Vv1m3d^mNxrUrwF!ln~61QsN6RGs73K zK%&r5gtc0IHUpgWPOW2Rl2g2ZUMTuERpSa@{NyJ>RQKT@ol*^(l9$iP1RHjm05ooHmv3QbZK`=`YEqq3!Lq0jeXDNOUWhqAn5)J5))Via)3~pIw1~Gh}#6o^0W1Jq&R8zD8D9>o2+Y+Lo*+} z`Z!rNi~Df1My~3%A>qoq{9(0uoKA5tsF^pj4@LQ;0Q!Wq3*z7pGl+6X@O`~tc(l-~ zbNG77kRJugDPXq`!PBk_&>xY*q(Nm)e-+(_=VAL+Hr7q#R^YaX7~KGjaKo>0QrPl@ zPkD%Kues$!*@M49MlmxOBaL&$%O!~MV2)@Inq{JvwHK;#TmUtG+(&xMVC*vM+2XuS zm8uA7h;TL>TMHjg1+@yTzyw)H8`w1E@q!?faG4D3RS5DpGoFA60^tH_gJO@l$Sod; z=uR<11=LQ(gT6}h;#sO|(+u8>?AIKup|#+Xu7Cxr227r(tc%-JrK{=zyjI>$*0BAd7;aJwe-<{fmHN|^$mfZ0MN9C_Vd0_bVv#wM z=%&Erij(lrNk~(rrWJW`ruh*@e*+gj>i}QD^J=7A=25N&C+)fTP1OE7C8*PyyEhjz z?Bc`6*vpXNFuBUqfuG7iJtfTE+xtN|;~R6H_oEEIS?ths`El|*q)e_b_}*w`7f>m^ zW!M(Q?GBs8$%K2+ry!LhyL7tPAN$K ziBvCHX1R0kcvi7oPA;Vav5>U;{*F)XGjrqM0jb{1SGv{+pa6(kKCQg&-gXNkV<|n( z)9>%^N~@RyU&=Ok<#;j(DI{t(GJc4S&5*95W%Kz>hIO&mYWDL`Z`P(95y-ZPhXPtc z6yU{ViH(its9Ir7M~VS(c27QpZFF5#Q}8q=&KlNj-9?uAc-#rNIX4ACtUAZnQU(Yt z4Q)l;z%T~WV$=R~wL59iTHtJ2UvhHas?o5UTv;FB=uC9+6Mi*?P`xecJ^gUU(-yXq z?NH1%9&x-xoa8i*Z`XbYjCjhpkJh~cc?+(%!yI`BG-+o`2+KP@X~c}yf^JTVLrtD> zz2GMAx9h(MN&UN9*9N#ATz^UY!l8Ug9P$lhqEz5*{AGp6jMN+$>~=bFvb-8wa=6*7 zCq$KceJy_FoUEZ?3|f9dH+(t{H}VpV4h|*BFTKWHc9|K&Ac!D^z$bc(=H_JICVcc# z_iueeoES5g`t6Ku)aI|!FiBjfAQIt)Eo2UEG`)gqwK#O9vY3X(Tun16GsY zo++InEh^Gw(!6}CE-J25Y}gYKDXZU&4K{bM97ucTwS}y;H`!Lm!U1f!fJP-3eypd` zStktn794$E;Qb+tg1TCTy8|(M3T3?dOX*SFbm7H!y$vNTa(`XHUYJ<{Rd%SXq;8hl z9cT+8F5CSPGwS$u!IMPM3b(@FuFXPl=t@eTh*nIXayIg)Dyc|}EV3Y_nBwYM{ns;D z-=cAPG&K96UCce{Biwx+|Ffk4u!`XOpV(NDDY!RIP9P3UWB~XFU^j|iceW7#VbN^g z1n7;4w$5W~L`Z<|v%DSZjZ4mZ*G}o~-+|2=7M7WEoo@bK3h4>m?^fu z_)O&b(J^_Kn@>l4J^Yo5PjvPODILj8BvRcwTas^>q3v937Mk}| zAfqR}^|@HB_oq3zIXR+Xa)~2miIWzZlU9Jza#p5a;s*)(^6&jx*@wQzrYCk&V&iZT zcFYIF@_3;*s5NSuRRwSoqWB9KeH!9GyDq3NjB9mtN(dSjn|SW7uZht;P6FbXW7*81 zM(BzsSXP9Tg{D>_hUIfa4Q$B(1nXIq@F@obHY5H14BQk2k!hoLZeO;o&Nr*k$uREf0yuvm z`wuK+k39_23<3dxh<}S{c{aTv!aP!cGC27iCoQus9nuW$T*b#XQXq0=v$P&kuH=i%2U{-YkeE8HC^J?Ga&@ znwjov?AJ98$W%Z`@#pu&nI}#pe)0q`D(p*zE#oSL#pYsCE9AZ+mk(J^73siZR0mUm z3{$qwpWMk4b#-rtGX|R;GFFOYGF?V5Np^XJc55v=;%>>#`&T&8+>}-dCkoT)@hu9< z%9cPJ*U#xk=poW=fjC1f$#{({tmV3N<9dsUO9N7=EjDPL}uqH zxz?MW3lUYXLpt`i+`d&sqAf(q(z)$zPqu-EkKr1;v>raBa!#hu$R8%6yjqo8bdPD3FIbWUo-n4Rix;9zf ze%I`M=pjS*a7ubi`=leEfeGXb8LCF!ig+H##)&_vMfOhg^75C{Y&~>n>pNAH(clPA z+Q}1%>_m!VnT~+vtut~%j0*chMb#kr86LV^NN(n#B&G)C&3bamfGA`16!FQ;t3|Ff z>D8L&f?8mA|DC=n7@e(oypg~$rwt3$Sb3)whe%`kGRdpw&!pM>V7?H3EW`fogo*Q8 zi~e&8gvz8}LtEOcJf?Yy%lyNfr*{g2gnO)kW7k*>(m#Ik`JWO`>a;MwNPu)7-iD{w zJl{=#8M;rt_l8AX|1gKP+e#VMH<;v}ZFsf!_1PUi$&z>8E}#bDA;>*TyX^=#!18KL z|1QFyC??CDYm?U!{R_EN#cXNOwd+?6{4bNYYa+3!MX7;BufB;}10E7qt zA^`mV=RE)b1^7Y$f5bopFwpR$$CHxrQ!)<`#sd_cHr;qNM zpsR5g!3QIN^c{V|(TM-)kc+8tRhqlmiQiWvxIQ4e8M3Nb$uw3FRz5fZ3Tcd*8dmFze3fY6&TBKK}+x0UxN?GXAdqJ zXJwLOtauQZb)1&4_{^BM8T*FHSq=&SzOg$#>S@`fS(&|`Wov8_(x38{4|0I%}BusTad@I`{n3ruUa!x6MCm!XlZa_fS_g zxp@>oV^ubBfjH!dQ6ByrFSR>tWI!>l^BYIN=^6WPmJxaMhiT1%Pk4dg4^KQV>guaI zKik8iTKPM$F}H}|egLBvp+yVD+cZsd*{s1j0elIQ7iw^I2q7}(;PyA@_-vzBm6A62 zbCp5dr^i`h{Q0_HkM}lcl2H*HkBS#gvZ~HAM(7Wz`o=7u<*3{2$5GDfZ8{> z|KJo5HTHi(6@=J&#q}RZKW@Y23%|cX{14Wb_N~g(s6M}Z+d9qLAr8281#;RH4$qZO zWIo?)<^8$Gf1Ug+H2hbJFXXF^+R_`Zh`1=okt#Ce%AYUqrf7lc#vDsC${&L&RfCt$ z!kPtI)nd z%bQIl@2kyivOq}0^BWylzkVPjjYZeHV7pkP;I&P+TRvix7j##zMcZ|*EP%ZwgHl@A zpKq2ZTX2G)p2??6OU`Hym8#{ljrNOR?XvR6y!=y=)?P%OkrVi@2Jh4U8PjCT*cP9w zK&$S}N;bOKF*5bLzLDFt#Z4KbbUeDe3{A!QobSRupDW%=wF)_}=szyr4)eW_s-vW* zTt=052OG^#JXR>5CUtOaddBynmil<@#y@dv>!mT7nJ$3(sIZp1iQwQgzc@VdpSYG! zo3ey<7iqp%h_n}!5C?NB2dRL2(%zTU`j6;o)va~P5&EuT9G?;Yd}TZDFh^cmh{EO^ zJ;Xv&c#$ME?Cn&kxEwM#Kh4)&Z9_bIm>H#M7?L<^`t;Lj4#fZ?_5^G3FoSBeVMvbU z+Im>xQ$QbaXPM8}clkms>PStNDH_VY2Zsdfp27{m$Z#Y*iRCPBHKdrpi%HG-t76n^ zibz(w^H0?l3r*g?X~>k;sB!r8XB@gk?bLEJV%&UGWh#Q#BC(Ru#TxD!34fP{FaCC} zyb+xD3%jD%TOHIg3+uwyU)(>#GXoj5am4$yH$@Aitp@lNk$ALrROhI}>WvvcCp{h9 zKyL1f(!rtoo4&Y_mbRQ>!kJaHU64J@ngl{bnj0%NbKxj^bL{r8Dr8{19$4dL-Fw@d zGu3equk4UJ zb;NBf&=6!iL3@GRhq$y;_}bp~Dk#-$x14={@{aB-=VF4Uy(6|9u29 zPHy#5Nnueddo-68KmoSEA+h@ZF!R6ssF37_@NXKKI7)iz&@gg}+dAoqhw=!3h-s zq2A5%8uZ=E5r@)&HYUFx=|dT9c9Zl6BbdMLGOzG~v@qo-(RtV%V#J6Yjok9Keuv=R z<1r$K%YjJmmDA5PLfgj39TjDZO)G+x(X~QlJX0->vtc&<{upV838t;;*BVQS?hEiqPM- zL;ow#6^y;dyi`{NS9>?=dgVIZ1fY&L5@>ugXuxEjW1Gb(B5Ee&<7vATu>tBNLQnzA z67)v*rZ2^NOcC-8+3zqnzk|6BBf?;T^jt$2dg=N}VtiQ#)*T^E&nO;S+#z zvGYcdANRn_?{?f%QZ2Ezuz_XqwZKXK$1QT;Y@;P&M&OLeSwU-Cbh8%@BO8L5!Vwlt zyl=q4eryS$nwxcPDprtKl+vmnC1)eX$H>zWL0zGTOmgJ(HEiH${TE1&pUiCaYo-glPW+S@KRC)`68R2hP*-fmS-}zMrN&4g2 zm9-SKzw21D;s9n!^DXix@0Xu8mf^CRe6E6$xekIo%@F3L2?Gs^Y8z6V?Q2#vNp(8- z(Ngy_5oBM#vhzPBVO{8&n4cf4FG@ik);e6?zh8Gg;S3zX|8ai*P+r6R;$Y~@`tWAW zMG}TNs}u)wQe6^sE^o4f$|nOlCGJrE;cd^;g5cA}*n6~qbhi5iCh_w0VCbtG>-VD0 z`&%vL+gdH5ZF<=UU)s@H*cE(uJP)ZUHV8|xO_;y=Rt<w5*N!f{7PELIbg}O^Gekamas4WKb*!C-ebPOhq3|BB+A0&n@AeiZy zaDK43Fsh6(w##t#IXl>~I#~Q3+BZn|T#}$4^6mV-(v?~7_2L+faxha}a`t+#I4O|+ zcEI*LSlpjEIok?_^L2+gmgbuW(vq|6nzGls5;;VfQ{mq2sqR)axOkXGo$on8JJU` zVVs#tC~Z_{J@N7JSiz)p=xDjdF%_hpQ*`JiQLk4pTq*W&;nrL-)0l?$JFpD8OmGNO z9E>3J-!Iw!T5+Zl%wTK!jbS9yxMP%*q6hN^WpfTIUSRZn*xf}R`%I$*8Vcu2xcA8c zlZ(ZTnfEBMFzP!}&*xT+a#umcavZ_JnM!xi_DLi7DyqDDi%5Q%hTB#c5GuYVs*n=t zIwzqPS@ktS1`_ zFVJbyA)Za7Vt1;W-Di0sNNx;a`ielV2saQhx0$unZS9pOEJLFW!e-G=!E$F{X~G5S z9m>-*3}cFe-cByER=p%}0HQHx%UHZHxNIs;zO-^vk#-U3!V7?h{CIjtCICh)74^Bi z#QQ-t>MC<|)ObVnSvd^8%8cOlQD9unU1d;j^AV-B$j#0Z|})5Db{Xein)1`SG|ijJ%~)rFnIW55-wk+y{e z-_vXtvGx&2%`00fi{&ldy1DF>(sAQRaZ);$tzcBuaFp`NtTH7*NCtQBCtD}kmy6wD z?izJ|eUB6WvYn$m(St{f`a2RH%FpgqqPK*f$X1Vt*jayUpyXLU&g0!{CNXGf7WHQy z{h3Vbon|j475&6trTd=BHIc7#|7dxrsjKguPFHH5!xKV95MtnlrP8onH-N~yvi?E@ zyD1X-ft7+1%a&4}Qn79w|J86l$G^_42q*+E)XMRk>GO3KD)0Y8LwCMR1 zQ6uDZ_xC?$jRFVl)CYAgR&6%>`hI2iq?c;>Q3Sem^$S)-H4G!~f`F<-?J z#0}%YcefRXY+qT^Za+IMnj|sv!~h$KnwjYYwIUWu%D9n3^B{9RmsIh^205QIvvN5y&6E-4^$_sd=~!ZsJ}FAwIkIPS$) z4#dRd%V!&}x+r=?b>Qg*%kUr|GPNeVxch!J|0`=m4wezze&{Zny*WEUHL}u=i-Qsq zERGu}iRn-%xOZfh7NhGrNh)@<7_Vsp`NNKXNAgi=Z1Lhr_JhckWQr??0s#$-pXumw z2;**{?9sa>GTwz8uZ{`RkaiU;6>gW#=J{&RoG+|!@A}{b(Ji3dn|uF6f?v)4WD|`# zgp1kw6Vv!W=!&9<<F5$W zwm)!g<2nGN22ydxbre8?FZe{C+pCSCz_--Sas4dY)z2g|N`XGeL38bF-JFPw8vPgJ z-A?ky*JsX?5V%{s@g)5Q>UKLz#AzJwWl>dy;oSDX!?APFPsV^gt!BvdQN69$=JS~V z$u@jQfE2u(yrn^x{Fi(0ZQiIm9u#TZV|` zx8!rau4qmrHl$j`3n~5+g9VyK9lbO6uvlAWm4fF-{TW$Pm zJO3of>gZMFd2P+X7$XZSe&9cqD##b10eKlOAc+QE7}==md?uT#xvZ0!P9uY6L=IY< z!;xI%0QRJ0tJ_<9vv5*(1||zF=YG1uX?ki3*P;}$SVzL7fe$f8x;nA77YmR78ESt1 zfXO^nFOTx3Ps2{6j1N{ym2{9Q(wJNBxv0s5ngpV?4?v*?*NQl<0Xx zdwy$qn`qHY%|~~tm)KQIyDAgkFre3w8MWG=LD=$GXR7z*qPVzW8TE75w(Sk`dw7AE zdWu~7HDWqJlaTL)X${XfUJ?|9Y$SR*3A$}%rzM26=jrsV5gHY+kd|pf*zVpIQDc4a z(1OnZerT~VCI0G_WszA{-8G&%J8s~^U4*j22fL4?KsjSWnE&8vugmW*HkoF&; z%GRF^Yd{Trl|%A4pq zUdGeqLji{YKb~|pCNt!`36%c}S-ZG_Ll8ENehM16Ex%i1mmJ1PU7H2m8V)9gCOD4C zls6OQWgFZV_OL~Jf4qI&Qhy*hFQS!fs_y&juQ4$30-qb{DPHRVE}^E)SK*No1Eunn zPsI}c0xI13sXvu}N2L{q>*TK*R2Tm^$A3J;_z}eX8}2$|{EX}P(D{gI@Pb9n99+(xTK8CiGM+Hdbt67z zvJRTnh+U+3%~naw^ge}ysN?Jgr?j!SAv}Nr{=Qi7(zVG<7r7P4-LOkS$3@KvPfrRc zUnJLTR+)PM$KUs@D8=VY&%8q=`Q@d#V?E8H8OR_d6D%kjVMo~;^k?lzy1l~T@-mVNr&e0@}@QMSk9W}+G={@nZl&33) z`KaHCLZW{>R%~16Jf8ca3eo^DZL;V5TQk5AcX@vc2rq&fsXGA6unRtp6$O02*#bYC0m!#kj{-h%33!728Bk{Go|x;BDQ z$G_C5^H1RjqWeCbr6dGG@`Q`J-*ia|{7zFO!!U$*Vzj~hnNhND0-=BlY0E~bj0gxr zi1C``&H2ZhqV`9P$AMwgNPkq&0|5KceB#NO?2^`545) zr%2+T5{{i&G>sIu%o{Vs6fb+M>IQpvW(O+GMa?4iS!&B|QNIsg!p^=Im;yw<5OXs+z z?fMzsdVY~jjzR!ZwZRFH0;+O9Yy%l9|!)Fb1U{km*_D+LexKT^#{U^oh+r)d1thxh1@Ac?_*w`vI` zU#Ds!5dnTA0RL#>XCs)uA0iQBh~Ia2=!(ADkfQ#tbJ>4SZ24mU9NhkUXxoZ&<_Gxq z>A&LtDH4Vs_yYbu^!-=qe-Q!75k3MD14(=@_M!hd9RAyp*3WsG$o?YeSz?#ILg8X= zEdMp2t|X+a#u2AKrc)mf`03qjHR~;{$zVxX_}C2FFu`Q?*W;&?)sD>uqOBGTTW=wv z*w>N=wXzT~Mn9ct#_J@##zhuKdP#_H#%I`p?MREdd#n%yoE~E7((;Xpe{`6bhz&2A zzouWV05Q*2@M<~v3&Q(NM!(j5u}qEiP?6_XdG!u-eO=2%2J~e+~@9#l!-U5TH zTBJ*@m_t+R_Qt;&-VxjGcIhR@ve%<0ZhPClp#*_CW$u&smA$CsA-FXb%V|Zy#7&%M zdavVlv4N?EU5rI_IZsnop*>=0_@HMwpYMn#U7bxzglfm^Lf0$=x~}j2RiA;S-D5Y6 z)(LiwL`HA4-)5)iv5mFGo(A+tey!j8cGF`rRQy7r#{K==IZp{M2NwIFAhJBNAx2k-U9dm ziuTioNhz4szteD1x^z3=zlwZ8wp$y$5P>@#y_x8Iq)_DQg+iW~(Q6Bz&i6bkY(>Ht6p z0suq{Mu_JW52TK>7hmR(0aOPG@gN$F!UDo@wsS1U(BUlTr{Q4 zT{K->%w1fdn=bUy=0wm=rnsf1qO`oc{8M#R zPF`_lPGMj~GCCw-aCByCV{Ky{Z^30;&uIDQuMHQt*^AkxxwJGrbPmq9W2M6{*H1{u z7jHHku06M?LL56OW>blmPb^|m%*^7CT3mRw47pIY_wPSP85+8|xq1IF2A zq6{UtjEp#hd^-Sx93cSj4g6zQa#mKl*Vp}WSJyiI*1P?M0AQuVcNvFSTFPF|@tyy* zR@2nif*Z;${o)mxZHG?YKRAj>tKQq$ocr<1^i51?Tyb4Xf7K)wD{d|>juppZ6J}@A z^hBEQ4i?cfGBQd{O-*Y`6Vb!-(0Y1!hokXrE)74%#{iy#ze>w75L(I6U%_uI1fWGk z@SUEXUefFrv|d^hT5olI8_?I^-rd&&gsiQt+f`MS_0h^U5#o5G{xu_F1$o)Y_?*RZD9ttwZr!L>t z;>IE;`RQbz`AfcN795@gtZf2b-yp}Lf@D~2%q7qEUS0luKxj1GBbMbJJzIa)LOrzh z0q($8Zo)e6xOMtK9)rPFUm{}l!@#AHur$z56)Tkm-}!CXI7F%*n}y4v#&agfH=W^` zPyMh|1a+TO;Ed@dy3m?6IgWX?SE49K{`X2+zS6BYzEtmEYtDIUOe6fro)fvJa6ODl zE!_UI0-Terz*By&45JDMq-g)mCB-0DoJwK zL0p@Tt)*A5a6nly0o8-UyrWBr zz6k8C8}kH4BNkQr2~-=QU|29AOyDX5SD5}C69KcGG``k1o^;R}5AK~D{ma!q-}}~3 z)%vSqf7OQ&A@g%+{4l0>(i!2cpDXEYrV-rX>k(!pge;-+4b=1LGH-VpKPMjXw~4rR zhc&-v@WaeqVsFGe_7!x8M|*tGc$@N8`dWVp_gl~TW37VKa-K<*uQ%`FXJ0`Y;hb5i zWHk@;iPWxC_)V0qtyfsjDQCrSemxE6(reE}8Yq2tQn&Lw`bLFJP+c?Q#a1J2Px zFD_jPANDQBfMc_tWygN9OLBUjUZ(cL;G&L^9`Y8KjtiUE&EkDaw&4Y%tqSCJCD*b( z?L;YA5FwKv)ZTp-|COWVm7>8TI2qf;c#iFEz;h7DnE8diPo9aUNc_6Rz$OI=$v7H< zsTYLs-T#&Nkm$@du$^jdzyxbyun)i>sYf8<7ztniC;%zs(nBzOac?K9!uRWaC_;cea3 zbAp4S3O!x5b(Sw3Dv{_kRQ3*O$nMm9T_ zL`qm;rD~jC=3YPXf~Qg$c(nyK?17DNlvl?ORTp^bGtMvb6@2$$*r?9;Uxy-bFW5^g zJ7q}-;(6ZmGIiRs!&=05Lgm_BtCV$Sj)H_vm4$L+6DfQPYbcyOdntEa;;!r*ec)nd z5p;ZgZdfp1UN}-7oA&##V!im*zPk6&$o&3u##g$2%CTztTXp^~W!_c(s)G2n5eC5p z%OBlm8&&IbvfB7=@@5!5C=8kn38i#eKBupy`t{6GQMlqBJ@Eh{G^y=#*^W3dHm;*1 zPp#!8t1*k@j}^Di)hjYRL-jTUc^|G!M1H{;T!oX$<$bXyBxpAFe_y&5*AYw#mi68= zCf~f?Ay>50I9-Dsp#X<3(zFm21n}3sK(p~W<)+VgISm{XkX+GCT&XKQ+`sO)b*wc) z#%F3hK79=|6QtcUr_Ez%M6|~)V7Ze38UH-sXukA`+?D`$L&1{~Wy8)AhH zElOlp#>3Qs*V(BB%5|Ybw|6BhQcY0W9ky!`@cDkhuoN9gxOVsyQ9hia%YA{4!NOOBTrdwj3ZhE-Z>`{{A#}Zw+1mZ((=-&(epP(roNOq zZ#JAahu4X;p-XMl+VqH6rM;AMHZ8@&Z}^!6FDbE-!&VtanAC+>Y5RL;ZsLLl)=6M* z37Qzvbh^Ho1s)k8v=WVUGP>{>L+4!sY{0Hw$FKld%W@DuLT~B!XVbx+&2_K8YiH08v*NSCE(X4hYF&a zdgqW3t5r*mZiO2Q)skfLe`xA}>+RuQltw4>*mx4VB4%hWMC1Bz5-&@2 zfAQRf*`m(evm%43&+B$*e$0_{e>Kd!F6|d!L67(s*#a2IkBV^F{?-@wNw# zkJGF|RvvFFty9-OH>w!mS+Hvp42Dx_xAx|6?Hf?z}Xsm70N@ zR$j*s16oMCmYDZy?lbHw`|Nq*e=USPO>&-KUC212+tIs<@s8#MZId;~Lk|}kX=z*f&4c=#FB{>vBU zqtS3-(CWMAlBDxI^Z_0iByGDHj5*P&n|p0a z&7Yif?$&w)#X<-Yvn}*&oKHm-4wYF9^$d>WMt@NOrzDo~^kQH7B;!?bzE~kh0NPgo z<2ICEf^ZrR@wqU#6aK|FDj5!AROR+2DT4W+iMCVEdiLPduyKCqf^Ft=5DIzpN@u|~ zeO(>~?Pm1d5$)%vfe<_l@D=~9s6s`^3g_q0SUE0mDPm)Vmu5d6-o7k`a$|!+4urB` z-ahgZa0o$EZ*6LG3MR`y35qowd?YItZ=k@`OiG^VADAEG>K~#hf6#X&r6Dnu7*Gu$-ESw^;Ufx zUUM|*?det)tZ8;t-HChP9MU0xY3AXxk_qxm@i@e|9Jrach?J05 zH{`0>|DodpVi%t(IHt{yUh+v)03k^5;yfjT(DHFi#brgCdN_`gM7@`C{$f=ZtaAmk zqOFJr!zjju3tB%t2gw7k4N3?14$Rk2aSKY^)=MWR;c^vSwX+!*dzaT;ar$)edg{F! z5kK#xwwm?ZVjl^nWRjdWU1NkUEnBN|NA+}R1c4qrQ$C-OOd9pi1LiC2xk26fOGPWHj(>KA1mj=NV1v^gYZ%IJ-Sx+QLL7v!WM`HGDo z`=#-=Z}bFXQ$qWRk6E6z|4y}V^DUIrskY`2sTH->FJt8odUaDU=qJPW^l)N7_@c0E zf0exXbY8X+S#eWT{z#qKw2?eY|u!2vZOtC%ppAp|xeR#?fKsDkao*?>4@I2Ra+_s3HKMz6}q9BcX6z~GO zmyv$Ag4gy&`LNL0T_KAPUmHulTDU*~)`-EqVIV>A~iHdK8m6 zB)<2+RFF}xP3x#N))Zr9saE_W;CP;RV32rug;>i_vzo$4IDGBHk&+wFtt39&nBBw= zE(W8%ZL;ATq|zUnLU{}5uiZJc*_@I*l+izI3-%bwN}E~9D%w*gycKWufTv|!B?)T$M9v< z@nuwYUc8{P$($xxu_FA?; zL~!2=x_h?^wH`EqcL=r`X>aIC^z8(b)0lq9re{D^Fz$xRQau#r@|UUTGwwI&>O7M_ zh`t)~$SV_BQGtA?$c1H*@{Ko2XxmR=k}25m)RS6l30YU*7{*1UAW@-HT*_C6iI3Ir z#*x33_Q>mu{3V1!z+RPE=FgAS8q30AjqS~(>4+>@n{Lx?1>op`yxShWwfjf{#6Zov zrm^W+-LXlQ_decQ`*)yaybzyyetppH0~7)w@In6zx={R(8V38*{exTj{6BAJ_Np-Q zllV;45r=}7R;hYp}?Y&m~M-~1UHZK>1+C~s~piOvi?P`!NT<@w)?hmeN75Hw2WQ+VP zEs~DDmCxhq3@5!Cpu2rM`inAh%B~)9HmUY*shroXwG_qj_&T|jirkNmUlCd{GIlMS zK?a`{6Kjdk!*zx8Z+dLp4Q_SrU!oz2ZRAU?dSkHL$aGPo)SLO9Ae9r)mN)u7bQzNO z&C3ol?*a%~{$9N`okMg|QyRXkLq;=1NiFk0iBq=53;BDC+`T%eJmt20$P?buJ~@f;I+p=7w@N$%UDz_L zJ2jk=9wVTp3s8dnp@vP7}FsPtbY>547+jNjA`X;7D6}dL4L8M0ci0F8V z;`hhLBoF}LhxKa5xl7fJFS=$^3ow7a#l4`H{AG8v@?o&S(p0 zm{XUtBpjtQea(HEbD+N-;JbgiMS|Z<+J%;!N$FcD4_&DN+0+m10=u{ml}KE_s1{Cd}I4E$3_W#jM>lEaoVjSj}ZX-I=48 z!O+O<;V0Xg0&=Npj)N!(3dmji0wm+_T;j;HO>1~+_-sGON_nA6;lSkg*z1HYkyyKB zBFx!6D<RUMQAF;?Dm(GDv_V;@ zL=A^eS79ZQEPY`NH!6!TL*=$Kyw_u=cu(>bKiC~ARq{T&^_%%vIr==K0L5t9f~>=gf3m@m#E^y4s0irNQHop z>+*~uHVqV#JseuD-MwjmqTFtOyIY)x?0m`IOL}0^ti(a~J@%zGCZx~Rzl@5g>OG}q z_PsI8O*sGWJgmy95H20*f^+rGLZv>bYQO-HOdB;!*oys%9YW zMiA=7j)`w;t7Tgd!guS)A>*9gjGkHnD;d3|bJMn&Z=|f1lH#2XN90xPU{A5>crsa5 zDyhs#_bY6Litk#ftABGAa@6jC?9G=KSW45ednXnjn4rD)myV!BY!RS{T)YHDvu7I? z;cGWPYUo=U5+v57cGLRNr!1Y=70MC+vM(cG$5uWO2Ygvjsr#H*?1CskQNg$H``TKJ zC>$?vsN5ghsQ8YL`h&;b`m93+N5nIkZqjrpWy3~7fS+zI zpHfUn$~*FCPlJzK*C+(cT}Ci8L3?o%pkGo6i?5dzSpUY@I-J=Pl0N#VMHi!@Q~b6Vf8-K{F%bZ(>ke@>f2u~`j9qv zoHcf{%eQb()*?mE?!QntU&0UyTyI1cdnojhrz9SK2%)?cX}bmqpNjPKv)1qk;+ zbR{3NSF6d`f2zSuLN1@&&v@Gi(F}W16?LzyONr9wc?%|p){Z_?RRg$Uk-_$zPREq$ z(v!j^!@hFe1Fn3uZ?N2~=Di}XO{Z$T5teDe-1_m zd)htz*AXldPQWTFu3u7|EP%H~SstTRtAYVAOGmH^Nw2L56$y^p2c1!7M@zX};Vd(7 znV6fnRIb-sz_1SdoVNP>z-x`gW>d+>v{@KQdFMb~OOiy@7TFvi%ADzQjZ3LSOV&u# z-&YN=a>U4ErO8Qb;ikDJHv`(Db_iHfH!y3rlR=-?nCLL*#0*s%>#k4$MI2Ps3c1(+ z9FuFinYTBf+;3GZC)ZBZrDW-vl>B?-xt2YLiznb6a?tib%9hA4#X2i}X2&((i1uC%DJ>JyN>Ep*x& zRu0$n{;X|tC=s3Akl(fC5q(Dt*_mXkc4_}-k(J1@KgTUO@<#VFY{N)cLd||1^T<)w zq}RwLpHbJx_DMY7HhWL1%DYOx^41!2H*m?TH?P%-VgSw{;hRATi$hZ!NJWx-OzV~~ za?v7*QBCs+?`6{m((q3am?g3-irJIxoMI(S-sv2+(ds1qfFXLNP)eat zY`>$5^w40+6@+u-v_q;Bze4ltcd*?=nFoiVL!__P+b64*@9kuqLZch0H&$kA!YNSeDm~D zX#i)Cj)ENruGnqtF?K61&`E2^0z1MP1Pylj>`YKNsmrSkPeEeHBUqyQz_jVZLAM)cL_YDF#-;{TcT|Svzt6eOqNm%^cAyg@KKbEv|G|uKt zI?K1JSn@c5`^zy=GJ~dJT4}EZJ4^eA+OxCE@*W!gY}bw}-L%(-(*sjUphc34{S}L% z-5J}QUz`?I3nN-zh3jY6{5-^3ZNy`hN9^B)O$E1g6gCsz*?SE=_R_X?l=s)H7l{1I z_0mw|5-_zO7_`23Jkh1eY3S)XQvjohKc3KqrLV>(d}UURe-#79BiWHtLMe+?2w8ZW zH39m*+uN9BPMi9?4H>Clx2;{fgq+;-^*-`)^Ta2xy4RHCC}wwPz|!whC+K`1jgWFm zqEUjX#qj}loZckC%ng;)lOXa;0i$L9V8_sVd_Y!-YFR{}JhPdl;3Nb7r2TsB<(tfR z4Bo3h9(u1?=B@r;XdS{u(@H+&=xePi2N(k(6&|BJJ!+$~Lm^O80 zN*zw3YVn_eJtz4$)c)h8mzw=g;QWQ2f05z%NjxsR|8x0Yi&KgJ>uj?B5@;>yTyLhE zdi!feBuLGGRSsN6#?L?-yN`h&7J=<7mKs0K4oDoF|a`1U%w|tZTpbrz;$?fg4wfIZlhklYQLlsb1_L;JPMN-5% z(cOnX-f}x4>sg9lf4q4wji%O^NDlZXco_bRPX7i^TfVD_8FE#-UA=(p318y{r+fx2 z`|@lS$e@xpOz1*IWMA}4AL=9O^Z&xjlue?|(bPMUuS_vEPA$-n{WWbhhP~3~3tD3; zlFam&D)rj!b1rfl6}P+39ZA-riVB-1g!rtiAays8-^!H_&&_9ry?dsG#wrW099El2 z7dd4oBrVC+RCCs=YTUF|W?G4NtxXfV@!pXq+mtE6P2^VVD-LF<-cG-sYdZ%W8ly%j zIcCDdbNZwXqrugM_M_sP=WpD+95goO0+_l!WgjLaVebTvS9W)HW;zuPH2pl?ez?*4 zV@kusF)KOjR)KZd!JX-Yf@G9MOqZq#?F$sz`HmzOJgoyT^b`9PdDz_{Y1<(zgx8P)2xdqN=sxynb9yT%p->+!!>k4o!w6KP%SK zFz;ZOziL7Bt?1;#q5*5rBiTXvu1c@7ywzy=n3xzI+VOax`uBbLsjVW4O1E$QeHbt{ zF5+yAW#thq%b>NXxtW>6^U$!j>3J?Hg@1?Qh38RhD3;zKrU7BJ1=h=UR!T)zi!TwA5`6jXPzbmP)TeBMEB7 z6)`toVkk$Wt^`h=J`S;Urn^cf%LUaE{gY}*^IX0z<#J13I>Th)uan4FG*+GsE$mJ_ z5ko2P&_V&+%M2%_VcMsBQ+0T)@pBRw!QvMt#nozug_7n3k+NBJW@K@V_E00RrQ0cw zI3NWo*bl1Ljf(K6AzcbG)1NZLB_X79d^_C&?#>p80IUuf3hJcR^9}ua7w#9e7@yYe zG4Mp^68tmgGWlO`6#)4yEd2b`Fs(x~P^V9vC$Ko#-ORG{sfpcvxXi%hX88z)@lXqn zfI|tOmewvQHTD{Be#4sL4m~gFmP{YLC}3vO)fd&!yt|<8@<6?YxGi{Z=r33a_+UNK zUMRm@y8P1MRMkZ6a?pZ^(Cx*-n=t|VLT^#p=k&$IT3;7GzC3+UN&}5^{WxKXPrb;6 zPK%GNHnM9hO}-ZxT%XL&vi8p2^2iLO`25;PWI0Cg?T190Y*Pb~=hH>S3(_$tx`2Sv z2U$#3Kin3p1D<*NKXqNNE?rS2wx!^dUv(X;`5RQ`Y8KB|^?&z`2PSbAC}Ziu7vB56 z@?+|$i@Z;*QTn3tqn?<$zqw;BOnYxKP1XXx%~)iJ0wc=EBr_O-2qpjf7$EZeO?8RB^h0EcT^a z`_r1XOhZUx|WDkzGBP84AK@I9|&yXkXov1{OeOsBGUZMI+BFoN%FVaj#R}$J8 zK5i1)UZPt5Cj2>nRW8(Xwt4hP5?g~5%W;X+8W)2A_tg|j0p?8Rp7w1qiPqtPA-{CD zM^*Ft`rdJuT7!A7Uar+<8XF=>C_CZFB#7ny^~cV3CHOFSGY;?)A-_f?6KX*)-Evdz zsWbmd;BYCx{$ecV9$MV@pQ)NN8*X+jsrp$ff=<@(e&(mE5!geE+AG-^VFA}giSh`kt(mhgHQ8#w}ccP$&9N#?W04k1fQkwxoPn*RPUl~i{NrppoZ zJV6hejv`G*tJiQQhmAk8zqT0;Vp_ofWoqB~Y3Rat$-$dU?6AoV7+Vl1WEuh{8E2w4 zC3{cyGk>`!_o{?%O6>0wFS9F)J6*d`R4Yd6Py?C zf3M)i=ssVf@7X1Qho`X`jgDH~R_m>Y5zyyP$eSQU$O#2>XB$&Yj`yWzlLSzFS0DSE zrqVUdc6o^VkSqdO2fiXE79`u*YUtA} z03rDMf*t$G$_GNg-v2zK*q;oFEeF|Gbg+M>oXtzXfJQ%<%a79WwtMWPR}iNxoVQ+eVl%)(j{j1vy}L9U+Ql;xtY2TE zGqZh(9tJwSpDwSdpJXO?7mac|v3Xv$z5dfp{C0_cYx$yy04l#_a^$1u{4NQ;W~+R8 zUorp0LF;!M!~022XWeH>ey~i5H|p2Mi+?BK3%<&+1L+>;5Bt$yqxQvLb+>E6ji}OMz-M|0s(3iEN|2L_`#*{C0(aL zLV`?sKN2Z0+!W(gPG^Rb+~#jWGWQun*6X literal 0 HcmV?d00001 From a4839144cc802ff2e5f49c64892ae13cf4e6099b Mon Sep 17 00:00:00 2001 From: HelpSeeker Date: Thu, 30 Jan 2020 21:46:35 +0100 Subject: [PATCH 02/40] Update README.md --- README.md | 26 +++++++++++++------------- 1 file changed, 13 insertions(+), 13 deletions(-) diff --git a/README.md b/README.md index b7618d0..edc2abb 100644 --- a/README.md +++ b/README.md @@ -6,22 +6,22 @@ CoubDownloader is a simple script to download videos (called coubs) from [Coub]( 1. [Usage](#usage) 2. [Requirements](#requirements) - 2.1 [Optional](#optional) + 1. [Optional](#optional) 3. [Input](#input) - 3.1. [Overview](#overview) - 3.2. [Direct coub links](#direct-coub-links) - 3.3. [Lists](#lists) - 3.4. [Channels](#channels) - 3.5. [Searches](#searches) - 3.6. [Random](#random) - 3.7. [Tags](#tags) - 3.8. [Communities](#communities) - 3.9. [Hot section](#hot-section) + 1. [Overview](#overview) + 2. [Direct coub links](#direct-coub-links) + 3. [Lists](#lists) + 4. [Channels](#channels) + 5. [Searches](#searches) + 6. [Random](#random) + 7. [Tags](#tags) + 8. [Communities](#communities) + 9. [Hot section](#hot-section) 4. [GUI](#gui) 5. [Misc. information](#misc-information) - 5.1. [Video resolution vs. quality](#video-resolution-vs-quality) - 5.2. [AAC audio](#aac-audio) - 5.3. ['share' videos](#share-videos) + 1. [Video resolution vs. quality](#video-resolution-vs-quality) + 2. [AAC audio](#aac-audio) + 3. ['share' videos](#share-videos) 6. [Changes since Coub's database upgrade (watermark & co)](#changes-since-coubs-database-upgrade-watermark--co) 7. [Changes since switching to Coub's API (previously used youtube-dl)](#changes-since-switching-to-coubs-api-previously-used-youtube-dl) From 3b74a29b1ed3edaa6fb9e93f647f629276c44b5c Mon Sep 17 00:00:00 2001 From: HelpSeeker Date: Sat, 1 Feb 2020 21:50:42 +0100 Subject: [PATCH 03/40] Make chunk size advanced option --- coub-gui.py | 1 + coub.py | 5 +++-- 2 files changed, 4 insertions(+), 2 deletions(-) diff --git a/coub-gui.py b/coub-gui.py index 4fff28c..ad1a78d 100755 --- a/coub-gui.py +++ b/coub-gui.py @@ -217,6 +217,7 @@ def parse_cli(): coubs_per_page=25, # allowed: 1-25 tag_sep="_", write_method="w", # w -> overwrite, a -> append + chunk_size=1024, ) args = parser.parse_args() diff --git a/coub.py b/coub.py index 71b2651..49203dc 100755 --- a/coub.py +++ b/coub.py @@ -1461,6 +1461,7 @@ def parse_cli(): coubs_per_page=25, # allowed: 1-25 tag_sep="_", write_method="w", # w -> overwrite, a -> append + chunk_size=1024, ) if not sys.argv[1:]: @@ -1805,7 +1806,7 @@ async def save_stream(link, path, session=None): async with session.get(link) as stream: with open(path, "wb") as f: while True: - chunk = await stream.content.read(1024) + chunk = await stream.content.read(opts.chunk_size) if not chunk: break f.write(chunk) @@ -1813,7 +1814,7 @@ async def save_stream(link, path, session=None): try: with urlopen(link) as stream, open(path, "wb") as f: while True: - chunk = stream.read(1024) + chunk = stream.read(opts.chunk_size) if not chunk: break f.write(chunk) From a8a51bc5e1e3cca2f690881a2cdaa297c00e13de Mon Sep 17 00:00:00 2001 From: HelpSeeker Date: Sat, 1 Feb 2020 22:11:23 +0100 Subject: [PATCH 04/40] coub-gui: Add help menu Only shows an 'About' dialog window for now, but eventually I'll add a few short help texts to cover topics like sorting. --- coub-gui.py | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/coub-gui.py b/coub-gui.py index ad1a78d..248b15e 100755 --- a/coub-gui.py +++ b/coub-gui.py @@ -82,6 +82,21 @@ def translate_to_cli(options): show_failure_modal=False, hide_progress_msg=False, terminal_font_family="monospace", # didn't work when I tested it on Windows + menu=[ + { + 'name': 'Help', + 'items': [ + { + 'type': 'AboutDialog', + 'menuTitle': 'About', + 'name': 'CoubDownloader', + 'description': 'A simple download script for coub.com', + 'website': 'https://github.com/HelpSeeker/CoubDownloader', + 'license': 'GPLv3', + } + ] + } + ] ) def parse_cli(): """Create Gooey GUI.""" From ffad786e6e44a40a160767a2746986b2215e6972 Mon Sep 17 00:00:00 2001 From: HelpSeeker Date: Mon, 3 Feb 2020 18:21:42 +0100 Subject: [PATCH 05/40] Define default advanced options in DefaultOptions --- coub.py | 14 ++++++++++---- 1 file changed, 10 insertions(+), 4 deletions(-) diff --git a/coub.py b/coub.py index 49203dc..b6b9e9e 100755 --- a/coub.py +++ b/coub.py @@ -156,6 +156,12 @@ class DefaultOptions: # Usage of an archive file is recommended in such an instance OUT_FORMAT = None + # Advanced options + COUBS_PER_PAGE=25 # allowed: 1-25 + TAG_SEP="_" + WRITE_METHOD="w" # w -> overwrite, a -> append + CHUNK_SIZE=1024 # in Bytes + # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ # Classes For Global Variables # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -1458,10 +1464,10 @@ def parse_cli(): # Advanced Options parser.set_defaults( - coubs_per_page=25, # allowed: 1-25 - tag_sep="_", - write_method="w", # w -> overwrite, a -> append - chunk_size=1024, + coubs_per_page=defaults.COUBS_PER_PAGE, + tag_sep=defaults.TAG_SEP, + write_method=defaults.WRITE_METHOD, + chunk_size=defaults.CHUNK_SIZE, ) if not sys.argv[1:]: From 3e8eb5a9040886f9de81220ff6961d5a179f4394 Mon Sep 17 00:00:00 2001 From: HelpSeeker Date: Mon, 3 Feb 2020 20:50:36 +0100 Subject: [PATCH 06/40] Add support for config files To change the default options one had to edit the script itself. All default settings are right at the beginning of the script accompanied by short descriptions, so it's by no means difficult to change them. There is one problem though, that interferes with some of my future plans. What if one doesn't run CoubDownloader as a script via the Python interpreter, but as a standalone executable (e.g. after bundling all dependencies with PyInstaller)? In that case it would be impossible to change the default options. So let's add the ability to change default options with a config file as well. The config file must be named "coub.conf" and be in the same location as the main script (although more paths could be easily added). Options are defined the same way as in DefaultOptions. White spaces are stripped from both the option name and value and lines starting with "#" are ignored as comments. Unknown option names or clearly invalid values (e.g. not an int value for RETRIES) are ignored. So are config files, which can't be read. Any errors will print warnings, but won't force a hard exit. However, it is important to note that (just like previously) there are no thorough validity checks as far as default options are concerned. All kinds of weird errors can occur, if one isn't careful to choose valid values. --- coub.py | 76 ++++++++++++++++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 72 insertions(+), 4 deletions(-) diff --git a/coub.py b/coub.py index b6b9e9e..cd24e28 100755 --- a/coub.py +++ b/coub.py @@ -157,10 +157,78 @@ class DefaultOptions: OUT_FORMAT = None # Advanced options - COUBS_PER_PAGE=25 # allowed: 1-25 - TAG_SEP="_" - WRITE_METHOD="w" # w -> overwrite, a -> append - CHUNK_SIZE=1024 # in Bytes + COUBS_PER_PAGE = 25 # allowed: 1-25 + TAG_SEP = "_" + WRITE_METHOD = "w" # w -> overwrite, a -> append + CHUNK_SIZE = 1024 # in Bytes + + def __init__(self): + # Only supports script's location for now, but write it to be extensible + config_dirs = [os.path.dirname(os.path.realpath(__file__))] + for d in config_dirs: + config_path = os.path.join(d, "coub.conf") + if os.path.exists(config_path): + self.read_from_config(config_path) + + def read_from_config(self, path): + """Change default options based on user config file.""" + try: + with open(path, "r") as f: + user_settings = [l for l in f + if "=" in l and not l.startswith("#")] + except (OSError, UnicodeError): + err(f"Error reading config file '{path}'!", color=fgcolors.WARNING) + user_settings = [] + + for setting in user_settings: + name = setting.split("=")[0].strip() + value = setting.split("=")[1].strip() + if hasattr(self, name): + try: + value = self.determine_value_type(name, value) + except ValueError: + err(f"{name}: Value ('{value}') has wrong type!", + color=fgcolors.WARNING) + continue + setattr(self, name, value) + else: + err(f"Unknown option in config file: {name}", + color=fgcolors.WARNING) + + @staticmethod + def determine_value_type(option, value): + """Convert values from config file (all strings) to the right type.""" + ints = [ + "VERBOSITY", + "REPEAT", + "CONNECT", + "RETRIES", + "MAX_COUBS", + "V_QUALITY", + "A_QUALITY", + "AAC", + "COUBS_PER_PAGE", + "CHUNK_SIZE", + ] + bools = [ + "KEEP", + "SHARE", + "RECOUBS", + "ONLY_RECOUBS", + "A_ONLY", + "V_ONLY", + ] + + if option in ints: + return int(value) + if option in bools: + if value == "True": + return True + if value == "False": + return False + raise ValueError + return value + # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ # Classes For Global Variables From dd885961923cc086fc99b2c7a5bb3fb32d6ea983 Mon Sep 17 00:00:00 2001 From: HelpSeeker Date: Mon, 3 Feb 2020 21:34:45 +0100 Subject: [PATCH 07/40] Change default output template from None to %id% For a while now parse_cli() makes sure that %id% is converted to the internal value None, which leads to faster skipping of existing files if no archive file is used. So we can change the default value to %id% without missing the increased skipping speed, which makes more sense from a user's perspective and allows us to fetch the actual default value for the help text. --- coub-gui.py | 6 ------ coub.py | 4 ++-- 2 files changed, 2 insertions(+), 8 deletions(-) diff --git a/coub-gui.py b/coub-gui.py index 248b15e..a44d9c3 100755 --- a/coub-gui.py +++ b/coub-gui.py @@ -21,12 +21,6 @@ class GuiDefaultOptions(coub.DefaultOptions): else: PATH = os.path.abspath(coub.DefaultOptions.PATH) - # "%id%" by itself gets replaced by None anyway and it's less confusing - # than just showing None as default value - # This might be worth changing in the main script as well - if not coub.DefaultOptions.OUT_FORMAT: - OUT_FORMAT = "%id%" - # Create special labels for dropdown menus # Some internally used values would cause confusion # Some menus also combine options diff --git a/coub.py b/coub.py index cd24e28..5ef8e0d 100755 --- a/coub.py +++ b/coub.py @@ -154,7 +154,7 @@ class DefaultOptions: # # Setting a custom value increases skip duration for existing coubs # Usage of an archive file is recommended in such an instance - OUT_FORMAT = None + OUT_FORMAT = "%id%" # Advanced options COUBS_PER_PAGE = 25 # allowed: 1-25 @@ -371,7 +371,7 @@ def format_help(self): Output: --ext EXTENSION merge output with the given extension (def: {self.get_default("merge_ext")}) ignored if no merge is required - -o, --output FORMAT save output with the given template (def: %id%) + -o, --output FORMAT save output with the given template (def: {self.get_default("out_format")}) Special strings: %id% - coub ID (identifier in the URL) From dd8df4b8421639aca6b3b3288075de8b729242f6 Mon Sep 17 00:00:00 2001 From: HelpSeeker Date: Mon, 3 Feb 2020 21:50:21 +0100 Subject: [PATCH 08/40] Interpret None correctly in config files --- coub.py | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/coub.py b/coub.py index 5ef8e0d..621296d 100755 --- a/coub.py +++ b/coub.py @@ -82,6 +82,7 @@ class DefaultOptions: RETRIES = 5 # Limit how many coubs can be downloaded during one script invocation + # 0, None, etc. to disable MAX_COUBS = None # What video/audio quality to download @@ -218,6 +219,12 @@ def determine_value_type(option, value): "A_ONLY", "V_ONLY", ] + none_allowed = [ + "DUR", + "PREVIEW", + "OUT_FILE", + "ARCHIVE_PATH", + ] if option in ints: return int(value) @@ -227,6 +234,9 @@ def determine_value_type(option, value): if value == "False": return False raise ValueError + if option in none_allowed: + if value == "None": + return None return value From 5f3f5767faa16dea2beb0c0fd39f7b4a95719c53 Mon Sep 17 00:00:00 2001 From: HelpSeeker Date: Mon, 3 Feb 2020 22:14:47 +0100 Subject: [PATCH 09/40] Add an example config --- .gitignore | 1 + example.conf | 123 +++++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 124 insertions(+) create mode 100644 example.conf diff --git a/.gitignore b/.gitignore index 258b66e..8c9999f 100644 --- a/.gitignore +++ b/.gitignore @@ -5,4 +5,5 @@ !README.md !coub.py !coub-gui.py +!example.conf !/images/ diff --git a/example.conf b/example.conf new file mode 100644 index 0000000..9e3e14c --- /dev/null +++ b/example.conf @@ -0,0 +1,123 @@ +# This is an example configuration file for CoubDownloader +# To use a custom config, place a file called 'coub.conf' in the same location as the script + +# Lines starting with '#' are treated as comments + +# Change verbosity of the script +# 0 -> only errors and warnings +# 1 -> normal verbosity +VERBOSITY = 1 + +# How to treat overwrite prompts +# prompt -> prompt the user +# yes -> always answer with yes +# no -> always answer with no +PROMPT = prompt + +# Default download destination +PATH = . + +# Keep individual video/audio streams +KEEP = False + +# How often to loop the video +# Only has an effect, if the looped video is shorter than the audio +# Otherwise the max. length is limited by the audio duration +REPEAT = 1000 + +# Max. coub duration (FFmpeg syntax) +# Can be used in combination with repeat, in which case the shorter duration will be used +DUR = None + +# Max no. of connections to use +# Raising this value can lead to shorter download times, but also increases the risk of Coub throttling or terminating your connections +# There's no benefit in higher values, if your connection is already fully utilized +CONNECT = 25 + +# How often to retry download when connection is lost +# >0 -> retry the specified number of times +# 0 -> don't retry +# <0 -> retry indefinitely +RETRIES = 5 + +# Limit how many links to parse during one run +# >0 -> limit to the specified number +# 0 -> no limit +MAX_COUBS = 0 + +# What video/audio quality to download +# 0 -> worst quality +# -1 -> best quality +# Everything else can lead to undefined behavior +V_QUALITY = -1 +A_QUALITY = -1 + +# Limits for the list of video streams +# V_MAX: limit what counts as best stream +# V_MIN: limit what counts as worst stream +# Supported values: +# med ( ~640px width) +# high (~1280px width) +# higher (~1600px width) +V_MAX = higher +V_MIN = med + +# How much to prefer AAC audio +# 0 -> never download AAC audio +# 1 -> rank it between low and high quality MP3 +# 2 -> prefer AAC, use MP3 fallback +# 3 -> either AAC or no audio +AAC = 1 + +# Download shared video+audio version instead of merging separate streams +SHARE = False + +# How to treat recoubs during channel downloads +# RECOUBS = False -> Only Original +# RECOUBS = True -> Original + Recoubs +# ONLY_RECOUBS = True -> Only Recoubs +# RECOUBS mustn't be False, when ONLY_RECOUBS is True +RECOUBS = True +ONLY_RECOUBS = False + +# Preview a downloaded coub with the given command +# Keyboard shortcuts may not work for CLI audio players +PREVIEW = None + +# Only download video/audio stream +# A_ONLY and V_ONLY are mutually exclusive +A_ONLY = False +V_ONLY = False + +# Write parsed links to the specified file and exit (don't download coubs) +OUT_FILE = None + +# Use an archive file to keep track of downloaded coubs +ARCHIVE_PATH = None + +# Container to merge separate video/audio streams into +# Must support AVC video and AAC/MP3 audio (e.g. mkv or mp4) +# See: https://en.wikipedia.org/wiki/Comparison_of_video_container_formats +MERGE_EXT = mkv + +# Output name formatting +# Supports the following special keywords: +# %id% - coub ID (identifier in the URL) +# %title% - coub title +# %creation% - creation date/time +# %community% - coub community +# %channel% - channel title +# %tags% - all tags (separated by TAG_SEP, see below) +# All other strings are interpreted literally. +# +# Setting a custom value increases skip duration for existing coubs +# Usage of an archive file is recommended +OUT_FORMAT = %id% + +# What character o string to use to separate tags in the output filename +TAG_SEP = _ + +# How to write parsed links to a file (see OUT_FILE) +# w -> overwrite +# a -> append +WRITE_METHOD = w From b28e22b583f0072a72fd1a56544bcafae39d1b3e Mon Sep 17 00:00:00 2001 From: HelpSeeker Date: Wed, 5 Feb 2020 16:19:20 +0100 Subject: [PATCH 10/40] Combine recoub-related options Instead of using 2 different booleans, just use a value range to define how much to prefer recoubs (similar to what is already used for AAC audio selection). The main motivation is that it removes a possible source for errors when using custom default options, but it also simplifies one of the workarounds in coub-gui.py. --- coub-gui.py | 18 +++++------------- coub.py | 28 +++++++++++++--------------- example.conf | 10 ++++------ 3 files changed, 22 insertions(+), 34 deletions(-) diff --git a/coub-gui.py b/coub-gui.py index a44d9c3..ba8d661 100755 --- a/coub-gui.py +++ b/coub-gui.py @@ -26,11 +26,7 @@ class GuiDefaultOptions(coub.DefaultOptions): # Some menus also combine options QUALITY_LABEL = ["Worst quality", "Best quality"] AAC_LABEL = ["Only MP3", "No Bias", "Prefer AAC", "Only AAC"] - RECOUB_LABEL = { - (True, False): "With Recoubs", - (False, False): "No Recoubs", - (True, True): "Only Recoubs", - } + RECOUB_LABEL = ["No Recoubs", "With Recoubs", "Only Recoubs"] SPECIAL_LABEL = { (True, False, False): "Share", (False, True, False): "Video only", @@ -44,11 +40,7 @@ def translate_to_cli(options): # Special dropdown menu labels and what they translate to QUALITY_LABEL = {"Worst quality": 0, "Best quality": -1} AAC_LABEL = {"Only MP3": 0, "No Bias": 1, "Prefer AAC": 2, "Only AAC": 3} - RECOUB_LABEL = { - "With Recoubs": (True, False), - "No Recoubs": (False, False), - "Only Recoubs": (True, True), - } + RECOUB_LABEL = {"No Recoubs": 0, "With Recoubs": 1, "Only Recoubs": 2} SPECIAL_LABEL = { "Share": (True, False, False), "Video only": (False, True, False), @@ -60,7 +52,7 @@ def translate_to_cli(options): options.v_quality = QUALITY_LABEL[options.v_quality] options.a_quality = QUALITY_LABEL[options.a_quality] options.aac = AAC_LABEL[options.aac] - options.recoubs, options.only_recoubs = RECOUB_LABEL[options.recoubs] + options.recoubs = RECOUB_LABEL[options.recoubs] options.share, options.v_only, options.a_only = SPECIAL_LABEL[options.special] return options @@ -115,8 +107,8 @@ def parse_cli(): input_.add_argument("--channels", default="", metavar="Channels", help="Download channels with the given names") input_.add_argument("--recoubs", metavar="Recoubs", - default=defs.RECOUB_LABEL[(defs.RECOUBS, defs.ONLY_RECOUBS)], - choices=["With Recoubs", "No Recoubs", "Only Recoubs"], + default=defs.RECOUB_LABEL[defs.RECOUBS], + choices=["No Recoubs", "With Recoubs", "Only Recoubs"], help="How to treat recoubs during channel downloads") input_.add_argument("--tags", default="", metavar="Tags", help="Download coubs with at least one of the given tags") diff --git a/coub.py b/coub.py index 621296d..f5ee6f4 100755 --- a/coub.py +++ b/coub.py @@ -114,12 +114,10 @@ class DefaultOptions: SHARE = False # How to treat recoubs during channel downloads - # RECOUBS = False -> Only Original - # RECOUBS = True -> Original + Recoubs - # ONLY_RECOUBS = True -> Only Recoubs - # RECOUBS mustn't be False, when ONLY_RECOUBS is True - RECOUBS = True - ONLY_RECOUBS = False + # 0 -> don't download recoubs + # 1 -> download recoubs + # 2 -> only download recoubs + RECOUBS = 1 # Preview a downloaded coub with the given command # Keyboard shortcuts may not work for CLI audio players @@ -208,14 +206,13 @@ def determine_value_type(option, value): "V_QUALITY", "A_QUALITY", "AAC", + "RECOUBS", "COUBS_PER_PAGE", "CHUNK_SIZE", ] bools = [ "KEEP", "SHARE", - "RECOUBS", - "ONLY_RECOUBS", "A_ONLY", "V_ONLY", ] @@ -657,9 +654,9 @@ def get_template(self): template = f"https://coub.com/api/v2/timeline/channel/{urlquote(self.id)}" template = f"{template}?per_page={opts.coubs_per_page}" - if not opts.recoubs: + if opts.recoubs == 0: template = f"{template}&type=simples" - elif opts.only_recoubs: + elif opts.recoubs == 2: template = f"{template}&type=recoubs" if self.sort in methods: @@ -1513,11 +1510,12 @@ def parse_cli(): default=defaults.AAC) # Channel Options recoub = parser.add_mutually_exclusive_group() - recoub.add_argument("--recoubs", action="store_true", default=defaults.RECOUBS) - recoub.add_argument("--no-recoubs", dest="recoubs", action="store_false", - default=defaults.RECOUBS) - recoub.add_argument("--only-recoubs", action="store_true", - default=defaults.ONLY_RECOUBS) + recoub.add_argument("--recoubs", action="store_const", + const=1, default=defaults.RECOUBS) + recoub.add_argument("--no-recoubs", dest="recoubs", action="store_const", + const=0, default=defaults.RECOUBS) + recoub.add_argument("--only-recoubs", dest="recoubs", action="store_const", + const=2, default=defaults.RECOUBS) # Preview Options player = parser.add_mutually_exclusive_group() player.add_argument("--preview", default=defaults.PREVIEW) diff --git a/example.conf b/example.conf index 9e3e14c..31ba62b 100644 --- a/example.conf +++ b/example.conf @@ -73,12 +73,10 @@ AAC = 1 SHARE = False # How to treat recoubs during channel downloads -# RECOUBS = False -> Only Original -# RECOUBS = True -> Original + Recoubs -# ONLY_RECOUBS = True -> Only Recoubs -# RECOUBS mustn't be False, when ONLY_RECOUBS is True -RECOUBS = True -ONLY_RECOUBS = False +# 0 -> don't download recoubs +# 1 -> download recoubs +# 2 -> only download recoubs +RECOUBS = 1 # Preview a downloaded coub with the given command # Keyboard shortcuts may not work for CLI audio players From 71d5bdedee3d6cefa23db8fcc780da96eb32fa4e Mon Sep 17 00:00:00 2001 From: HelpSeeker Date: Wed, 5 Feb 2020 22:01:42 +0100 Subject: [PATCH 11/40] Add validity test for default options Until now I followed a simple logic when it came to default options. If someone would take it upon themselves to change the default values in the script, then this person has the sole responsibility to ensure that the new values make sense. This stance becomes somewhat difficult to justify, now that reading custom default options (via coub.conf) is a central part of the script. Measures must be taken to prevent invalid default values. This is the goal of the new DefaultOptions.check_values() method. --- coub.py | 43 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 43 insertions(+) diff --git a/coub.py b/coub.py index f5ee6f4..bcf6d03 100755 --- a/coub.py +++ b/coub.py @@ -168,6 +168,7 @@ def __init__(self): config_path = os.path.join(d, "coub.conf") if os.path.exists(config_path): self.read_from_config(config_path) + self.check_values() def read_from_config(self, path): """Change default options based on user config file.""" @@ -194,6 +195,48 @@ def read_from_config(self, path): err(f"Unknown option in config file: {name}", color=fgcolors.WARNING) + def check_values(self): + """Test defaults for valid ranges and types.""" + checks = { + "VERBOSITY": (lambda x: x in [0, 1]), + "PROMPT": (lambda x: True), # Anything but yes/no will lead to prompt + "PATH": (lambda x: isinstance(x, str)), + "KEEP": (lambda x: isinstance(x, bool)), + "REPEAT": (lambda x: isinstance(x, int) and x > 0), + "DUR": (lambda x: isinstance(x, str) or x is None), + "CONNECT": (lambda x: isinstance(x, int) and x > 0), + "RETRIES": (lambda x: isinstance(x, int)), + "MAX_COUBS": (lambda x: not x or isinstance(x, int) and x > 0), + "V_QUALITY": (lambda x: x in [0, -1]), + "A_QUALITY": (lambda x: x in [0, -1]), + "V_MAX": (lambda x: x in ["higher", "high", "med"]), + "V_MIN": (lambda x: x in ["higher", "high", "med"]), + "AAC": (lambda x: x in [0, 1, 2, 3]), + "SHARE": (lambda x: isinstance(x, bool)), + "RECOUBS": (lambda x: x in [0, 1, 2]), + "PREVIEW": (lambda x: isinstance(x, str) or x is None), + "A_ONLY": (lambda x: isinstance(x, bool)), + "V_ONLY": (lambda x: isinstance(x, bool)), + "OUT_FILE": (lambda x: isinstance(x, str) or x is None), + "ARCHIVE_PATH": (lambda x: isinstance(x, str) or x is None), + "MERGE_EXT": (lambda x: x in ["mkv", "mp4", "asf", "avi", "flv", "f4v", "mov"]), + "OUT_FORMAT": (lambda x: isinstance(x, str) or x is None), + "COUBS_PER_PAGE": (lambda x: x in range(1, 26)), + "TAG_SEP": (lambda x: isinstance(x, str)), + "WRITE_METHOD": (lambda x: x in ["w", "a"]), + "CHUNK_SIZE": (lambda x: isinstance(x, int) and x > 0), + } + + errors = [] + for option in checks: + value = getattr(self, option) + if not checks[option](value): + errors.append((option, value)) + if errors: + for e in errors: + err(f"{e[0]}: invalid default value '{e[1]}'") + sys.exit(status.OPT) + @staticmethod def determine_value_type(option, value): """Convert values from config file (all strings) to the right type.""" From 641cff0ac3d24a52fb46f1ff3b9b35cc588df1c2 Mon Sep 17 00:00:00 2001 From: HelpSeeker Date: Wed, 5 Feb 2020 22:57:08 +0100 Subject: [PATCH 12/40] Simplify special string mapping for configs The previous approach was pretty convoluted, since I tried to catch some obvious user errors. This isn't necessary anymore, since invalid default options lead to a hard exit anyway. So instead of listing all options requiring special treatment, just convert special strings (i.e. Python keywords and integers) by default. There are some options which should interpret all values as strings. They will skip the integer conversion attempt, but "None", "True" and "False" will still be mapped to their respective Python keyword, as most of them are disabled via None. This could lead to some confusing error messages, if the user doesn't know about it. So let's also warn them in the example config. --- coub.py | 63 ++++++++++++++++++---------------------------------- example.conf | 3 ++- 2 files changed, 24 insertions(+), 42 deletions(-) diff --git a/coub.py b/coub.py index bcf6d03..993334b 100755 --- a/coub.py +++ b/coub.py @@ -82,7 +82,6 @@ class DefaultOptions: RETRIES = 5 # Limit how many coubs can be downloaded during one script invocation - # 0, None, etc. to disable MAX_COUBS = None # What video/audio quality to download @@ -184,12 +183,7 @@ def read_from_config(self, path): name = setting.split("=")[0].strip() value = setting.split("=")[1].strip() if hasattr(self, name): - try: - value = self.determine_value_type(name, value) - except ValueError: - err(f"{name}: Value ('{value}') has wrong type!", - color=fgcolors.WARNING) - continue + value = self.guess_string_type(name, value) setattr(self, name, value) else: err(f"Unknown option in config file: {name}", @@ -206,7 +200,7 @@ def check_values(self): "DUR": (lambda x: isinstance(x, str) or x is None), "CONNECT": (lambda x: isinstance(x, int) and x > 0), "RETRIES": (lambda x: isinstance(x, int)), - "MAX_COUBS": (lambda x: not x or isinstance(x, int) and x > 0), + "MAX_COUBS": (lambda x: isinstance(x, int) and x > 0 or x is None), "V_QUALITY": (lambda x: x in [0, -1]), "A_QUALITY": (lambda x: x in [0, -1]), "V_MAX": (lambda x: x in ["higher", "high", "med"]), @@ -238,46 +232,33 @@ def check_values(self): sys.exit(status.OPT) @staticmethod - def determine_value_type(option, value): + def guess_string_type(option, string): """Convert values from config file (all strings) to the right type.""" - ints = [ - "VERBOSITY", - "REPEAT", - "CONNECT", - "RETRIES", - "MAX_COUBS", - "V_QUALITY", - "A_QUALITY", - "AAC", - "RECOUBS", - "COUBS_PER_PAGE", - "CHUNK_SIZE", - ] - bools = [ - "KEEP", - "SHARE", - "A_ONLY", - "V_ONLY", - ] - none_allowed = [ + specials = { + "None": None, + "True": True, + "False": False, + } + # Some options should not follow the above directives + # Usually options which are supposed to ONLY take strings + exceptions = [ + "PATH", "DUR", "PREVIEW", "OUT_FILE", "ARCHIVE_PATH", + "OUT_FORMAT", + "TAG_SEP", ] - if option in ints: - return int(value) - if option in bools: - if value == "True": - return True - if value == "False": - return False - raise ValueError - if option in none_allowed: - if value == "None": - return None - return value + if string in specials: + return specials[string] + if option in exceptions: + return string + try: + return int(string) + except ValueError: + return string # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ diff --git a/example.conf b/example.conf index 31ba62b..b54205e 100644 --- a/example.conf +++ b/example.conf @@ -2,6 +2,7 @@ # To use a custom config, place a file called 'coub.conf' in the same location as the script # Lines starting with '#' are treated as comments +# True, False and None (case-sensitive) will be converted to Python keywords. They should be avoided as paths or output names. # Change verbosity of the script # 0 -> only errors and warnings @@ -112,7 +113,7 @@ MERGE_EXT = mkv # Usage of an archive file is recommended OUT_FORMAT = %id% -# What character o string to use to separate tags in the output filename +# What character or string to use to separate tags in the output filename TAG_SEP = _ # How to write parsed links to a file (see OUT_FILE) From e0d79648af4bae212d39831acb828f420ecafcd5 Mon Sep 17 00:00:00 2001 From: HelpSeeker Date: Wed, 5 Feb 2020 23:04:50 +0100 Subject: [PATCH 13/40] Use str.join() to replace %tags% in output name I don't even know how to excuse this. I just didn't know better when I initially wrote it and I guess it slipped my attention until now. --- coub.py | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-) diff --git a/coub.py b/coub.py index 993334b..0eee1dd 100755 --- a/coub.py +++ b/coub.py @@ -1721,10 +1721,7 @@ def get_name(req_json, c_id): except (KeyError, TypeError, IndexError): name = name.replace("%community%", "") - tags = "" - for t in req_json['tags']: - # Don't add tag separator after the last tag - tags += f"{t['title']}{opts.tag_sep if t != req_json['tags'][-1] else ''}" + tags = opts.tag_sep.join([t['title'] for t in req_json['tags']]) name = name.replace("%tags%", tags) # Strip/replace special characters that can lead to script failure (ffmpeg concat) From c54ccf41c95ae64750eab15812dace8ef16a8189 Mon Sep 17 00:00:00 2001 From: HelpSeeker Date: Wed, 5 Feb 2020 23:33:10 +0100 Subject: [PATCH 14/40] Add example extension to filename length test --- coub.py | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/coub.py b/coub.py index 0eee1dd..dc64680 100755 --- a/coub.py +++ b/coub.py @@ -1731,9 +1731,10 @@ def get_name(req_json, c_id): name = name.replace("\n", " ") try: - f = open(name, "w") + # Add example extension to simulate the full name length + f = open(f"{name}.ext", "w") f.close() - os.remove(name) + os.remove(f"{name}.ext") except OSError: err(f"Error: Filename invalid or too long! Falling back to '{c_id}'", color=fgcolors.WARNING) From a537f7f74d3f8b01825daa0d1f4a1fe33120b012 Mon Sep 17 00:00:00 2001 From: HelpSeeker Date: Thu, 6 Feb 2020 15:41:34 +0100 Subject: [PATCH 15/40] Update descriptions in example.conf --- example.conf | 105 ++++++++++++++++++++++++++++++--------------------- 1 file changed, 62 insertions(+), 43 deletions(-) diff --git a/example.conf b/example.conf index b54205e..5f8f86c 100644 --- a/example.conf +++ b/example.conf @@ -1,65 +1,76 @@ -# This is an example configuration file for CoubDownloader -# To use a custom config, place a file called 'coub.conf' in the same location as the script +# This is an example configuration file for CoubDownloader. +# To use a custom config, place a file called 'coub.conf' in the same location as the script. +# +# Lines starting with '#' are treated as comments. +# +# True, False and None (case-sensitive) will be converted to Python keywords. +# On their own they do NOT count as strings (important for paths and other string-only options). -# Lines starting with '#' are treated as comments -# True, False and None (case-sensitive) will be converted to Python keywords. They should be avoided as paths or output names. -# Change verbosity of the script -# 0 -> only errors and warnings + +# Verbosity of the command line output +# 0 -> quiet mode (only errors/warnings/prompts) # 1 -> normal verbosity +# Allowed values: 0, 1 VERBOSITY = 1 -# How to treat overwrite prompts -# prompt -> prompt the user -# yes -> always answer with yes -# no -> always answer with no +# How to answer overwrite prompts +# yes -> always answer yes +# no -> always answer no +# Allowed values: no restrictions (everything but yes/no, will lead to a prompt) PROMPT = prompt # Default download destination +# ~ and ~user do NOT get expanded and will be treated literally +# Allowed values: absolute/relative paths (don't have to exist yet) PATH = . -# Keep individual video/audio streams +# Whether to keep the individual video/audio streams after merging +# Allowed values: True, False KEEP = False # How often to loop the video -# Only has an effect, if the looped video is shorter than the audio -# Otherwise the max. length is limited by the audio duration +# Max. duration is still limited by the audio +# Allowed values: integers >0 REPEAT = 1000 -# Max. coub duration (FFmpeg syntax) -# Can be used in combination with repeat, in which case the shorter duration will be used +# Limit duration (FFmpeg syntax) +# https://ffmpeg.org/ffmpeg-utils.html#time-duration-syntax +# Between REPEAT and DUR the shorter option will decide the length +# Allowed values: FFmpeg time syntax, None (to disable) DUR = None -# Max no. of connections to use -# Raising this value can lead to shorter download times, but also increases the risk of Coub throttling or terminating your connections -# There's no benefit in higher values, if your connection is already fully utilized +# Max. number of connections to use +# High values can speed up downloads, but also increase the risk of being throttled or disconnected +# More connections only make sense, if your download speed isn't fully utilized yet +# Allowed values: integers >0 CONNECT = 25 -# How often to retry download when connection is lost +# How often to reestablish the connection to Coub if it is lost # >0 -> retry the specified number of times # 0 -> don't retry # <0 -> retry indefinitely +# Allowed values: integers RETRIES = 5 -# Limit how many links to parse during one run -# >0 -> limit to the specified number +# Limit how many links to parse during one script invocation +# >0 -> limit to the specified amount # 0 -> no limit -MAX_COUBS = 0 +# Allowed values: integers >0, None (to disable) +MAX_COUBS = None -# What video/audio quality to download +# What video/audio quality to download from the range of available streams # 0 -> worst quality # -1 -> best quality -# Everything else can lead to undefined behavior +# Allowed values: 0, -1 V_QUALITY = -1 A_QUALITY = -1 -# Limits for the list of video streams -# V_MAX: limit what counts as best stream -# V_MIN: limit what counts as worst stream -# Supported values: -# med ( ~640px width) -# high (~1280px width) -# higher (~1600px width) +# Limit the range of video streams to choose from +# med -> ~640px width +# high -> ~1280px width +# higher -> ~1600px width +# Allowed values: med, high, higher V_MAX = higher V_MIN = med @@ -68,35 +79,43 @@ V_MIN = med # 1 -> rank it between low and high quality MP3 # 2 -> prefer AAC, use MP3 fallback # 3 -> either AAC or no audio +# Allowed values: 0, 1, 2, 3 AAC = 1 -# Download shared video+audio version instead of merging separate streams +# Download 'shared' video (includes audio) instead of merging separate streams +# This provides the same videos as Coub's own 'Download' button +# Allowed values: True, False SHARE = False # How to treat recoubs during channel downloads # 0 -> don't download recoubs # 1 -> download recoubs # 2 -> only download recoubs +# Allowed values: 0, 1, 2 RECOUBS = 1 -# Preview a downloaded coub with the given command +# Call a program (i.e. media player) to display downloaded coubs as soon as they are ready # Keyboard shortcuts may not work for CLI audio players +# Allowed values: invokable command, None (to disable) PREVIEW = None -# Only download video/audio stream -# A_ONLY and V_ONLY are mutually exclusive +# Only download video or audio streams +# These options are mutually exclusive (i.e. both mustn't be True at the same time) +# Allowed values: True, False A_ONLY = False V_ONLY = False # Write parsed links to the specified file and exit (don't download coubs) +# How to open the file can be set with WRITE_METHOD (see further below) +# Allowed values: absolute paths, None (to disable) OUT_FILE = None # Use an archive file to keep track of downloaded coubs +# Allowed values: absolute/relative paths, None (to disable) ARCHIVE_PATH = None # Container to merge separate video/audio streams into -# Must support AVC video and AAC/MP3 audio (e.g. mkv or mp4) -# See: https://en.wikipedia.org/wiki/Comparison_of_video_container_formats +# Allowed values: mkv, mp4, asf, avi, flv, f4v, mov MERGE_EXT = mkv # Output name formatting @@ -108,15 +127,15 @@ MERGE_EXT = mkv # %channel% - channel title # %tags% - all tags (separated by TAG_SEP, see below) # All other strings are interpreted literally. -# -# Setting a custom value increases skip duration for existing coubs -# Usage of an archive file is recommended +# Setting a custom value increases skip duration for existing coubs (usage of an archive file is recommended) +# Allowed values: strings, None (same as %id%) OUT_FORMAT = %id% -# What character or string to use to separate tags in the output filename +# Character or string to separate tags in the output filename +# Allowed values: strings TAG_SEP = _ # How to write parsed links to a file (see OUT_FILE) -# w -> overwrite -# a -> append +# w -> overwrite OUT_FILE +# a -> append to OUT_FILE WRITE_METHOD = w From cb54ee6fc4e2c800b51ab287dc1c78959bcfe206 Mon Sep 17 00:00:00 2001 From: HelpSeeker Date: Thu, 6 Feb 2020 15:47:05 +0100 Subject: [PATCH 16/40] Obfuscate default options in script Using a config file should be the preferred way to change defaults and example.conf describes all options in detail (more so than the comments in the main script). That's why I'll stop treating the DefaultOptions class special. It's moved further down and grouped with other class definitions and the extensive descriptions were removed. Of course it's still easy to change default options within the script, but making it as easy as possible is not a priority anymore. --- coub.py | 208 +++++++++++++++++--------------------------------------- 1 file changed, 61 insertions(+), 147 deletions(-) diff --git a/coub.py b/coub.py index dc64680..d7ecfd6 100755 --- a/coub.py +++ b/coub.py @@ -36,129 +36,93 @@ colors = False # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -# Default Options +# Classes For Global Variables +# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +class ExitCodes: + """Store exit codes for non-successful execution.""" + + DEP = 1 # missing required software + OPT = 2 # invalid user-specified option + RUN = 3 # misc. runtime error + DOWN = 4 # failed to download all input links (existence == success) + INT = 5 # early termination was requested by the user (i.e. Ctrl+C) + CONN = 6 # connection either couldn't be established or was lost + + +class Colors: + """Store ANSI escape codes for colorized output.""" + + # https://en.wikipedia.org/wiki/ANSI_escape_code#Colors + ERROR = '\033[31m' # red + WARNING = '\033[33m' # yellow + SUCCESS = '\033[32m' # green + RESET = '\033[0m' + + def disable(self): + """Disable colorized output by removing escape codes.""" + # I'm not going to stop addressing these attributes as constants, just + # because Windows thinks it needs to be special + self.ERROR = '' + self.SUCCESS = '' + self.WARNING = '' + self.RESET = '' + +# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +# Global Variables +# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +status = ExitCodes() +fgcolors = Colors() +if not colors: + fgcolors.disable() + +total = 0 +count = 0 +done = 0 + +# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +# Classes # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ class DefaultOptions: """Define and store all import user settings.""" - # Change verbosity of the script - # 0 for quiet, >= 1 for normal verbosity + # Common defaults VERBOSITY = 1 - - # yes/no will answer prompts automatically - # Everything else will lead to a prompt PROMPT = None - - # Default download destination PATH = "." - - # Keep individual video/audio streams KEEP = False - - # How often to loop the video - # Only has an effect, if the looped video is shorter than the audio - # Otherwise the max. length is limited by the audio duration REPEAT = 1000 - - # Max. coub duration (FFmpeg syntax) - # Can be used in combination with repeat, in which case the shorter - # duration will be used DUR = None - - # Max no. of connections for aiohttp's ClientSession - # Raising this value can lead to shorter download times, but also - # increases the risk of Coub throttling or terminating your connections - # There's also no benefit in higher values, if your connection is already - # fully utilized + # Download defaults CONNECT = 25 - - # How often to retry download when connection is lost - # >0 -> retry the specified number of times - # 0 -> don't retry - # <0 -> retry indefinitely - # Retries happen through recursion, so the max. number is theoretically - # limited to 1000 retries (although Python's limit could be raised as well) RETRIES = 5 - - # Limit how many coubs can be downloaded during one script invocation MAX_COUBS = None - - # What video/audio quality to download - # 0 -> worst quality - # -1 -> best quality - # Everything else can lead to undefined behavior + # Format defaults V_QUALITY = -1 A_QUALITY = -1 - - # Limits for the list of video streams - # V_MAX: limits what counts as best stream - # V_MIN: limits what counts as worst stream - # Supported values: - # med ( ~640px width) - # high (~1280px width) - # higher (~1600px width) - V_MAX = 'higher' - V_MIN = 'med' - - # How much to prefer AAC audio - # 0 -> never download AAC audio - # 1 -> rank it between low and high quality MP3 - # 2 -> prefer AAC, use MP3 fallback - # 3 -> either AAC or no audio + V_MAX = "higher" + V_MIN = "med" AAC = 1 - - # Use shared video+audio instead of merging separate streams - # Leads to shorter videos, also no further quality selection SHARE = False - - # How to treat recoubs during channel downloads - # 0 -> don't download recoubs - # 1 -> download recoubs - # 2 -> only download recoubs + # Channel defaults RECOUBS = 1 - - # Preview a downloaded coub with the given command - # Keyboard shortcuts may not work for CLI audio players + # Preview defaults PREVIEW = None - - # Only download video/audio stream - # A_ONLY and V_ONLY are mutually exclusive + # Misc. defaults A_ONLY = False V_ONLY = False - - # Output parsed coubs to file instead of downloading - # Values other than None will terminate the script after the initial - # parsing process (i.e. no coubs will be downloaded) OUT_FILE = None - - # Use an archive file to keep track of downloaded coubs ARCHIVE_PATH = None - - # Container to merge separate video/audio streams into - # Must support AVC video and AAC/MP3 audio (e.g. mkv or mp4) - # See: https://en.wikipedia.org/wiki/Comparison_of_video_container_formats + # Output defaults MERGE_EXT = "mkv" - - # Output name formatting (default: %id%) - # Supports the following special keywords: - # %id% - coub ID (identifier in the URL) - # %title% - coub title - # %creation% - creation date/time - # %community% - coub community - # %channel% - channel title - # %tags% - all tags (separated by tag_sep, see below) - # All other strings are interpreted literally. - # - # Setting a custom value increases skip duration for existing coubs - # Usage of an archive file is recommended in such an instance OUT_FORMAT = "%id%" - - # Advanced options - COUBS_PER_PAGE = 25 # allowed: 1-25 + # Advanced defaults + COUBS_PER_PAGE = 25 TAG_SEP = "_" - WRITE_METHOD = "w" # w -> overwrite, a -> append - CHUNK_SIZE = 1024 # in Bytes + WRITE_METHOD = "w" + CHUNK_SIZE = 1024 def __init__(self): # Only supports script's location for now, but write it to be extensible @@ -239,7 +203,7 @@ def guess_string_type(option, string): "True": True, "False": False, } - # Some options should not follow the above directives + # Some options should not undergo integer conversion # Usually options which are supposed to ONLY take strings exceptions = [ "PATH", @@ -261,56 +225,6 @@ def guess_string_type(option, string): return string -# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -# Classes For Global Variables -# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -class ExitCodes: - """Store exit codes for non-successful execution.""" - - DEP = 1 # missing required software - OPT = 2 # invalid user-specified option - RUN = 3 # misc. runtime error - DOWN = 4 # failed to download all input links (existence == success) - INT = 5 # early termination was requested by the user (i.e. Ctrl+C) - CONN = 6 # connection either couldn't be established or was lost - - -class Colors: - """Store ANSI escape codes for colorized output.""" - - # https://en.wikipedia.org/wiki/ANSI_escape_code#Colors - ERROR = '\033[31m' # red - WARNING = '\033[33m' # yellow - SUCCESS = '\033[32m' # green - RESET = '\033[0m' - - def disable(self): - """Disable colorized output by removing escape codes.""" - # I'm not going to stop addressing these attributes as constants, just - # because Windows thinks it needs to be special - self.ERROR = '' - self.SUCCESS = '' - self.WARNING = '' - self.RESET = '' - -# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -# Global Variables -# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -status = ExitCodes() -fgcolors = Colors() -if not colors: - fgcolors.disable() - -total = 0 -count = 0 -done = 0 - -# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -# Classes -# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - class InputHelp(argparse.Action): """Custom action to print input help.""" From 5e911d422b4393ff42be05cf2dbf35be3a0ef6cf Mon Sep 17 00:00:00 2001 From: HelpSeeker Date: Thu, 6 Feb 2020 17:42:22 +0100 Subject: [PATCH 17/40] Change a selected few option names This gets rid of a few unnecessary abbreviations and generally confusing names (e.g. out_file for the path specified with --write-list), which would look odd as options in a configuration file. --- coub-gui.py | 34 +++++++++---------- coub.py | 92 ++++++++++++++++++++++++++-------------------------- example.conf | 20 ++++++------ 3 files changed, 73 insertions(+), 73 deletions(-) diff --git a/coub-gui.py b/coub-gui.py index ba8d661..6e3fc0d 100755 --- a/coub-gui.py +++ b/coub-gui.py @@ -28,10 +28,10 @@ class GuiDefaultOptions(coub.DefaultOptions): AAC_LABEL = ["Only MP3", "No Bias", "Prefer AAC", "Only AAC"] RECOUB_LABEL = ["No Recoubs", "With Recoubs", "Only Recoubs"] SPECIAL_LABEL = { + (False, False, False): "None", (True, False, False): "Share", (False, True, False): "Video only", (False, False, True): "Audio only", - (False, False, False): None, } @@ -42,10 +42,10 @@ def translate_to_cli(options): AAC_LABEL = {"Only MP3": 0, "No Bias": 1, "Prefer AAC": 2, "Only AAC": 3} RECOUB_LABEL = {"No Recoubs": 0, "With Recoubs": 1, "Only Recoubs": 2} SPECIAL_LABEL = { + "None": (False, False, False), "Share": (True, False, False), "Video only": (False, True, False), "Audio only": (False, False, True), - None: (False, False, False), } # Convert GUI labels to valid options for the main script @@ -130,13 +130,13 @@ def parse_cli(): metavar="Prompt Behavior", help="How to answer user prompts") common.add_argument("--repeat", type=coub.positive_int, default=defs.REPEAT, metavar="Loop Count", help="How often to loop the video stream") - common.add_argument("--dur", type=coub.valid_time, default=defs.DUR, + common.add_argument("--duration", type=coub.valid_time, default=defs.DURATION, metavar="Limit duration", help="Max. duration of the output (FFmpeg syntax)") common.add_argument("--preview", default=defs.PREVIEW, metavar="Preview Command", help="Command to invoke to preview each finished coub") - common.add_argument("--archive-path", type=coub.valid_archive, - default=defs.ARCHIVE_PATH, widget="FileSaver", + common.add_argument("--archive", type=coub.valid_archive, + default=defs.ARCHIVE, widget="FileSaver", metavar="Archive", gooey_options={'message': "Choose archive file"}, help="Use an archive file to keep track of already downloaded coubs") common.add_argument("--keep", action=f"store_{'false' if defs.KEEP else 'true'}", @@ -145,8 +145,8 @@ def parse_cli(): # Download Options download = parser.add_argument_group("Download", gooey_options={'columns': 1}) - download.add_argument("--connect", type=coub.positive_int, - default=defs.CONNECT, metavar="Number of connections", + download.add_argument("--connections", type=coub.positive_int, + default=defs.CONNECTIONS, metavar="Number of connections", help="How many connections to use (>100 not recommended)") download.add_argument("--retries", type=int, default=defs.RETRIES, metavar="Retry Attempts", @@ -173,14 +173,14 @@ def parse_cli(): formats.add_argument("--aac", default=defs.AAC_LABEL[defs.AAC], choices=["Only MP3", "No Bias", "Prefer AAC", "Only AAC"], metavar="Audio Format", help="How much to prefer AAC over MP3") - formats.add_argument("--special", choices=["Share", "Video only", "Audio only"], + formats.add_argument("--special", choices=["None", "Share", "Video only", "Audio only"], default=defs.SPECIAL_LABEL[(defs.SHARE, defs.V_ONLY, defs.A_ONLY)], metavar="Special Formats", help="Use a special format selection") # Output output = parser.add_argument_group("Output", gooey_options={'columns': 1}) - output.add_argument("--out-file", type=os.path.abspath, widget="FileSaver", - default=defs.OUT_FILE, metavar="Output to List", + output.add_argument("--output-list", type=os.path.abspath, widget="FileSaver", + default=defs.OUTPUT_LIST, metavar="Output to List", gooey_options={'message': "Save link list"}, help="Save all parsed links in a list (no download)") output.add_argument("--path", type=os.path.abspath, default=defs.PATH, @@ -195,7 +195,7 @@ def parse_cli(): choices=["mkv", "mp4", "asf", "avi", "flv", "f4v", "mov"], help="What extension to use for merged output files " "(has no effect if no merge is required)") - output.add_argument("--out-format", default=defs.OUT_FORMAT, + output.add_argument("--name-template", default=defs.NAME_TEMPLATE, metavar="Name Template", help=dedent(f"""\ Change the naming convention of output files @@ -237,15 +237,15 @@ def parse_cli(): args.input.append(coub.RandomCategory()) # Read archive content - if args.archive_path and os.path.exists(args.archive_path): - with open(args.archive_path, "r") as f: - args.archive = [l.strip() for l in f] + if args.archive and os.path.exists(args.archive): + with open(args.archive, "r") as f: + args.archive_content = [l.strip() for l in f] else: - args.archive = None + args.archive_content = None # The default naming scheme is the same as using %id% # but internally the default value is None - if args.out_format == "%id%": - args.out_format = None + if args.name_template == "%id%": + args.name_template = None return translate_to_cli(args) diff --git a/coub.py b/coub.py index d7ecfd6..e66df1c 100755 --- a/coub.py +++ b/coub.py @@ -94,9 +94,9 @@ class DefaultOptions: PATH = "." KEEP = False REPEAT = 1000 - DUR = None + DURATION = None # Download defaults - CONNECT = 25 + CONNECTIONS = 25 RETRIES = 5 MAX_COUBS = None # Format defaults @@ -113,11 +113,11 @@ class DefaultOptions: # Misc. defaults A_ONLY = False V_ONLY = False - OUT_FILE = None - ARCHIVE_PATH = None + OUTPUT_LIST = None + ARCHIVE = None # Output defaults MERGE_EXT = "mkv" - OUT_FORMAT = "%id%" + NAME_TEMPLATE = "%id%" # Advanced defaults COUBS_PER_PAGE = 25 TAG_SEP = "_" @@ -161,8 +161,8 @@ def check_values(self): "PATH": (lambda x: isinstance(x, str)), "KEEP": (lambda x: isinstance(x, bool)), "REPEAT": (lambda x: isinstance(x, int) and x > 0), - "DUR": (lambda x: isinstance(x, str) or x is None), - "CONNECT": (lambda x: isinstance(x, int) and x > 0), + "DURATION": (lambda x: isinstance(x, str) or x is None), + "CONNECTIONS": (lambda x: isinstance(x, int) and x > 0), "RETRIES": (lambda x: isinstance(x, int)), "MAX_COUBS": (lambda x: isinstance(x, int) and x > 0 or x is None), "V_QUALITY": (lambda x: x in [0, -1]), @@ -175,10 +175,10 @@ def check_values(self): "PREVIEW": (lambda x: isinstance(x, str) or x is None), "A_ONLY": (lambda x: isinstance(x, bool)), "V_ONLY": (lambda x: isinstance(x, bool)), - "OUT_FILE": (lambda x: isinstance(x, str) or x is None), - "ARCHIVE_PATH": (lambda x: isinstance(x, str) or x is None), + "OUTPUT_LIST": (lambda x: isinstance(x, str) or x is None), + "ARCHIVE": (lambda x: isinstance(x, str) or x is None), "MERGE_EXT": (lambda x: x in ["mkv", "mp4", "asf", "avi", "flv", "f4v", "mov"]), - "OUT_FORMAT": (lambda x: isinstance(x, str) or x is None), + "NAME_TEMPLATE": (lambda x: isinstance(x, str) or x is None), "COUBS_PER_PAGE": (lambda x: x in range(1, 26)), "TAG_SEP": (lambda x: isinstance(x, str)), "WRITE_METHOD": (lambda x: x in ["w", "a"]), @@ -207,11 +207,11 @@ def guess_string_type(option, string): # Usually options which are supposed to ONLY take strings exceptions = [ "PATH", - "DUR", + "DURATION", "PREVIEW", - "OUT_FILE", - "ARCHIVE_PATH", - "OUT_FORMAT", + "OUTPUT_LIST", + "ARCHIVE", + "NAME_TEMPLATE", "TAG_SEP", ] @@ -280,7 +280,7 @@ def format_help(self): -d, --duration TIME specify max. coub duration (FFmpeg syntax) Download options: - --connections N max. number of connections (def: {self.get_default("connect")}) + --connections N max. number of connections (def: {self.get_default("connections")}) --retries N number of retries when connection is lost (def: {self.get_default("retries")}) 0 to disable, <0 to retry indefinitely --limit-num LIMIT limit max. number of downloaded coubs @@ -316,7 +316,7 @@ def format_help(self): Output: --ext EXTENSION merge output with the given extension (def: {self.get_default("merge_ext")}) ignored if no merge is required - -o, --output FORMAT save output with the given template (def: {self.get_default("out_format")}) + -o, --output FORMAT save output with the given template (def: {self.get_default("name_template")}) Special strings: %id% - coub ID (identifier in the URL) @@ -551,7 +551,7 @@ async def process(self, quantity=None): msg(f" {pages} out of {self.pages} pages") tout = aiohttp.ClientTimeout(total=None) - conn = aiohttp.TCPConnector(limit=opts.connect) + conn = aiohttp.TCPConnector(limit=opts.connections) async with aiohttp.ClientSession(timeout=tout, connector=conn) as session: tasks = [parse_page(req, session) for req in requests] links = await asyncio.gather(*tasks) @@ -889,14 +889,14 @@ def check_existence(self): if self.erroneous(): return - if opts.archive and self.id in opts.archive: + if opts.archive_content and self.id in opts.archive_content: self.exists = True return old_file = None # Existence of self.name indicates whether API request was already # made (i.e. if 1st or 2nd check) - if not opts.out_format: + if not opts.name_template: if not self.name: old_file = exists(self.id) else: @@ -1013,8 +1013,8 @@ def merge(self): "-f", "concat", "-safe", "0", "-i", f"file:{t_name}", "-i", f"file:{self.a_name}", ] - if opts.dur: - command.extend(["-t", opts.dur]) + if opts.duration: + command.extend(["-t", opts.duration]) command.extend(["-c", "copy", "-shortest", f"file:temp_{m_name}"]) subprocess.run(command) @@ -1037,7 +1037,7 @@ def archive(self): if self.erroneous(): return - with open(opts.archive_path, "a") as f: + with open(opts.archive, "a") as f: print(self.id, file=f) def preview(self): @@ -1073,7 +1073,7 @@ async def process(self, session=None): # 2nd existence check # Handles custom names exclusively (slower since API request necessary) - if opts.out_format: + if opts.name_template: self.check_existence() # Download @@ -1088,7 +1088,7 @@ async def process(self, session=None): # of valid streams with special format options (e.g. --video-only) self.done = True - if opts.archive_path: + if opts.archive: self.archive() if opts.preview: self.preview() @@ -1419,11 +1419,11 @@ def parse_cli(): repeat.add_argument("-r", "--repeat", type=positive_int, default=defaults.REPEAT) parser.add_argument("-p", "--path", type=os.path.abspath, default=defaults.PATH) parser.add_argument("-k", "--keep", action="store_true", default=defaults.KEEP) - parser.add_argument("-d", "--duration", dest="dur", type=valid_time, - default=defaults.DUR) + parser.add_argument("-d", "--duration", type=valid_time, + default=defaults.DURATION) # Download Options - parser.add_argument("--connections", dest="connect", type=positive_int, - default=defaults.CONNECT) + parser.add_argument("--connections", type=positive_int, + default=defaults.CONNECTIONS) parser.add_argument("--retries", type=int, default=defaults.RETRIES) parser.add_argument("--limit-num", dest="max_coubs", type=positive_int, default=defaults.MAX_COUBS) @@ -1466,15 +1466,15 @@ def parse_cli(): stream.add_argument("--video-only", dest="v_only", action="store_true", default=defaults.V_ONLY) stream.add_argument("--share", action="store_true", default=defaults.SHARE) - parser.add_argument("--write-list", dest="out_file", type=os.path.abspath, - default=defaults.OUT_FILE) - parser.add_argument("--use-archive", dest="archive_path", type=valid_archive, - default=defaults.ARCHIVE_PATH) + parser.add_argument("--write-list", dest="output_list", type=os.path.abspath, + default=defaults.OUTPUT_LIST) + parser.add_argument("--use-archive", dest="archive", type=valid_archive, + default=defaults.ARCHIVE) # Output parser.add_argument("--ext", dest="merge_ext", default=defaults.MERGE_EXT, choices=["mkv", "mp4", "asf", "avi", "flv", "f4v", "mov"]) - parser.add_argument("-o", "--output", dest="out_format", - default=defaults.OUT_FORMAT) + parser.add_argument("-o", "--output", dest="name_template", + default=defaults.NAME_TEMPLATE) # Advanced Options parser.set_defaults( @@ -1496,15 +1496,15 @@ def parse_cli(): else: args.input = args.raw_input # Read archive content - if args.archive_path and os.path.exists(args.archive_path): - with open(args.archive_path, "r") as f: - args.archive = [l.strip() for l in f] + if args.archive and os.path.exists(args.archive): + with open(args.archive, "r") as f: + args.archive_content = [l.strip() for l in f] else: - args.archive = None + args.archive_content = None # The default naming scheme is the same as using %id% # but internally the default value is None - if args.out_format == "%id%": - args.out_format = None + if args.name_template == "%id%": + args.name_template = None return args @@ -1611,19 +1611,19 @@ def parse_input(sources): def write_list(links): """Output parsed links to a list and exit.""" - with open(opts.out_file, opts.write_method) as f: + with open(opts.output_list, opts.write_method) as f: for l in links: print(l, file=f) - msg(f"\nParsed coubs written to '{opts.out_file}'!", + msg(f"\nParsed coubs written to '{opts.output_list}'!", color=fgcolors.SUCCESS) def get_name(req_json, c_id): """Assemble final output name of a given coub.""" - if not opts.out_format: + if not opts.name_template: return c_id - name = opts.out_format + name = opts.name_template name = name.replace("%id%", c_id) name = name.replace("%title%", req_json['title']) @@ -1879,7 +1879,7 @@ async def process(coubs): """Call the process function of all parsed coubs.""" if aio: tout = aiohttp.ClientTimeout(total=None) - conn = aiohttp.TCPConnector(limit=opts.connect) + conn = aiohttp.TCPConnector(limit=opts.connections) try: async with aiohttp.ClientSession(timeout=tout, connector=conn) as session: tasks = [c.process(session) for c in coubs] @@ -1938,7 +1938,7 @@ def main(): msg("\n### Parse Input ###") links = parse_input(opts.input) - if opts.out_file: + if opts.output_list: write_list(links) sys.exit(0) total = len(links) diff --git a/example.conf b/example.conf index 5f8f86c..fbed3b2 100644 --- a/example.conf +++ b/example.conf @@ -38,13 +38,13 @@ REPEAT = 1000 # https://ffmpeg.org/ffmpeg-utils.html#time-duration-syntax # Between REPEAT and DUR the shorter option will decide the length # Allowed values: FFmpeg time syntax, None (to disable) -DUR = None +DURATION = None # Max. number of connections to use # High values can speed up downloads, but also increase the risk of being throttled or disconnected # More connections only make sense, if your download speed isn't fully utilized yet # Allowed values: integers >0 -CONNECT = 25 +CONNECTIONS = 25 # How often to reestablish the connection to Coub if it is lost # >0 -> retry the specified number of times @@ -108,11 +108,16 @@ V_ONLY = False # Write parsed links to the specified file and exit (don't download coubs) # How to open the file can be set with WRITE_METHOD (see further below) # Allowed values: absolute paths, None (to disable) -OUT_FILE = None +OUTPUT_LIST = None + +# How to write parsed links to a file (see OUT_FILE) +# w -> overwrite OUT_FILE +# a -> append to OUT_FILE +WRITE_METHOD = w # Use an archive file to keep track of downloaded coubs # Allowed values: absolute/relative paths, None (to disable) -ARCHIVE_PATH = None +ARCHIVE = None # Container to merge separate video/audio streams into # Allowed values: mkv, mp4, asf, avi, flv, f4v, mov @@ -129,13 +134,8 @@ MERGE_EXT = mkv # All other strings are interpreted literally. # Setting a custom value increases skip duration for existing coubs (usage of an archive file is recommended) # Allowed values: strings, None (same as %id%) -OUT_FORMAT = %id% +NAME_TEMPLATE = %id% # Character or string to separate tags in the output filename # Allowed values: strings TAG_SEP = _ - -# How to write parsed links to a file (see OUT_FILE) -# w -> overwrite OUT_FILE -# a -> append to OUT_FILE -WRITE_METHOD = w From 1b077c3c98b71a96c1bf7bf3d90d399734942094 Mon Sep 17 00:00:00 2001 From: HelpSeeker Date: Sun, 16 Feb 2020 19:31:46 +0100 Subject: [PATCH 18/40] Check explicitly for certificate verification fail While testing standalone binaries in different VMs (notably the most recent Debian and Ubuntu versions) there were ambiguous connection failure errors, despite a working network connection. As it turned out the CA certificate verification failed, which was easy enough to fix, but only after I knew what was going on. So let's print a special message for this particular error. --- coub.py | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/coub.py b/coub.py index e66df1c..93174fb 100755 --- a/coub.py +++ b/coub.py @@ -8,6 +8,7 @@ import sys from math import ceil +from ssl import SSLCertVerificationError from textwrap import dedent import urllib.error @@ -1154,8 +1155,11 @@ def check_connection(): """Check if user can connect to coub.com.""" try: urlopen("https://coub.com/") - except urllib.error.URLError: - err("Unable to connect to coub.com! Please check your connection.") + except urllib.error.URLError as error: + if isinstance(error.reason, SSLCertVerificationError): + err("Certificate verification failed! Please update your CA certificates.") + else: + err("Unable to connect to coub.com! Please check your connection.") sys.exit(status.CONN) From 4ea8551f2a750ee799c2e8c3a8b033a0ac00d686 Mon Sep 17 00:00:00 2001 From: HelpSeeker Date: Sun, 16 Feb 2020 20:46:44 +0100 Subject: [PATCH 19/40] Remove support for synchronous download via urllib I always wanted to avoid dependencies outside the standard library or at least make them optional. The main motivation was to enable easier setup for people, who weren't familiar with Python environments. Once this development branch gets merged into master, I plan to provide standalone binaries however, which allows me to not worry (as much) about using 3rd party libraries. This commit promotes aiohttp to a necessary requirement. --- README.md | 2 +- coub.py | 107 +++++++++++++++++------------------------------------- 2 files changed, 34 insertions(+), 75 deletions(-) diff --git a/README.md b/README.md index edc2abb..95f6b47 100644 --- a/README.md +++ b/README.md @@ -113,11 +113,11 @@ Output: ## Requirements * Python >= 3.7 +* [aiohttp](https://aiohttp.readthedocs.io/en/stable/) * [FFmpeg](https://www.ffmpeg.org/) ### Optional -* [aiohttp](https://aiohttp.readthedocs.io/en/stable/) for asynchronous execution **(recommended)** * [colorama](https://github.com/tartley/colorama) for colorized terminal output on Windows * [Gooey](https://github.com/chriskiehl/Gooey) to run `coub-gui.py` diff --git a/coub.py b/coub.py index 93174fb..78ccef5 100755 --- a/coub.py +++ b/coub.py @@ -16,11 +16,7 @@ from urllib.parse import quote as urlquote from urllib.parse import unquote as urlunquote -try: - import aiohttp - aio = True -except ModuleNotFoundError: - aio = False +import aiohttp # ANSI escape codes don't work on Windows, unless the user jumps through # additional hoops (either by using 3rd-party software or enabling VT100 @@ -547,22 +543,14 @@ async def process(self, quantity=None): msg(f"\nDownloading {self.type} info" f"{f': {self.id}' if self.id else ''}" f" (sorted by '{self.sort}')") + msg(f" {pages} out of {self.pages} pages") - if aio: - msg(f" {pages} out of {self.pages} pages") - - tout = aiohttp.ClientTimeout(total=None) - conn = aiohttp.TCPConnector(limit=opts.connections) - async with aiohttp.ClientSession(timeout=tout, connector=conn) as session: - tasks = [parse_page(req, session) for req in requests] - links = await asyncio.gather(*tasks) - links = [l for page in links for l in page] - else: - links = [] - for i in range(pages): - msg(f" {i+1} out of {self.pages} pages") - page = await parse_page(requests[i]) - links.extend(page) + tout = aiohttp.ClientTimeout(total=None) + conn = aiohttp.TCPConnector(limit=opts.connections) + async with aiohttp.ClientSession(timeout=tout, connector=conn) as session: + tasks = [parse_page(req, session) for req in requests] + links = await asyncio.gather(*tasks) + links = [l for page in links for l in page] if quantity: return links[:quantity] @@ -912,18 +900,9 @@ async def parse(self, session=None): if self.erroneous(): return - if aio: - async with session.get(self.req) as resp: - resp_json = await resp.read() - resp_json = json.loads(resp_json) - else: - try: - with urlopen(self.req) as resp: - resp_json = resp.read() - resp_json = json.loads(resp_json) - except (urllib.error.HTTPError, urllib.error.URLError): - self.unavailable = True - return + async with session.get(self.req) as resp: + resp_json = await resp.read() + resp_json = json.loads(resp_json) v_list, a_list = stream_lists(resp_json) if v_list: @@ -1530,14 +1509,9 @@ def resolve_paths(): async def parse_page(req, session=None): """Request a single timeline page and parse its content.""" - if aio: - async with session.get(req) as resp: - resp_json = await resp.read() - resp_json = json.loads(resp_json) - else: - with urlopen(req) as resp: - resp_json = resp.read() - resp_json = json.loads(resp_json) + async with session.get(req) as resp: + resp_json = await resp.read() + resp_json = json.loads(resp_json) ids = [ c['recoub_to']['permalink'] if c['recoub_to'] else c['permalink'] @@ -1824,24 +1798,13 @@ def stream_lists(resp_json): async def save_stream(link, path, session=None): """Download a single media stream.""" - if aio: - async with session.get(link) as stream: - with open(path, "wb") as f: - while True: - chunk = await stream.content.read(opts.chunk_size) - if not chunk: - break - f.write(chunk) - else: - try: - with urlopen(link) as stream, open(path, "wb") as f: - while True: - chunk = stream.read(opts.chunk_size) - if not chunk: - break - f.write(chunk) - except (urllib.error.HTTPError, urllib.error.URLError): - return + async with session.get(link) as stream: + with open(path, "wb") as f: + while True: + chunk = await stream.content.read(opts.chunk_size) + if not chunk: + break + f.write(chunk) def valid_stream(path, attempted_fix=False): @@ -1881,22 +1844,18 @@ def valid_stream(path, attempted_fix=False): async def process(coubs): """Call the process function of all parsed coubs.""" - if aio: - tout = aiohttp.ClientTimeout(total=None) - conn = aiohttp.TCPConnector(limit=opts.connections) - try: - async with aiohttp.ClientSession(timeout=tout, connector=conn) as session: - tasks = [c.process(session) for c in coubs] - await asyncio.gather(*tasks) - except aiohttp.ClientConnectionError: - err("\nLost connection to coub.com!") - raise - except aiohttp.ClientPayloadError: - err("\nReceived malformed data!") - raise - else: - for c in coubs: - await c.process() + tout = aiohttp.ClientTimeout(total=None) + conn = aiohttp.TCPConnector(limit=opts.connections) + try: + async with aiohttp.ClientSession(timeout=tout, connector=conn) as session: + tasks = [c.process(session) for c in coubs] + await asyncio.gather(*tasks) + except aiohttp.ClientConnectionError: + err("\nLost connection to coub.com!") + raise + except aiohttp.ClientPayloadError: + err("\nReceived malformed data!") + raise def clean(coubs): From e8902126be4cfd9a5808472767560bafd54988c7 Mon Sep 17 00:00:00 2001 From: HelpSeeker Date: Sun, 16 Feb 2020 21:09:23 +0100 Subject: [PATCH 20/40] Return coub IDs instead of URLs after parsing This is done in preparation of the next commit, where using URLs would force the script to convert links to IDs and then back to links again. Otherwise there's not much difference between the two approaches. --- coub-gui.py | 2 +- coub.py | 53 ++++++++++++++++++++++++----------------------------- 2 files changed, 25 insertions(+), 30 deletions(-) diff --git a/coub-gui.py b/coub-gui.py index 6e3fc0d..36a6b55 100755 --- a/coub-gui.py +++ b/coub-gui.py @@ -224,7 +224,7 @@ def parse_cli(): args = parser.parse_args() args.input = [] args.input.extend([coub.mapped_input(u) for u in args.urls.split(",") if u]) - args.input.extend([f"https://coub.com/view/{i}" for i in args.ids.split(",") if i]) + args.input.extend([i for i in args.ids.split(",") if i]) args.input.extend([coub.LinkList(l) for l in args.lists.split(",") if l]) args.input.extend([coub.Channel(c) for c in args.channels.split(",") if c]) args.input.extend([coub.Tag(t) for t in args.tags.split(",") if t]) diff --git a/coub.py b/coub.py index 78ccef5..cc34bd8 100755 --- a/coub.py +++ b/coub.py @@ -549,12 +549,12 @@ async def process(self, quantity=None): conn = aiohttp.TCPConnector(limit=opts.connections) async with aiohttp.ClientSession(timeout=tout, connector=conn) as session: tasks = [parse_page(req, session) for req in requests] - links = await asyncio.gather(*tasks) - links = [l for page in links for l in page] + ids = await asyncio.gather(*tasks) + ids = [i for page in ids for i in page] if quantity: - return links[:quantity] - return links + return ids[:quantity] + return ids class Channel(BaseContainer): @@ -841,7 +841,10 @@ async def process(self, quantity=None): content = content.replace(" ", "\n") content = content.splitlines() - links = [l for l in content if "https://coub.com/view/" in l] + links = [ + l.partition("https://coub.com/view/")[2] + for l in content if "https://coub.com/view/" in l + ] msg(f" {len(links)} link{'s' if len(links) != 1 else ''} found") if quantity: @@ -852,9 +855,9 @@ async def process(self, quantity=None): class Coub: """Store all relevant infos and methods to process a single coub.""" - def __init__(self, link): - self.link = link - self.id = link.split("/")[-1] + def __init__(self, c_id): + self.id = c_id + self.link = f"https://coub.com/view/{self.id}" self.req = f"https://coub.com/api/v2/coubs/{self.id}" self.v_link = None @@ -1149,13 +1152,6 @@ def no_url(string): return string -def direct_link(string): - """Convert string provided by parse_cli() to a direct coub link.""" - coub_id = no_url(string) - link = f"https://coub.com/view/{coub_id}" - return link - - def positive_int(string): """Convert string provided by parse_cli() to a positive int.""" try: @@ -1329,7 +1325,7 @@ def mapped_input(string): link = normalize_link(string) if "https://coub.com/view/" in link: - source = link + source = link.partition("https://coub.com/view/")[2] elif "https://coub.com/tags/" in link: name = link.partition("https://coub.com/tags/")[2] source = Tag(name) @@ -1370,7 +1366,7 @@ def parse_cli(): # Input parser.add_argument("raw_input", nargs="*", type=mapped_input) parser.add_argument("-i", "--id", dest="input", action="append", - type=direct_link) + type=no_url) parser.add_argument("-l", "--list", dest="input", action="append", type=LinkList) parser.add_argument("-c", "--channel", dest="input", action="append", @@ -1517,8 +1513,7 @@ async def parse_page(req, session=None): c['recoub_to']['permalink'] if c['recoub_to'] else c['permalink'] for c in resp_json['coubs'] ] - - return [f"https://coub.com/view/{i}" for i in ids] + return ids def remove_container_dupes(containers): @@ -1538,14 +1533,14 @@ def remove_container_dupes(containers): def parse_input(sources): """Handle the parsing process of all provided input sources.""" - links = [s for s in sources if isinstance(s, str)] + directs = [s for s in sources if isinstance(s, str)] containers = [s for s in sources if not isinstance(s, str)] containers = remove_container_dupes(containers) if opts.max_coubs: - parsed = links[:opts.max_coubs] + parsed = directs[:opts.max_coubs] else: - parsed = links + parsed = directs if parsed: msg("\nReading command line:") @@ -1587,11 +1582,11 @@ def parse_input(sources): return parsed -def write_list(links): +def write_list(ids): """Output parsed links to a list and exit.""" with open(opts.output_list, opts.write_method) as f: - for l in links: - print(l, file=f) + for i in ids: + print(f"https://coub.com/view/{i}", file=f) msg(f"\nParsed coubs written to '{opts.output_list}'!", color=fgcolors.SUCCESS) @@ -1900,12 +1895,12 @@ def main(): check_connection() msg("\n### Parse Input ###") - links = parse_input(opts.input) + ids = parse_input(opts.input) if opts.output_list: - write_list(links) + write_list(ids) sys.exit(0) - total = len(links) - coubs = [Coub(l) for l in links] + total = len(ids) + coubs = [Coub(i) for i in ids] msg("\n### Download Coubs ###\n") try: From 3ed9b250ddd76a472f6de7f3bb6813a01444921d Mon Sep 17 00:00:00 2001 From: HelpSeeker Date: Sun, 16 Feb 2020 22:54:05 +0100 Subject: [PATCH 21/40] Check against archive during the parsing process For normal users the process of checking coub IDs against an archive file was efficient enough. But let's assume an "enthusiast" scrapes communities on a regular basis. In that case they're not only likely to work with several thousand input IDs per script invocation, but also to have an archive file with an even greater magnitude. To put it into numbers: Skipping 99% of 90.000 IDs based on an archive file with 400.000 IDs takes approximately 160 seconds. That's not acceptable. Now we can do 2 things to make this faster. 1.) Use a set to store the archive's content 2.) Check the parsed ID list against the archive 1. is done for obvious reasons. 2. prevents the unnecessary creation of Coub objects, all the existence checks during the processing chain and the printing of an "exists" messages. Going back to the previous example with the new approach: Skipping 99% of 90.000 IDs based on an archive file with 400.000 IDs takes approximately 0,8 seconds. --- coub-gui.py | 4 ++-- coub.py | 40 +++++++++++++++++++++------------------- 2 files changed, 23 insertions(+), 21 deletions(-) diff --git a/coub-gui.py b/coub-gui.py index 36a6b55..23b73e2 100755 --- a/coub-gui.py +++ b/coub-gui.py @@ -239,9 +239,9 @@ def parse_cli(): # Read archive content if args.archive and os.path.exists(args.archive): with open(args.archive, "r") as f: - args.archive_content = [l.strip() for l in f] + args.archive_content = {l.strip() for l in f} else: - args.archive_content = None + args.archive_content = set() # The default naming scheme is the same as using %id% # but internally the default value is None if args.name_template == "%id%": diff --git a/coub.py b/coub.py index cc34bd8..0f753e5 100755 --- a/coub.py +++ b/coub.py @@ -881,10 +881,6 @@ def check_existence(self): if self.erroneous(): return - if opts.archive_content and self.id in opts.archive_content: - self.exists = True - return - old_file = None # Existence of self.name indicates whether API request was already # made (i.e. if 1st or 2nd check) @@ -1477,9 +1473,9 @@ def parse_cli(): # Read archive content if args.archive and os.path.exists(args.archive): with open(args.archive, "r") as f: - args.archive_content = [l.strip() for l in f] + args.archive_content = {l.strip() for l in f} else: - args.archive_content = None + args.archive_content = set() # The default naming scheme is the same as using %id% # but internally the default value is None if args.name_template == "%id%": @@ -1566,13 +1562,16 @@ def parse_input(sources): before = len(parsed) parsed = list(set(parsed)) # Weed out duplicates + dupes = before - len(parsed) + parsed = [i for i in parsed if i not in opts.archive_content] + archived = before - dupes - len(parsed) after = len(parsed) - dupes = before - after - if dupes: + if dupes or archived: msg(dedent(f""" Results: {before} input link{'s' if before != 1 else ''} {dupes} duplicate{'s' if dupes != 1 else ''} + {archived} found in archive file {after} final link{'s' if after != 1 else ''}""")) else: msg(dedent(f""" @@ -1896,17 +1895,20 @@ def main(): msg("\n### Parse Input ###") ids = parse_input(opts.input) - if opts.output_list: - write_list(ids) - sys.exit(0) - total = len(ids) - coubs = [Coub(i) for i in ids] - - msg("\n### Download Coubs ###\n") - try: - attempt_process(coubs) - finally: - clean(coubs) + if ids: + if opts.output_list: + write_list(ids) + sys.exit(0) + total = len(ids) + coubs = [Coub(i) for i in ids] + + msg("\n### Download Coubs ###\n") + try: + attempt_process(coubs) + finally: + clean(coubs) + else: + msg("\nAll coubs present in archive file!", color=fgcolors.WARNING) msg("\n### Finished ###\n") From cbfe6574b6605eadaa72fe31c2eef81bc382fbf0 Mon Sep 17 00:00:00 2001 From: HelpSeeker Date: Sun, 16 Feb 2020 23:04:27 +0100 Subject: [PATCH 22/40] Set check=False for subprocess.run() pylint warns against not setting "check" explicitly and who am I to disagree? --- coub.py | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/coub.py b/coub.py index 0f753e5..29789d1 100755 --- a/coub.py +++ b/coub.py @@ -996,7 +996,7 @@ def merge(self): command.extend(["-t", opts.duration]) command.extend(["-c", "copy", "-shortest", f"file:temp_{m_name}"]) - subprocess.run(command) + subprocess.run(command, check=False) finally: if os.path.exists(t_name): os.remove(t_name) @@ -1123,7 +1123,7 @@ def check_prereq(): """Test if all required 3rd-party tools are installed.""" try: subprocess.run(["ffmpeg"], stdout=subprocess.DEVNULL, \ - stderr=subprocess.DEVNULL) + stderr=subprocess.DEVNULL, check=False) except FileNotFoundError: err("Error: FFmpeg not found!") sys.exit(status.DEP) @@ -1809,7 +1809,7 @@ def valid_stream(path, attempted_fix=False): "-t", "1", "-f", "null", "-", ] - out = subprocess.run(command, capture_output=True, text=True) + out = subprocess.run(command, capture_output=True, text=True, check=False) # Fix broken video stream if "moov atom not found" in out.stderr and not attempted_fix: From 5127ab65976577a19183863e7f3f09cabd7b22c7 Mon Sep 17 00:00:00 2001 From: HelpSeeker Date: Wed, 26 Feb 2020 19:24:54 +0100 Subject: [PATCH 23/40] Add advanced option for custom FFmpeg path/command Allows the user to call FFmpeg without the command "ffmpeg" being available. Alternatively it also makes it easier to switch between different version. The main motivation is that dynamically linked FFmpeg builds (as provided by most Linux distros) don't play nice with standalone Python binaries (created with PyInstaller). It either clutters the terminal output with warnings or may even fail entirely. The easiest solution is to use a static FFmpeg build, which can be done now without having to fiddle with one's PATH. --- coub.py | 16 +++++++++++----- example.conf | 3 +++ 2 files changed, 14 insertions(+), 5 deletions(-) diff --git a/coub.py b/coub.py index 29789d1..77d1f6d 100755 --- a/coub.py +++ b/coub.py @@ -116,6 +116,7 @@ class DefaultOptions: MERGE_EXT = "mkv" NAME_TEMPLATE = "%id%" # Advanced defaults + FFMPEG_PATH = "ffmpeg" COUBS_PER_PAGE = 25 TAG_SEP = "_" WRITE_METHOD = "w" @@ -176,6 +177,7 @@ def check_values(self): "ARCHIVE": (lambda x: isinstance(x, str) or x is None), "MERGE_EXT": (lambda x: x in ["mkv", "mp4", "asf", "avi", "flv", "f4v", "mov"]), "NAME_TEMPLATE": (lambda x: isinstance(x, str) or x is None), + "FFMPEG_PATH": (lambda x: isinstance(x, str)), "COUBS_PER_PAGE": (lambda x: x in range(1, 26)), "TAG_SEP": (lambda x: isinstance(x, str)), "WRITE_METHOD": (lambda x: x in ["w", "a"]), @@ -209,6 +211,7 @@ def guess_string_type(option, string): "OUTPUT_LIST", "ARCHIVE", "NAME_TEMPLATE", + "FFMPEG_PATH", "TAG_SEP", ] @@ -988,7 +991,7 @@ def merge(self): # Loop footage until shortest stream ends # Concatenated video (via list) counts as one long stream command = [ - "ffmpeg", "-y", "-v", "error", + opts.ffmpeg_path, "-y", "-v", "error", "-f", "concat", "-safe", "0", "-i", f"file:{t_name}", "-i", f"file:{self.a_name}", ] @@ -1122,8 +1125,10 @@ def msg(*args, color=fgcolors.RESET, **kwargs): def check_prereq(): """Test if all required 3rd-party tools are installed.""" try: - subprocess.run(["ffmpeg"], stdout=subprocess.DEVNULL, \ - stderr=subprocess.DEVNULL, check=False) + subprocess.run([opts.ffmpeg_path], + stdout=subprocess.DEVNULL, + stderr=subprocess.DEVNULL, + check=False) except FileNotFoundError: err("Error: FFmpeg not found!") sys.exit(status.DEP) @@ -1162,7 +1167,7 @@ def positive_int(string): def valid_time(string): """Test valditiy of time syntax with FFmpeg.""" command = [ - "ffmpeg", "-v", "quiet", + opts.ffmpeg_path, "-v", "quiet", "-f", "lavfi", "-i", "anullsrc", "-t", string, "-c", "copy", "-f", "null", "-", @@ -1453,6 +1458,7 @@ def parse_cli(): # Advanced Options parser.set_defaults( + ffmpeg_path=defaults.FFMPEG_PATH, coubs_per_page=defaults.COUBS_PER_PAGE, tag_sep=defaults.TAG_SEP, write_method=defaults.WRITE_METHOD, @@ -1804,7 +1810,7 @@ async def save_stream(link, path, session=None): def valid_stream(path, attempted_fix=False): """Test a given stream for eventual corruption with a test remux (FFmpeg).""" command = [ - "ffmpeg", "-v", "error", + opts.ffmpeg_path, "-v", "error", "-i", f"file:{path}", "-t", "1", "-f", "null", "-", diff --git a/example.conf b/example.conf index fbed3b2..05d3387 100644 --- a/example.conf +++ b/example.conf @@ -136,6 +136,9 @@ MERGE_EXT = mkv # Allowed values: strings, None (same as %id%) NAME_TEMPLATE = %id% +# Custom path or command to invoke FFmpeg +FFMPEG_PATH = ffmpeg + # Character or string to separate tags in the output filename # Allowed values: strings TAG_SEP = _ From bad351a3166ac395f795a30fe0740e51de784214 Mon Sep 17 00:00:00 2001 From: HelpSeeker Date: Thu, 27 Feb 2020 21:49:33 +0100 Subject: [PATCH 24/40] Change program name for standalone binaries Using staticx on PyInstaller archives ensures statically linked binaries. However, staticx changes the value of sys.argv[0] to .staticx.prog, which is in itself only a cosmetic problem (e.g. usage messages), but still annoying. So let's change the way how the program's name is located, depending on whether the script is executed directly or as part of a standalone binary. --- coub-gui.py | 8 ++++++++ coub.py | 9 ++++++++- 2 files changed, 16 insertions(+), 1 deletion(-) diff --git a/coub-gui.py b/coub-gui.py index 23b73e2..842adfc 100755 --- a/coub-gui.py +++ b/coub-gui.py @@ -1,6 +1,7 @@ #!/usr/bin/env python3 import os +import sys from textwrap import dedent from gooey import Gooey, GooeyParser @@ -87,7 +88,14 @@ def translate_to_cli(options): def parse_cli(): """Create Gooey GUI.""" defs = GuiDefaultOptions() + if getattr(sys, "frozen", False): + # workaround since staticx changes sys.argv[0] to .staticx.prog + prog = __file__ + else: + # argparse default + prog = os.path.basename(sys.argv[0]) parser = GooeyParser( + prog=prog, description="Download videos from coub.com", usage="%(prog)s [OPTIONS] INPUT [INPUT]..." ) diff --git a/coub.py b/coub.py index 77d1f6d..ce2f8f9 100755 --- a/coub.py +++ b/coub.py @@ -1362,7 +1362,14 @@ def mapped_input(string): def parse_cli(): """Parse the command line.""" defaults = DefaultOptions() - parser = CustomArgumentParser(usage="%(prog)s [OPTIONS] INPUT [INPUT]...") + if getattr(sys, "frozen", False): + # workaround since staticx changes sys.argv[0] to .staticx.prog + prog = __file__ + else: + # argparse default + prog = os.path.basename(sys.argv[0]) + parser = CustomArgumentParser(prog=prog, + usage="%(prog)s [OPTIONS] INPUT [INPUT]...") # Input parser.add_argument("raw_input", nargs="*", type=mapped_input) From 539b7efe491d17a1a22423f980fc1f7bb18d1ffe Mon Sep 17 00:00:00 2001 From: HelpSeeker Date: Thu, 27 Feb 2020 23:15:57 +0100 Subject: [PATCH 25/40] Add config support for GUI version Some small adjustments had to be made to DefaultOptions' initialization in the process. When importing coub.py inside a coub-gui standalone binary, __file__ points to the PyInstaller extraction location and not the location of the binary. So the binary location must be determined inside coub-gui.py and then passed to the __init__ function. This commit also makes sure that any custom FFmpeg path is converted to an absolute path (wasn't yet the case for coub.py). --- coub-gui.py | 14 ++++++++++---- coub.py | 9 +++++---- 2 files changed, 15 insertions(+), 8 deletions(-) diff --git a/coub-gui.py b/coub-gui.py index 842adfc..0fd9eb6 100755 --- a/coub-gui.py +++ b/coub-gui.py @@ -35,6 +35,11 @@ class GuiDefaultOptions(coub.DefaultOptions): (False, False, True): "Audio only", } + def __init__(self): + # Necessary as __file__ within coub.py would point to the extracted + # PyInstaller archive for standalone coub-gui binaries + config_dirs = [os.path.dirname(os.path.realpath(__file__))] + super(GuiDefaultOptions, self).__init__(config_dirs=config_dirs) def translate_to_cli(options): """Make GUI-specific options object compatible with the main script.""" @@ -223,10 +228,11 @@ def parse_cli(): # Advanced Options parser.set_defaults( verbosity=1, - coubs_per_page=25, # allowed: 1-25 - tag_sep="_", - write_method="w", # w -> overwrite, a -> append - chunk_size=1024, + ffmpeg_path=os.path.abspath(defs.FFMPEG_PATH), + coubs_per_page=defs.COUBS_PER_PAGE, # allowed: 1-25 + tag_sep=defs.TAG_SEP, + write_method=defs.WRITE_METHOD, # w -> overwrite, a -> append + chunk_size=defs.CHUNK_SIZE, ) args = parser.parse_args() diff --git a/coub.py b/coub.py index ce2f8f9..f6cc51d 100755 --- a/coub.py +++ b/coub.py @@ -122,9 +122,10 @@ class DefaultOptions: WRITE_METHOD = "w" CHUNK_SIZE = 1024 - def __init__(self): - # Only supports script's location for now, but write it to be extensible - config_dirs = [os.path.dirname(os.path.realpath(__file__))] + def __init__(self, config_dirs=None): + if not config_dirs: + # Only supports script's location as default for now + config_dirs = [os.path.dirname(os.path.realpath(__file__))] for d in config_dirs: config_path = os.path.join(d, "coub.conf") if os.path.exists(config_path): @@ -1465,7 +1466,7 @@ def parse_cli(): # Advanced Options parser.set_defaults( - ffmpeg_path=defaults.FFMPEG_PATH, + ffmpeg_path=os.path.abspath(defaults.FFMPEG_PATH), coubs_per_page=defaults.COUBS_PER_PAGE, tag_sep=defaults.TAG_SEP, write_method=defaults.WRITE_METHOD, From 149b0f18cdbb3d8baad7255cb1ea6666f4f87b90 Mon Sep 17 00:00:00 2001 From: HelpSeeker Date: Fri, 28 Feb 2020 17:29:49 +0100 Subject: [PATCH 26/40] Revert absolute path conversion for FFmpeg path Converting the value to an absolute path obviously interfered with the default case, in which the command "ffmpeg" is invoked. I don't see a reasonable way to allow commands as well as relative and absolute paths, so FFMPEG_PATH must be provided as an absolute path after all. --- coub-gui.py | 2 +- coub.py | 2 +- example.conf | 1 + 3 files changed, 3 insertions(+), 2 deletions(-) diff --git a/coub-gui.py b/coub-gui.py index 0fd9eb6..fcaddc8 100755 --- a/coub-gui.py +++ b/coub-gui.py @@ -228,7 +228,7 @@ def parse_cli(): # Advanced Options parser.set_defaults( verbosity=1, - ffmpeg_path=os.path.abspath(defs.FFMPEG_PATH), + ffmpeg_path=defs.FFMPEG_PATH, coubs_per_page=defs.COUBS_PER_PAGE, # allowed: 1-25 tag_sep=defs.TAG_SEP, write_method=defs.WRITE_METHOD, # w -> overwrite, a -> append diff --git a/coub.py b/coub.py index f6cc51d..1e184dc 100755 --- a/coub.py +++ b/coub.py @@ -1466,7 +1466,7 @@ def parse_cli(): # Advanced Options parser.set_defaults( - ffmpeg_path=os.path.abspath(defaults.FFMPEG_PATH), + ffmpeg_path=defaults.FFMPEG_PATH, coubs_per_page=defaults.COUBS_PER_PAGE, tag_sep=defaults.TAG_SEP, write_method=defaults.WRITE_METHOD, diff --git a/example.conf b/example.conf index 05d3387..dc9be61 100644 --- a/example.conf +++ b/example.conf @@ -137,6 +137,7 @@ MERGE_EXT = mkv NAME_TEMPLATE = %id% # Custom path or command to invoke FFmpeg +# Paths must be absolute FFMPEG_PATH = ffmpeg # Character or string to separate tags in the output filename From b62876fa985cdec8f55ffcbb69abb4841a53a642 Mon Sep 17 00:00:00 2001 From: HelpSeeker Date: Fri, 28 Feb 2020 17:53:13 +0100 Subject: [PATCH 27/40] Revert "Change program name for standalone binaries" This reverts commit bad351a3166ac395f795a30fe0740e51de784214. I give up. Life is too short to try to fix all the problems that arise while trying to make a universal Linux build out of this script. And since only Linux binaries would've used staticx, this code became pointless. --- coub-gui.py | 8 -------- coub.py | 9 +-------- 2 files changed, 1 insertion(+), 16 deletions(-) diff --git a/coub-gui.py b/coub-gui.py index fcaddc8..fe6269d 100755 --- a/coub-gui.py +++ b/coub-gui.py @@ -1,7 +1,6 @@ #!/usr/bin/env python3 import os -import sys from textwrap import dedent from gooey import Gooey, GooeyParser @@ -93,14 +92,7 @@ def translate_to_cli(options): def parse_cli(): """Create Gooey GUI.""" defs = GuiDefaultOptions() - if getattr(sys, "frozen", False): - # workaround since staticx changes sys.argv[0] to .staticx.prog - prog = __file__ - else: - # argparse default - prog = os.path.basename(sys.argv[0]) parser = GooeyParser( - prog=prog, description="Download videos from coub.com", usage="%(prog)s [OPTIONS] INPUT [INPUT]..." ) diff --git a/coub.py b/coub.py index 1e184dc..351d7b4 100755 --- a/coub.py +++ b/coub.py @@ -1363,14 +1363,7 @@ def mapped_input(string): def parse_cli(): """Parse the command line.""" defaults = DefaultOptions() - if getattr(sys, "frozen", False): - # workaround since staticx changes sys.argv[0] to .staticx.prog - prog = __file__ - else: - # argparse default - prog = os.path.basename(sys.argv[0]) - parser = CustomArgumentParser(prog=prog, - usage="%(prog)s [OPTIONS] INPUT [INPUT]...") + parser = CustomArgumentParser(usage="%(prog)s [OPTIONS] INPUT [INPUT]...") # Input parser.add_argument("raw_input", nargs="*", type=mapped_input) From 0a8258c4575ea928e7802c706d55f2410caccfdb Mon Sep 17 00:00:00 2001 From: HelpSeeker Date: Fri, 3 Apr 2020 22:09:49 +0200 Subject: [PATCH 28/40] Update README.md --- README.md | 45 +++++++++++++++++++++++++++++++++++++++------ 1 file changed, 39 insertions(+), 6 deletions(-) diff --git a/README.md b/README.md index 95f6b47..55deb07 100644 --- a/README.md +++ b/README.md @@ -5,9 +5,11 @@ CoubDownloader is a simple script to download videos (called coubs) from [Coub]( ## Contents 1. [Usage](#usage) -2. [Requirements](#requirements) +2. [Standalone builds](#standalone-builds) +3. [Requirements](#requirements) 1. [Optional](#optional) -3. [Input](#input) +4. [Configuration](#configuration) +5. [Input](#input) 1. [Overview](#overview) 2. [Direct coub links](#direct-coub-links) 3. [Lists](#lists) @@ -17,13 +19,13 @@ CoubDownloader is a simple script to download videos (called coubs) from [Coub]( 7. [Tags](#tags) 8. [Communities](#communities) 9. [Hot section](#hot-section) -4. [GUI](#gui) -5. [Misc. information](#misc-information) +6. [GUI](#gui) +7. [Misc. information](#misc-information) 1. [Video resolution vs. quality](#video-resolution-vs-quality) 2. [AAC audio](#aac-audio) 3. ['share' videos](#share-videos) -6. [Changes since Coub's database upgrade (watermark & co)](#changes-since-coubs-database-upgrade-watermark--co) -7. [Changes since switching to Coub's API (previously used youtube-dl)](#changes-since-switching-to-coubs-api-previously-used-youtube-dl) +8. [Changes since Coub's database upgrade (watermark & co)](#changes-since-coubs-database-upgrade-watermark--co) +9. [Changes since switching to Coub's API (previously used youtube-dl)](#changes-since-switching-to-coubs-api-previously-used-youtube-dl) ## Usage @@ -110,17 +112,47 @@ Output: This option has no influence on the file extension. ``` +## Standalone builds + +Starting with v3.7 standalone Windows builds are provided on the [release page](https://github.com/HelpSeeker/CoubDownloader/releases). These executables don't require an existing Python environment and the [GUI version](#gui) is a standalone application. FFmpeg is still a mandatory requirement. + +#### Why no Mac builds? + +Because I lack the necessary build environment. + +#### Why no Linux builds? + +Because I lack the resources to target a variety of distros individually and my attempts to create universal Linux builds were unsuccessful. + +The main problem is the usage of external tools like FFmpeg (but also media players with the `--preview` option) via `subprocess.run`. They either spam the terminal output with warnings or outright fail because of library version mismatches between my build environment and the user's system. It would've required me to ship Linux builds with static FFmpeg binaries and either remove the `--preview` option or make it the user's responsibility to get it working. And even then there's still a plethora of problems regarding the GUI version, which I couldn't reliably get to work across a handful of mainstream distros (Debian, Ubuntu, CentOS, Linux Mint, etc.). + ## Requirements * Python >= 3.7 * [aiohttp](https://aiohttp.readthedocs.io/en/stable/) * [FFmpeg](https://www.ffmpeg.org/) +Standalone builds only require FFmpeg. + ### Optional * [colorama](https://github.com/tartley/colorama) for colorized terminal output on Windows * [Gooey](https://github.com/chriskiehl/Gooey) to run `coub-gui.py` +## Configuration + +CoubDownloader's defaults can be changed via a configuration file. + +Simply create a text file with the name `coub.conf` in the same location as the script and specify custom defaults in the following format: + +``` +OPTION = value +``` + +Lines starting with `#` get treated as comments. + +For all available options and their allowed values, please take a look at the [example configuration file](example.conf). + ## Input #### Overview @@ -489,6 +521,7 @@ Coub started to massively overhaul their database and API. Of course those chang - [x] Download random coubs - [x] Option to change the container format for stream remuxing - [x] Basic GUI frontend +- [x] Read custom defaults from config file ## Changes since switching to Coub's API (previously used youtube-dl) From 4e29e6c08accd03d8a1d40631b7f439acc292ea8 Mon Sep 17 00:00:00 2001 From: HelpSeeker Date: Wed, 29 Apr 2020 21:53:51 +0200 Subject: [PATCH 29/40] Revert "Remove support for synchronous download via urllib" This reverts commit 4ea8551f2a750ee799c2e8c3a8b033a0ac00d686. My excuse to promote aiohttp from an optional to a mandatory requirement was that I planned to provide standalone binaries for both Windows and Linux after this development cycle, which would include aiohttp by default. However, my efforts to create universal Linux builds failed. As a result I once again make it my goal to avoid mandatory requirements outside the standard library. --- README.md | 2 +- coub.py | 107 +++++++++++++++++++++++++++++++++++++----------------- 2 files changed, 75 insertions(+), 34 deletions(-) diff --git a/README.md b/README.md index 55deb07..ed1cad3 100644 --- a/README.md +++ b/README.md @@ -129,13 +129,13 @@ The main problem is the usage of external tools like FFmpeg (but also media play ## Requirements * Python >= 3.7 -* [aiohttp](https://aiohttp.readthedocs.io/en/stable/) * [FFmpeg](https://www.ffmpeg.org/) Standalone builds only require FFmpeg. ### Optional +* [aiohttp](https://aiohttp.readthedocs.io/en/stable/) for asynchronous execution **(strongly recommended)** * [colorama](https://github.com/tartley/colorama) for colorized terminal output on Windows * [Gooey](https://github.com/chriskiehl/Gooey) to run `coub-gui.py` diff --git a/coub.py b/coub.py index 351d7b4..1ee5122 100755 --- a/coub.py +++ b/coub.py @@ -16,7 +16,11 @@ from urllib.parse import quote as urlquote from urllib.parse import unquote as urlunquote -import aiohttp +try: + import aiohttp + aio = True +except ModuleNotFoundError: + aio = False # ANSI escape codes don't work on Windows, unless the user jumps through # additional hoops (either by using 3rd-party software or enabling VT100 @@ -547,14 +551,22 @@ async def process(self, quantity=None): msg(f"\nDownloading {self.type} info" f"{f': {self.id}' if self.id else ''}" f" (sorted by '{self.sort}')") - msg(f" {pages} out of {self.pages} pages") - tout = aiohttp.ClientTimeout(total=None) - conn = aiohttp.TCPConnector(limit=opts.connections) - async with aiohttp.ClientSession(timeout=tout, connector=conn) as session: - tasks = [parse_page(req, session) for req in requests] - ids = await asyncio.gather(*tasks) - ids = [i for page in ids for i in page] + if aio: + msg(f" {pages} out of {self.pages} pages") + + tout = aiohttp.ClientTimeout(total=None) + conn = aiohttp.TCPConnector(limit=opts.connections) + async with aiohttp.ClientSession(timeout=tout, connector=conn) as session: + tasks = [parse_page(req, session) for req in requests] + ids = await asyncio.gather(*tasks) + ids = [i for page in ids for i in page] + else: + ids = [] + for i in range(pages): + msg(f" {i+1} out of {self.pages} pages") + page = await parse_page(requests[i]) + ids.extend(page) if quantity: return ids[:quantity] @@ -903,9 +915,18 @@ async def parse(self, session=None): if self.erroneous(): return - async with session.get(self.req) as resp: - resp_json = await resp.read() - resp_json = json.loads(resp_json) + if aio: + async with session.get(self.req) as resp: + resp_json = await resp.read() + resp_json = json.loads(resp_json) + else: + try: + with urlopen(self.req) as resp: + resp_json = resp.read() + resp_json = json.loads(resp_json) + except (urllib.error.HTTPError, urllib.error.URLError): + self.unavailable = True + return v_list, a_list = stream_lists(resp_json) if v_list: @@ -1508,9 +1529,14 @@ def resolve_paths(): async def parse_page(req, session=None): """Request a single timeline page and parse its content.""" - async with session.get(req) as resp: - resp_json = await resp.read() - resp_json = json.loads(resp_json) + if aio: + async with session.get(req) as resp: + resp_json = await resp.read() + resp_json = json.loads(resp_json) + else: + with urlopen(req) as resp: + resp_json = resp.read() + resp_json = json.loads(resp_json) ids = [ c['recoub_to']['permalink'] if c['recoub_to'] else c['permalink'] @@ -1799,13 +1825,24 @@ def stream_lists(resp_json): async def save_stream(link, path, session=None): """Download a single media stream.""" - async with session.get(link) as stream: - with open(path, "wb") as f: - while True: - chunk = await stream.content.read(opts.chunk_size) - if not chunk: - break - f.write(chunk) + if aio: + async with session.get(link) as stream: + with open(path, "wb") as f: + while True: + chunk = await stream.content.read(opts.chunk_size) + if not chunk: + break + f.write(chunk) + else: + try: + with urlopen(link) as stream, open(path, "wb") as f: + while True: + chunk = stream.read(opts.chunk_size) + if not chunk: + break + f.write(chunk) + except (urllib.error.HTTPError, urllib.error.URLError): + return def valid_stream(path, attempted_fix=False): @@ -1845,18 +1882,22 @@ def valid_stream(path, attempted_fix=False): async def process(coubs): """Call the process function of all parsed coubs.""" - tout = aiohttp.ClientTimeout(total=None) - conn = aiohttp.TCPConnector(limit=opts.connections) - try: - async with aiohttp.ClientSession(timeout=tout, connector=conn) as session: - tasks = [c.process(session) for c in coubs] - await asyncio.gather(*tasks) - except aiohttp.ClientConnectionError: - err("\nLost connection to coub.com!") - raise - except aiohttp.ClientPayloadError: - err("\nReceived malformed data!") - raise + if aio: + tout = aiohttp.ClientTimeout(total=None) + conn = aiohttp.TCPConnector(limit=opts.connections) + try: + async with aiohttp.ClientSession(timeout=tout, connector=conn) as session: + tasks = [c.process(session) for c in coubs] + await asyncio.gather(*tasks) + except aiohttp.ClientConnectionError: + err("\nLost connection to coub.com!") + raise + except aiohttp.ClientPayloadError: + err("\nReceived malformed data!") + raise + else: + for c in coubs: + await c.process() def clean(coubs): From 4831efa24764f97bbca0cc3e23d68da843c5744e Mon Sep 17 00:00:00 2001 From: HelpSeeker Date: Wed, 29 Apr 2020 23:09:00 +0200 Subject: [PATCH 30/40] Fix catching aiohttp exceptions without aiohttp Of course being able to catch aiohttp exceptions depended on aiohttp being available. Now that urllib is back as default library, we need to differentiate between urllib/aiohttp usage in this exception block as well. --- coub.py | 20 +++++++++++--------- 1 file changed, 11 insertions(+), 9 deletions(-) diff --git a/coub.py b/coub.py index 1ee5122..46bba7d 100755 --- a/coub.py +++ b/coub.py @@ -1160,8 +1160,8 @@ def check_connection(): """Check if user can connect to coub.com.""" try: urlopen("https://coub.com/") - except urllib.error.URLError as error: - if isinstance(error.reason, SSLCertVerificationError): + except urllib.error.URLError as e: + if isinstance(e.reason, SSLCertVerificationError): err("Certificate verification failed! Please update your CA certificates.") else: err("Unable to connect to coub.com! Please check your connection.") @@ -1920,13 +1920,15 @@ def attempt_process(coubs, level=0): try: asyncio.run(process(coubs), debug=False) - except (aiohttp.ClientConnectionError, aiohttp.ClientPayloadError): - check_connection() - # Reduce the list of coubs to only those yet to finish - coubs = [c for c in coubs if not c.done] - level += 1 - attempt_process(coubs, level) - + except Exception as e: + if aio and isinstance(e, (aiohttp.ClientConnectionError, aiohttp.ClientPayloadError)): + check_connection() + # Reduce the list of coubs to only those yet to finish + coubs = [c for c in coubs if not c.done] + level += 1 + attempt_process(coubs, level) + else: + raise # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ # Main Function From 0aafe914c7fa27b0b728fe5632420c1a5e5a7be1 Mon Sep 17 00:00:00 2001 From: HelpSeeker Date: Thu, 30 Apr 2020 01:11:21 +0200 Subject: [PATCH 31/40] Replace forbidden characters in filename This was already done on a very basic level, but now it replaces the forbidden characters (instead of just removing them) and it covers a slightly broader collection of forbidden characters, including the most common pitfalls for filenames on Windows. The latter is especially important because %creation% always led to the fallback filename as it includes colons. Still not perfect, but it should prevent many titles from failing. --- coub-gui.py | 1 + coub.py | 42 ++++++++++++++++++++++++++---------------- example.conf | 4 ++++ 3 files changed, 31 insertions(+), 16 deletions(-) diff --git a/coub-gui.py b/coub-gui.py index fe6269d..d19229f 100755 --- a/coub-gui.py +++ b/coub-gui.py @@ -223,6 +223,7 @@ def parse_cli(): ffmpeg_path=defs.FFMPEG_PATH, coubs_per_page=defs.COUBS_PER_PAGE, # allowed: 1-25 tag_sep=defs.TAG_SEP, + fallback_char=defs.FALLBACK_CHAR, write_method=defs.WRITE_METHOD, # w -> overwrite, a -> append chunk_size=defs.CHUNK_SIZE, ) diff --git a/coub.py b/coub.py index 46bba7d..008f844 100755 --- a/coub.py +++ b/coub.py @@ -123,6 +123,7 @@ class DefaultOptions: FFMPEG_PATH = "ffmpeg" COUBS_PER_PAGE = 25 TAG_SEP = "_" + FALLBACK_CHAR = "-" WRITE_METHOD = "w" CHUNK_SIZE = 1024 @@ -185,6 +186,7 @@ def check_values(self): "FFMPEG_PATH": (lambda x: isinstance(x, str)), "COUBS_PER_PAGE": (lambda x: x in range(1, 26)), "TAG_SEP": (lambda x: isinstance(x, str)), + "FALLBACK_CHAR": (lambda x: isinstance(x, str)), "WRITE_METHOD": (lambda x: x in ["w", "a"]), "CHUNK_SIZE": (lambda x: isinstance(x, int) and x > 0), } @@ -218,6 +220,7 @@ def guess_string_type(option, string): "NAME_TEMPLATE", "FFMPEG_PATH", "TAG_SEP", + "FALLBACK_CHAR", ] if string in specials: @@ -1483,6 +1486,7 @@ def parse_cli(): ffmpeg_path=defaults.FFMPEG_PATH, coubs_per_page=defaults.COUBS_PER_PAGE, tag_sep=defaults.TAG_SEP, + fallback_char=defaults.FALLBACK_CHAR, write_method=defaults.WRITE_METHOD, chunk_size=defaults.CHUNK_SIZE, ) @@ -1628,26 +1632,32 @@ def get_name(req_json, c_id): if not opts.name_template: return c_id - name = opts.name_template - - name = name.replace("%id%", c_id) - name = name.replace("%title%", req_json['title']) - name = name.replace("%creation%", req_json['created_at']) - name = name.replace("%channel%", req_json['channel']['title']) + specials = { + '%id%': c_id, + '%title%': req_json['title'], + '%creation%': req_json['created_at'], + '%channel%': req_json['channel']['title'], + '%tags%': opts.tag_sep.join([t['title'] for t in req_json['tags']]), + } # Coubs don't necessarily belong to a community (although it's rare) try: - name = name.replace("%community%", req_json['communities'][0]['permalink']) + specials['%community%'] = req_json['communities'][0]['permalink'] except (KeyError, TypeError, IndexError): - name = name.replace("%community%", "") + specials['%community%'] = "undefined" - tags = opts.tag_sep.join([t['title'] for t in req_json['tags']]) - name = name.replace("%tags%", tags) - - # Strip/replace special characters that can lead to script failure (ffmpeg concat) - # ' common among coub titles - # Newlines can be occasionally found as well - name = name.replace("'", "") - name = name.replace("\n", " ") + name = opts.name_template + for to_replace in specials: + name = name.replace(to_replace, specials[to_replace]) + + # An attempt to remove the most blatant problematic characters + # Linux supports all except /, but \n and \t are only asking for trouble + # https://dwheeler.com/essays/fixing-unix-linux-filenames.html + # ' is problematic as it causes issues with FFmpeg's concat muxer + forbidden = ["\n", "\t", "'", "/"] + if os.name == "nt": + forbidden.extend(["<", ">", ":", "\"", "\\", "|", "?", "*"]) + for to_replace in forbidden: + name = name.replace(to_replace, opts.fallback_char) try: # Add example extension to simulate the full name length diff --git a/example.conf b/example.conf index dc9be61..5931eb2 100644 --- a/example.conf +++ b/example.conf @@ -143,3 +143,7 @@ FFMPEG_PATH = ffmpeg # Character or string to separate tags in the output filename # Allowed values: strings TAG_SEP = _ + +# Character to replace forbidden characters in the filename (newlines, tabs, /, \, etc.) +# Allowed values: strings +FALLBACK_CHAR = - From d330660e1769b660cc4c00aa5609aba489452a6f Mon Sep 17 00:00:00 2001 From: HelpSeeker Date: Thu, 30 Apr 2020 20:16:15 +0200 Subject: [PATCH 32/40] Fix invalid options for GUI being accepted GUI and CLI behave the same regarding their options, but there are 2 notable exceptions. 1. Default overwrite prompt answer 2. Default output location There's no way for the user to input a prompt answer, so using anything but "yes" and "no" as default answer is pointless. Likewise it doesn't make a lot of sense to use relative default paths with a GUI. coub-gui.py already overwrote the main script's defaults in those cases, but it did that BEFORE custom defaults were potentially read from a config file. This could lead to outright failure (GUI doesn't accept prompt answers besides yes/no) or confusing output locations. Now those special options get overwritten - if necessary - AFTER the config file parsing. Additionally all relative paths (not just the current location) fall back to ~/coubs (and whatever the equivalent for Windows is). --- coub-gui.py | 21 ++++++++++----------- example.conf | 10 +++++++--- 2 files changed, 17 insertions(+), 14 deletions(-) diff --git a/coub-gui.py b/coub-gui.py index d19229f..7647047 100755 --- a/coub-gui.py +++ b/coub-gui.py @@ -10,17 +10,6 @@ class GuiDefaultOptions(coub.DefaultOptions): """Custom default option class to reflect the differences between CLI and GUI.""" - # There's no way for the user to enter input if a prompt occurs - # So only "yes" or "no" make sense - PROMPT = "no" - - # Outputting to the current dir is a viable strategy for a CLI tool - # Not so much for a GUI - if coub.DefaultOptions.PATH == ".": - PATH = os.path.join(os.path.expanduser("~"), "coubs") - else: - PATH = os.path.abspath(coub.DefaultOptions.PATH) - # Create special labels for dropdown menus # Some internally used values would cause confusion # Some menus also combine options @@ -40,6 +29,16 @@ def __init__(self): config_dirs = [os.path.dirname(os.path.realpath(__file__))] super(GuiDefaultOptions, self).__init__(config_dirs=config_dirs) + # There's no way for the user to enter input if a prompt occurs + # So only "yes" or "no" make sense + if self.PROMPT not in {"yes", "no"}: + self.PROMPT = "no" + + # Outputting to the current dir (or any relative path) is a viable strategy for a CLI tool + # Not so much for a GUI + if not os.path.isabs(self.PATH): + self.PATH = os.path.join(os.path.expanduser("~"), "coubs") + def translate_to_cli(options): """Make GUI-specific options object compatible with the main script.""" # Special dropdown menu labels and what they translate to diff --git a/example.conf b/example.conf index 5931eb2..57ac134 100644 --- a/example.conf +++ b/example.conf @@ -17,12 +17,16 @@ VERBOSITY = 1 # How to answer overwrite prompts # yes -> always answer yes # no -> always answer no -# Allowed values: no restrictions (everything but yes/no, will lead to a prompt) +# Allowed values: +# coub.py: no restrictions (everything but yes/no will lead to a prompt) +# coub-gui.py: yes, no (everything but yes/no will lead to no) PROMPT = prompt -# Default download destination +# Default download destination (will be created if it doesn't exist) # ~ and ~user do NOT get expanded and will be treated literally -# Allowed values: absolute/relative paths (don't have to exist yet) +# Allowed values: +# coub.py: absolute/relative paths +# coub-gui.py: absolute paths (relative paths will lead to fallback path) PATH = . # Whether to keep the individual video/audio streams after merging From 63985c5e8dd1f99bc484b7fbe6145d240d0b8d89 Mon Sep 17 00:00:00 2001 From: HelpSeeker Date: Sat, 2 May 2020 21:31:45 +0200 Subject: [PATCH 33/40] Adjust copyright/license notices This is done in accordance with the FSF's instructions on how to use GNU licenses for one's software. --- .gitignore | 2 +- LICENSE => COPYING | 0 coub-gui.py | 36 +++++++++++++++++++++++++++++++++++- coub.py | 17 +++++++++++++++++ 4 files changed, 53 insertions(+), 2 deletions(-) rename LICENSE => COPYING (100%) diff --git a/.gitignore b/.gitignore index 8c9999f..ee167d1 100644 --- a/.gitignore +++ b/.gitignore @@ -1,7 +1,7 @@ /* !.gitignore -!LICENSE +!COPYING !README.md !coub.py !coub-gui.py diff --git a/LICENSE b/COPYING similarity index 100% rename from LICENSE rename to COPYING diff --git a/coub-gui.py b/coub-gui.py index 7647047..8f92b43 100755 --- a/coub-gui.py +++ b/coub-gui.py @@ -1,5 +1,22 @@ #!/usr/bin/env python3 +""" +Copyright (C) 2018-2020 HelpSeeker + +This program is free software: you can redistribute it and/or modify +it under the terms of the GNU General Public License as published by +the Free Software Foundation, either version 3 of the License, or +(at your option) any later version. + +This program is distributed in the hope that it will be useful, +but WITHOUT ANY WARRANTY; without even the implied warranty of +MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +GNU General Public License for more details. + +You should have received a copy of the GNU General Public License +along with this program. If not, see . +""" + import os from textwrap import dedent @@ -82,7 +99,24 @@ def translate_to_cli(options): 'name': 'CoubDownloader', 'description': 'A simple download script for coub.com', 'website': 'https://github.com/HelpSeeker/CoubDownloader', - 'license': 'GPLv3', + 'license': dedent( + """ + Copyright (C) 2018-2020 HelpSeeker + + This program is free software: you can redistribute it and/or modify + it under the terms of the GNU General Public License as published by + the Free Software Foundation, either version 3 of the License, or + (at your option) any later version. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with this program. If not, see . + """ + ), } ] } diff --git a/coub.py b/coub.py index 008f844..5492c1b 100755 --- a/coub.py +++ b/coub.py @@ -1,5 +1,22 @@ #!/usr/bin/env python3 +""" +Copyright (C) 2018-2020 HelpSeeker + +This program is free software: you can redistribute it and/or modify +it under the terms of the GNU General Public License as published by +the Free Software Foundation, either version 3 of the License, or +(at your option) any later version. + +This program is distributed in the hope that it will be useful, +but WITHOUT ANY WARRANTY; without even the implied warranty of +MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +GNU General Public License for more details. + +You should have received a copy of the GNU General Public License +along with this program. If not, see . +""" + import argparse import asyncio import json From 012570ae05f36f30a3538065a5467908e28aa180 Mon Sep 17 00:00:00 2001 From: HelpSeeker Date: Sat, 2 May 2020 21:58:12 +0200 Subject: [PATCH 34/40] Update README.md --- README.md | 54 ++++++++---------------------------------------------- 1 file changed, 8 insertions(+), 46 deletions(-) diff --git a/README.md b/README.md index ed1cad3..2c598b5 100644 --- a/README.md +++ b/README.md @@ -24,8 +24,7 @@ CoubDownloader is a simple script to download videos (called coubs) from [Coub]( 1. [Video resolution vs. quality](#video-resolution-vs-quality) 2. [AAC audio](#aac-audio) 3. ['share' videos](#share-videos) -8. [Changes since Coub's database upgrade (watermark & co)](#changes-since-coubs-database-upgrade-watermark--co) -9. [Changes since switching to Coub's API (previously used youtube-dl)](#changes-since-switching-to-coubs-api-previously-used-youtube-dl) + 4. [Known problems](#known-problems) ## Usage @@ -402,6 +401,8 @@ The default sort order (most popular coubs of the month) may provide less result A basic GUI, powered by [Gooey](https://github.com/chriskiehl/Gooey), is provided via `coub-gui.py`. +Please note that this is only a quickly thrown together frontend for convenience. The main focus of this project is the CLI tool. + ![Settings window on Windows](/images/coub-gui_input_Windows.png) ![Progress window on Linux](/images/coub-gui_execution_Linux.png) It provides the same functionality as the main CLI tool, with a few notable exceptions: @@ -497,47 +498,8 @@ Also because of the special property of the *share* version, there are some pitf There's no fallback for *share* videos. If the *share* version is not yet available, then the script will count the coub as unavailable. -## Changes since Coub's database upgrade (watermark & co) - -Coub started to massively overhaul their database and API. Of course those changes aren't documented (why would you document API changes anyway?). - -- [x] Only repair video streams that are actually broken -- [x] Remove mobile option (they now come with a watermark and are the exact same as html5 med) -- [x] Add AAC mobile audio as another possible audio version (ranked between low and high quality MP3 audio) -- [x] Add options to prefer AAC or only download AAC audio -- [x] Add shared option (video+audio already combined) -- [x] Download coubs from the hot section -- [x] Download coubs from communities (incl. Featured & Coub of the Day) -- [x] Asynchronous coub processing -- [x] Asynchronous timeline parsing -- [x] Detect stream corruption (incl. old Coub storage method) -- [x] Workspace cleanup (incomplete coubs) after user interrupt -- [x] Colorized terminal output -- [x] Download retries -- [x] URL input without input options -- [x] Autocompletion of incomplete/malformed URLs (to some extent) -- [x] Advanced sorting per input -- [x] Support for sort order related URLs -- [x] Download random coubs -- [x] Option to change the container format for stream remuxing -- [x] Basic GUI frontend -- [x] Read custom defaults from config file - -## Changes since switching to Coub's API (previously used youtube-dl) - -- [x] Download all coubs from a channel -- [x] Download all recoubs from a channel -- [x] Limit number of downloaded coubs -- [x] ~~Wait x seconds between downloads~~ (not supported anymore due to async execution) -- [x] ~~Limit download speed~~ (was only possible in the Bash version) -- [x] Download all coubs with a certain tag -- [x] Check for the existence of a coub before downloading -- [x] Specify max. coub duration (FFmpeg syntax) -- [x] Keep track of already downloaded coubs -- [x] Export parsed coub links (from channels or tags) to a file for later usage -- [x] Different verbosity levels -- [x] Choose download order for channels and tags -- [x] Custom output formatting -- [x] Download all coubs from a search query -- [x] Choose what video/audio quality to download -- [x] ~~Download videos for mobile devices to avoid watermarks~~ (not possible anymore) +### Known problems + +* no error handling or retries if server disconnects during input parsing +* losing your internet connection while downloading will stall the script indefinitely +* (GUI only) progress messages don't use monospace fonts on Windows From 198f49f1be603f03bc947411ae2b998fd38039d9 Mon Sep 17 00:00:00 2001 From: HelpSeeker Date: Sat, 2 May 2020 21:45:09 +0000 Subject: [PATCH 35/40] Update README.md --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 2c598b5..d2efe38 100644 --- a/README.md +++ b/README.md @@ -136,7 +136,7 @@ Standalone builds only require FFmpeg. * [aiohttp](https://aiohttp.readthedocs.io/en/stable/) for asynchronous execution **(strongly recommended)** * [colorama](https://github.com/tartley/colorama) for colorized terminal output on Windows -* [Gooey](https://github.com/chriskiehl/Gooey) to run `coub-gui.py` +* [Gooey](https://github.com/chriskiehl/Gooey) to run `coub-gui.py` (be sure to install it with wxPython < 4.1.0) ## Configuration From 018636c4b6e8d7484f4d7b250bad35c679cb6f8a Mon Sep 17 00:00:00 2001 From: HelpSeeker Date: Sun, 3 May 2020 10:40:32 +0200 Subject: [PATCH 36/40] Add special keywords for tag_sep and fallback_char In particular it wasn't possible to use a whitespace as tag separator or fallback character or to only remove disallowed characters from the filename, if specified via a config file. This is remedied by introducing keywords that the script will interpret accordingly space -> whitespace (tag separator and fallback character) None -> empty string (i.e. removal) (fallback character) Not the most elegant solution, but we're only dealing with 2 keywords anyway. --- coub-gui.py | 8 ++++++++ coub.py | 10 +++++++++- example.conf | 4 ++-- 3 files changed, 19 insertions(+), 3 deletions(-) diff --git a/coub-gui.py b/coub-gui.py index 8f92b43..60f0ce3 100755 --- a/coub-gui.py +++ b/coub-gui.py @@ -286,6 +286,14 @@ def parse_cli(): # but internally the default value is None if args.name_template == "%id%": args.name_template = None + # Defining whitespace or an empty string in the config isn't possible + # Instead translate appropriate keywords + if args.tag_sep == "space": + args.tag_sep = " " + if args.fallback_char is None: + args.fallback_char = "" + elif args.fallback_char == "space": + args.fallback_char = " " return translate_to_cli(args) diff --git a/coub.py b/coub.py index 5492c1b..4f2079e 100755 --- a/coub.py +++ b/coub.py @@ -203,7 +203,7 @@ def check_values(self): "FFMPEG_PATH": (lambda x: isinstance(x, str)), "COUBS_PER_PAGE": (lambda x: x in range(1, 26)), "TAG_SEP": (lambda x: isinstance(x, str)), - "FALLBACK_CHAR": (lambda x: isinstance(x, str)), + "FALLBACK_CHAR": (lambda x: isinstance(x, str) or x is None), "WRITE_METHOD": (lambda x: x in ["w", "a"]), "CHUNK_SIZE": (lambda x: isinstance(x, int) and x > 0), } @@ -1529,6 +1529,14 @@ def parse_cli(): # but internally the default value is None if args.name_template == "%id%": args.name_template = None + # Defining whitespace or an empty string in the config isn't possible + # Instead translate appropriate keywords + if args.tag_sep == "space": + args.tag_sep = " " + if args.fallback_char is None: + args.fallback_char = "" + elif args.fallback_char == "space": + args.fallback_char = " " return args diff --git a/example.conf b/example.conf index 57ac134..656374d 100644 --- a/example.conf +++ b/example.conf @@ -145,9 +145,9 @@ NAME_TEMPLATE = %id% FFMPEG_PATH = ffmpeg # Character or string to separate tags in the output filename -# Allowed values: strings +# Allowed values: strings, space (to separate with whitespace) TAG_SEP = _ # Character to replace forbidden characters in the filename (newlines, tabs, /, \, etc.) -# Allowed values: strings +# Allowed values: strings, space (to replace with whitespace), None (to only remove) FALLBACK_CHAR = - From 23788281620737328e2a92cc928fa6dbd6c06a3e Mon Sep 17 00:00:00 2001 From: HelpSeeker Date: Sun, 3 May 2020 16:13:47 +0200 Subject: [PATCH 37/40] Fix error when using --duration An annoying oversight. valid_time() and other test functions for argparse get called before the global options object was created. That means we can't yet access the (potentially custom) FFmpeg path. Fortunately advanced defaults and advanced options always have the same values, so in a dirty workaround we can assemble the defaults again in valid_time() to know what FFmpeg path to use. Doing that and perhaps even reading the config file again feels wasteful but at the same time the alternatives aren't more pleasant. Moving the test after the parsing process would lead to incoherent error messages (can't let the parser report the error, when the parser is already finished). And making the defaults global would open a whole can of worms in coub-gui.py, not to mention that creating another global options object just to get access once outside of parse_cli() would feel even dirtier. --- coub.py | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/coub.py b/coub.py index 4f2079e..045e51b 100755 --- a/coub.py +++ b/coub.py @@ -1208,8 +1208,11 @@ def positive_int(string): def valid_time(string): """Test valditiy of time syntax with FFmpeg.""" + # Gets called in parse_cli, so opts.ffmpeg_path isn't available yet + # Exploits the fact that advanced defaults and options are always the same + defaults = DefaultOptions() command = [ - opts.ffmpeg_path, "-v", "quiet", + defaults.FFMPEG_PATH, "-v", "quiet", "-f", "lavfi", "-i", "anullsrc", "-t", string, "-c", "copy", "-f", "null", "-", From 57ea31e86fcd4d39955f22b2f75b9a3dd4c56dbd Mon Sep 17 00:00:00 2001 From: HelpSeeker Date: Sun, 3 May 2020 18:27:18 +0200 Subject: [PATCH 38/40] Fix confusing checkbox for --keep (but not really) When setting "KEEP = True" via a config file, a checked box actually meant False (don't keep), while an unchecked box meant True (keep). This fixes the confusing selection, but then again not really, because the option now suffers the same problem as with the CLI version, that it can only be used to activate keeping and not deactivate it. So to summarize: KEEP = False: Checked -> True, Unchecked -> False KEEP = True: Checked -> True, Unchecked -> True The only real solution would be to introduce a --no-keep option (similar to --preview/--no-preview) for the CLI version and write a proper GUI for the GUI version. The latter won't be happening any time soon and I don't see the point of introducing yet another CLI option that nobody will use (including myself). --- coub-gui.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/coub-gui.py b/coub-gui.py index 60f0ce3..bca52a8 100755 --- a/coub-gui.py +++ b/coub-gui.py @@ -177,7 +177,7 @@ def parse_cli(): default=defs.ARCHIVE, widget="FileSaver", metavar="Archive", gooey_options={'message': "Choose archive file"}, help="Use an archive file to keep track of already downloaded coubs") - common.add_argument("--keep", action=f"store_{'false' if defs.KEEP else 'true'}", + common.add_argument("--keep", action="store_const", const=True, default=defs.KEEP, widget="BlockCheckbox", metavar="Keep streams", help="Whether to keep the individual streams after merging") From 4bdf6f84619265adc11943c3c5f71581f9cf38c2 Mon Sep 17 00:00:00 2001 From: HelpSeeker Date: Sun, 3 May 2020 19:23:11 +0200 Subject: [PATCH 39/40] Update README.md --- README.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/README.md b/README.md index d2efe38..fe95031 100644 --- a/README.md +++ b/README.md @@ -502,4 +502,9 @@ There's no fallback for *share* videos. If the *share* version is not yet availa * no error handling or retries if server disconnects during input parsing * losing your internet connection while downloading will stall the script indefinitely +* some enabling switches (e.g. --keep) don't have a disabling counterpart, so activating their options via a config means you can't disable them at runtime +* using a CLI media player to preview audio files may lead to its keyboard shortcuts not working +* Python 3.8 and newer will print tracebacks (only warnings) when the user interrupts the program (see the Wiki for more infos) * (GUI only) progress messages don't use monospace fonts on Windows +* (GUI only) having wrong values defined in your config will prevent the GUI from opening (open it from the command line to see the error) +* (GUI only) Gooey seems to be broken for WxPython 4.1.0 From c0d1f29a5e0c635755daa81be236c6960ea22369 Mon Sep 17 00:00:00 2001 From: HelpSeeker Date: Sun, 3 May 2020 21:11:13 +0200 Subject: [PATCH 40/40] Update README.md --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index fe95031..d5be694 100644 --- a/README.md +++ b/README.md @@ -135,7 +135,7 @@ Standalone builds only require FFmpeg. ### Optional * [aiohttp](https://aiohttp.readthedocs.io/en/stable/) for asynchronous execution **(strongly recommended)** -* [colorama](https://github.com/tartley/colorama) for colorized terminal output on Windows +* [colorama](https://github.com/tartley/colorama) for colorized terminal output on Windows (you should also install it if you want to use `coub-gui.py`) * [Gooey](https://github.com/chriskiehl/Gooey) to run `coub-gui.py` (be sure to install it with wxPython < 4.1.0) ## Configuration