V2.8.7 Features: Masking sensitive data in logs. No delete option. Fixes & Refactoring. Code coverage and Python 3.7 testing.

Fixes/Features

• Setup via PyPi
• Fixed #73: Weird issue on Synology with python installed from default packages (DO NOT INSTALL IT) . Updated README.
• Fixed #74: Initial logging has an erratic behaviour (Low priority)
• Fixed/Feature: uploadr.cron: Proper use of PREFIX and PATH settings
• Feature: At the end prints the total for (countflickr - countlocal) for easier reading.
• Feature #71: Suggestion: no.delete: New option --no-delete-from-flickr (under testing.Use at your own risk for now!)
• Feature #78: Enhancement: Mask sensitive data (like file name and Albums) on Logs. New option: -m/-mask-sensitive
• Cleaned up debugging of external modules namely multiprocessing logging.
• Testing with python 2.7, 3.6 and 3.7
• Refactored all DB function calls. Protected by error code hangling and lock usage whenm applicable

Setup options:

• (For alpha versions like this one) You can install from pip:
  pip install --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple flickr-uploader==2.8.7a36 --prefix=~/apps/Python/ --no-cache-dir
• (For regular versions) You can install from pip:
  pip install flickr-uploader --prefix=~/apps/Python
• You can download and run install with command:
  python3 setup.py install --prefix=~/apps/Python --old-and-unmanageable
• You can download and run from local folder.
• uploadr.ini: Note that uploadr.ini definition for path search is changed:
  • Sets base folder with FOLDER = os.path.abspath(os.getcwd())
  • DB_PATH, LOCK_PATH, TOKEN_CACHE and TOKEN_PATH are now relative to FOLDER, for example: os.path.join(%(FOLDER)s, "flickrdb")
• From v2.8.6 onwards it looks for uploadr.ini as follows:
  1. Use --config-file argument option to indicate where to find the file. Example --config_file uploadr.ini
  2. If not, os.path.dirname(sys.argv[0]). Example: ~/apps/Python/bin/uploadr.ini or ./uploadr.ini
  3. If not, os.path.dirname(sys.argv[0]), '..', 'etc', 'uploadr.ini'. Example: ~/apps/Python/etc/uploadr.ini

Output Messages

• Adjustments on output and logging messages.
• As a reference from version 2.8.6: New set of options related to Rotating Error Logging added to uploadr.ini:

################################################################################
#   Output logging information into a rotating set of log file(s).
#       ROTATING_LOGGING to Enable/Disable
#       ROTATING_LOGGING_PATH location of folder/main logging filename
#       ROTATING_LOGGING_FILE_SIZE for maximum file size of each log file
#       ROTATING_LOGGING_FILE_COUNT for maximum count of old log files to keep
#       ROTATING_LOGGING_LEVEL Level Logging 
#           Check LOGGING_LEVEL setting for options.
#           Normally set ROTATING_LOGGING_LEVELto lower than LOGGING_LEVEL
################################################################################
ROTATING_LOGGING = True
ROTATING_LOGGING_PATH = os.path.join(os.path.dirname(sys.argv[0]), "logs", "uploadr.err")
ROTATING_LOGGING_FILE_SIZE = 25*1024*1024  # 25 MBytes
ROTATING_LOGGING_FILE_COUNT = 3
ROTATING_LOGGING_LEVEL = 30

Environment and Coding

• Python 2.7 + 3.6 + 3.7 compatibility: use of "# noqa" as applicable
• Development now use coveralls.io to check source code coverage
• autopep8, PEP8, flakes and pylint3 (not all!) adjustments. Several pylint reaching now a rating of 9.45/10.00.
• pytest --flakes & pytest --doctest-modules
• Runs several unittests
• For installation:
  • pip install flickr-uploader
  • setup.py (optional use)
  • or manual
• NicePrint:
  • use of staticmethods
  • New argument logalso
  • niceprocessedfiles new argument for adaptable output message
• On upload failure: sleep for(configured on Konstants.py) 20s (instead of 10) to avoid duplicated uploaded files.

V2.8.7-alpha36 Features: Masking sensitive data in logs. No delete option. Fixes & Refactoring. Code coverage and Python 3.7 testing.

Fixes/Features

• Setup via PyPi
• Fixed #73: Weird issue on Synology with python installed from default packages (DO NOT INSTALL IT) . Updated README.
• Fixed #74: Initial logging has an erratic behaviour (Low priority)
• Fixed/Feature: uploadr.cron: Proper use of PREFIX and PATH settings
• Feature: At the end prints the total for (countflickr - countlocal) for easier reading.
• Feature #71: Suggestion: no.delete: New option --no-delete-from-flickr (under testing.Use at your own risk for now!)
• Feature #78: Enhancement: Mask sensitive data (like file name and Albums) on Logs. New option: -m/-mask-sensitive
• Cleaned up debugging of external modules namely multiprocessing logging.
• Testing with python 2.7, 3.6 and 3.7
• Refactored all DB function calls. Protected by error code hangling and lock usage whenm applicable
• NicePrint use of staticmethods

Setup options:

• (For alpha versions like this one) You can install from pip:
  pip install --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple flickr-uploader==2.8.7a36 --prefix=~/apps/Python/ --no-cache-dir
• (For regular versions) You can install from pip:
  pip install flickr-uploader --prefix=~/apps/Python
• You can download and run install with command:
  python3 setup.py install --prefix=~/apps/Python --old-and-unmanageable
• You can download and run from local folder.
• uploadr.ini: Note that uploadr.ini definition for path search is changed:
  • Sets base folder with FOLDER = os.path.abspath(os.getcwd())
  • DB_PATH, LOCK_PATH, TOKEN_CACHE and TOKEN_PATH are now relative to FOLDER, for example: os.path.join(%(FOLDER)s, "flickrdb")
• From v2.8.6 onwards it looks for uploadr.ini as follows:
  1. Use --config-file argument option to indicate where to find the file. Example --config_file uploadr.ini
  2. If not, os.path.dirname(sys.argv[0]). Example: ~/apps/Python/bin/uploadr.ini or ./uploadr.ini
  3. If not, os.path.dirname(sys.argv[0]), '..', 'etc', 'uploadr.ini'. Example: ~/apps/Python/etc/uploadr.ini

Output Messages

• Adjustments on output and logging messages.
• As a reference from version 2.8.6: New set of options related to Rotating Error Logging added to uploadr.ini:

################################################################################
#   Output logging information into a rotating set of log file(s).
#       ROTATING_LOGGING to Enable/Disable
#       ROTATING_LOGGING_PATH location of folder/main logging filename
#       ROTATING_LOGGING_FILE_SIZE for maximum file size of each log file
#       ROTATING_LOGGING_FILE_COUNT for maximum count of old log files to keep
#       ROTATING_LOGGING_LEVEL Level Logging 
#           Check LOGGING_LEVEL setting for options.
#           Normally set ROTATING_LOGGING_LEVELto lower than LOGGING_LEVEL
################################################################################
ROTATING_LOGGING = False
ROTATING_LOGGING_PATH = os.path.join(os.path.dirname(sys.argv[0]), "logs", "uploadr.err")
ROTATING_LOGGING_FILE_SIZE = 25*1024*1024  # 25 MBytes
ROTATING_LOGGING_FILE_COUNT = 3
ROTATING_LOGGING_LEVEL = 30

Environment and Coding

• Python 2.7 + 3.6 + 3.7 compatibility: use of "# noqa" as applicable
• Development now use coveralls.io to check source code coverage
• autopep8, PEP8, flakes and pylint3 (not all!) adjustments. Several pylint reaching now a rating of 9.45/10.00.
• pytest --flakes & pytest --doctest-modules
• Runs several unittests
• For installation:
  • pip install flickr-uploader
  • setup.py (optional use)
  • or mannual

V2.8.6 Features: install via PyPi. New settings on uploadr.ini. Under the hood improvements/refactoring.

Fixes/Features

• Setup via PyPi
• Several under the hood code refactoring, pylint3 check and more.
  • Drop unused setting TOKEN_PATH.
  • Added verbosity control with argument (-v, -vv, etc...) on NicePrint class.
• Fix #46 Use current working directory to look for INI (uploadr.ini) file
• Now support for multiprocessing for feature: #68 Option to have Rotating Error Log written to files to keep size under control.
• Setup options:
  • You can install from pip:
    pip install flickr-uploader --prefix=~/apps/Python
  • You can download and run install with command:
    python3 setup.py install --prefix=~/apps/Python --old-and-unmanageable
  • You can download and run from local folder.
• uploadr.ini: Note that uploadr.ini definition for path search is changed:
  • Sets base folder with FOLDER = os.path.abspath(os.getcwd())
  • DB_PATH, LOCK_PATH, TOKEN_CACHE and TOKEN_PATH are now relative to FOLDER, for example: os.path.join(%(FOLDER)s, "flickrdb")
• From v2.8.6 onwards it looks for uploadr.ini as follows:
  1. Use --config-file argument option to indicate where to find the file. Example --config_file uploadr.ini
  2. If not, os.path.dirname(sys.argv[0]). Example: ~/apps/Python/bin/uploadr.ini or ./uploadr.ini
  3. If not, os.path.dirname(sys.argv[0]), '..', 'etc', 'uploadr.ini'. Example: ~/apps/Python/etc/uploadr.ini

Output Messages

• Adjustments on output and logging messages.
• Feature: #68 Option to have Rotating Error Log written to files to keep size under control.
• New set of options related to Rotating Error Logging added to uploadr.ini:

################################################################################
#   Output logging information into a rotating set of log file(s).
#       ROTATING_LOGGING to Enable/Disable
#       ROTATING_LOGGING_PATH location of folder/main logging filename
#       ROTATING_LOGGING_FILE_SIZE for maximum file size of each log file
#       ROTATING_LOGGING_FILE_COUNT for maximum count of old log files to keep
#       ROTATING_LOGGING_LEVEL Level Logging 
#           Check LOGGING_LEVEL setting for options.
#           Normally set ROTATING_LOGGING_LEVELto lower than LOGGING_LEVEL
################################################################################
ROTATING_LOGGING = False
ROTATING_LOGGING_PATH = os.path.join(os.path.dirname(sys.argv[0]), "logs", "uploadr.err")
ROTATING_LOGGING_FILE_SIZE = 25*1024*1024  # 25 MBytes
ROTATING_LOGGING_FILE_COUNT = 3
ROTATING_LOGGING_LEVEL = 30

Environment and Coding

• Python 2.7 + 3.6 compatibility: use of "# noqa" as applicable
• autopep8, PEP8, flakes and pylint3 (not all!) adjustments
• pytest --flakes & pytest --doctest-modules
• Runs several unittests
• For installation:
  • pip install flickr-uploader
  • setup.py (optional use)
  • or mannual

V2.8.5 Features: authentication, rotating, FlickrAPI wrapper module, setup. Under the hood improvements/refactoring.

Fixes/Features

• Several under the hood code refactoring, pylint3 check and more.
• Fix: #66 fix Flickr error: "ValueError: invalid literal for int() with base 10"
• Feature: #60 Initial authentication option (-a)
• Internal improvement: #64 Isolate and wrap around retry and try/except sequence to flickrapi function calls
• Feature: #68 Option to have Rotating Error Log written to files to keep size under control.
• Setup (optional):
  • You can run install with command:
    python3 setup.py install --prefix=~/apps/Python
  • You can run installcfg to copy config files (.ini and .cron) to a designated folder with command:
    python3 setup.py installcfg --folder ~/apps/flickr-uploader
• From v2.7.6 uploadr.ini configuration is searched from:
  • --config-file argument
  • from the sys.argv[0] path (compatible with previous versions)

Questions & Answers

• Note that uploadr.ini definition for path search of DB_PATH, LOCK_PATH, TOKEN_CACHE and TOKEN_PATH still uses sys.argv[0] location as a basis.
• (to be) Updated installation notes on Readme.

Output Messages

• Adjustments on output and logging messages.
• Feature: #68 Option to have Rotating Error Log written to files to keep size under control.
  • New set of options related to Rotating Error Logging added to uploadr.ini:

################################################################################
#   Output logging information into a rotating set of log file(s).
#       ROTATING_LOGGING to Enable/Disable
#       ROTATING_LOGGING_PATH location of folder/main logging filename
#       ROTATING_LOGGING_FILE_SIZE for maximum file size of each log file
#       ROTATING_LOGGING_FILE_COUNT for maximum count of old log files to keep
#       ROTATING_LOGGING_LEVEL Level Logging 
#           Check LOGGING_LEVEL setting for options.
#           Normally set ROTATING_LOGGING_LEVELto lower than LOGGING_LEVEL
################################################################################
ROTATING_LOGGING = False
ROTATING_LOGGING_PATH = os.path.join(os.path.dirname(sys.argv[0]), "logs", "uploadr.err")
ROTATING_LOGGING_FILE_SIZE = 25*1024*1024  # 25 MBytes
ROTATING_LOGGING_FILE_COUNT = 3
ROTATING_LOGGING_LEVEL = 30

Environment and Coding

• Python 2.7 + 3.6 compatibility: use of "# noqa" as applicable
• autopep8, PEP8, flakes and pylint3 (not all!) adjustments
• pytest --flakes & pytest --doctest-modules
• Runs several unittests
• setup.py (optional use)
• setup.py installcfg (optional use)

V2.8.1 Under the hood improvements/refactoring. Standardize multiprocessing processes.

Fixes/Features

• Several under the hood code refactoring, avoid use of globals, use of (more) modules, pylint3 check, order of imports, remove unused import, docstrs, use logging % instead of .format; and more.
• #61 Use mprocessing function to setup multiprocessing and simplify code.
• Setup (optional):
  • You can run install with python3 setup.py install --prefix=~/apps/Python
• From v2.7.6 uploadr.ini configuration is searched from:
  • --config-file argument
  • from the sys.argv[0] path (compatible with previous versions)
• Initial lock control (Linux/Windows compatible) to ensure only one copy is running was corrected.
• Exception handling on (more) sql functions.

Questions & Answers

• Note that uploadr.ini definition for path search of DB_PATH, LOCK_PATH, TOKEN_CACHE and TOKEN_PATH still uses sys.argv[0] location as a basis.
• (to be) Updated installation notes on Readme.

Output Messages

• Adjustments on output and logging messages.

Environment and Coding

• Python 2.7 + 3.6 compatibility: use of "# noqa" as applicable
• autopep8, PEP8, flakes and pylint3 (not all!) adjustments
• pytest --flakes & pytest --doctest-modules
• Runs several unittests
• setup.py (optional use)

V2.8.0 Improved performance on Add Pics to Sets/Albums running in Multiprocessing mode

Fixes/Features

• #11 Added feature to improve performance on Add Pics to Sets(Albums)
• #57 Used control of closing connection to Database.
• Setup CONVERT_RAW_FILES to True on uploadr.ini configuration file.
  • Uses external tool: exiftool by Phil Harvey.
• Dry run option (-n) also works wit convert RAW files.
• Setup (optional):
  • You can run install with python3 setup.py install --prefix=~/apps/Python
• From v2.7.6 uploadr.ini configuration is searched from:
  • --config-file argument
  • from the sys.argv[0] path (compatible with previous versions)
• moved niceprocessedfiles function to niceprint module
• created mprocessing function for multiprocessing scenarios
• initial lock control to ensure only one copy is running was corrected
• exception handling on (more) sql functions

Questions & Answers

• Note that uploadr.ini definition for path search of DB_PATH, LOCK_PATH, TOKEN_CACHE and TOKEN_PATH still uses sys.argv[0] location as a basis.
• (to be) Updated installation notes on Readme.

Output Messages

• Small adjustments on output messages.

Environment and Coding

• Python 2.7 + 3.6 compatibility: use of "# noqa" as applicable
• autopep8, PEP8, flakes adjustments
• pytest --flakes & pytest --doctest-modules
• Runs several unittests
• Created setup.py (optional use)

V2.7.8 convert Raw Files option with exiftool

Fixes/Features

• Added feature to allow convert RAW files into JPG for loading.
• Setup CONVERT_RAW_FILES to True on uploadr.ini configuration file.
  • Uses external tool: exiftool by Phil Harvey.
• Dry run option (-n) also works wit convert RAW files.
• Setup (optional):
  • You can run install with python3 setup.py install --prefix=~/apps/Python
• From v2.7.6 uploadr.ini configuration is searched from:
  • --config-file argument
  • from the sys.argv[0] path (compatible with previous versions)

Questions & Answers

• Note that uploadr.ini definition for path search of DB_PATH, LOCK_PATH, TOKEN_CACHE and TOKEN_PATH still uses sys.argv[0] location as a basis.
• (to be) Updated installation notes on Readme.

Output Messages

• Small adjustments on output messages.

Environment and Coding

• Python 2.7 + 3.6 compatibility: use of "# noqa" as applicable
• autopep8, PEP8, flakes adjustments
• pytest --flakes & pytest --doctest-modules
• Runs several unittests
• Created setup.py (optional use)

V2.7.7 Compatibility with Windows (use of portalocker, if available), Fix #56

Fixes/Features

• For locking on Windows: portalocker (when available) for added compatibility (For Windows support). Use fcntl otherwise (thanks @belidzs and others!!)
• Fix for #56 (3 attempts to load file with exception "038".... aborts!)
• Setup (optional):
  • You can run install with python3 setup.py install --prefix=~/apps/Python
• From v2.7.6 uploadr.ini configuration is searched from:
  • --config-file argument
  • from the sys.argv[0] path (compatible with previsous versions)

Questions & Answers

• Note that uploadr.ini definition for path search of DB_PATH, LOCK_PATH, TOKEN_CACHE and TOKEN_PATH still uses sys.argv[0] location as a basis.
• (to be) Updated installation notes on Readme.

Output Messages

• Small adjustments on output messages.

Environment and Coding

• Python 2.7 + 3.6 compatibility: use of "# noqa" as applicable
• autopep8, PEP8, flakes adjustments
• pytest --flakes & pytest --doctest-modules
• Runs several unittests
• Created setup.py (optional use)

V2.7.6 Enhanced control on INI parameters. Improved UploadFile function. New setup.py (optional)

Fixes/Features

• Enhancement #7 apply further control on INI parameters. New configuration mechanism. Default values are included in the app. Ignores wrong values in INI file. Except for api_key and secret which you have to adjust, all others are preset.
• Enhancement #28 Revise upload sequence.
• More clear output messages on execution progress.
• Setup (optional):
  • You can run install with python3 setup.py install --prefix=~/apps/Python
• From v2.7.6 uploadr.ini configuration is searched from:
  • --config-file argument
  • from the sys.argv[0] path (compatible with previsous versions)

Questions & Answers

• Note that uploadr.ini definition for path search of DB_PATH, LOCK_PATH, TOKEN_CACHE and TOKEN_PATH still uses sys.argv[0] location as a basis.
• (to be) Updated installation notes on Readme.

Output Messages

• More clear output messages on excution progress.

Environment and Coding

• Python 2.7 + 3.6 compatibility: use of "# noqa" as applicable
• autopep8, PEP8, flakes adjustments
• pytest --flakes & pytest --doctest-modules
• Runs several unittests
• Created setup.py (optional)

V2.7.3 Replace Video (#45), Code cleanup (autopep8, flakes adjustments, use of lib modules)

Fixes/Features

• I'm including a distribution tarball flickr-uploader-2.7.3.tar.gz for faster/easier download with the essential files and no test related files. The full; source code is still available of course!
• From v2.7.0 onwards, uploadr adds to Flickr an album tag to each pic. uploadr v2.7.0 will automatically update the previously loaded pics with such album tag, upon first run.
  • Version 2.7.1, 2.7.2 and 2.7.3 run in multiprocessing mode (use option -p) to add album tag. It limits function calls to flickr to 5 per second to avoid reaching the 3600 requests/hour of Flickr.
  • New option: --add-albums-migrate is only available to re-run assigning album tags to previously uploaded pics, should it be necessary and in case the automated process fails.
  • PLEASE NOTE FOR LARGE LIBRARIES THIS 1st RUN:
    • MAY TAKE A WHILE. IN MY CASE ABOUT 10min FOR EACH 1000 PICS OR SO...
    • MAY SURPASS THE Flickr 3600 requests/hour LIMIT. SIMPLY RESTART AFTER SOME TIME & UNTIL COMPLETION
• Confirmed resolution of issue #28
• #45 Delete/Upload to be able to replace Videos on Flickr

Questions & Answers

• Updated installation notes on Readme.
• Readme includes master test result status (from travisCI)

Output Messages

• Removed (some) multiprocessing messages from verbose

Environment and Coding

• Python 2.7 + 3.6 compatibility: use of "# nova" as applicable
• autopep8, PEP8, flakes adjustments (thank you @cpb8010 for your sugestions)
• pytest --flakes & pytest --doctest-modules
• Reorganized order of Init message and import.
• Moved sections from file (uploadr.py) into CONTRIBUTING, TODO, README
• Included LICENSE.txt
• Version number defined in lib/version.py
• Created classes lib/niceprint.py, lib/UPLDRConstants.py, lib/rate_limited.py
• Setup several unittests
• Created setup.py (in trial)
• Addressed #23 (split into modules)