Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add update_config_file() and unittests #91

Open
wants to merge 2 commits into
base: master
Choose a base branch
from
Open

Add update_config_file() and unittests #91

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
cb27bdb
Add update_config_file() and unittests
luandy64 Nov 16, 2018
f479a53
Add missing dependency
luandy64 Nov 16, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions singer/utils.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -185,6 +185,9 @@ def check_config(config, required_keys):
if missing_keys:
raise Exception("Config is missing required keys: {}".format(missing_keys))

def update_config_file(config_path, new_config):
with open(config_path, 'w') as output:
output.write(json.dumps(new_config, indent=2))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm a bit concerned about the API here, due to the possibility of accidentally zeroing out the config.

I wonder if it would make more sense to take a key and a value, then add or update that in the file, instead of replacing the file itself? That way there is no way to delete things from the config, just add them as it needs.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree. I like the idea of mirroring the metadata API here.

singer-python/singer/metadata.py

Lines 10 to 23 in 2cdb93f

def delete(compiled_metadata, breadcrumb, k):
del compiled_metadata[breadcrumb][k]
def write(compiled_metadata, breadcrumb, k, val):
if val is None:
raise Exception()
if breadcrumb in compiled_metadata:
compiled_metadata.get(breadcrumb).update({k: val})
else:
compiled_metadata[breadcrumb] = {k: val}
return compiled_metadata
def get(compiled_metadata, breadcrumb, k):
return compiled_metadata.get(breadcrumb, {}).get(k)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

While we're talking about API design, one possible extension to the metadata approach, would be to have a 3-arity write that assumes that breadcrumb = tuple(). Could be nicer for simple uses of the API, at the cost of a bit more complexity in the code (type checking the second arg, etc).

I'm not familiar with the Pythonic way to do multi-arity, maybe just defining the function twice is enough, or it might have to be *args with explicit run-time checking code.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think pythonic here would be a default value for breadcrumb breadcrumb=[].


def backoff(exceptions, giveup):
"""Decorates a function to retry up to 5 times using an exponential backoff
Expand Down
21 changes: 20 additions & 1 deletion tests/test_utils.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
import pytz
import logging
import singer.utils as u

import json

class TestFormat(unittest.TestCase):
def test_small_years(self):
Expand Down Expand Up @@ -33,3 +33,22 @@ def test_exception_fn(self):
def foo():
raise RuntimeError("foo")
self.assertRaises(RuntimeError, foo)

class TestUpdateConfig(unittest.TestCase):
def test_file_changed(self):
original_config = {'key_a': 'val_a'}

with open('config.json', 'w') as file:
file.write(json.dumps(original_config))

changed_config = {'key_b': 'val_b'}

u.update_config_file('config.json', changed_config)

read_config = u.load_json('config.json')

self.assertEqual(changed_config.get('key_b'), read_config.get('key_b'))

def tearDown(self):
import os
os.remove('config.json')