0.8.0

FEATURES

1. Added <extend> tag to support loading template content from files or from TTP_templates repository and use them within other templates
2. Added filters to <extend> tag to support groups, inputs, vars, lookups, outputs tag filtering
3. Added support for nested <extend> tag handling with child groups
4. Added Environment variable TTPCACHEFOLDER for lazy import system to support caching TTP dictionary at user defined path
5. Added new match variable function - raise, to raise RuntimeError error if it is called, useful for testing text data - if pattern matched, error raised
6. Added support for group function itemize to support dynamic path
7. Added group items2dict function to combine match values in a dictionary
8. Added quick_parse utility function to allow parsing data in one line of code

CHANGES

1. Modified group macro function to check result of each macro execution at each step to exit immediately if receive False.
2. Improved handling of empty characters between words - prior to this change all spaces were replaced with "\ +" regex, now tabs also included in that regex - "[ \t]+" allowing to have tabs and space mixed between words within line.
3. Added documentation for doc <tag>

BUGS

1. Encountered issue with results processing when child group being parsed even when parent group does not have start re matched, improved parsing logic to account for edge cases were it can be useful, for all other cases child groups no longer parsed - that fixed the issue and should improve processing speed.
2. Bug with file loader not joining OS path properly if path itself is a bit malformed, updated to use os.path.join to account for it.
3. Bug - when group macro returns None, TTP assigns None to results, this was fixed to make sure that original data processed further instead of replacing it with None
4. If last start regex span is bigger than last end regex span TTP skipping some text data to parse, fixed it and made sure that end span set to the end of document in this case

0.7.2

CHANGES

1. ttp.lazy_import_functions changes log level to warning if fails to save ttp_dict.pickle to the file system #54
2. Fixed support for group set function to retrieve variable values from _ttp_["global_vars"]

0.7.1

FEATURE

1. Added support for table formatters (excel, tabulate) to support "strict" keyword for traverse function to not fail if no path present in result

CHANGES

1. ttp.lazy_import_functions added explicit check for ttp dictionary cache existence instead of relying on try/except
2. ttp.lazy_import_functions added encoding keyword to open() function while loading TTP files

BUGS

1. Added empty tag handling for Python loader to not throw an error.

0.7.0

FEATURES

1. Added support for file returner to format filename using ttp["vars"] and ttp["global_vars"] dictionaries, as a result filenames can now be formatted on a per results basis #41
2. Added support to gethostname to extract hostname from Nokia SROS prompt #43
3. Made regexes to account for \r\n ending in addition to \n to detect a new line #44
4. Added support for excel formatter to update existing spreadsheets using "update" parameter
5. Made ## comments to support indentation without generating log message
6. Added outputters "condition" attribute to be able to run outputters conditionally based on variable in vars passed to TTP object

BUGS

1. Improved behavior of results selection and fixed bug with same group name/path results selection by adding grp_id in self.record to track previous group identification #37
2. Added support for \r\n line ending in addition to \n #44
3. Fixed joinmatches to properly handle sanitized match variable names
4. Fixed BUG with match result selection for multiple start re matches

CHANGES

1. TTP Lazy importing system modified to use pickle module to cache ttp dictionary and to scan subfolders' files one by one,
   that is to avoid "OSError: [Errno 24] Too many open files" problem when TTP scans through folders to find all function;
   additionally, added caching functions to speed up initialization process roughly by 60ms.
2. Output traverse function added "strict=True" boolean parameter, if True - traverse raises KeyError if no path item found
   in results, if False, will return empty dictionary if no item found in results
3. Improved Syslog returner - wrapped sending logs to try-except blocks with logging of errors on failure
4. Improved results path selection to use group.group_id instead of path and index attributes
5. Generalized anonymous group name, path and group_id formation to account for nested/child anonymous groups
6. Modified yangson validate behavior to return error if no yangson installed, added option to just return results without metadata if validation was successful

0.6.0

BUGS fixes

1. Fixed headers to not match a lot of garbage, even when line is much shorter than headers string
2. Fixed bug with 'add' adding results for previous group in next group due to false matches #37

FEATURES

1. Added support to match Huawei style MAC addresses in MAC pattern
2. Group default attribute can reference template variable, template variable can be a dictionary or a string
3. Added support for parsing results YANG modules validation using yangson library
4. TTP Templates can be referenced using "ttp://" syntaxis

0.5.0

NEW FEATURES

1. Added TTP object add_function method to add reference to custom functions for groups, variables, outputs, macro etc.
2. Added support for output deepdiff to reference template variable content as a data to compare with.
3. Inputs can accept structured data, dictionary or list or etc. This data need to be pre-processed within templates using input macro functions
4. Added support to sanitize match variables names so it can contain any symbols or start with digit
5. Added inter-log interval to syslog returner, set default to 1ms

CHANGES

1. form_result template function logic simplified, to produce consistent results structure across normal and anonymous groups
2. made name="/" not to produce empty key
3. if output formatter is jinja2, loader for Outputter automatically set to text
4. passing self.attributes to returners and formatters in outputs to unpack them with **kwargs, useful for custom functions
5. Removed support for adding file objects to inputs, instead string with OS path to file should be used
6. Added "vars" keyword to _ttp_dictionary to store in run variables in addition to results_object.vars reference, updated built-in functions to use ttp["vars"]

BUGS fixes

1. Make file returner to convert non string data to text and save it
2. Fixed bug with headers indicator when last column value is shorter then headers or last column is empty
3. Fixed csv loader include statement, now lookups and variables can be loaded from csv files
4. Fixed/Added support for joinmatches to have "ignore" indicator in line

0.4.0

NEW FEATURES

1. resub and resuball function can reference regex patterns from template vars
2. exact_space match variable indicator
3. Added group record function
4. Added group set function
5. Added support to groups for absolute path
6. Added support for filter list to ttp.add_template method, to allow filtering of child templates that need to be loaded from text file that contains several templates or from template that has child tags defined, filter contains names of child templates to be loaded
7. Added filter support for parent templates names, now they can be filtered as well - if no name of template specified in filter, that template content will be ignored and no results produced
8. Added gpvlookup to lookup match results against glob patterns
9. Added support for geoip2 db lookups - geoip_lookup match variable function
10. Added support for lookup tag databse loaders
11. Added match variable geoip_lookup function
12. Added support for group contains_val and exclude_val to reference variable;
13. Added support for match variable equal, non_equal and cidr_match to reference variable;
14. Added cli tool -v/--vars keyword to supply json string-dictionary of variables to add to ttp parser object
15. Added colouring to terminal returner using colorama
16. Added group validate function that used Cerberus to check group result
17. Modified outputter validate function to have features similar to group validate function
18. Match variable function equal/notequal/exclude/append/prepend/contains/replaceall can reference Template variables
19. Added TTP variable get_timestamp_iso function
20. Added support for flat_list result structure to flatten overall results
21. Added group 'expand' function to expand dot separated variables names to nested dictionary
22. Added 'clear_result' method to ttp parser object
23. Added outputter support for N2G formatter to produce xml diagrams
24. Added group lookup function
25. added support for "doc" tag - TTP will not complain about it and this tag can be used to put template description in it
26. Added support for group functions to reference list of functions in template variable using chain or functions attribute
27. Added default name to templates - "root_template", added template_name attribute to add_input, set_input, clear_input methods; get_inout_load method now returns dictionary of templates and input names.
28. Added syslog returner to outputs using logging syslog handler to return results to syslog server over udp
29. Added support for netmiko as an input source function
30. Added support for "headers" indicator to dynamically construct regexes line to match tabulated data using exact number of spaces
31. Added support for Nornir as an input source
32. Added reference to variables for match variable conditional functions to introduce dynamic filtering capabilities

CHANGES

1. Modified variable and pattern line escaping behavior, to fix bug with several match variables with digits in name e.g. d1_1, d1_2
2. Modified behavior of DEFAULTS handling in results object to add them at the very end and make them the least preferred in comparison to results values.
3. Modified per_template results method handling to flatten results structure by converting overall result to dictionary for named groups and using single list for anonymous group
4. Results object processgrp now adds default to results instead of save_results method
5. Updated template object form results to handle anonymous and named groups so that results for both groups included in output
6. Deleted get_input_commands_list and get_input_commands_dict parser objects, as they no longer deemed to be useful
7. Data on add associated with templates straight away instead of being saved in ttp.__data list. As a result, templates must be added before data will be added, if data added before template, that data will not be processed, as there are not templates/inputs to associate it with.
8. Outputters now load tag data and merge it with attributes if its a dictionary.

TESTS

1. test1-48 for variable with digits in name and for exact indicator
2. Test 1-50-x for group record and set functions
3. Test 1-51 to test group absolute path handling
4. Test 1-52 to test let vs set behavior when defaults must be overridden by match values
5. Test 1-53 for exact_space indicator
6. Test 29-1 and 29-2 gpvlookup tests
7. Test 1-50 for group macro attribute with comma separated macro names
8. Added tests 1-56, 1-57 to test match cidr_match and group contains_val functions referencing to variable
9. Test 1-58 to test group validate function
10. Results forming for template with anonymous and named group to test change-5
11. Started moving tests to pytest - 31 tests added so far

DOCS

1. Added HOWTO for parsing text data structured in tables
2. Added docs for group record and set functions
3. Added docs for group absolute path
4. Added docs to 'ignore' indicator saying that regex can come from re patterns or from template variables
5. Added docs to cli tool about new keywords options -ot and -s
6. Added docs about exact_space indicator
7. Added docs for gpvlookup match variable function
8. Added docs for geoip_lookup function
9. Updated docs for group contains_val and exclude_val, for match variable equal, non_equal and cidr_match to mention that they can reference variable
10. Added docs for group 'expand' function
11. Updated docstring for 'flat_list' structure
12. Flattened docs structure, created separate section for results structure
13. Wrote docs for headers indicator
14. Updated summary notes to installation page for other packages to be installed for certain features to work
15. Documented N2G output formatter
16. Changed output formatters to include all supported arguments within each of them, rather then having separate section for formatter's attributes.
17. Added new cli tool -v/--vars argument in the docs
18. Added docs about coloring terminal returner output using colorama
19. Modified inputs docs - merged attributes and parameters together
20. Added docs for syslog returner
21. Added docs for netmiko source
22. Added docs about Norninr source
23. Added docs about group validate function

BUGS

1. fixed bug with get_attributes function when empty match variable item give e.g. {{ var | | func }}
2. Fixed bug for variables having digit in name, that digit was substituted with \d+ pattern, resulting in match varibales have the same name, which is prohibited for regex groups
3. Fixed bug for py2.7 struct load for exec() function when text contains "True" of "False" by skipping the error, as exec chokes on it while eval used to load it. Not an issue in py3 though.
4. Fixed problem with set overriding let values due to default values override results dictionary when group method="table" and we have dynamic path in use to form dictionary, override happen due to the fact that each match was written to results straight away together with defaults values and if we have let function and unconditional set (set to defaults) values for same variable, let value was overridden by defaults from set.
5. Fixed handling of absolute path together with anonymous parent group
6. Fixed template form_results function to handle the case when anonymous group produces dictionary results in per_template mode
7. Load struct function was fixed to correctly handle the case when template vars tag has include attribute but has not text defined in XML element
8. Fixed traceback display issue with get_attributes function, log string had wrong formatting
9. Fixed truncate function and added test to test it
   10.Fixed issue with group macro having several macro names separated by comma, added test to test it
10. TTP duplicate logs when used as a module - changed log config function
11. Fixed bug with anonymous template introduced by adding r'' raw strings
12. Fixed csv formatter results handling when they contains integers
13. Fixed this bug:
interface {{ interface }} description {{ description | ORPHRASE }} was returning no results for interfaces without descriptions 15. Fixed CSV formatter to handle "," properly in matches, by vrapping items in '"' or "'" characters

v0.3.0

New features

1. -tn CLI tool keyword, to provide name of template within templates file to use for parsing
2. New template variable "get_time", "get_date", "get_timestamp" and "get_timestamp_ms" time functions
3. Implemented "mac_eui" match variable function to convert mac into EUI like format
4. Macro functions support to inputs for custom data preprocessing
5. Added capability to guess function names on error using difflib.get_close_matches function, to make errors output more user friendly
6. Added support for '_' null path group name attribute
7. Added "get_time_ns" function to variable functions, to get time in nanoseconds since epoch
8. Added "count" function to match variables
9. Added ttp get_input_load method to retrieve inputs parameters so that arbitrary data can be passed from input to upper programs
10. Added support for Cerberus validation engine for groups using "cerberus" function
11. Modified args and kwargs extraction behavior, so that simpler syntax can be used, e.g. instead of func_name="'arg1', kwarg1='kval1'" it can be func_name="arg1, kwarg1=kval1"
12. Added match variable and group 'void' functions to unconditionally invalidate results
13. Added to_float and to_unicode match variable function
14. Implemented str_to_unicode group function to convert str string in Unicode string for python2
15. Added groups 'equal' function to check certain key for value
16. Added deepdiff output function to find difference between input results
17. Added support for lookup to search for values in template and group results
18. Added group to_int function to convert match result to integers
19. Added support for deepdiff to reference template name to compare with
20. Added "contains_val" and "exclude_val" group function to filter based on certain values
21. Added -ot CLI tool keyword to reference template name to output results for

Changes

1. TTP re patterns moved to patterns dir for the ease of adding new patterns
2. Templates folder removed from TTP repo, instead of it -tn keyword added to CLI tool
3. removed input "preference" attribute and simplified input handling logic
4. Made changes to input class and template class methods to make input handling more straightforward
5. Changed groups default to not modify provided default and always be equal to provided value
6. Added ttp dictionary to globals of all loaded macro, so that macro now can cross reference ttp built in functions
7. Modified logic for match selection - normal versus start versus line re - to make line always less preferred
8. Simplified logic for Group Lock handling
9. Updated groups/input/output functions extraction logic to follow same approach
10. Updated logic for results selection when same match result produced by several regexes i.e. start vs normal vs line re preference
11. Added support for several patterns in match variable contains function to match variable
12. Added support for inputs to use relative OS path in url parameter, relative to directory where TTP invoked

New Tests

1. test15 - group with no matches but with start re defaults
2. test16 - template test for line indicator
3. test1-33 - tests filtering of groups based on start re contains
4. test18 - to test macro that used ttp function
5. test1-34 - test for mac_eui function
6. test1-35 - input commands function
7. test1-36 - one more input commands functions test
8. test20 - null '_' path handling
9. test21 - tests count match variable function and output macro function
10. test1-37 to test group Cerberus validation function
11. test1-38 to test match variable and group 'void' functions
12. test1-40 str_to_unicode group function
13. test1-41 to test group 'equal' function
14. test22 - test deepdiff function
15. test23 - test lookup across template results
16. test24 - test lookup across group results
17. Test 1-42-1 and 1-42-2 for group to_int function
18. test 1-45 and 1-46 to test null path handling when it is a group right after start re
19. test28 - test child groups with start re defaults only

Docs

1. Added docs for group with no matches but with start re defaults
2. Update Quick Start section to be more concise
3. Updated CLI tool section with -tn keyword usage
4. Drafted docs for template variable "get_time", "get_date", "get_timestamp" and "get_timestamp_ms" time functions
5. Updated performance section with more details
6. Split input section in several sections
7. Added docs for match variable mac_eui function
8. Add docs for input functions
9. Added match variable count function docs
10. Added docs for group 'cerberus' function
11. Added docs for match variable and group 'void' function
12. Added docs for Null '_' path handling
13. Added docs for to_float and to_unicode match variable functions
14. Documented str_to_unicode group function
15. Added group 'equal' function documentation
16. Updated docs for match variable lookup function describing cross group/template lookups
17. Added docs for group to_int function
18. Added docs for input url relative os path and for variables relative include statements

Bug fixes

1. Fixed load_files utils function to check if path is of the type "Unicode" string in case if py2 running the script
2. Fixed bug when adding data to input when using ttp as a module, it will ad input groups again and again, fixed by passing overall input group_indexes through list(set()) functions
3. Fixed bug with extracting line indicator
4. Fixed but with input commands function by adding \n to match results, without it TTP will fail to match last line
5. Fixed handling of end re together with group "table" method
6. Fixed bug with line and end indicators when match PATH is not the same as current group PATH, added test19 to test this behavior
7. Fixed load_py_exec to support loading recursive functions
8. Modified start vs normal vs line re selection logic to properly select results matched by several start regexes
9. Modified behaviuor for TTP to save first match in group, as opposed to saving last match when same variable produced multiple matches
10. Fixed group dynamic path forming when match was converted to integer by converting replacement patter to str type
11. Fixed null path handling issue when it is a group right after start re

v0.2.0

New features

1 Inputs order now preserved by the use of OrderedDict
2 Added support for global variables for record/set operations, now variables recorded during one input processing can be used by set function during other inputs processing
3 Moved template variable functions in separate sub dictionary to optimize templates variables functions execution
4 In addition to ignoring trailing spaces, trailing tabs are ignore as well
5 Added support for input functions handling prior to starting data parsing
6 Added ttp input functions - test, functions and commands
7 Added template get_input_commands_list method to get a set of commands that input expects
7.1 Added template get_input_commands_list method to get a mapping of commands to input name
8 Added group "del" function to delete certain keys out of group results
9 Added uptimeparse match variable function to convert strings like "4 weeks, 3days, 5 hours, 10 seconds" into seconds
10 Added group itemize function to produce list of items out of group match results
11 Added group sformat function to format strings based on group match results or template vars or global vars
12 Added support for default only groups - if group start re has default value and no results matches by that group, group default values will be added to results, if any

New Tests

1 tes1-1 now tests trailing tabs as well
2 test11 to test global variable record/set functions
3 test12 - input "commands" functions test
4 test1-31 - Added group "del" delete function
5 test 1-32-1 and 1-32-2 for uptimeparse function
6 test14 - group itemize function test
7 test13 - group sformat function tests

Docs updates

1 Updated "Writing Templates" notes on leading/trailing spaces and tabs handling
2 added drafts for input functions
3 added draft for group 'del' function
4 added docs for uptimeparse function
5 added draft for itemize function
6 added group sformat function docs draft

Bug fixes