Generate JSON output #194

devdanzin · 2023-07-09T03:16:07Z

This proof of concept adds a --json option to diff, index, rank and report, allowing to generate output in JSON format.

Example output (excerpt from wily report -n 11 --json src\wily\commands\report.py):

[
 {
    "Revision": "f1e8225",
    "Author": "Anthony Shaw",
    "Date": "2023-03-12",
    "Cyclomatic Complexity": "24 (0)",
    "Unique Operands": "8 (0)",
    "Maintainability Index": "56.8982 (0.0)",
    "Lines of Code": "206 (0)"
  },
  {
    "Revision": "e9921dd",
    "Author": "Christian Clauss",
    "Date": "2023-03-11",
    "Cyclomatic Complexity": "24 (0)",
    "Unique Operands": "8 (0)",
    "Maintainability Index": "56.8982 (0)",
    "Lines of Code": "206 (0)"
  }
]

Pasting this here I just realized it might be a good idea to add file name to each JSON entry. Oh well, back to the drawing board.

Still needs tests and docstring updates.

Fixes part of #92.

…ate output in JSON format.

devdanzin · 2023-07-09T12:41:31Z

The CI failures could be fixed by extracting a function from the kinda duplicated code that prints the table or JSON, something like below:

def print_result(as_json, console_format, data, headers, path):
    if as_json:
        json_data = [{headers[x]: d[x] for x in range(len(headers))} for d in data]
        for entry in json_data:
            entry["Filename"] = path
        print(json.dumps(json_data, indent=2))
    else:
        print(
            tabulate.tabulate(
                headers=headers, tabular_data=data, tablefmt=console_format
            )
        )

However, it's not clear to me where such a function should live. Maybe in a new module in wily.helper?

codecov-commenter · 2023-07-09T13:08:00Z

Codecov Report

Attention: 2 lines in your changes are missing coverage. Please review.

Comparison is base (2590691) 95.86% compared to head (74d2813) 95.83%.

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #194      +/-   ##
==========================================
- Coverage   95.86%   95.83%   -0.03%     
==========================================
  Files          25       26       +1     
  Lines        1403     1442      +39     
  Branches      296      315      +19     
==========================================
+ Hits         1345     1382      +37     
- Misses         33       34       +1     
- Partials       25       26       +1

Files	Coverage Δ
src/wily/__main__.py	`94.92% <100.00%> (+0.10%)`	⬆️
src/wily/commands/index.py	`100.00% <100.00%> (ø)`
src/wily/commands/rank.py	`96.82% <100.00%> (+0.15%)`	⬆️
src/wily/commands/report.py	`97.45% <100.00%> (+0.20%)`	⬆️
src/wily/helper/output.py	`100.00% <100.00%> (ø)`
src/wily/commands/diff.py	`88.77% <88.88%> (-0.76%)`	⬇️

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

… code.

sorasful · 2023-10-22T21:22:03Z

This feature could be really useful, do you need help on something ?

devdanzin · 2023-10-22T23:55:56Z

Yes, thank you, I'd love to get some validation on the chosen output format/entries. I'll update this PR this week and show what the output currently looks like.

…ata in commands.

devdanzin · 2023-10-29T11:19:00Z

I've updated the PR and added some tests. I'm open to all suggestions: if you see ways to improve the output or the code, please let me know.

Here's some sample output for report:

[
  {
    "Revision": "d0ad384",
    "Message": "Add some more typing (#221)\n\n* Update typing info ",
    "Author": "devdanzin",
    "Date": "2023-09-22",
    "Cyclomatic Complexity": "23 (0)",
    "Unique Operands": "6 (0)",
    "Maintainability Index": "62.7366 (-0.30257)",
    "Lines of Code": "176 (1)",
    "Filename": "src\\wily\\commands\\graph.py"
  },
  {
    "Revision": "bdc825f",
    "Message": "Standalone plotly.min.js for graph (fix #189) (#20",
    "Author": "devdanzin",
    "Date": "2023-09-22",
    "Cyclomatic Complexity": "23 (0)",
    "Unique Operands": "6 (0)",
    "Maintainability Index": "63.0391 (0.624665)",
    "Lines of Code": "175 (3)",
    "Filename": "src\\wily\\commands\\graph.py"
  },
  {
    "Revision": "6f5cd5d",
    "Message": "Restore displaying graph markers. (#215)\n\n",
    "Author": "devdanzin",
    "Date": "2023-09-04",
    "Cyclomatic Complexity": "23 (0)",
    "Unique Operands": "6 (0)",
    "Maintainability Index": "62.4145 (0.0997117)",
    "Lines of Code": "172 (0)",
    "Filename": "src\\wily\\commands\\graph.py"
  },
  {
    "Revision": "07fc855",
    "Message": "Allow passing multiple file names to graph, to get",
    "Author": "devdanzin",
    "Date": "2023-08-25",
    "Cyclomatic Complexity": "23 (3)",
    "Unique Operands": "6 (1)",
    "Maintainability Index": "62.3148 (-2.57913)",
    "Lines of Code": "172 (15)",
    "Filename": "src\\wily\\commands\\graph.py"
  }
]

Here's sample output for report, diff, rank and index (only the JSON goes to stdout):

Output for wily report --json

wily report src\wily\commands\graph.py -c -m --json
Using default metrics ['raw.loc', 'maintainability.mi', 'halstead.h1', 'cyclomatic.complexity']
-----------History for ['cyclomatic.complexity', 'halstead.h1', 'maintainability.mi', 'raw.loc']------------
[
  {
    "Revision": "d0ad384",
    "Message": "Add some more typing (#221)\n\n* Update typing info ",
    "Author": "devdanzin",
    "Date": "2023-09-22",
    "Cyclomatic Complexity": "23 (0)",
    "Unique Operands": "6 (0)",
    "Maintainability Index": "62.7366 (-0.30257)",
    "Lines of Code": "176 (1)",
    "Filename": "src\\wily\\commands\\graph.py"
  },
  {
    "Revision": "bdc825f",
    "Message": "Standalone plotly.min.js for graph (fix #189) (#20",
    "Author": "devdanzin",
    "Date": "2023-09-22",
    "Cyclomatic Complexity": "23 (0)",
    "Unique Operands": "6 (0)",
    "Maintainability Index": "63.0391 (0.624665)",
    "Lines of Code": "175 (3)",
    "Filename": "src\\wily\\commands\\graph.py"
  },
  {
    "Revision": "6f5cd5d",
    "Message": "Restore displaying graph markers. (#215)\n\n",
    "Author": "devdanzin",
    "Date": "2023-09-04",
    "Cyclomatic Complexity": "23 (0)",
    "Unique Operands": "6 (0)",
    "Maintainability Index": "62.4145 (0.0997117)",
    "Lines of Code": "172 (0)",
    "Filename": "src\\wily\\commands\\graph.py"
  },
  {
    "Revision": "07fc855",
    "Message": "Allow passing multiple file names to graph, to get",
    "Author": "devdanzin",
    "Date": "2023-08-25",
    "Cyclomatic Complexity": "23 (3)",
    "Unique Operands": "6 (1)",
    "Maintainability Index": "62.3148 (-2.57913)",
    "Lines of Code": "172 (15)",
    "Filename": "src\\wily\\commands\\graph.py"
  }
]

Output of wily diff --json

wily diff src\wily\commands\report.py -r bdc825f --json
Using default metrics ['raw.loc', 'maintainability.mi', 'halstead.h1', 'cyclomatic.complexity']
Comparing current with bdc825f by devdanzin on 2023-09-22.
[
  {
    "File": "src\\wily\\commands\\report.py",
    "Lines of Code": "216 -> 222",
    "Maintainability Index": "52.2582 -> 51.1045",
    "Unique Operands": "10 -> 10",
    "Cyclomatic Complexity": "27 -> 29"
  },
  {
    "File": "src\\wily\\commands\\report.py:report",
    "Lines of Code": "-",
    "Maintainability Index": "-",
    "Unique Operands": "10 -> 10",
    "Cyclomatic Complexity": "27 -> 29"
  },
  {
    "File": "src\\wily\\commands\\report.py:report",
    "Lines of Code": "-",
    "Maintainability Index": "-",
    "Unique Operands": "10 -> 10",
    "Cyclomatic Complexity": "27 -> 29"
  }
]

Output for wily rank --json

wily rank --json
-----------Rank for Maintainability Index for f57006e by devdanzin on 2023-10-28.------------
[
  {
    "File": "test\\unit\\test_report_unit.py",
    "Maintainability Index": 32.32190502991561
  },
  {
    "File": "src\\wily\\__main__.py",
    "Maintainability Index": 34.30472174464306
  },
[...]
  {
    "File": "test",
    "Maintainability Index": 53.8019265092112
  },
  {
    "File": "src\\wily\\commands\\diff.py",
    "Maintainability Index": 54.453194689759016
  },
  {
    "File": "test\\unit\\test_cache.py",
    "Maintainability Index": 56.010024060239985
  },
  {
    "File": "test\\integration\\test_archiver.py",
    "Maintainability Index": 56.92174441952554
  },
  {
    "File": "test\\integration",
    "Maintainability Index": 58.06125906375679
  },
  {
    "File": "test\\unit\\test_operators.py",
    "Maintainability Index": 58.341013499953924
  },
  {
    "File": "src\\wily\\operators\\__init__.py",
    "Maintainability Index": 58.88435086943125
  },
  {
    "File": "test\\integration\\test_all_operators.py",
    "Maintainability Index": 60.037337074043776
  },
  {
    "File": "src\\wily\\commands\\build.py",
    "Maintainability Index": 61.21595278201455
  },
  {
    "File": "test\\unit\\test_list_metrics_unit.py",
    "Maintainability Index": 61.88150602294518
  },
  {
    "File": "src\\wily\\cache.py",
    "Maintainability Index": 62.37760644177526
  },
  {
    "File": "src\\wily\\commands\\graph.py",
    "Maintainability Index": 62.736559310518224
  },
  {
    "File": "src\\wily\\state.py",
    "Maintainability Index": 62.91391505052898
  },
  {
    "File": "src\\wily\\archivers\\git.py",
    "Maintainability Index": 64.88920153537751
  },
  {
    "File": "",
    "Maintainability Index": 66.68375918601492
  },
[...]
    "File": "docs",
    "Maintainability Index": 100.0
  },
  {
    "File": "docs\\source",
    "Maintainability Index": 100.0
  },
  {
    "File": "Total",
    "Maintainability Index": 68.52496920011775
  }
]

Output of wily index --json

wily index -m --json --------Configuration--------- Path: ~\PycharmProjects\wily Archiver: git Operators: {'cyclomatic', 'raw', 'maintainability', 'halstead'}

-----------History------------
[
{
"Revision": "f57006e",
"Author": "devdanzin",
"Message": "Increase ruff max branches, statements and complex",
"Date": "2023-10-28"
},
[...]
{
"Revision": "be1fb28",
"Author": "devdanzin",
"Message": "Add as_json to command docstrings, fix docstring p",
"Date": "2023-10-27"
},
{
"Revision": "3ef2985",
"Author": "devdanzin",
"Message": "Make as_json in rank() a keyword argument defaulti",
"Date": "2023-10-27"
},
{
"Revision": "8f17212",
"Author": "devdanzin",
"Message": "Merge master.\n",
"Date": "2023-10-27"
},
{
"Revision": "e4fdec9",
"Author": "devdanzin",
"Message": "Change print_result() into print_json(), restore p",
"Date": "2023-10-27"
},
{
"Revision": "2590691",
"Author": "Anthony Shaw",
"Message": "update gitignore\n",
"Date": "2023-10-11"
},
{
"Revision": "4ac2334",
"Author": "Anthony Shaw",
"Message": "Merge branch 'master' of github.com:tonybaloney/wi",
"Date": "2023-10-11"
},
{
"Revision": "ee7014a",
"Author": "Anthony Shaw",
"Message": "Update version data\n",
"Date": "2023-10-11"
},
{
"Revision": "a79b62e",
"Author": "Anthony Shaw",
"Message": "Test support for Python 3.12 (#224)\n\n* Test suppor",
"Date": "2023-10-11"
},
{
"Revision": "4663329",
"Author": "devdanzin",
"Message": "Fix typing of tests and make pyright check them on",
"Date": "2023-09-25"
},
[...]
{
"Revision": "8eb4416",
"Author": "Anthony Shaw",
"Message": "Annotate core functions and remove sphinx style ar",
"Date": "2023-08-03"
}
]

src/wily/helper/output.py

separated columns in report() JSON output.

…rating JSON.

…ain to allow report() to pass.

devdanzin added 2 commits July 9, 2023 00:03

Add --json option for diff, index, rank and report, allowing to gener…

b185a34

…ate output in JSON format.

Add filename to JSON entries in report().

431e332

devdanzin mentioned this pull request Jul 9, 2023

standardized, reusable output formats (e.g json and png) #92

Open

Move printing logic of commands to wily.helper.output.print_result().

1481953

devdanzin added 10 commits July 9, 2023 10:48

Add tests for JSON output support.

6d793af

Remove blank line.

fbc1cd3

black fixes, even though one of them is clearly wrong.

33dbd3f

Move expected data out of test functions so black doesn't create ugly…

b9f6cbd

… code.

Avoid black weirdness.

89075f5

Merge master.

3e7cef3

Merge branch 'tonybaloney:master' into json_output

c46e1df

Merge branch 'tonybaloney:master' into json_output

d3949b0

Merge master.

1dd904c

Merge branch 'tonybaloney:master' into json_output

828dcba

devdanzin added 11 commits October 27, 2023 08:54

Change print_result() into print_json(), restore printing tabulated d…

e4fdec9

…ata in commands.

Merge master.

8f17212

Make as_json in rank() a keyword argument defaulting to False.

3ef2985

Add as_json to command docstrings, fix docstring punctuation.

be1fb28

Raise ruff's allowed number of function arguments.

7ba3429

Format __main__.py.

b95a7e6

Change ruff option '--format' to '--output-format'.

0d94aa8

Don't use terminal colors for JSON output.

7c89f4e

Only strip colors in report() if format isn't HTML, so HTML colors work.

467c87c

Add unit tests for JSON output in index(), rank() and report().

5cfcc75

Increase ruff max branches, statements and complexity for report().

f57006e

devdanzin changed the title ~~WIP: generate JSON output~~ Generate JSON output Oct 29, 2023

devdanzin added enhancement New feature or request help wanted Extra attention is needed labels Oct 29, 2023

sorasful reviewed Oct 29, 2023

View reviewed changes

src/wily/helper/output.py Outdated Show resolved Hide resolved

devdanzin added 7 commits November 2, 2023 10:07

Tighten typing info in print_json().

a2268c4

Rename test_print_result_* to test_print_json_*.

c4cbc86

Make deltas

180436b

separated columns in report() JSON output.

Remove unused branch in report().

64b7448

Add more typing info to print_json().

7a05c06

Output current and new values as separate columns in diff() when gene…

af81cfa

…rating JSON.

Increase ruff's max values for complexity, branches and statements ag…

74d2813

…ain to allow report() to pass.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Generate JSON output #194

Generate JSON output #194

devdanzin commented Jul 9, 2023 •

edited

Loading

devdanzin commented Jul 9, 2023

codecov-commenter commented Jul 9, 2023 •

edited

Loading

sorasful commented Oct 22, 2023

devdanzin commented Oct 22, 2023

devdanzin commented Oct 29, 2023

Generate JSON output #194

Are you sure you want to change the base?

Generate JSON output #194

Conversation

devdanzin commented Jul 9, 2023 • edited Loading

devdanzin commented Jul 9, 2023

codecov-commenter commented Jul 9, 2023 • edited Loading

Codecov Report

sorasful commented Oct 22, 2023

devdanzin commented Oct 22, 2023

devdanzin commented Oct 29, 2023

devdanzin commented Jul 9, 2023 •

edited

Loading

codecov-commenter commented Jul 9, 2023 •

edited

Loading