Configure project to use pyproject.toml and consistent CI and test tooling

[DonnchaC]

Restructure the project to make development tooling more consistent and approachable in future.

• Consolidate all tool settings in a pyproject.toml file
• Move to a src project layout
• Remove black - use ruff's formatter
• Remove flake8 - use ruff's linter but enable the same ruleset as the previous flake8 config
• Try to use Make targets as much as possible from CI
• Remove unused dev/* scripts

Makefile changes

• make install installs as an editable package which can be used for development.
• make test-requirements installs the test requirements
• make check runs all linting with ruff and type checks with mypy
• make autofix autofixes any fixable linting/formatting errors with ruff.
• make test runs all tests

Note: latest pip and build packages are required.

[github-actions]

Coverage Report

File	Stmts	Miss	Cover	Missing
src/mvt/android
cli.py	157	54	66%	49, 57, 92–124, 164–188, 228–229, 236, 288–289, 296, 354–355, 362, 389–396, 408–409
cmd_check_adb.py	10	3	70%	26–37
cmd_check_androidqf.py	34	5	85%	57–60, 64
cmd_check_backup.py	63	18	71%	60, 69–70, 78–79, 85–88, 95–108, 112
cmd_check_bugreport.py	39	7	82%	51, 54–57, 70, 76
cmd_download_apks.py	86	68	21%	38–42, 51–53, 63–97, 103–110, 114–170, 173–175, 178–182
utils.py	16	5	69%	11–20
src/mvt/android/artifacts
dumpsys_appops.py	90	6	93%	23, 89–90, 138–140, 146
dumpsys_battery_daily.py	42	2	95%	42–43
dumpsys_battery_history.py	48	7	85%	47–55, 68
dumpsys_dbinfo.py	42	5	88%	60–65
dumpsys_package_activities.py	40	4	90%	12, 64, 70–71
dumpsys_packages.py	121	5	96%	18–24, 161, 168
dumpsys_receivers.py	57	6	89%	24, 29, 34, 97, 103–104
getprop.py	31	3	90%	40, 48, 51
processes.py	34	9	74%	20, 24–25, 31, 55, 59, 63–65
src/mvt/android/modules/adb
base.py	147	111	24%	51–61, 66–73, 77–138, 142, 146–148, 157, 166–167, 171–172, 185, 198–200, 218–224, 234–269, 282–306, 309–352, 356
chrome_history.py	36	23	36%	30–38, 41, 50–55, 63–97, 100–109
dumpsys_accessibility.py	15	8	47%	26, 36–47
dumpsys_activities.py	14	7	50%	28–37, 40–45
dumpsys_appops.py	14	6	57%	28, 38–44
dumpsys_battery_daily.py	13	6	54%	26, 36–42
dumpsys_battery_history.py	13	6	54%	26, 36–42
dumpsys_dbinfo.py	14	6	57%	28, 38–44
dumpsys_full.py	16	9	44%	25, 35–45
dumpsys_receivers.py	14	7	50%	26–35, 38–44
files.py	72	57	21%	37–45, 48–56, 59–70, 73–88, 93–121, 124–155
getprop.py	14	7	50%	26–35, 38–43
logcat.py	21	14	33%	25, 35–57
packages.py	168	148	12%	39–47, 50–72, 75–109, 113–162, 166–176, 181–212, 215–312
processes.py	13	6	54%	26, 36–42
root_binaries.py	25	18	28%	24, 34–36, 39–70
selinux_status.py	17	10	41%	26–35, 38–48
settings.py	25	18	28%	26–35, 38–58
sms.py	76	57	25%	56–65, 68–69, 77–89, 97–124, 134–148, 151–176
whatsapp.py	52	38	27%	31, 41–42, 50–59, 67–102, 105–112
src/mvt/android/modules/androidqf
dumpsys_packages.py	24	3	88%	52–53, 56
packages.py	40	3	92%	44–50
settings.py	24	2	92%	51–52
sms.py	49	13	73%	48, 51, 56–57, 73–78, 81, 85–91, 96–97
src/mvt/android/modules/backup
base.py	33	2	94%	64–65
helpers.py	22	1	95%	26
sms.py	32	3	91%	40, 44, 47
src/mvt/android/modules/bugreport
accessibility.py	17	3	82%	38–42, 51
activities.py	16	2	88%	42–46
appops.py	15	2	87%	38–42
base.py	52	17	67%	48–49, 54–55, 62–67, 71, 86–95
battery_daily.py	15	2	87%	38–42
battery_history.py	15	2	87%	38–42
dbinfo.py	16	2	88%	40–44
getprop.py	26	7	73%	40–44, 51–52, 57–60
packages.py	24	3	88%	39–43, 56
receivers.py	16	2	88%	40–44
src/mvt/android/parsers
backup.py	107	9	92%	62, 102–103, 109, 129, 132, 175, 190–191
src/mvt/common
artifact.py	10	2	80%	22, 28
cmd_check_iocs.py	38	29	24%	26–36, 39–80
command.py	132	47	64%	67–73, 79–97, 103–109, 118–146, 152–156, 159–161, 167, 201, 218, 222–223
indicators.py	258	61	76%	37–39, 53, 116, 120, 124, 128, 153–158, 227, 269–272, 289, 303–321, 325–340, 348, 370, 401, 420, 435–441, 454–462, 473, 497, 522, 531–538, 553, 578–591, 602, 649
logo.py	33	28	15%	14–60, 64–71
module.py	119	36	70%	71–75, 80–84, 99–119, 160, 169, 174, 184, 203–204, 220–221, 237–246
options.py	13	3	77%	27–33
updates.py	143	118	17%	26–33, 38–51, 56–64, 67–69, 72–80, 83–85, 88–100, 103–118, 121–159, 166–195, 198–208, 211–241
url.py	25	6	76%	327, 370, 376–380
utils.py	107	37	65%	48–50, 63, 95–96, 110, 124–125, 164–181, 196–197, 210–211, 218–227, 242, 252, 260
virustotal.py	23	13	43%	25–52
src/mvt/ios
cli.py	139	62	55%	52, 60, 97–141, 167–191, 234–235, 272–293, 320–327, 339–340
cmd_check_fs.py	13	4	69%	28–40, 43
decrypt.py	114	92	19%	33–36, 39, 48–56, 61–64, 73–123, 131–181, 192–221, 227–231, 244–255
versions.py	32	3	91%	21, 30, 48
src/mvt/ios/modules
base.py	88	22	75%	60, 67–92, 110, 118, 125, 133–134, 153–156, 191–192
net_base.py	119	46	61%	74–75, 156–211, 226–237, 249, 294–295, 303–304, 308
src/mvt/ios/modules/backup
backup_info.py	29	2	93%	43, 79
configuration_profiles.py	67	46	31%	43–48, 58, 61–88, 103–178
manifest.py	81	8	90%	59, 66, 110–117, 122, 168–169
profile_events.py	51	32	37%	44, 55, 58–67, 71–97, 103–110, 113
src/mvt/ios/modules/fs
analytics.py	67	52	22%	34, 44, 52–83, 86–144, 147–150, 153–159
analytics_ios_versions.py	36	26	28%	30, 40, 48–86
cache_files.py	46	35	24%	24, 34–45, 48–62, 65–80, 92–99
filesystem.py	42	8	81%	52, 56–57, 61, 77–78, 89–90
net_netusage.py	18	10	44%	34, 44–57
safari_favicon.py	37	26	30%	31, 41, 50–60, 63–115, 118–124
shutdownlog.py	65	54	17%	30, 40, 49–69, 72–127, 130–133
version_history.py	20	9	55%	32, 42, 50–65
webkit_base.py	22	16	27%	17–24, 27–38
webkit_indexeddb.py	13	4	69%	34, 44, 53–54
webkit_localstorage.py	12	4	67%	32, 42, 51–52
webkit_safariviewservice.py	10	3	70%	32, 42–43
src/mvt/ios/modules/mixed
applications.py	74	46	38%	44–51, 55–93, 99–107, 113–118, 128–140, 146, 148–150
calendar.py	49	2	96%	75–78
calls.py	22	10	55%	41, 53–82
chrome_favicon.py	35	22	37%	42, 50–60, 66–104
chrome_history.py	30	17	43%	44, 54–61, 67–102
contacts.py	28	17	39%	45–75
firefox_favicon.py	32	19	41%	43, 52–62, 68–106
firefox_history.py	30	17	43%	47, 55–62, 68–101
global_preferences.py	25	1	96%	45
idstatuscache.py	56	39	30%	46, 55–72, 75–105, 110–120
interactionc.py	55	40	27%	251–275, 281–320
locationd.py	83	65	22%	58–70, 73–133, 136–155, 160–172
osanalytics_addaily.py	31	17	45%	45, 56–63, 70–98
safari_browserstate.py	73	27	63%	68, 71–75, 96–98, 112–132, 167, 173–180
safari_history.py	70	44	37%	48, 59–98, 107, 110–113, 116–151, 163–171
shortcuts.py	69	52	25%	47–55, 71–78, 84–152
sms.py	69	13	81%	74, 86, 110–127, 140, 154
sms_attachments.py	42	24	43%	44, 57–74, 97–128
tcc.py	80	28	65%	68, 85, 108–125, 140–143, 163–205
webkit_resource_load_statistics.py	52	10	81%	67–68, 93–94, 128–135
webkit_session_resource_log.py	75	52	31%	56–66, 70, 73–113, 119–149, 156–173
whatsapp.py	51	38	25%	43–48, 56–63, 69–135
TOTAL	5732	2391	58%

Tests	Skipped	Failures	Errors	Time
97	0 💤	0 ❌	0 🔥	7.725s ⏱️

[DonnchaC]

The code coverage is broken in this PR. Pytest is run using the installed packaged and coverage is shown for the installed package rather than the source directory. I'm making a note here as I don't have time to dig into this currently. @roaree Have you seen this before?

/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/mvt/android

[DonnchaC]

The last two commits appear to have fixed the code coverage collection and reporting.