Skip to content

Commit

Permalink
Merge pull request #847 from hyperledger/aath-summary-results
Browse files Browse the repository at this point in the history
  • Loading branch information
@swcurran
swcurran authored Jul 10, 2024
2 parents 92e6a08 + 3fc9f6b commit 9548eb7
Show file tree
Hide file tree
Showing 4 changed files with 53 additions and 46 deletions.
6 changes: 3 additions & 3 deletions docs/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -24,15 +24,15 @@ Want to add your Aries component to this page? You need to add a runset to the

| Test Agent | Scope | Exceptions | ACA-Py | AFJ | VCX | **All Tests** |
| ----- | ----- | ----- | :----: | :----: | :----: | :----: |
| [ACA-Py](acapy.md)| AIP 1, 2 | None | 96 / 96<br>100% | 48 / 67<br>71% | 36 / 38<br>94% | **180 / 201<br>89%** |
| [ACA-Py](acapy.md)| AIP 1, 2 | None | 96 / 96<br>100% | 48 / 67<br>71% | 20 / 56<br>35% | **164 / 219<br>74%** |
| [AFJ](javascript.md)| AIP 1 | Revocation | 48 / 67<br>71% | 12 / 28<br>42% | 3 / 18<br>16% | **63 / 113<br>55%** |
| [VCX](aries-vcx.md)| AIP 1 | Revocation | 36 / 38<br>94% | 3 / 18<br>16% | 17 / 20<br>85% | **56 / 76<br>73%** |
| [VCX](aries-vcx.md)| AIP 1 | Revocation | 20 / 56<br>35% | 3 / 18<br>16% | 27 / 32<br>84% | **50 / 106<br>47%** |

- Where the row and column are the same Test Agent, the results include only the tests where the Test Agent plays ALL of the roles (ACME, Bob, Faber and Mallory)
- The results in the "All Tests" column include tests involving the "Test Agent" in ANY of the roles.
- Wondering what the results mean? Please read the brief [introduction to Aries interoperability](aries-interop-intro.md) for some background.
- Select the "Test Agent" links to drill down into the tests being run for each Test Agent.


*Results last updated: Wed Jul 3 05:38:40 UTC 2024*
*Results last updated: Wed Jul 10 04:39:26 UTC 2024*

43 changes: 23 additions & 20 deletions docs/acapy.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4,12 +4,12 @@

| Runset | ACME<br>(Issuer) | Bob<br>(Holder) | Faber<br>(Verifier) | Mallory<br>(Holder) | Scope | Results |
| ------ | :--------------: | :-------------: | :----------------: | :-----------------: | ----- | :-----: |
| [acapy-afj](#runset-acapy-afj) | acapy-main<br>0.12.1 | javascript<br>0.5.0-alpha.97 | acapy-main<br>0.12.1 | acapy-main<br>0.12.1 | AIP 1.0 | [**37 / 39<br>94%**](https://allure.vonx.io/api/allure-docker-service/projects/acapy-b-javascript/reports/latest/index.html?redirect=false#behaviors) |
| [acapy-aip10](#runset-acapy-aip10) | acapy-main<br>0.12.1 | acapy-main<br>0.12.1 | acapy-main<br>0.12.1 | acapy-main<br>0.12.1 | AIP 1.0 | [**35 / 35<br>100%**](https://allure.vonx.io/api/allure-docker-service/projects/acapy-aip10/reports/latest/index.html?redirect=false#behaviors) |
| [acapy-aip20](#runset-acapy-aip20) | acapy-main<br>0.12.1 | acapy-main<br>0.12.1 | acapy-main<br>0.12.1 | acapy-main<br>0.12.1 | AIP 2.0 | [**61 / 61<br>100%**](https://allure.vonx.io/api/allure-docker-service/projects/acapy-aip20/reports/latest/index.html?redirect=false#behaviors) |
| [acapy-ariesvcx](#runset-acapy-ariesvcx) | acapy-main<br>0.12.1 | aries-vcx<br>1.0.0 | acapy-main<br>0.12.1 | acapy-main<br>0.12.1 | AIP 1.0 | [**18 / 18<br>100%**](https://allure.vonx.io/api/allure-docker-service/projects/acapy-b-aries-vcx/reports/latest/index.html?redirect=false#behaviors) |
| [afj-acapy](#runset-afj-acapy) | javascript<br>0.5.0-alpha.97 | acapy-main<br>0.12.1 | javascript<br>0.5.0-alpha.97 | javascript<br>0.5.0-alpha.97 | AIP 1.0 | [**11 / 28<br>39%**](https://allure.vonx.io/api/allure-docker-service/projects/javascript-b-acapy/reports/latest/index.html?redirect=false#behaviors) |
| [ariesvcx-acapy](#runset-ariesvcx-acapy) | aries-vcx<br>1.0.0 | acapy-main<br>0.12.1 | aries-vcx<br>1.0.0 | aries-vcx<br>1.0.0 | AIP 1.0 | [**18 / 20<br>90%**](https://allure.vonx.io/api/allure-docker-service/projects/aries-vcx-b-acapy/reports/latest/index.html?redirect=false#behaviors) |
| [acapy-afj](#runset-acapy-afj) | acapy-main<br>1.0.0rc4 | javascript<br>0.5.0-alpha.97 | acapy-main<br>1.0.0rc4 | acapy-main<br>1.0.0rc4 | AIP 1.0 | [**37 / 39<br>94%**](https://allure.vonx.io/api/allure-docker-service/projects/acapy-b-javascript/reports/latest/index.html?redirect=false#behaviors) |
| [acapy-aip10](#runset-acapy-aip10) | acapy-main<br>1.0.0rc4 | acapy-main<br>1.0.0rc4 | acapy-main<br>1.0.0rc4 | acapy-main<br>1.0.0rc4 | AIP 1.0 | [**35 / 35<br>100%**](https://allure.vonx.io/api/allure-docker-service/projects/acapy-aip10/reports/latest/index.html?redirect=false#behaviors) |
| [acapy-aip20](#runset-acapy-aip20) | acapy-main<br>1.0.0rc4 | acapy-main<br>1.0.0rc4 | acapy-main<br>1.0.0rc4 | acapy-main<br>1.0.0rc4 | AIP 2.0 | [**61 / 61<br>100%**](https://allure.vonx.io/api/allure-docker-service/projects/acapy-aip20/reports/latest/index.html?redirect=false#behaviors) |
| [acapy-ariesvcx](#runset-acapy-ariesvcx) | acapy-main<br>1.0.0rc4 | aries-vcx<br>0.64.0 | acapy-main<br>1.0.0rc4 | acapy-main<br>1.0.0rc4 | AIP 1.0 | [**18 / 28<br>64%**](https://allure.vonx.io/api/allure-docker-service/projects/acapy-b-aries-vcx/reports/latest/index.html?redirect=false#behaviors) |
| [afj-acapy](#runset-afj-acapy) | javascript<br>0.5.0-alpha.97 | acapy-main<br>1.0.0rc4 | javascript<br>0.5.0-alpha.97 | javascript<br>0.5.0-alpha.97 | AIP 1.0 | [**11 / 28<br>39%**](https://allure.vonx.io/api/allure-docker-service/projects/javascript-b-acapy/reports/latest/index.html?redirect=false#behaviors) |
| [ariesvcx-acapy](#runset-ariesvcx-acapy) | aries-vcx<br>0.64.0 | acapy-main<br>1.0.0rc4 | aries-vcx<br>0.64.0 | aries-vcx<br>0.64.0 | AIP 1.0 | [**2 / 28<br>7%**](https://allure.vonx.io/api/allure-docker-service/projects/aries-vcx-b-acapy/reports/latest/index.html?redirect=false#behaviors) |

## Runset Notes

Expand All @@ -21,7 +21,7 @@ Runset Name: ACA-PY to AFJ
**Latest results: 37 out of 39 (94%)**
*Last run: Wed Jul 3 00:47:46 UTC 2024*
*Last run: Wed Jul 10 00:47:39 UTC 2024*
```

#### Current Runset Status
Expand All @@ -44,7 +44,7 @@ Runset Name: ACA-PY to ACA-Py
**Latest results: 35 out of 35 (100%)**
*Last run: Wed Jul 3 01:10:17 UTC 2024*
*Last run: Wed Jul 10 01:09:53 UTC 2024*
```

#### Current Runset Status
Expand All @@ -67,7 +67,7 @@ Runset Name: ACA-PY to ACA-Py
**Latest results: 61 out of 61 (100%)**
*Last run: Wed Jul 3 01:42:30 UTC 2024*
*Last run: Wed Jul 10 01:42:21 UTC 2024*
```

#### Current Runset Status
Expand All @@ -87,16 +87,16 @@ All of the tests being executed in this runset are passing.
Runset Name: acapy to aries-vcx

```tip
**Latest results: 18 out of 18 (100%)**
**Latest results: 18 out of 28 (64%)**
*Last run: Wed Jul 3 02:17:59 UTC 2024*
*Last run: Wed Jul 10 02:00:06 UTC 2024*
```

#### Current Runset Status
```warning
No test status note is available for this runset. Please update: .github/workflows/test-harness-acapy-ariesvcx.yml.
```

RFC0023 is disabled due to inconsistent results. RFC0793 is also being investigated: https://github.com/hyperledger/aries-vcx/issues/1252
*Status Note Updated: 2024.07.05*

#### Runset Details

Expand All @@ -112,7 +112,7 @@ Runset Name: AFJ to ACA-PY
**Latest results: 11 out of 28 (39%)**
*Last run: Wed Jul 3 03:16:15 UTC 2024*
*Last run: Wed Jul 10 02:57:59 UTC 2024*
```

#### Current Runset Status
Expand All @@ -132,16 +132,19 @@ All AIP10 tests are currently running.
Runset Name: aries-vcx to acapy

```tip
**Latest results: 18 out of 20 (90%)**
**Latest results: 2 out of 28 (7%)**
*Last run: Wed Jul 3 04:05:55 UTC 2024*
*Last run: Wed Jul 10 03:44:25 UTC 2024*
```

#### Current Runset Status
```warning
No test status note is available for this runset. Please update: .github/workflows/test-harness-ariesvcx-acapy.yml.
```

Most tests are currently struggling, due to aries-vcx reporting the wrong connection state to the
backchannel. Being resolved here: https://github.com/hyperledger/aries-vcx/issues/1253
@RFC0793 has relatively low success due to aries-vcx not supporting full range of DID methods in
these tests.
*Status Note Updated: 2024.07.05*

#### Runset Details

Expand Down
40 changes: 22 additions & 18 deletions docs/aries-vcx.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4,9 +4,9 @@

| Runset | ACME<br>(Issuer) | Bob<br>(Holder) | Faber<br>(Verifier) | Mallory<br>(Holder) | Scope | Results |
| ------ | :--------------: | :-------------: | :----------------: | :-----------------: | ----- | :-----: |
| [acapy-ariesvcx](#runset-acapy-ariesvcx) | acapy-main<br>0.12.1 | aries-vcx<br>1.0.0 | acapy-main<br>0.12.1 | acapy-main<br>0.12.1 | AIP 1.0 | [**18 / 18<br>100%**](https://allure.vonx.io/api/allure-docker-service/projects/acapy-b-aries-vcx/reports/latest/index.html?redirect=false#behaviors) |
| [ariesvcx-acapy](#runset-ariesvcx-acapy) | aries-vcx<br>1.0.0 | acapy-main<br>0.12.1 | aries-vcx<br>1.0.0 | aries-vcx<br>1.0.0 | AIP 1.0 | [**18 / 20<br>90%**](https://allure.vonx.io/api/allure-docker-service/projects/aries-vcx-b-acapy/reports/latest/index.html?redirect=false#behaviors) |
| [ariesvcx-ariesvcx](#runset-ariesvcx-ariesvcx) | aries-vcx<br>1.0.0 | aries-vcx<br>1.0.0 | aries-vcx<br>1.0.0 | aries-vcx<br>1.0.0 | AIP 1.0 | [**17 / 20<br>85%**](https://allure.vonx.io/api/allure-docker-service/projects/aries-vcx/reports/latest/index.html?redirect=false#behaviors) |
| [acapy-ariesvcx](#runset-acapy-ariesvcx) | acapy-main<br>1.0.0rc4 | aries-vcx<br>0.64.0 | acapy-main<br>1.0.0rc4 | acapy-main<br>1.0.0rc4 | AIP 1.0 | [**18 / 28<br>64%**](https://allure.vonx.io/api/allure-docker-service/projects/acapy-b-aries-vcx/reports/latest/index.html?redirect=false#behaviors) |
| [ariesvcx-acapy](#runset-ariesvcx-acapy) | aries-vcx<br>0.64.0 | acapy-main<br>1.0.0rc4 | aries-vcx<br>0.64.0 | aries-vcx<br>0.64.0 | AIP 1.0 | [**2 / 28<br>7%**](https://allure.vonx.io/api/allure-docker-service/projects/aries-vcx-b-acapy/reports/latest/index.html?redirect=false#behaviors) |
| [ariesvcx-ariesvcx](#runset-ariesvcx-ariesvcx) | aries-vcx<br>0.64.0 | aries-vcx<br>0.64.0 | aries-vcx<br>0.64.0 | aries-vcx<br>0.64.0 | AIP 1.0 | [**27 / 32<br>84%**](https://allure.vonx.io/api/allure-docker-service/projects/aries-vcx/reports/latest/index.html?redirect=false#behaviors) |
| [javascript-ariesvcx](#runset-javascript-ariesvcx) | javascript<br>0.5.0-alpha.97 | aries-vcx<br>1.0.0 | javascript<br>0.5.0-alpha.97 | javascript<br>0.5.0-alpha.97 | AIP 1.0 | [**3 / 18<br>16%**](https://allure.vonx.io/api/allure-docker-service/projects/javascript-b-aries-vcx/reports/latest/index.html?redirect=false#behaviors) |

## Runset Notes
Expand All @@ -16,16 +16,16 @@
Runset Name: acapy to aries-vcx

```tip
**Latest results: 18 out of 18 (100%)**
**Latest results: 18 out of 28 (64%)**
*Last run: Wed Jul 3 02:17:59 UTC 2024*
*Last run: Wed Jul 10 02:00:06 UTC 2024*
```

#### Current Runset Status
```warning
No test status note is available for this runset. Please update: .github/workflows/test-harness-acapy-ariesvcx.yml.
```

RFC0023 is disabled due to inconsistent results. RFC0793 is also being investigated: https://github.com/hyperledger/aries-vcx/issues/1252
*Status Note Updated: 2024.07.05*

#### Runset Details

Expand All @@ -38,16 +38,19 @@ No test status note is available for this runset. Please update: .github/workflo
Runset Name: aries-vcx to acapy

```tip
**Latest results: 18 out of 20 (90%)**
**Latest results: 2 out of 28 (7%)**
*Last run: Wed Jul 3 04:05:55 UTC 2024*
*Last run: Wed Jul 10 03:44:25 UTC 2024*
```

#### Current Runset Status
```warning
No test status note is available for this runset. Please update: .github/workflows/test-harness-ariesvcx-acapy.yml.
```

Most tests are currently struggling, due to aries-vcx reporting the wrong connection state to the
backchannel. Being resolved here: https://github.com/hyperledger/aries-vcx/issues/1253
@RFC0793 has relatively low success due to aries-vcx not supporting full range of DID methods in
these tests.
*Status Note Updated: 2024.07.05*

#### Runset Details

Expand All @@ -60,16 +63,17 @@ No test status note is available for this runset. Please update: .github/workflo
Runset Name: aries-vcx to aries-vcx

```tip
**Latest results: 17 out of 20 (85%)**
**Latest results: 27 out of 32 (84%)**
*Last run: Wed Jul 3 04:36:55 UTC 2024*
*Last run: Wed Jul 10 03:57:14 UTC 2024*
```

#### Current Runset Status
```warning
No test status note is available for this runset. Please update: .github/workflows/test-harness-ariesvcx-ariesvcx.yml.
```

@RFC0793 has some failures due to aries-vcx not supporting full range of DID methods in
these tests.
*Status Note Updated: 2024.07.05*

#### Runset Details

Expand Down
10 changes: 5 additions & 5 deletions docs/javascript.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4,8 +4,8 @@

| Runset | ACME<br>(Issuer) | Bob<br>(Holder) | Faber<br>(Verifier) | Mallory<br>(Holder) | Scope | Results |
| ------ | :--------------: | :-------------: | :----------------: | :-----------------: | ----- | :-----: |
| [acapy-afj](#runset-acapy-afj) | acapy-main<br>0.12.1 | javascript<br>0.5.0-alpha.97 | acapy-main<br>0.12.1 | acapy-main<br>0.12.1 | AIP 1.0 | [**37 / 39<br>94%**](https://allure.vonx.io/api/allure-docker-service/projects/acapy-b-javascript/reports/latest/index.html?redirect=false#behaviors) |
| [afj-acapy](#runset-afj-acapy) | javascript<br>0.5.0-alpha.97 | acapy-main<br>0.12.1 | javascript<br>0.5.0-alpha.97 | javascript<br>0.5.0-alpha.97 | AIP 1.0 | [**11 / 28<br>39%**](https://allure.vonx.io/api/allure-docker-service/projects/javascript-b-acapy/reports/latest/index.html?redirect=false#behaviors) |
| [acapy-afj](#runset-acapy-afj) | acapy-main<br>1.0.0rc4 | javascript<br>0.5.0-alpha.97 | acapy-main<br>1.0.0rc4 | acapy-main<br>1.0.0rc4 | AIP 1.0 | [**37 / 39<br>94%**](https://allure.vonx.io/api/allure-docker-service/projects/acapy-b-javascript/reports/latest/index.html?redirect=false#behaviors) |
| [afj-acapy](#runset-afj-acapy) | javascript<br>0.5.0-alpha.97 | acapy-main<br>1.0.0rc4 | javascript<br>0.5.0-alpha.97 | javascript<br>0.5.0-alpha.97 | AIP 1.0 | [**11 / 28<br>39%**](https://allure.vonx.io/api/allure-docker-service/projects/javascript-b-acapy/reports/latest/index.html?redirect=false#behaviors) |
| [afj](#runset-afj) | javascript<br>0.5.0-alpha.97 | javascript<br>0.5.0-alpha.97 | javascript<br>0.5.0-alpha.97 | javascript<br>0.5.0-alpha.97 | AIP 1.0 | [**12 / 28<br>42%**](https://allure.vonx.io/api/allure-docker-service/projects/javascript/reports/latest/index.html?redirect=false#behaviors) |
| [javascript-ariesvcx](#runset-javascript-ariesvcx) | javascript<br>0.5.0-alpha.97 | aries-vcx<br>1.0.0 | javascript<br>0.5.0-alpha.97 | javascript<br>0.5.0-alpha.97 | AIP 1.0 | [**3 / 18<br>16%**](https://allure.vonx.io/api/allure-docker-service/projects/javascript-b-aries-vcx/reports/latest/index.html?redirect=false#behaviors) |

Expand All @@ -19,7 +19,7 @@ Runset Name: ACA-PY to AFJ
**Latest results: 37 out of 39 (94%)**
*Last run: Wed Jul 3 00:47:46 UTC 2024*
*Last run: Wed Jul 10 00:47:39 UTC 2024*
```

#### Current Runset Status
Expand All @@ -42,7 +42,7 @@ Runset Name: AFJ to ACA-PY
**Latest results: 11 out of 28 (39%)**
*Last run: Wed Jul 3 03:16:15 UTC 2024*
*Last run: Wed Jul 10 02:57:59 UTC 2024*
```

#### Current Runset Status
Expand All @@ -65,7 +65,7 @@ Runset Name: AFJ to AFJ
**Latest results: 12 out of 28 (42%)**
*Last run: Wed Jul 3 03:31:50 UTC 2024*
*Last run: Wed Jul 10 03:13:58 UTC 2024*
```

#### Current Runset Status
Expand Down

0 comments on commit 9548eb7

Please sign in to comment.