Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

HIP-980: Add shadow fork support as a rejected idea #1058

Open
wants to merge 15 commits into
base: main
Choose a base branch
from
+64 −29
Open

HIP-980: Add shadow fork support as a rejected idea #1058

Changes from all commits
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
1660a67
Create HIP for extending mirorr node APIs with timestamp query
IvanKavaldzhiev Jun 5, 2024
5889bb3
Update HIP number and discussion link
IvanKavaldzhiev Jun 5, 2024
ad63f04
Add requestor
IvanKavaldzhiev Jun 5, 2024
f11dce0
nit: rename HIP file
IvanKavaldzhiev Jun 25, 2024
5df8d05
nit: resolve PR comments
IvanKavaldzhiev Jun 27, 2024
e750582
nit: resolve PR comments
IvanKavaldzhiev Jun 28, 2024
602f37b
nit: resolve PR comments
IvanKavaldzhiev Jul 1, 2024
cc0e585
nit: fix typo
IvanKavaldzhiev Jul 9, 2024
74e83b5
feat: enhance description about timestamp query parameter
IvanKavaldzhiev Jul 10, 2024
abf02d8
Merge branch 'main' of github.com:hashgraph/hedera-improvement-propos…
IvanKavaldzhiev Jul 10, 2024
5a60cbe
Merge branch 'main' of github.com:hashgraph/hedera-improvement-propos…
IvanKavaldzhiev Jul 10, 2024
cd7762d
feat: enhance HIP description with response enhancement
IvanKavaldzhiev Jul 10, 2024
bca0aa9
Merge remote-tracking branch 'hedera/main' into hip-xxx-extend-apis-w…
IvanKavaldzhiev Sep 11, 2024
a13df90
Merge remote-tracking branch 'hedera/main' into hip-xxx-extend-apis-w…
IvanKavaldzhiev Oct 8, 2024
0bd83bd
feat: remove shadow fork proposal and move it to rejected ideas
IvanKavaldzhiev Oct 9, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
93 changes: 64 additions & 29 deletions HIP/hip-980.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,55 +11,67 @@ last-call-date-time: 2024-07-23T07:00:00Z
created: 2024-06-03
requested-by: Solo Labs
discussions-to: https://github.com/hashgraph/hedera-improvement-proposal/discussions/981
updated: 2024-08-16
updated: 2024-10-09
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mgarbs and @steven-sheehy should we be updating the Status to be Rejected?

---

## Abstract

This HIP enhances part of the mirror node REST APIs to support querying by timestamp. This will allow users to query for the historical state of any entity info on the network that currently misses this functionality.
This HIP enhances part of the mirror node REST APIs to support querying by timestamp. This will allow users to query for
the historical state of any entity info on the network that currently misses this functionality.

## Motivation

To enhance user experience and allow users or developers to query the state of an entity at a specific point in time we will add a timestamp query parameter to all of the entity-based REST APIs that currently don't have one.
This change will be especially valuable for building the foundation of the so-called shadow fork functionality, where a user or developer can use a local mirror node instance but run it against a production environment state.
Having this support, users or developers will have the ability to use public state data to simulate and troubleshoot existing issues or replay historical transactions using local mirror node instance against production state.
To enhance user experience and allow users or developers to query the state of an entity at a specific point in time we
will add a timestamp query parameter to all of the entity-based REST APIs that currently don't have one.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Q: why was the motivation section cut from 3 points to 1?
Seems strange for motivation to have changed now.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's because the shadow fork initiative is being rejected in this HIP. Most of the points in the motivation section were describing the motivation of having shadow fork functionality. If we remove this part of the section what is left is the extension of reading historical data for all entity types via the REST API.


## Rationale

Users may want to run historical transactions on a local mirror node instance using public state data. To make this possible, some of the existing REST APIs should be enhanced with a timestamp query parameter, so that historical public data can be fetched and used
Users may want to run historical transactions on a local mirror node instance using public state data. To make this
possible, some of the existing REST APIs should be enhanced with a timestamp query parameter, so that historical public
data can be fetched and used
for the simulations.

## User stories

- As a user of mirror node, I want to be able to query for the historical state of nfts.
- As a user of mirror node, I want to be able to query for the historical state of token relationships for an account.
- As a user of mirror node, I want to be able to query for the historical state of allowances for an account.
- As a user of mirror node, I want to be able to run locally an `eth_call` request that fails on a production environment against the corresponding public state.
- As a user of mirror node, I want to be able to run locally an `eth_estimateGas` request that fails on a production environment against the corresponding public state.
- As a user of mirror node, I want to be able to run locally an `eth_debugTraceTransaction` request that fails on a production environment against the corresponding public state.
- As a mirror node developer, I want to be able to test new features locally against production state, so that I have better understanding of the feature behavior.

## Specification

The proposed enhancement involves adding a query parameter `timestamp` to the existing routes:

- `/api/v1/accounts/{idOrAliasOrEvmAddress}/tokens` to return correct historical information about an account and their associated fungible tokens for a specific timestamp
- `/api/v1/accounts/{idOrAliasOrEvmAddress}/nfts` to return correct historical information about an account and their associated nfts for a specific timestamp
- `/api/v1/accounts/{idOrAliasOrEvmAddress}/allowances/crypto` to return correct historical information about an account's crypto allowances for a specific timestamp
- `/api/v1/accounts/{idOrAliasOrEvmAddress}/allowances/nfts` to return correct historical information about an account's NFT allowances for a specific timestamp
- `/api/v1/accounts/{idOrAliasOrEvmAddress}/allowances/tokens` to return correct historical information about an account's token allowances for a specific timestamp
- `/api/v1/tokens/{tokenId}/nfts/{serialNumber}` to return correct historical information about an NFT for a specific timestamp
- `/api/v1/accounts/{idOrAliasOrEvmAddress}/tokens` to return correct historical information about an account and their
associated fungible tokens for a specific timestamp
- `/api/v1/accounts/{idOrAliasOrEvmAddress}/nfts` to return correct historical information about an account and their
associated nfts for a specific timestamp
- `/api/v1/accounts/{idOrAliasOrEvmAddress}/allowances/crypto` to return correct historical information about an
account's crypto allowances for a specific timestamp
- `/api/v1/accounts/{idOrAliasOrEvmAddress}/allowances/nfts` to return correct historical information about an account's
NFT allowances for a specific timestamp
- `/api/v1/accounts/{idOrAliasOrEvmAddress}/allowances/tokens` to return correct historical information about an
account's token allowances for a specific timestamp
- `/api/v1/tokens/{tokenId}/nfts/{serialNumber}` to return correct historical information about an NFT for a specific
timestamp

**Timestamp Query Parameter**

A new `timestamp` query parameter will be added to search for all entities and entity related information that are missing its support. It will be with the format of `array[string]`.
This query parameter will represent a single or multiple consensus timestamps in a Unix format in `seconds.nanoseconds` format and will have an optional comparison operator.
That is, it will search for results that are `equal`, `greater` or `less` than the timestamp itself or its seconds component.
A new `timestamp` query parameter will be added to search for all entities and entity related information that are
missing its support. It will be with the format of `array[string]`.
This query parameter will represent a single or multiple consensus timestamps in a Unix format in `seconds.nanoseconds`
format and will have an optional comparison operator.
That is, it will search for results that are `equal`, `greater` or `less` than the timestamp itself or its seconds
component.

The exact names of the operators will be `eq`, `gt`, `gte`, `lt` and `lte`. The user doesn't need to know the exact timestamp at which an entity was created or updated for it to function. The API will return the results that match the timestamp or the closest one to it.
For example, if we have an `asc` ordering of the results, then `?timestamp=X`, `timestamp=eq:X`, or `timestamp=lte:X` will have the same behaviour and vice versa, where `X` is the desired timestamp.
The exact names of the operators will be `eq`, `gt`, `gte`, `lt` and `lte`. The user doesn't need to know the exact
timestamp at which an entity was created or updated for it to function. The API will return the results that match the
timestamp or the closest one to it.
For example, if we have an `asc` ordering of the results, then `?timestamp=X`, `timestamp=eq:X`, or `timestamp=lte:X`
will have the same behaviour and vice versa, where `X` is the desired timestamp.

The `timestamp` query parameter will support having a range of timestamps. For example, `timestamp=gt:X&timestamp=lt:Y` will return a single result with the entity or entity related info in the most recent state within the range that is greater than `X` and less than `Y`, where `X` and `Y` are different timestamps.
The `timestamp` query parameter will support having a range of timestamps. For example, `timestamp=gt:X&timestamp=lt:Y`
will return a single result with the entity or entity related info in the most recent state within the range that is
greater than `X` and less than `Y`, where `X` and `Y` are different timestamps.

Here are the affected APIs with the query parameter added:

Expand All @@ -72,27 +84,35 @@ Here are the affected APIs with the query parameter added:
/api/v1/tokens/{tokenId}/nfts/{serialNumber}?timestamp={value}
```

Note: There is an existing API with a `timestamp` query parameter, which is `/api/v1/accounts/{idOrAliasOrEvmAddress}?timestamp={value}`. While parameter changes won't be applied for it, it's
behaviour will reflect to actually return modification changes of all account fields. Currently, the timestamp filter is applied only to the transaction list. The balance info section will continue to have granularity of 15 minutes.
Note: There is an existing API with a `timestamp` query parameter, which is
`/api/v1/accounts/{idOrAliasOrEvmAddress}?timestamp={value}`. While parameter changes won't be applied for it, it's
behaviour will reflect to actually return modification changes of all account fields. Currently, the timestamp filter is
applied only to the transaction list. The balance info section will continue to have granularity of 15 minutes.

**Request**

```
/api/v1/accounts/{idOrAliasOrEvmAddress}/tokens?timestamp={value}
```

**Example Request**

```
GET api/v1/accounts/4408244/tokens?order=asc&timestamp=lte:1725334849.603493001
```

Note that the following two requests will return the same response as the query above:

```
GET api/v1/accounts/4408244/tokens?order=asc&timestamp=1705334449.803393003
GET api/v1/accounts/4408244/tokens?order=asc&timestamp=eq:1705334449.803393003
```

**Response**

The response format of all the mentioned APIs will be enhanced with additional timestamp range with `from` and `to` fields, reflecting the period in which the returned data state was valid. The `to` field will be null if the result represent the latest updated state.
The response format of all the mentioned APIs will be enhanced with additional timestamp range with `from` and `to`
fields, reflecting the period in which the returned data state was valid. The `to` field will be null if the result
represent the latest updated state.

```json
{
Expand All @@ -119,22 +139,37 @@ The response format of all the mentioned APIs will be enhanced with additional t

## **Backwards Compatibility**

The change is backwards compatible as it is simply adding a query parameter to an existing route. The data required to filter by timestamp is already stored in the mirror node.
The change is backwards compatible as it is simply adding a query parameter to an existing route. The data required to
filter by timestamp is already stored in the mirror node.

If a timestamp query parameter is not provided, the API will use the latest state of the entity as it does currently.

## Security Implications

Historical timestamp filtering can slow down some of the queries, especially if the timestamp is not indexed or the data set is very large.
Historical timestamp filtering can slow down some of the queries, especially if the timestamp is not indexed or the data
set is very large.

## Rejected Ideas

None
Initially, the HIP aimed to allow the grounding steps for supporting shadow fork on a local mirror node. This means
users could run a local instance of the mirror node
and use a public state of the network, including historical one. However, this idea was rejected, since the
implementation of this feature would have meant having
a lot of REST API requests against the public mirror nodes, which would have generated a heavy load on the network.
Additionally, it would have been impractical
for many users accustomed to rely on a public service, having this functionality (e.g. call eth_call or
eth_debugTraceTransaction with a specific block number against a
public service, which may be paid), to spend additional efforts of spinning their own local mirror node and sending
their call against it.

As a summary, this idea was rejected in favour of a public mirror node provider to allow this functionality as a
service.

## References

https://mainnet-public.mirrornode.hedera.com/api/v1/docs/

## Copyright/license

This document is licensed under the Apache License, Version 2.0 -- see [LICENSE](https://www.notion.so/LICENSE) or (https://www.apache.org/licenses/LICENSE-2.0)
This document is licensed under the Apache License, Version 2.0 -- see [LICENSE](https://www.notion.so/LICENSE)
or (https://www.apache.org/licenses/LICENSE-2.0)
Loading