Merge pull request #187 from tablelandnetwork/dtb/lt-fork

Local Tableland chain forking

config/sidebars.ts

@@ -387,6 +387,7 @@ const localTableland = [
   "local-tableland/cli",
   "local-tableland/methods",
   "local-tableland/testing",
+  "local-tableland/fork-chain",
 ];
 
 // Gateway REST API

docs/fundamentals/about/glossary.md

@@ -79,7 +79,7 @@ Sending a transaction is not free; you have to pay [gas](https://ethereum.org/en
 
 ### IPFS
 
-[IPFS](https://ipfs.tech/) stands for Interplanetary File System. It is a p2p protocol and is used as a common way to store files but in a distributed manner, which enables content addressing by way of [CIDs](#cid). IPFS has less of a "guaranteed" persistence mechanism but does offer a feature for "[pinning](https://docs.ipfs.tech/how-to/pin-files/)" files to help tell the network to keep the file around (but it's not incentive driven like its related protocol [Filecoin](#filecion)).
+[IPFS](https://ipfs.tech/) stands for Interplanetary File System. It is a p2p protocol and is used as a common way to store files but in a distributed manner, which enables content addressing by way of [CIDs](#cid). IPFS has less of a "guaranteed" persistence mechanism but does offer a feature for "[pinning](https://docs.ipfs.tech/how-to/pin-files/)" files to help tell the network to keep the file around (but it's not incentive driven like its related protocol [Filecoin](#filecoin)).
 
 ## L
 

docs/local-tableland/fork-chain.md

@@ -0,0 +1,128 @@
+---
+title: Forking chain state
+description: Fork a live testnet or mainnet chain and use its state on Local Tableland.
+keywords:
+  - fork
+  - fork chain
+  - local tableland
+---
+
+A common development pattern is to use an existing chain's state as a starting point for testing. Local Tableland supports forking a live testnet or mainnet chain, and the full chain's state is available for use. This includes any smart contract or transactions that occurred on the chain, and the Tableland table state materialized is also materialized on Local Tableland. In other words, you can interact with some table deployed on the forked, live chain as if it were a local table.
+
+## Overview
+
+The required parameters to fork a chain are passed during `LocalTableland` instantiation:
+
+- `forkUrl`: The provider URL of the chain to fork, such as an Alchemy or Infura URL.
+- `forkBlockNumber`: The block number to fork from, which recreates the chain state starting at that block.
+- `forkChainId`: The chain ID of the chain to fork.
+
+Once these values are set, a few things will happen:
+
+- The chain's forked state will be initialized at the starting block number.
+- Events/logs will be replayed from the forked chain, starting from the block number specified.
+- The Tableland validator will process _all_ of the Tableland registry's events/logs from the forked chain up until the forked block, and the state of the tables will be materialized.
+
+When you initially start Local Tableland, it will take a bit of time to backfill the forked chain's state. The `forkUrl` you specify for the provider will make API calls to `eth_getLogs` in batches, and the validator replays these one-by-one. Keep in mind this will continue hitting the API until the state is fully backfilled, but it _should_ be cached after the first time you run it. But, **caching only works if you use the same** `forkBlockNumber` value each time you start Local Tableland.
+
+:::note
+Forking testnet chains _is_ possible. However, due to the large volume of data on chains like Polygon Mumbai, the backfilling process can take a **very long time**—potentially, up to an hour. We're researching ways to optimize this process (e.g., trusted bootstrap for backfilled state), but for now, it's best to fork mainnet chains with less data.
+:::
+
+## Setup
+
+This example will use `mocha`, `chai` and `chai-as-promised`, so make sure these are installed, along with Local Tableland:
+
+```bash npm2yarn
+npm install -D @tableland/local mocha chai chai-as-promised
+```
+
+Import the necessary functions and classes from Local Tableland and the testing libraries:
+
+```js
+import { after, before, describe } from "mocha";
+import { LocalTableland, getAccounts, getDatabase } from "@tableland/local";
+import chai from "chai";
+import chaiAsPromised from "chai-as-promised";
+
+chai.use(chaiAsPromised);
+const expect = chai.expect;
+
+describe("fork", function () {
+  // Set up Local Tableland and tests...
+});
+```
+
+Now, let's set up the `LocalTableland` instance with the forked chain parameters. The example uses Polygon mainnet:
+
+- `forkUrl`: The Alchemy URL for Polygon mainnet `https://polygon-mainnet.g.alchemy.com/v2/<your_alchemy_api_key>`, but be sure to replace the path parameter your API key.
+- `forkBlockNumber`: The block `53200000` occurs in [early 2024](https://polygonscan.com/block/53200000).
+- `forChainId`: The Polygon mainnet chain ID is `137`.
+
+The top-level `this.timeout` is set to `30000` milliseconds, which is used for each test. But, the `before` hook that starts Local Tableland need a longer timeout, so it's set to `90000` milliseconds. After `lt.start()` is called, API calls are made to the `forkUrl` to get historical state information, and then this data is materialized by the validator, which is why an additional `setTimeout` is used to wait for the state to be fully backfilled. Depending on which chain you fork, this process can take a while—e.g., a testnet might take tens of minutes or more, whereas mainnets might take a minute or two.
+
+```js
+describe("fork", function () {
+  this.timeout(30000);
+
+  const lt = new LocalTableland({
+    silent: false,
+    forkUrl: "https://polygon-mainnet.g.alchemy.com/v2/<your_alchemy_api_key>",
+    forkBlockNumber: "53200000",
+    forkChainId: "137",
+  });
+  const [, signer] = getAccounts(lt);
+  const db = getDatabase(signer);
+
+  before(async function () {
+    // Depending on the chain, this could take a while—adjust timeout for Local
+    // Tableland startup, in case it's longer than the top-level timeout
+    this.timeout(90000);
+    await lt.start();
+    // After calling `start`, the forked chain data must be materialized—you
+    // must set this timeout to wait until all state is materialized
+    await new Promise((resolve) =>
+      setTimeout(() => {
+        resolve(undefined);
+      }, 60000)
+    );
+  });
+
+  after(async function () {
+    await lt.shutdown();
+  });
+
+  // Tests here...
+});
+```
+
+The first time you run this test, it will take a bit longer because the chain state is not cached. After the first run, the state should be cached (i.e., no additional API calls needed if you use the same block number). Thus, the tests should run faster in subsequent runs; however, the validator will still need to process the state from the forked chain. In other words, after each test run, the validator's state is entirely cleared, but the Hardhat node's state is cached and reused.
+
+## Testing forked chain data
+
+Now, let's run a couple of tests. The first test makes a read query on the `healthbot_137_1` table, which is a health check table that was deployed by the Tableland team upon launching on Polygon mainnet. The second test creates a new table on the forked chain, and the table name will be suffixed with the `chainId` of `137` and use the next available `tableId` of `245`.
+
+```js
+describe("fork", function () {
+  // Exiting code...
+
+  it("should read existing table created on forked chain", async function () {
+    // The "healthbot" table is created on the forked chain
+    // It always has value `1` on mainnets, whereas it's incremented on testnets
+    const { results } = await db.prepare("select * from healthbot_137_1").all();
+    expect(results[0].counter).to.be.equal(1);
+  });
+
+  it("should create a new table on forked chain", async function () {
+    // Create a table on the forked chain
+    const { meta } = await db
+      .prepare("create table my_table (id int primary key, val text)")
+      .run();
+    await meta.txn?.wait();
+    const [table] = meta.txn?.names ?? [];
+    // Since the forked chain is Polygon, the table name will be suffixed with
+    // with `chainId` of `137` and use the next available `tableId` of `245`
+    expect(table).to.be.equal("my_table_137_245");
+  });
+});
+```

docs/local-tableland/methods.md

@@ -43,6 +43,9 @@ The `LocalTableland` constructor accepts a single argument, an object with the f
 - `verbose`: Output verbose logs to stdout.
 - `silent`: Silence all output to stdout.
 - `registryPort`: Use a custom registry hardhat port, e.g., `http://127.0.0.1:8545` is the default with port `8545`, but.
+- `forkUrl`: If forking a chain, the provider URL of the testnet or mainnet chain, such as an Alchemy or Infura URL.
+- `forkBlockNumber`: If forking a chain, the block number to fork from, which recreates the chain state up to that block.
+- `forkChainId`: If forking a chain, the chain ID of the chain.
 
 Most uses cases only use the `verbose` or `silent` options, which control logging behavior during tests.
 
@@ -52,6 +55,8 @@ const lt = new LocalTableland({ silent: false });
 
 In more complex cases, the registry and validator options allow you to point to custom instances. A monorepo setup, for example, might want separately test things in parallel, so the registry and validator need to be unique to each environment—check out the [`tableland-js` monorepo](https://github.com/tablelandnetwork/tableland-js/tree/main/packages) for how this might look.
 
+There's also a feature for _forking_ an existing testnet or mainnet chain, which will allow you to interact with the chain and table state in a sandboxed local environment. See the [forking chain state](/local-tableland/fork-chain) documentation for more information.
+
 ### Start
 
 The `start` method starts the Tableland network with the Hardhat node and the Tableland validator node. This should be ran before any tests are executed.

docs/studio/web/import-table.md

@@ -13,13 +13,13 @@ import importedTables from "@site/static/assets/studio/16_imported-tables.png";
 
 ## Importing an existing table
 
-To import an existing table, you'll need to know the table's full name in the form `{prefix}_{chainId}_{tableId}` since these are parameters required during the import. Typically, you'll also want to make sure that whatever wallet you're connected to in the Studio is _the same account_ that created the table outside of the Studio. This example will use some "messages" table that we've already created; however, it _is_ possible to import a table that someone else created, even if you don't have write permissions on the table.
+To import an existing table, you'll need to know the table's full name in the form `{prefix}_{chainId}_{tableId}` since these are parameters required during the import. Typically, you'll also want to make sure that whatever wallet you're connected to in the Studio is _the same account_ that created the table outside of the Studio. This example will use some "users" table that we've already created; however, it _is_ possible to import a table that someone else created, even if you don't have write permissions on the table.
 
 1. Navigate to your project homepage and click the **Import Table** button in the top right corner.
 
   <img src={importTable} width='80%'/>
 
-2. Then, you'll need to enter the information from the universally unique table name—the prefix, chain ID, and table ID.
+1. Then, you'll need to enter the information from the universally unique table name—the prefix, chain ID, and table ID. E.g., if the original globally unique of the table is `users_31337_15`, then you'd enter `users` in the "name" section, `31337` for the chain ID (aka Local Tableland / Hardhat), and `15` for the table ID.
 
   <img src={importTableInfo} width='80%'/>
 

docs/validator/api/query.md

@@ -10,15 +10,42 @@ keywords:
 
 ## Endpoint
 
-`GET /query`
+The `/query` endpoint supports both `GET` and `POST` requests, returning the results of a SQL read query, along with options for formatting the response.
 
-The following describes the usage with placeholder parameters on a testnet gateway URL:
+`GET /query` or `POST /query`
+
+### GET request
+
+The following describes the `GET` usage with placeholder parameters on a testnet gateway URL:
 
 ```bash
 curl -X GET https://testnets.tableland.network/api/v1/query?format={objects|table}&extract={boolean}&unwrap={boolean}&statement={sql_select_statement} \
   -H 'Accept: application/json'
 ```
 
+For example, to query the `healthbot_80001_1` table with default formatting options:
+
+```bash
+curl -X GET https://testnets.tableland.network/api/v1/query?statement=select%20%2A%20from%20healthbot_80001_1 \
+  -H 'Accept: application/json'
+```
+
+### POST request
+
+The following describes the `POST` usage with example parameters on a testnet gateway URL:
+
+```bash
+curl -L 'https://testnets.tableland.network/api/v1/query' \
+  -H 'Content-Type: application/json' \
+  -H 'Accept: application/json' \
+  -d '{
+    "statement": "select * from healthbot_80001_1",
+    "format": "objects",
+    "extract": false,
+    "unwrap": false
+  }'
+```
+
 ## Parameters
 
 | Name      | In    | Type    | Required | Description                                                                                                      |
@@ -48,6 +75,17 @@ curl -X GET https://testnets.tableland.network/api/v1/query?statement=select%20%
   -H 'Accept: application/json'
 ```
 
+Or:
+
+```bash
+curl -L 'https://testnets.tableland.network/api/v1/query' \
+  -H 'Content-Type: application/json' \
+  -H 'Accept: application/json' \
+  -d '{
+    "statement": "select * from healthbot_80001_1"
+  }'
+```
+
 Returns a successful (200) response:
 
 ```json