Consistent Appends

[scottrblock]

Feature Proposal: Consistent String Append Operation

Dexie Cloud currently provides consistency guarantees for various update operations

Proposal:

Implement a consistent string append operation that would work similarly to the existing add() operation for sets or the mathematical add() for numbers.

Potential Implementation

await db.someTable.where({ id: someId }).modify({
  description: {
    append: " This is additional text."
  }
})

Rationale

1. Additive Nature: String append is fundamentally an additive operation, similar to adding elements to a set or numbers.
2. Non-destructive: Appending doesn't overwrite existing data, aligning with conflict-avoidance principles.
3. Use Cases: This would be useful for collaborative text editing, log entries, or any scenario where multiple clients need to add to a string concurrently.
4. Consistency with Existing Features: This would extend the consistency model already in place for tree structures and other data types to string operations.

Potential Issues

• Order Dependency: How would concurrent appends from multiple clients be resolved?

Is this a reasonable request? let me know any questions or next steps if I can help at all

[dfahlander]

It would be very easy to implement this requirement. One thing that worries me though: Could it be over-used and lead to extreme performance issues if very long strings are kept being appended to?

Also, I'm not sure about any use case that wouldn't be better if by adding new rows instead of extending an existing string. Logging is one such example.

[scottrblock]

Thanks @dfahlander, I'm curious what counts as very long? I'm trying to think through the best way to store the content of a text document from CodeMirror.

Their data model

treats the document as a flat string. It stores this in a tree-shaped data structure to allow cheap updates anywhere in the document (and efficient indexing by line number).

It sounds like I am better off storing the string the data in Dexie as an array of lines, or a tree as opposed to a flat string.

The tree seems most complex, but would be most performant if I could get it it to work.

The array would help with appends (this is a direct feature of the app I'm building, a "quick capture" that appends to your note). But for regular updates, I would either need to blow away the whole array and re-insert all of it, or compute which lines to update. The latter sounds as difficult as just storing it in a tree.

Asking about what size this would start to matter because I think almost all notes would be less than 1mb, most less than 50kb.

For what its worth, this was a difficult issue I was trying to solve without Dexie.js, so I am at least excited to see Dexie.js has some supprot for storing trees!

[dfahlander]

For rich-text collaboration, I'm currently building support for Y.js in #1926 so I suggest to use y-codemirror once #1926 has been completed and released (which would be within some weeks).

Basically, Dexie.js will support Y.Doc instances to be stored in props just like we support Blob, Date, etc. There will be a new DexieYProvider() available that can be used for syncing and awareness. In the examples of y-codemirror, the difference will be that the Y.Doc instance will be fetched from your Dexie database instead of being new:ed and that you will use DexieYProvider instead of any other kind of provider.

If translating the code sample from y-codemirror into using Dexie Cloud, this is what it will look something like this:

import * as Y from 'yjs'
import { CodemirrorBinding } from 'y-codemirror'
import { DexieYProvider } from 'dexie'
import CodeMirror from 'codemirror'
import dexieCloud from 'dexie-cloud-addon';

const db = new Dexie('mydb', {
  Y, // Important to provide the Y library here to Dexie constructor
  addons: [ dexieCloud]
});
db.version(1).stores({
  items: '@id, codeContent:Y'. // Declare the Y.Doc property as ":Y"
});
...

const obj = await db.someTable.get(someId); // Fetch a db row
const ydoc = obj.codecontent;
const provider = new DexieYProvider(ydoc); // Connects with Dexie.js and Dexie Cloud.
const yText = ydoc.getText('codemirror')
const yUndoManager = new Y.UndoManager(yText)

const editor = CodeMirror(editorDiv, {
  mode: 'javascript',
  lineNumbers: true
})

const binding = new CodemirrorBinding(yText, editor, provider.awareness, { yUndoManager })

[scottrblock]

Interesting, I was shying away from CRDTs because of their storage overhead and potential performance issues with that. I am building mostly around single player note taking, though collaborative use cases of course start coming up because people have multiple devices.

In any case, I think I am able to store and update the text from codemirror as a tree somewhat easily:

export interface NoteNode {
  id: string;
  parentId: string | null;
  content: string;
  children: string[];
}

export class DB extends Dexie {
  noteNodes!: DexieCloudTable<NoteNode, "id">;

  constructor() {
    super("ElaterDBCloud1", {
      addons: [dexieCloud],
    });
    this.version(3).stores({
      noteNodes: "@id, parentId, *children",
      realms: "@realmId",
      members: "@id,[realmId+email]",
      roles: "[realmId+name]",
    });

    // Configure cloud:
    this.cloud.configure({
      databaseUrl: import.meta.env.VITE_DEXIE_URL!,
      requireAuth: false,
      customLoginGui: true,
    });
  }
}

const db = new DB();

export { db };

async function updateNoteTree(changes: ChangeSet, oldLength: number) {
    const updateQueue: {
      nodeId: string;
      content: string;
      children?: string[];
    }[] = [];

    const processNode = async (
      node: NoteNode,
      nodeStart: number,
      nodeEnd: number
    ): Promise<number> => {
      let nodeContent = node.content;
      let contentChanged = false;
      let newNodeEnd = nodeEnd;

      changes.iterChanges((fromA, toA, fromB, toB, inserted) => {
        if (toB <= nodeStart) return;
        if (fromB >= newNodeEnd) return;

        const changeStart = Math.max(fromB, nodeStart) - nodeStart;
        const changeEnd = Math.min(toA, nodeEnd) - nodeStart;
        const insertedContent = inserted.sliceString(
          Math.max(0, nodeStart - fromB),
          Math.min(inserted.length, newNodeEnd - fromB)
        );

        nodeContent =
          nodeContent.slice(0, changeStart) +
          insertedContent +
          nodeContent.slice(changeEnd);

        const delta = toB - fromB - (toA - fromA);
        newNodeEnd += delta;
        contentChanged = true;
      });

      if (contentChanged) {
        console.log("Node content changed", node.id, nodeContent);
        updateQueue.push({
          nodeId: node.id,
          content: nodeContent,
        });
      }

      let childStart = nodeStart;
      const newChildren: string[] = [];
      for (const childId of node.children) {
        const childNode = await db.noteNodes.get(childId);
        if (childNode) {
          const newChildEnd = await processNode(
            childNode,
            childStart,
            childStart + childNode.content.length
          );
          if (newChildEnd > childStart) {
            newChildren.push(childId);
            childStart = newChildEnd;
          }
        }
      }

      if (newChildren.length !== node.children.length) {
        updateQueue.push({
          nodeId: node.id,
          content: nodeContent,
          children: newChildren,
        });
      }

      return newNodeEnd;
    };

    const rootNode = await db.noteNodes.get(rootNodeId()!);
    if (!rootNode) return;

    await processNode(rootNode, 0, oldLength);

    // Batch update all changed nodes
    if (updateQueue.length > 0) {
      await db.transaction("rw", db.noteNodes, async () => {
        for (const update of updateQueue) {
          await db.noteNodes.update(update.nodeId, update);
        }
      });
    }

    setSyncState(LocalSyncState.Synced);
  }

What you're describing sounds nice with the existence of y-codemirror, so might still give that a shot one 1926 is done.

I'd say for now I definitely don't need string appeneds, so happy to close this. Would definitely appreciate any feedback on the update code above, I generated it with claude, but it seems to be working.

[scottrblock]

I am noticing some issues with my editor currently when QA/stress testing it, so I likely will switch to yjs shortly