Skip to content

Commit

Permalink
added PostgreSQL Caveats (#518)
Browse files Browse the repository at this point in the history
  • Loading branch information
roblaszczak authored Nov 23, 2024
1 parent a0cfc22 commit 15d765b
Showing 1 changed file with 21 additions and 4 deletions.
25 changes: 21 additions & 4 deletions docs/content/pubsubs/sql.md
Original file line number Diff line number Diff line change
@@ -1,15 +1,13 @@
+++
title = "SQL"
title = "SQL (PostgreSQL, MySQL)"
description = "Pub/Sub based on MySQL or PostgreSQL."
date = 2019-07-06T22:30:00+02:00
bref = "Pub/Sub based on MySQL or PostgreSQL."
weight = 120
+++

SQL Pub/Sub executes queries on any SQL database, using it like a messaging system. At the moment, **MySQL** and **PostgreSQL** are supported.

While the performance of this approach isn't the best, it fits many use cases, where eventual consistency is acceptable.
It can also be useful for projects that are not using any specialized message queue at the moment, but have access to a SQL database.
It be useful for projects that are not using any specialized message queue at the moment, but have access to a SQL database.

The SQL subscriber runs a `SELECT` query within short periods, remembering the position of the last record. If it finds
any new records, they are returned. One handy use case is consuming events from a database table, that can be later published
Expand Down Expand Up @@ -122,3 +120,22 @@ Currently, this schema is supported only for PostgreSQL.
{{% load-snippet-partial file="src-link/watermill-sql/pkg/sql/queue_schema_adapter_postgresql.go" first_line_contains="// PostgreSQLQueueSchema" last_line_contains="}" %}}

{{% load-snippet-partial file="src-link/watermill-sql/pkg/sql/queue_offsets_adapter_postgresql.go" first_line_contains="// PostgreSQLQueueOffsetsAdapter" last_line_contains="}" %}}

## Caveats

### Using last processed transaction ID in PostgreSQL to ensure no messages are lost

In some cases, PostgreSQL `SERIAL` is not incremental.
The `SERIAL` value is generated while the transaction is in progress, not when it is committed.
If transactions are committed in a different order than they were started, message offsets based on `SERIAL` values will not be incremental.

To keep storing acknowledgment information efficient, Watermill keeps only the last message's acknowledgment information.
To ensure no messages are missed when a message order is not kept, Watermill also uses the transaction ID to ensure no message is lost.
For more details, see [Watermill#311](https://github.com/ThreeDotsLabs/watermill/issues/311).

It is important to note that very long-running transactions may result in delayed message delivery.
For instance, if a transaction is running for an hour, no messages will be delivered until the transaction is committed.
While we do not recommend the use of such long transactions, **if they are necessary, we advise the use of the [Queue schema adapter](#queue), which does not depend on the transaction ID.**
You have nothing to worry about if you don't have such long transactions.

If you are migrating your data to a new database, you may need to set `last_processed_transaction_id` in your offsets table.

0 comments on commit 15d765b

Please sign in to comment.