Hey HN,

For the past few months, as part of my job at Supabase, I have been working on pg_replicate. pg_replicate lets you very easily build applications which can copy data (full table copies and cdc) from Postgres to any other data system. Around six months back I was figuring out what can be built by tailing Postgres' WAL. pg_replicate grew organically out of that effort. Many similar tools, like Debezium, exist already which do a great job, but pg_replicate is much simpler and focussed only on Postgres. Rust was used in the project because I am most comfortable with it. pg_replicate abstracts over the Postgres logical replication protocol[0] and lets you work with higher level concepts. There are three main concepts to understand pg_replicate: source, sink and pipeline.

1/ A source is a Postgres db from which data is to be copied. 2/ A sink is a data system into which data will be copied. 3/ A pipeline connects a source to a sink.

Currently pg_replicate supports BigQuery, DuckDb local file and, MotherDuck as sinks. More sinks will be added in future. To support a new data system, you just need to implement the BatchSink trait (older Sink trait will be deprecated soon).

pg_replicate is still under heavy development and is a little thin on documentation. Performance is another area which hasn't received much attention. We are releasing this to get feedback from the community and are still evaluating how (or if) we can integrate it with the Supabase platform. Comments and feedback are welcome.

[0] Postgres logical replication protocol: [https://www.postgresql.org/docs/current/protocol-logical-rep...)

Postgres + Rust is one of the most exciting intersections of tech I've seen in a while.

There's external tooling like his project, but postgres extensions in Rust are exciting.

Full extensions via pgrx have been cool to see, but plrust + pg_tle is also starting to show up.

If you aren't familiar with TLE (Trusted Language Extensions), it is a postgres extension from AWS that created some privileged interfaces for procedural languages (used for user-defined functions) to do some extra stuff. Right now it's mostly auth-related hooks but my hope is that it expands in the future.

Plrust is a procedural language extension for Rust, allowing user defined functions written in Rust.

The combination of those two could open up a world of rich extensions usable in managed hosted environments like RDS.

This is super timely.

Windmill (https://windmill.dev) used to only support webhooks to trigger code and flow jobs. We have just added email support building our own MX server, and wanted to add CDC change. We were gonna do it on Debezium but this will allow us to remove the need for a third-party service and just add this as a crate. Thank you supabase for open-sourcing this.

I would love to see this become the [Litestream](https://litestream.io/) equivalent to SQLite: a very cheap option for simple deployments (ones with a single database) to get continuous backups in a cheap storage from which one can restore when needed.

My specific use-case would be: Single Postgres in my cluster, replicated via something based on pg_replicate running as a side-car and writing to my NAS running Minio.

I was trying out the stdout example. Could not get it to log anything. DuckDB example worked so I went digging into the source. Apparently the stdout sink is using tracing and I did not have a `RUST_LOG` env var set.

Might be a good idea to have it documented or have the default level set to info for the stdout example.

Maybe this is common Rust knowledge and I just don't know what I'm doing though.

Nice! I'm one of the authors of pg-bifrost which is in the same space. Have you thought about / have solved sharding consumption across multiple slots / multi consumers to increase throughput? This is on my radar but not something I've investigated yet.

The issue we've ran into is some team at work decides to re-write an entire table and things get backed up until they stop updating rows.

This is great! We've been using PostgREST along with a PostgreSQL-based queue to handle side-effects like sending webhooks after database operations (inserts/updates/deletes). The queue feeds into a node server that processes these tasks. However, this setup is becoming a performance bottleneck as we scale.

I'm exploring an alternative way to run logic asynchronously after db operations without the overhead, and I think using cdc to export jobs into an external queue is the way to go here. Essentially a lightweight alternative to Debezium with a better developer experience that is easier to manage. This crate could serve as the core of such a service.

Do you do anything special with toast not changed values, this always seemed problematic to me with doing cdc as either you have to maintain the latest value in your own state or hit the read connection to pull the value

A SQLite3 sink would be very nice. Granted, there are impedance mismatches between PG and SQLite3, but the onus for making the schemas compatible should be on the user.

Wow I think i build one of the first pg-rust-cdc replication module for postgresql and now it seems mainstream. It was a part of my https://github.com/posix4e/rpgffi project, which is garbage code I'm now embarrassed about. Rust programming has come a long way and we didn't have the best practices. Now it seems actual grownup engineering teams are living the dream. We truly live in the best of all universes.

Very cool! Thanks for sharing. I am hacking on something similar too and I think there are a lot of interesting use cases to unlock with this model

Wow, Rust meets SQL! Excited to see seamless Postgres replication with pg_replicate.

This is awesome! Rust + Postgres = SQL powerhouse! Great tool for robust replication apps