Why providing non-compressed version at all? This is new protocol, no need for backwards compatibility. Dictionary may be baked into protocol itself, being fixed for specific version. E.g. protocol v1 uses that fixed v1 dictionary. Useful for replaying stored events on both sides

I gotta say, I am not very excited about "let's throw away all the security properties for performance!" (and also "CBOR is too hard!")

If everyone is on one server (remains to be seen), and all the bots blindly trust it because they are cheap and lazy, what the hell is the point?

Nice feat!

I wonder if a rewrite of this in C++ would even bump further the performance and optimise the overall system.

The "bring it all home" screenshot shows a CPU Utilization graph, and the units of measurements on the vertical axis appears to be milliseconds. Could someone help me understand what that measurement might be?

>Before this new surge in activity, the firehose would produce around 24 GB/day of traffic. After the surge, this volume jumped to over 232 GB/day!

>Jetstream is a streaming service that consumes an AT Proto com.atproto.sync.subscribeRepos stream and converts it into lightweight, friendly JSON.

So let me get this straight. if you did want to run Jetstream yourself you'd still need to be able to handle the 232 GB/day of bandwidth?

This always has been my issue with Bluesky/AT Protocol, For all the talk about their protocol being federated, It really doesn't seem realistic for anyone to run any of the infrastructure themselves. You're always going to be reliant on a big player that has the capital to keep everything running smoothly. At this point I don't really see how it's any different then being on any of the old centralized social media.

I thought this was going to be about NATS Jetstream, but it is not.

https://docs.nats.io/nats-concepts/jetstream

I thought this was going to be a strange read about the Hayes command set[1] at first glance.

1: https://en.m.wikipedia.org/wiki/Hayes_AT_command_set

I'm never not going to look for Hayes command set topics when people talk about BlueSky.

> Before this new surge in activity, the firehose would produce around 24 GB/day of traffic.

The firehose is all public data going into the network right?

Isn't that pretty tiny for a worldwide social network?

And the fact one country can cause a 10x surge in traffic also suggests its worldwide footprint must be tiny...

Given that “AT Protocol” already has a definition in IT that’s as old as OP’s grandma, what is this AT Protocol they are talking about here?

Introduce your jargon before expositing, please.

Was expecting Nats Jetstream but this is also cool

[flagged]

[flagged]

[flagged]

[flagged]

I'm just popping in here to say this (and BlueSky and atproto) are two of the coolest technical feats in today's tech world.

Server-Sent Events (SSE) with standard gzip compression could be a simpler solution -- or maybe I'm missing something about the websocket + zstd approach.

SSE Benefits: Standard HTTP protocol, Built-in gzip compression, Simpler client implementation

I find it baffling that the difference in cost of serving 41GB/day vs 232GB/day is worth spending any dev time on. We're talking about a whopping 21.4Mbps on average, which costs me roughly CAD$3.76/month in transit (and my transit costs are about to be cut in half for 2 x 10Gbps links thanks to contracts being up and the market being very competitive). 1 hour of dev time is upwards of 2 years of bandwidth usage at that rate.