Very cool that even at this scale the right vanilla SQL database just works. No fancy document store, map-reduce, or GPU implementations needed.

This blog post seems to blame GC heavily, but if you look back at their earlier blog post [0], it seems to be more shortcomings in either how they're using Cassandra or how Cassandra handles heavy deletes, or some combination:

"It was at that moment that it became obvious they deleted millions of messages using our API, leaving only 1 message in the channel. If you have been paying attention you might remember how Cassandra handles deletes using tombstones (mentioned in Eventual Consistency). When a user loaded this channel, even though there was only 1 message, Cassandra had to effectively scan millions of message tombstones (generating garbage faster than the JVM could collect it)."

And although the blog post talks about GC tuning, there's mention here [1] that they didn't do much tuning and were actually running on an old version of Cassandra (and presumably JVM) - having just switched over from CMS (!).

  0) https://discord.com/blog/how-discord-stores-billions-of-messages
  1) https://news.ycombinator.com/item?id=33136453

Needs (2023)

That services layer reminds be of a big, fancy, distributed Varnish Cache... they don't mention caching and they chose the word coalesce so I assume it doesn't do much actual caching. But made me think of Varnish's "grace mode" and it's use to prevent the thundering herd problem (which is where I first heard of 'request coalescing') https://varnish-cache.org/docs/6.1/users-guide/vcl-grace.htm...

Also love to see consistent hashing come up again and again. It's a great piece of duct tape that has proven useful in many similar situations. If you know where something should be then you know where everything is gonna come look for it!

Very well-written article. I'm happy for them that part of the solution was switching from Cassandra to drop-in replacement Scylla, rather than having to deal with something entirely different.

Discussed (a bit) at the time:

How Discord Stores Trillions of Messages - https://news.ycombinator.com/item?id=35048410 - March 2023 (10 comments)

Storing is one thing. Performing data mining on them is another

Having used discord in the past. Most of the conversations were just shit posts. Nothing serious. Why even bother storing a trillion messages of garbage in the first place?

My takeaway from this is maybe somewhat different from what the authors intended:

> The last one? Our friend, cassandra-messages. [...] To start with, it’s a big cluster. With trillions of messages and nearly 200 nodes, any migration was going to be an involved effort.

To me, that's a surprisingly small amount of nodes for message storage, given the size of discord. I had honestly expected a much more intricate architecture, engineered towards quick scalability, involving a lot more moving parts. I'm sure the complexity is higher than stated in the article, but it makes me wonder, given that I've been partially responsible for more than 200 physical nodes that did less, how much of modern cloud architecture is over engineered.

> The ScyllaDB team prioritized improvements and implemented performant reverse queries, removing the last database blocker in our migration plan.

I wonder how much they paid ScyllaDB to do this before even using ScyllaDB.

I see more people mixing up past and present tense randomly, as in this post. It’s confusing to read. Is the concept of tense starting to disappear entirely in US English, I wonder?

Fun article. Also fun to think about how many people have decided to document their crimes in these Cassandra nodes.

Anyone else reading this and being quite happy that they're not working at this scale?

Is there a fundamental reason you wouldn't use postgres for something like this? Scale certainly wouldn't be it.

My love of embedded stuff is growing. I'm self teaching C and assembly....to get better at low level programming and interactions with hardware but it all seems much simpler than the big data systems. Granted I'm sure it call be broken down into steps and issues to solve like any programming issue but I'm happy focusing on low level stuff for now.

Did they go with ScyllaDB just because it was compatible with Cassandra? Would it make sense to use a totally different solution altogether if they didn't start with that.

Until they don't, or they can't, and they need to start deleting.

(Not trying to undermine the engineering efforts, or the welcoming engineering blog posts though! I really think all these is needed)

Pretty fun read, even tho I'll never work at such scale lol

[flagged]

I think it’s annoying they interview engineers like they are Google and reading the blog they made it up and learned some basic “pitfalls” as they went along