"Brewer’s Conjecture and the Feasibility of Consistent, Available, Partition-Tolerant Web Services" (The formalization of CAP, which caused a ton of very poor trade-off thinking in the decade that followed by defining Availability in a very goofy way. Still a classic.)
"Practical Byzantine fault tolerance" (Moved the conversation on Byzantine faults forward significantly).
This is just a short selection. There's so much good stuff going back in the 70s and 80s distributed database literature, for example (and in the modern systems and DB literature too).
Lamport lists his collected works on his site[0] along with some commentary for each one, and directly mentions, The Maintenance of Duplicate Databases, as the origin for logical clock paper. Many of the notes have interesting commentary.
> Many computer scientists claim to have read it. But I have rarely encountered anyone who was aware that the paper said anything about state machines. People seem to think that it is about either the causality relation on events in a distributed system, or the distributed mutual exclusion problem. People have insisted that there is nothing about state machines in the paper. I've even had to go back and reread it to convince myself that I really did remember what I had written.[1]
I gradually realized that Lamport is more of the godfather of distributed system than Hinton is to deep learning. Lamport is less prominent than Shannon is to information theory. Shannon is the closest to any title of "gold-like" figure to a scientific discipline of universal relevance in modern society.
Lamport specifically revealed a philosophical connections between computer systems and physics, in the parallel of distributed consensus to relativity theory. To me, the enlightenment is that, the relations between events happening in a distributed system, is more fundamental than their absolute ordering, thus the central role of an "observer". I haven't really analyzed if this realization is from Lamport's papers, or my general reading and thinking, but I am moderately confident that general readings are heavily influenced by Lamport's papers, or can be traced back to be compatible with Lamport's thinking. I have not seriously study if this connection is valid in depth, which might be another pure amateur speculation of mine.
One thing I think Lamport falls short is that his writing is not easy to read and understand. I unconciously feel that Lamport (and Dario from Anthropic) probably share a hidden sense of intellectual supriority grew from their own experience throughout their career. So their writing (and Dario's gospel) all share a unchangable sense of narration from their own delicate and graceful ideas, much less of faciliating the understanding to their audience. In this cateogry, Shannon is abosolutely superior in any measure, in his writing, ideas are so naturally presented, although the implications of the ideas remain elusive due to the inherent depth.
Also, among the 3 prominent figures of modern AI: Hinton/Bengio are more like Shannon, Lecun is closer to Lamport.
Enough random rambling. Lamport, as indicated by the outweight presence in this list, is no doubt the single most important scientist in distributed systems.
> the relations between events happening in a distributed system, is more fundamental than their absolute ordering
The important thing in most distributed systems is having an order. Having a single observer serialize events as it receives them is so much more tractable than trying to use absolute order. Using absolute order requires very precise time synchronization which is hard; using absolute order requires knowing when you have received all the reports of events that already happened which is hard. Determining a designated observer isn't typically easy, but having it determine the order it observes events is easy. If two events happen at a similar time it's typically not a big deal which one is considered first as long as all nodes will agree on the result --- let the designated observer just pick the first one it sees works pretty well. If your report takes an unexpectedly long time to make it to the designated observer, then it won't be first and you'll deal.
Much better than trying to figure out unknowable questions of relativity. :P
Computer scientists are in denial about it, but CS is a branch of theoretical physics, not mathematics. You can point to this or that model of computation, such as lambda calculus or mu-recursive functions and try to claim its abstracted well beyond the particular laws of physics for some specific universe, but they all have some kind of rate limit built into them... and where does the motivation for this idea, that it takes something (time, space, work) to compute something ultimately come from? That's right - from underlying physics itself[1] - from the Bekenstein bound or Bremermann's limit or the like.
Even apparently non-physically-realizable models of computation like non-deterministic Turing machines are ultimately informed by and motivated by concepts in physics... otherwise they would just be examples of chmess[2] and of no interest to anyone. Computer science is of course somewhat abstracted from the details, but no more so than, say, thermodynamics, where concepts like entropy or Gibbs free energy can be studied in the abstract without reference to whether we are talking about a gas of non-interacting molecules or the spins of a bunch of electrons trapped in a lattice.
So, it's of no surprise whatsoever that the fundamental problems of distributed computing are ultimately the same as those found in the relativity of simultaneity[3]. You've all been studying the same things all along, just with different tools and at different levels of abstraction.
> One thing I think Lamport falls short is that his writing is not easy to read and understand
Interesting; I actually grew to be a fellow admirer of Lamport primarily because I actually found his papers to be a lot more approachable and relatively straightforward.
No disrespect meant to Mr. Armstrong, but it's possible it's never listed because it's basically a textbook. It's 295 pages. The rough average of all papers in OP is like 15 pages.
In distributed systems the fist lesson is resilience is more important.
Spend enough time in a computer lab, and you will see things halt and catch fire on occasion. Especially if it has a bunch of GPUs pinning the utilization 24/7, or a cheap power supply in the cluster. =3
It's the thing that baffles me the most with those people, the Knuths and Lamports and Carmarcks, and whatnot.
It's not like they have light schedules - they do research, they have classes, they attendconference, and they have jobs, etc...
And some benefit from tenure, which give them time to deeply focus on some topics.
But I can't imagine how I would write tex in ten years in Pascal on archaic machines if I had all the time to myself.
Doing that in the middle of understanding more computer science than my eyes would ever read ?
What do they do with their time ? What do they don't do ? Are they able to skip the eating and pooping and sleeping and cleaning their house and picking up parcels at the post office and grocery shopping parts ?
Or a proto Unix being written in three weeks. Concretely, what are the days of those three weeks made of ? Especially since it's famously when the spouse was NOT there to handle the housekeeping...
I see not amount of "life hacks" and "atomic habits" and whatnot helping with that. But it's probably just as natural to them as biking - and you can't explain biking, I guess ?
I implemented a library for rendezvous hashing with skeleton in order to get a similar performance as consistent hashing; it gets much more complicated and IMO loses some of the elegance once you add the skeleton. Most of the challenge is that the skeleton structure relies on a tree and so has to be rebalanced sometimes and this means that sometimes you have to move around more keys than ideal. The startegy you can use to perform the rebalancing can become very complicated.
Though the 'virtual nodes' in consistent hashing also add a lot of complexity there.
Obligatory shout out to Morris and Kaashoek's class at MIT - they update the reading list every year, and it's always great: https://pdos.csail.mit.edu/6.824/
This is not a bad list for sure. Here are some deeper cuts for those looking for something a bit less mainstream:
"The Maintenance of Duplicate Databases" https://datatracker.ietf.org/doc/html/rfc677 (AFAIK the genesis of the use of logical clocks in distributed systems).
"Chain Replication for Supporting High Throughput and Availability" https://www.usenix.org/legacy/event/osdi04/tech/full_papers/... (Chain replication is how a huge percentage of real-world cloud-scale data replication is done).
"Brewer’s Conjecture and the Feasibility of Consistent, Available, Partition-Tolerant Web Services" (The formalization of CAP, which caused a ton of very poor trade-off thinking in the decade that followed by defining Availability in a very goofy way. Still a classic.)
"Paxos Made Live" https://research.google/pubs/paxos-made-live-an-engineering-... (Brought a much-needed engineering perspective to a conversation that was largely theoretical up until this time.)
"Practical Byzantine fault tolerance" (Moved the conversation on Byzantine faults forward significantly).
This is just a short selection. There's so much good stuff going back in the 70s and 80s distributed database literature, for example (and in the modern systems and DB literature too).
Lamport lists his collected works on his site[0] along with some commentary for each one, and directly mentions, The Maintenance of Duplicate Databases, as the origin for logical clock paper. Many of the notes have interesting commentary.
> Many computer scientists claim to have read it. But I have rarely encountered anyone who was aware that the paper said anything about state machines. People seem to think that it is about either the causality relation on events in a distributed system, or the distributed mutual exclusion problem. People have insisted that there is nothing about state machines in the paper. I've even had to go back and reread it to convince myself that I really did remember what I had written.[1]
0: https://lamport.azurewebsites.net 1: https://lamport.azurewebsites.net/pubs/pubs.html#time-clocks
> Chain replication is how a huge percentage of real-world cloud-scale data replication is done
Is this still true today? I know that Facebook does [1], but I don't have references for any other places which use it.
[1] https://engineering.fb.com/2022/05/04/data-infrastructure/de...
Question from a recent CS grad here, who just learnt about CAP: what do you mean by “very poor trade-off thinking in the decade that followed”?
I am a Lamport admirer.
I gradually realized that Lamport is more of the godfather of distributed system than Hinton is to deep learning. Lamport is less prominent than Shannon is to information theory. Shannon is the closest to any title of "gold-like" figure to a scientific discipline of universal relevance in modern society.
Lamport specifically revealed a philosophical connections between computer systems and physics, in the parallel of distributed consensus to relativity theory. To me, the enlightenment is that, the relations between events happening in a distributed system, is more fundamental than their absolute ordering, thus the central role of an "observer". I haven't really analyzed if this realization is from Lamport's papers, or my general reading and thinking, but I am moderately confident that general readings are heavily influenced by Lamport's papers, or can be traced back to be compatible with Lamport's thinking. I have not seriously study if this connection is valid in depth, which might be another pure amateur speculation of mine.
One thing I think Lamport falls short is that his writing is not easy to read and understand. I unconciously feel that Lamport (and Dario from Anthropic) probably share a hidden sense of intellectual supriority grew from their own experience throughout their career. So their writing (and Dario's gospel) all share a unchangable sense of narration from their own delicate and graceful ideas, much less of faciliating the understanding to their audience. In this cateogry, Shannon is abosolutely superior in any measure, in his writing, ideas are so naturally presented, although the implications of the ideas remain elusive due to the inherent depth.
Also, among the 3 prominent figures of modern AI: Hinton/Bengio are more like Shannon, Lecun is closer to Lamport.
Enough random rambling. Lamport, as indicated by the outweight presence in this list, is no doubt the single most important scientist in distributed systems.
> the relations between events happening in a distributed system, is more fundamental than their absolute ordering
The important thing in most distributed systems is having an order. Having a single observer serialize events as it receives them is so much more tractable than trying to use absolute order. Using absolute order requires very precise time synchronization which is hard; using absolute order requires knowing when you have received all the reports of events that already happened which is hard. Determining a designated observer isn't typically easy, but having it determine the order it observes events is easy. If two events happen at a similar time it's typically not a big deal which one is considered first as long as all nodes will agree on the result --- let the designated observer just pick the first one it sees works pretty well. If your report takes an unexpectedly long time to make it to the designated observer, then it won't be first and you'll deal.
Much better than trying to figure out unknowable questions of relativity. :P
Hot take of the day:
Computer scientists are in denial about it, but CS is a branch of theoretical physics, not mathematics. You can point to this or that model of computation, such as lambda calculus or mu-recursive functions and try to claim its abstracted well beyond the particular laws of physics for some specific universe, but they all have some kind of rate limit built into them... and where does the motivation for this idea, that it takes something (time, space, work) to compute something ultimately come from? That's right - from underlying physics itself[1] - from the Bekenstein bound or Bremermann's limit or the like.
Even apparently non-physically-realizable models of computation like non-deterministic Turing machines are ultimately informed by and motivated by concepts in physics... otherwise they would just be examples of chmess[2] and of no interest to anyone. Computer science is of course somewhat abstracted from the details, but no more so than, say, thermodynamics, where concepts like entropy or Gibbs free energy can be studied in the abstract without reference to whether we are talking about a gas of non-interacting molecules or the spins of a bunch of electrons trapped in a lattice.
So, it's of no surprise whatsoever that the fundamental problems of distributed computing are ultimately the same as those found in the relativity of simultaneity[3]. You've all been studying the same things all along, just with different tools and at different levels of abstraction.
[1]: https://en.wikipedia.org/wiki/Limits_of_computation
[2]: https://link.springer.com/article/10.1007/s11245-006-0005-2
[3]: https://en.wikipedia.org/wiki/Relativity_of_simultaneity
> One thing I think Lamport falls short is that his writing is not easy to read and understand
Interesting; I actually grew to be a fellow admirer of Lamport primarily because I actually found his papers to be a lot more approachable and relatively straightforward.
Some other classics (applied dist systems):
- DDB: https://www.allthingsdistributed.com/files/amazon-dynamo-sos...
- MapReduce: https://www.cs.princeton.edu/courses/archive/fall13/cos518/p...
- Spark/RDDs: https://www.usenix.org/system/files/conference/nsdi12/nsdi12...
- BigTable: https://static.googleusercontent.com/media/research.google.c...
Worth mentioning that Dynamo (the classic 2007 paper) and DynamoDB (the modern AWS product) have fairly little in common architecturally.
For DynamoDB, check out https://www.usenix.org/conference/atc22/presentation/elhemal... https://www.usenix.org/conference/atc23/presentation/idziore... and my analysis of the differences here https://brooker.co.za/blog/2025/08/15/dynamo-dynamodb-dsql.h...
As always, lists like these don’t include Joe Armstrong's PhD thesis “Making reliable distributed systems in the presence of software errors” - http://erlang.org/download/armstrong_thesis_2003.pdf
No disrespect meant to Mr. Armstrong, but it's possible it's never listed because it's basically a textbook. It's 295 pages. The rough average of all papers in OP is like 15 pages.
Are there any other works that should also be included that you know of?
Does anyone use Elixir/Erlang anymore?
reliable? we all wish...
In distributed systems the fist lesson is resilience is more important.
Spend enough time in a computer lab, and you will see things halt and catch fire on occasion. Especially if it has a bunch of GPUs pinning the utilization 24/7, or a cheap power supply in the cluster. =3
Here is my list: https://muratbuffalo.blogspot.com/2021/02/foundational-distr...
Lamport is the author of more than half of these, it's crazy how influential he is in this space
In addition to making LaTex which has almost nothing to do with it
It's the thing that baffles me the most with those people, the Knuths and Lamports and Carmarcks, and whatnot.
It's not like they have light schedules - they do research, they have classes, they attendconference, and they have jobs, etc...
And some benefit from tenure, which give them time to deeply focus on some topics.
But I can't imagine how I would write tex in ten years in Pascal on archaic machines if I had all the time to myself.
Doing that in the middle of understanding more computer science than my eyes would ever read ?
What do they do with their time ? What do they don't do ? Are they able to skip the eating and pooping and sleeping and cleaning their house and picking up parcels at the post office and grocery shopping parts ?
Or a proto Unix being written in three weeks. Concretely, what are the days of those three weeks made of ? Especially since it's famously when the spouse was NOT there to handle the housekeeping...
I see not amount of "life hacks" and "atomic habits" and whatnot helping with that. But it's probably just as natural to them as biking - and you can't explain biking, I guess ?
Some of these probably aren't exactly classics but I do love:
- rendezvous hashing / consistent hashing
- Hybrid logical clocks
- "Don’t Settle for Eventual: Scalable Causal Consistency for Wide-Area Storage with COPS"
- "Scaling Replicated State Machines with Compartmentalization" (much more recent but really one of my favorite papers)
Rendezvous hashing is so simple and elegant.
I implemented a library for rendezvous hashing with skeleton in order to get a similar performance as consistent hashing; it gets much more complicated and IMO loses some of the elegance once you add the skeleton. Most of the challenge is that the skeleton structure relies on a tree and so has to be rebalanced sometimes and this means that sometimes you have to move around more keys than ideal. The startegy you can use to perform the rebalancing can become very complicated.
Though the 'virtual nodes' in consistent hashing also add a lot of complexity there.
I haven't updated this in quite a while, so probably more than a few broken links: my list of distributed systems reading lists.
https://gist.github.com/macintux/6227368
Obligatory shout out to Morris and Kaashoek's class at MIT - they update the reading list every year, and it's always great: https://pdos.csail.mit.edu/6.824/
I am going through this right now. I finished Lab 3. I also made my own notes about some of the papers they discuss, specifically Spanner.
https://sudk1896.github.io/2026/09/03/Spanner-notes.html
Highly recommend this course.
I want to work on coding for a service large enough to involve distributed systems someday. I keep studying the theory, but still...
Try doing the assignments from MIT 6.5840 Distributed Systems course by Morris and Kaashoek.