On crypto outages

What to do when your decentralized, scalable, performant blockchain turns out to be not so scalable, sort-of-centralized and not so performant? 

Crypto’s selling point is robustness that is built on decentralization. No single points of failures should mean no downtime right? Right?

Turns out, crypto’s weaknesses are same as those of other, more mundane technologies. Bad code, bad actors and the fact that building scalable, distributed (and decentralized) systems is hard!

Solana, a Layer 1 blockchain, suffered a long outage over the weekend. This happened when the crypto markets are melting down.. 

Solana is supposed to be the answer to Ethereum’s performance and scalability issues. And yet, Solana has been plagued by performance issues and outages over the last few months. 

This weekend’s issue was caused by “program cache exhaustion” due to “excessive duplicate transactions”. Solana developers released an emergency patch to resolve this issue and begged every validator to upgrade.

Where there is code, there are bugs.. 

Welcome to the brave new world, where the problems are the same as the ones in the old world. They just cost you a lot of funny money.

Explaining Bitcoin (through terrible science fiction)

On a Planet far away..

The planet of Isthar, in the outer reaches of the Delta Quadrant, was a dusty, arid, and sparsely populated world. Over the millennia, the original colonists split into tightly knit tribes and spread across the deserts and rocky canyons of the planet. What technology they had brought from their homeworld was soon forgotten in the face of the ongoing struggle to live in Isthar’s hostile environment.

When Isthar’s largest moon was full, the tribes would meet at an oasis to trade. A long history of bad blood, warfare, and mistrust had meant that these were often tense occasions. A fiercely independent set of people, they had little need for a central government or any sort of written laws. Trade meant barter without contracts, banks, or currency, with plenty of misunderstandings, violence, and theft.

During one of these gatherings, out of the western desert emerged a group of monks called themselves the Order of the Great Lizard. They claimed to have a way that would allow the tribes to trade peacefully without sacrificing their independence or indeed have to trust each other.

The monks produced a tablet made of stone. They called the tablet the Ledger. They claimed the Ledger would provide a tamper-proof way of recording what was traded and by whom. This would help reduce the sort of misunderstandings that often lead to bloodshed.

The Order of the Great Lizard had placed a spell on the Ledger that gave it some unique properties.

  1. Only a monk of their Order could write on the Ledger.
  2. Things once written could never be erased.
  3. While every monk could write to the Ledger, only one chosen monk could do so at a time.
  4. The chosen monk must have had a unique vision of the Great Lizard in a pose that a majority of other monks agreed was magnificent and one befitting the splendor of the Great Lizard.
  5. Once the monk wrote on the Ledger, the spell would cause a coin to appear. It was a token of benevolence from the Great Lizard, and out of respect, these were called Lizard Coins. The monk could do what they wished with their Lizard Coin.

The tribespeople were a little bemused. But the Order seemed harmless and willing to help, so they agreed to give this new way of trading a try.

The monks all sat in a large circle around the Ledger. Tribespeople would approach and shout out the trade they just had done.

“I Troopz of the tribe Gooners bartered my sand goat for a sack of salt and a bottle cap with Bubbles of the Hammers.”

Each monk would then contemplate the Great Lizard until the first one had a vision.

“I, Brother Chili, see the Great Lizard lying down with its right arm raised up.”

As long as more than half of the other monks agreed that it was indeed a splendid and unique vision, Brother Chili could record Troopz and Bubbles’ trade on the Ledger. He would then be rewarded with a Lizard Coin by the most magnificent and benevolent Great Lizard.

Once there was a record of Troopz bartering his sand goat to Bubbles on the Ledger, he would not be able to trade it to anyone else. Bubble’s ownership of the said sand goat was now backed by the Ledger.

And so, the Order of the Great Lizard made trade possible for the people of Isthar. There were some interesting consequences:

  • The visions could only include the Grand Lizard and nothing else. Hence it would become more challenging to develop new and unique visions as time passed. This meant the total number of Lizard Coins would also be limited.
  • Some folks tried bribing one or more monks to see if they could write false transactions on the Ledger, but this would not be possible without convincing a majority of the monks.
  • The Order of the Great Lizard was a genuinely open group. Anyone could join and become a monk provided they committed to contemplating the magnificence of the Great Lizard.
  • The monks used their coins to buy goods and services from the tribes. As the number of Lizard Coins increased, they became a useful and widely accepted currency on Isthar.
  • Very soon most transactions recorded on the ledger were transfers of Lizard Coin from one person to another.

Through their stone tablet and their magic spell, the Order of the Great Lizard had given the tribes a way to trade without having to trust each other. The Order was also open, so anyone could join provided they would spend their time and energy contemplating the magnificence of the Great Lizard.

The End.


Bitcoin Explained

Bitcoin was created to solve the problem of coordination in a trustless environment. This problem is also known as the Byzantine Generals problem.

In our story, the Order of the Great Lizard attempts to solve a similar problem for the tribes that did not have reason to trust one another.

The Ledger plays the same function as the blockchain in Bitcoin. It is an immutable (cannot be modified), append-only data structure. In a given period of time, only the first monk to come up with a unique, verified vision of the great lizard could write a new line to the ledger.

Miners play the same role in Bitcoin as the monks of the Order. The first miner to solve a complex mathematical puzzle that can be verified by a majority of other miners can write a new block to the Bitcoin chain. The monks get rewarded by a Lizard Coin and the miners get rewarded by a Bitcoin. This concept is also known as Proof of Work. The idea is to incentivize the miners to compete to solve puzzles to be able to generate and write a block to the Blockchain. The intrinsic value of Bitcoin comes from the energy expended in solving mathematical puzzles.

Transactions on the Bitcoin blockchain are limited to moving a fraction of Bitcoin from one address to another. On Ishtar, the transactions could be anything but eventually become mostly moving Lizard Coin from one person to another.

The Bitcoin protocol is represented by the spell cast on the Ledger. It provides the rules for recording transactions as well as rewards for the monks (miners) who are the fastest to have a unique vision of the Great Lizard. In Bitcoin, the magic spell is written in publically available and vetted code.

Our story also doesn’t cover the important roles that cryptography plays in Bitcoin. While we watched Troopz and Bubbles exchange a goat for salt and bottle caps, participants on the Bitcoin network can only be identified by the public key corresponding to their wallets. Hence, participants are pseudonymous.


But.. why bother?

Bitcoin and cryptocurrencies are complex. Skeptics often call them technologies looking for problems. My attempt at using terrible science fiction to explain how Bitcoin works is an attempt to explain how the technology works at a high level to folks who are not in the crypto ecosystem 24X7. I hope you were amused and somewhat enlightened. Drop me a line if you have any comments or questions!


Further Reading

The problem with Web 3.0 – It’s the Stories

We tell stories and learn by analogy. A good story maps the abstract to the concrete. For the story to function, it has to fall back to a base of shared understanding.

When learning a new technology, I try to map what I am learning to my mental model of the world. I tell myself how this new technology fits into the stories I know and try to imagine what other stories I would tell once I learn it.


I have been struggling with learning about cryptocurrencies and Web 3.0.

As someone looking to explore the nascent Web 3.0 developer landscape, I keep getting lost in layers, tokens, protocols, DApps, and DAOs. Applications (technically Protocols) like Aave have access to billions of dollars in liquidity but trying to understand how things work leads to a maze of smart contracts, oracles, and tokens interspersed with more familiar words like liquidity, interest, collateral, virtual machines, etc. Like trying to make sense of a world through a fogged-up window.

Besides me being a little slow, I think the reason is that the stories are terrible. There are plenty of grand visions of censorship-resistant platforms, the possibilities of generating life-changing wealth, but these are built on self-referential and confusing foundations. Turtles all the way down.

It is early days and the world of cryptocurrencies is still a frontier. This frontier is being explored by a rag-tag bunch of clever programmers, mathematicians, financial wizards, resourceful scammers, and brazen hustlers. 

But in order for a frontier to be settled, you need not just explorers but also settlers willing to uproot their lives to claim their 160 acres. You need developers to build mundane applications that solve mundane, but important problems. You need salespeople who can articulate the value proposition of building on this new frontier. You need good stories.

To stretch this tenuous analogy: we are now in the gold rush, but along with the gold rush you also need good weather and fertile land in order for the Wild West to turn into Sunny California.

Those passionate about the emerging world of cryptocurrencies and decentralized applications need to do a better job in bringing the rest of us plebs along. Otherwise, the gold rush will be over soon and all that will be left is a barren wilderness of abandoned protocols, orphaned DAOs, and blind oracles.


I am hopeful though. The infrastructure of Web 3.0 is still under construction. I hope strong, secure, and performant platforms emerge from the current Cambrian explosion of web technology. I am inspired by the likes of Chris Dixon, Balaji Srinivasan, and others who are bringing Web 3.0 concepts to wider audiences. But will it translate into wider developer adoption and mindshare? Time will tell.

Adventures in Web 3.0 #1 – Say My Name!

I bought my first NFT today – I am the proud owner of rushiluhar.eth. It was an interesting experience. Quite similar, in some ways, to creating my first website almost twenty years ago.

Some observations:

1. You need to know what you are doing 🤔.
The world of Web 3.0 is confusing and the UX .. leaves a lot to be desired. I used the most popular wallet – Metamask and bought rushiluhar.eth from ens.domains. None of these applications are for the faint hearted.

2. You need to be patient ⏳. 
I wanted to use the domain to point to my newly created Ethereum wallet address. This involved two separate steps. First buying the NFT (yes, each eth domain address is a NFT), and then another step to link the domain to my ETH wallet. Each step involves a transaction, and each transaction takes at least a minute to complete. And given the costs involved (see below), the lack of feedback or clarity is .. perplexing.

3. You need to be rich 🤑. 
ENS is run as a non profit, but you have to pay transaction fees. Which are crazy high, and change all the time. Buying a domain (like rushiluhar.eth) costs 0.001 ETH + the number of years you want to register the domain. So for 5 years, it costs 0.006 ETH – roughly $25 today. Not bad! *But* – the gas fees were (quoted at most) 0.043 ETH – almost $200! Making it my primary ENS domain involved another ~$85 in transaction fees. Ouch.

4. You need to be an exhibitionist 😬. 
The blockchain is public! Every single transaction is visible. If you want to laugh at me paying exorbitant gas fees for buying a worthless vanity NFT, just hop on over to Etherscan and search for rushiluhar.eth. 

5. You need to be slightly delusional 🤪.
The poster child of Web 3.0 (Ethereum) claims to become the World Computer. *But* current transaction fees put doing anything interesting on it out of reach of 99.9% of the world’s population. Something does not compute.. 

Also feel free to send a couple of ETH my way, you know the address..


Some Notes

Domain – You can buy your very own eth domain at ENS Domains but you will need a wallet, some crypto and a basic understanding of how ENS works and why you should bother.

WalletMetamask has integrations with a bunch of different Web3.0 sites like OpenSea, Foundation, ens.domains, etc. Its the easiest way to setup a wallet, just don’t forget your password.

Crypto – I would suggest using a reputed exchange like Coinbase to buy or sell crypto and then transfer a small amount to your Metamask wallet.

Why bother – A brief explainer of what else you can do with an ENS domain.

If you are interested in why I decided to spend a good bit of money on a ENS domain (apart from bragging rights on LinkedIn..) , check out this thread by Balaji Srinivasan.