Crypto and Transaction Costs

You live in the up-and-coming suburb of Cryptoville and you want to buy a house. It costs $1m. 

There might be some transaction fees involved, but you won’t actually know how much the fees will be until you complete the transaction. Oh, you are not competing with anyone to buy the house, it’s just a transaction fee. Can’t be too bad right? 

On the day of closing, the transaction goes through. The transaction fees are $250,000! And there was no way to tell until you tried to buy the house. It’s just the way things work in Cryptoville.. 

This is pretty much what happened on Saturday when Yuga Labs, the company behind the Bored Ape Yacht Club, held a much anticipated virtual land / NFT sale on the Ethereum network. Gas fees (i.e. transaction fees on Ethereum) spiked as the network coped with thousands of ApeCoin holders looking to buy some virtual land for their virtual Apes. 

The shocking thing was that it caused the entire Ethereum network to clog up – raising transaction costs for everyone – not just those looking to buy virtual land. Folks looking to buy NFTs valued at under a dollar were seeing transaction fees of $3,500! 

This points to a serious, and well-known, issue with throughput on Ethereum. It does not scale under load. Perhaps the long-delayed migration to Proof of Stake may change this – when it happens.

But – do you know what happened to the “high-performance” blockchain Solana on Saturday? You see where this going..

Links:
Ethereum Gas Prices Spike
Solana Performance Issues
Introduction to Ethereum Scaling

Footnote
Ethereum can only process about 15 transactions per second. It is just the way it is designed. However, miners can be incentivized to process transactions by increasing gas (transaction) fees. This is what happened on Saturday – as the demand to mint NFTs skyrocketed, so did the transaction fees. Gas fees have since come down, but it shows the big issues that Ethereum continues to face as it remains the de-facto standard for blockchain development.

Elon Musk & The Twitter Algorithm

I have been trying to avoid the whole Elon Musk / Twitter drama, but it has been challenging. I am ambivalent about whether Mr. Musk’s takeover of Twitter is a good or bad thing. My vibe is 🤷🏾‍♂️.

But, I do have an issue with one of Mr. Musk’s ideas: open-sourcing the Twitter algorithm to ensure there is no “bias.”

I think this is disingenuous, and Mr. Musk is playing to his (adoring) audience a little bit. 

It is improbable that there is the “one true algorithm” at Twitter. They probably use a combination of machine learning-based recommendation models with other systems such as entity and intent detection. Take a look at Twitter’s engineering blog to see how much ML drives recommendations on the social network.

So, if the intention is to look at the code and delete any (left-wing | right-wing) bias, things will be.. difficult. 

Now, a discussion should be had about how the ML models are trained and if there are any biases in the labeled datasets that are used to drive recommendations, detect abusive content, etc. This is a complex problem, however! 

An important effect of the pervasive deployment of ML technologies is that it makes computing *probabilistic* instead of *deterministic*. i.e., we know what is likely to happen, but it is difficult to predict what *will* happen.

This paradigm shift makes it very difficult to point the finger at one or more woke/radical/reactionary programmer who decides to censor or advocate for free speech. 

Mr. Musk knows all this, of course. The entire Tesla “full self-driving” stack is built on ML. So, perhaps, a little bit of intellectual honesty might lead to a more interesting discourse about bias.

Links:
Why Elon Musk Wants to Open Source Twitter

Elon Musk’s Poll on whether the Twitter “algorithm” should be open-sourced: https://twitter.com/elonmusk/status/1507041396242407424

Twitter Engineering Blog: https://blog.twitter.com/engineering/en_us

MIT Technology Review has a good writeup about this: https://www.technologyreview.com/2022/04/27/1051472/the-problems-with-elon-musks-plan-to-open-source-the-twitter-algorithm/

Between Rock and a.. podcast?

Just because you can do it doesn’t make it a great business model. Take music streaming, for example.

Image by Chloe Ridgeway on Unsplash

Spotify, the world’s most popular streaming service, has been the target of some Internet ire in the last week or so. Neil Young, the creator of the legendary Pono digital media player (apparently he made some music too?), decided he didn’t want anything to do with Spotify. 

Why all the righteous indignation?

Spotify pays Joe Rogan, a media personality / MMA commentator / master of “doing his own research,” over $100m to have exclusive rights to his wildly popular podcast. 

Apparently, Mr. Rogan has some interesting ideas around COVID, vaccinations, and horse de-worming medication. Not particularly controversial topics 😬. 

Why is this a big deal for Spotify?

Music streaming is a terrible business. Spotify has been bleeding cash for years and only recently turned a meager profit. The company had an operating margin of 1.4% in the first nine months of last year. No hockey sticks in sight.

The reason? It has to pay royalties to music labels for each music stream. The value from streaming accrues to the music companies, not to the streamers or artists.

Spotify makes its money not from streaming but from selling subscriptions and advertising. 

This is where podcasts come in. Spotify pays millions to Joe Rogan because he brings in a massive audience in the highly desirable 18-34 demographic. Spotify offers targeted advertising on podcasts to its most important customers, advertisers. This makes much more economic sense than making tiny margins on each stream of, let’s say, “Rockin’ in the Free World.” 

The risk to Spotify in this, slightly ridiculous, situation is not losing access to rock & roll; its not being able to monetize their investments in podcasting. 

Spotify would rather you come for the music and stay for Elon Musk smoking some fine herb  with his buddy Joe Rogan. 

They have set up expectations for their users that they can stream any song at any time. So they have to double down on more economically viable content like the Joe Rogan Experience. 

I am sure there is a Neil Young song about rocks and hard places..

On crypto outages

What to do when your decentralized, scalable, performant blockchain turns out to be not so scalable, sort-of-centralized and not so performant? 

Crypto’s selling point is robustness that is built on decentralization. No single points of failures should mean no downtime right? Right?

Turns out, crypto’s weaknesses are same as those of other, more mundane technologies. Bad code, bad actors and the fact that building scalable, distributed (and decentralized) systems is hard!

Solana, a Layer 1 blockchain, suffered a long outage over the weekend. This happened when the crypto markets are melting down.. 

Solana is supposed to be the answer to Ethereum’s performance and scalability issues. And yet, Solana has been plagued by performance issues and outages over the last few months. 

This weekend’s issue was caused by “program cache exhaustion” due to “excessive duplicate transactions”. Solana developers released an emergency patch to resolve this issue and begged every validator to upgrade.

Where there is code, there are bugs.. 

Welcome to the brave new world, where the problems are the same as the ones in the old world. They just cost you a lot of funny money.

On Roblox’s Outage

Roblox is one of the world’s biggest game platforms. With over fifty million daily users, it is a wildly popular platform to build and play games. 

In October last year, they had an outage where the entire platform was down for over 72 hours. This was all over the news at the time..

Today, Roblox published a post mortem about the incident. It is fascinating reading for anyone interested in distributed systems, DevOps, and Engineering (link below). I will write up a more detailed note in a couple of days.

Summary
– The outage was due to an issue in their service discovery infrastructure which is implemented in Consul
– Roblox is deployed on-premise(!!) on 18,000 servers which run 170,000 service instances
– These services rely on Consul (from HashiCorp) for service discovery and configuration
– An upgrade to Consul and the resulting switch to the way services interact with Consul lead to a cascading set of failures resulting in the outage

Some Initial Thoughts
– Distributed systems are hard, and the use of service-oriented architectures come with costs of coordination and service discovery
– Microservice architectures do not reduce complexity, just move it up a layer of abstraction
– The complexity of the modern software stack comes not just from your code, but also from your dependencies. 
– Leader election is one of the hardest problems in Computer Science 🙂 

On Forgiveness in UX Design

As engineers and designers, we need to focus on building products that have empathy and forgiveness for their users. 

Software is eating the world, but as it optimizes for engagement and retention, it leaves behind confused and exhausted users. 

Companies raise millions of dollars at billion-dollar valuations. With those valuations comes a drive to add new features. With the move to SaaS for everything, user interfaces and modes of interaction seem to change overnight.

Perhaps we could take inspiration from the consumer packaged goods industry. 

As a new father, I have changed diapers in various circumstances. In the dark, in the park, trying to mitigate a full-on meltdown and sometimes just trying to stem an avalanche of 💩. 

And yet, the diaper works as intended. Forgiveness is built into the design. I can operate it one-handed if I have to, and it gives some protection even when not used correctly. I can be confident that the design won’t change dramatically in the next iteration.

So, dear UX designer, next time you fire up Figma, think of the humble diaper, and a poor sleep-deprived dad dealing with a poop 🌋 at 3am. 

Think of the mistakes a user may make and design your application to forgive them and not punish them when they make those mistakes when addled, distracted, or simply exhausted.

Explaining Bitcoin (through terrible science fiction)

On a Planet far away..

The planet of Isthar, in the outer reaches of the Delta Quadrant, was a dusty, arid, and sparsely populated world. Over the millennia, the original colonists split into tightly knit tribes and spread across the deserts and rocky canyons of the planet. What technology they had brought from their homeworld was soon forgotten in the face of the ongoing struggle to live in Isthar’s hostile environment.

When Isthar’s largest moon was full, the tribes would meet at an oasis to trade. A long history of bad blood, warfare, and mistrust had meant that these were often tense occasions. A fiercely independent set of people, they had little need for a central government or any sort of written laws. Trade meant barter without contracts, banks, or currency, with plenty of misunderstandings, violence, and theft.

During one of these gatherings, out of the western desert emerged a group of monks called themselves the Order of the Great Lizard. They claimed to have a way that would allow the tribes to trade peacefully without sacrificing their independence or indeed have to trust each other.

The monks produced a tablet made of stone. They called the tablet the Ledger. They claimed the Ledger would provide a tamper-proof way of recording what was traded and by whom. This would help reduce the sort of misunderstandings that often lead to bloodshed.

The Order of the Great Lizard had placed a spell on the Ledger that gave it some unique properties.

  1. Only a monk of their Order could write on the Ledger.
  2. Things once written could never be erased.
  3. While every monk could write to the Ledger, only one chosen monk could do so at a time.
  4. The chosen monk must have had a unique vision of the Great Lizard in a pose that a majority of other monks agreed was magnificent and one befitting the splendor of the Great Lizard.
  5. Once the monk wrote on the Ledger, the spell would cause a coin to appear. It was a token of benevolence from the Great Lizard, and out of respect, these were called Lizard Coins. The monk could do what they wished with their Lizard Coin.

The tribespeople were a little bemused. But the Order seemed harmless and willing to help, so they agreed to give this new way of trading a try.

The monks all sat in a large circle around the Ledger. Tribespeople would approach and shout out the trade they just had done.

“I Troopz of the tribe Gooners bartered my sand goat for a sack of salt and a bottle cap with Bubbles of the Hammers.”

Each monk would then contemplate the Great Lizard until the first one had a vision.

“I, Brother Chili, see the Great Lizard lying down with its right arm raised up.”

As long as more than half of the other monks agreed that it was indeed a splendid and unique vision, Brother Chili could record Troopz and Bubbles’ trade on the Ledger. He would then be rewarded with a Lizard Coin by the most magnificent and benevolent Great Lizard.

Once there was a record of Troopz bartering his sand goat to Bubbles on the Ledger, he would not be able to trade it to anyone else. Bubble’s ownership of the said sand goat was now backed by the Ledger.

And so, the Order of the Great Lizard made trade possible for the people of Isthar. There were some interesting consequences:

  • The visions could only include the Grand Lizard and nothing else. Hence it would become more challenging to develop new and unique visions as time passed. This meant the total number of Lizard Coins would also be limited.
  • Some folks tried bribing one or more monks to see if they could write false transactions on the Ledger, but this would not be possible without convincing a majority of the monks.
  • The Order of the Great Lizard was a genuinely open group. Anyone could join and become a monk provided they committed to contemplating the magnificence of the Great Lizard.
  • The monks used their coins to buy goods and services from the tribes. As the number of Lizard Coins increased, they became a useful and widely accepted currency on Isthar.
  • Very soon most transactions recorded on the ledger were transfers of Lizard Coin from one person to another.

Through their stone tablet and their magic spell, the Order of the Great Lizard had given the tribes a way to trade without having to trust each other. The Order was also open, so anyone could join provided they would spend their time and energy contemplating the magnificence of the Great Lizard.

The End.


Bitcoin Explained

Bitcoin was created to solve the problem of coordination in a trustless environment. This problem is also known as the Byzantine Generals problem.

In our story, the Order of the Great Lizard attempts to solve a similar problem for the tribes that did not have reason to trust one another.

The Ledger plays the same function as the blockchain in Bitcoin. It is an immutable (cannot be modified), append-only data structure. In a given period of time, only the first monk to come up with a unique, verified vision of the great lizard could write a new line to the ledger.

Miners play the same role in Bitcoin as the monks of the Order. The first miner to solve a complex mathematical puzzle that can be verified by a majority of other miners can write a new block to the Bitcoin chain. The monks get rewarded by a Lizard Coin and the miners get rewarded by a Bitcoin. This concept is also known as Proof of Work. The idea is to incentivize the miners to compete to solve puzzles to be able to generate and write a block to the Blockchain. The intrinsic value of Bitcoin comes from the energy expended in solving mathematical puzzles.

Transactions on the Bitcoin blockchain are limited to moving a fraction of Bitcoin from one address to another. On Ishtar, the transactions could be anything but eventually become mostly moving Lizard Coin from one person to another.

The Bitcoin protocol is represented by the spell cast on the Ledger. It provides the rules for recording transactions as well as rewards for the monks (miners) who are the fastest to have a unique vision of the Great Lizard. In Bitcoin, the magic spell is written in publically available and vetted code.

Our story also doesn’t cover the important roles that cryptography plays in Bitcoin. While we watched Troopz and Bubbles exchange a goat for salt and bottle caps, participants on the Bitcoin network can only be identified by the public key corresponding to their wallets. Hence, participants are pseudonymous.


But.. why bother?

Bitcoin and cryptocurrencies are complex. Skeptics often call them technologies looking for problems. My attempt at using terrible science fiction to explain how Bitcoin works is an attempt to explain how the technology works at a high level to folks who are not in the crypto ecosystem 24X7. I hope you were amused and somewhat enlightened. Drop me a line if you have any comments or questions!


Further Reading

The problem with Web 3.0 – It’s the Stories

We tell stories and learn by analogy. A good story maps the abstract to the concrete. For the story to function, it has to fall back to a base of shared understanding.

When learning a new technology, I try to map what I am learning to my mental model of the world. I tell myself how this new technology fits into the stories I know and try to imagine what other stories I would tell once I learn it.


I have been struggling with learning about cryptocurrencies and Web 3.0.

As someone looking to explore the nascent Web 3.0 developer landscape, I keep getting lost in layers, tokens, protocols, DApps, and DAOs. Applications (technically Protocols) like Aave have access to billions of dollars in liquidity but trying to understand how things work leads to a maze of smart contracts, oracles, and tokens interspersed with more familiar words like liquidity, interest, collateral, virtual machines, etc. Like trying to make sense of a world through a fogged-up window.

Besides me being a little slow, I think the reason is that the stories are terrible. There are plenty of grand visions of censorship-resistant platforms, the possibilities of generating life-changing wealth, but these are built on self-referential and confusing foundations. Turtles all the way down.

It is early days and the world of cryptocurrencies is still a frontier. This frontier is being explored by a rag-tag bunch of clever programmers, mathematicians, financial wizards, resourceful scammers, and brazen hustlers. 

But in order for a frontier to be settled, you need not just explorers but also settlers willing to uproot their lives to claim their 160 acres. You need developers to build mundane applications that solve mundane, but important problems. You need salespeople who can articulate the value proposition of building on this new frontier. You need good stories.

To stretch this tenuous analogy: we are now in the gold rush, but along with the gold rush you also need good weather and fertile land in order for the Wild West to turn into Sunny California.

Those passionate about the emerging world of cryptocurrencies and decentralized applications need to do a better job in bringing the rest of us plebs along. Otherwise, the gold rush will be over soon and all that will be left is a barren wilderness of abandoned protocols, orphaned DAOs, and blind oracles.


I am hopeful though. The infrastructure of Web 3.0 is still under construction. I hope strong, secure, and performant platforms emerge from the current Cambrian explosion of web technology. I am inspired by the likes of Chris Dixon, Balaji Srinivasan, and others who are bringing Web 3.0 concepts to wider audiences. But will it translate into wider developer adoption and mindshare? Time will tell.

Adventures in Web 3.0 #1 – Say My Name!

I bought my first NFT today – I am the proud owner of rushiluhar.eth. It was an interesting experience. Quite similar, in some ways, to creating my first website almost twenty years ago.

Some observations:

1. You need to know what you are doing 🤔.
The world of Web 3.0 is confusing and the UX .. leaves a lot to be desired. I used the most popular wallet – Metamask and bought rushiluhar.eth from ens.domains. None of these applications are for the faint hearted.

2. You need to be patient ⏳. 
I wanted to use the domain to point to my newly created Ethereum wallet address. This involved two separate steps. First buying the NFT (yes, each eth domain address is a NFT), and then another step to link the domain to my ETH wallet. Each step involves a transaction, and each transaction takes at least a minute to complete. And given the costs involved (see below), the lack of feedback or clarity is .. perplexing.

3. You need to be rich 🤑. 
ENS is run as a non profit, but you have to pay transaction fees. Which are crazy high, and change all the time. Buying a domain (like rushiluhar.eth) costs 0.001 ETH + the number of years you want to register the domain. So for 5 years, it costs 0.006 ETH – roughly $25 today. Not bad! *But* – the gas fees were (quoted at most) 0.043 ETH – almost $200! Making it my primary ENS domain involved another ~$85 in transaction fees. Ouch.

4. You need to be an exhibitionist 😬. 
The blockchain is public! Every single transaction is visible. If you want to laugh at me paying exorbitant gas fees for buying a worthless vanity NFT, just hop on over to Etherscan and search for rushiluhar.eth. 

5. You need to be slightly delusional 🤪.
The poster child of Web 3.0 (Ethereum) claims to become the World Computer. *But* current transaction fees put doing anything interesting on it out of reach of 99.9% of the world’s population. Something does not compute.. 

Also feel free to send a couple of ETH my way, you know the address..


Some Notes

Domain – You can buy your very own eth domain at ENS Domains but you will need a wallet, some crypto and a basic understanding of how ENS works and why you should bother.

WalletMetamask has integrations with a bunch of different Web3.0 sites like OpenSea, Foundation, ens.domains, etc. Its the easiest way to setup a wallet, just don’t forget your password.

Crypto – I would suggest using a reputed exchange like Coinbase to buy or sell crypto and then transfer a small amount to your Metamask wallet.

Why bother – A brief explainer of what else you can do with an ENS domain.

If you are interested in why I decided to spend a good bit of money on a ENS domain (apart from bragging rights on LinkedIn..) , check out this thread by Balaji Srinivasan.

Obesity, Second-Order Consequences & ..Molds?

I came across this series of posts on the root causes of Obesity by Slime Mold Time Mold, a delightfully weird pseudonym for the team behind the posts. The posts are long, well researched, and, despite the weighty content (haha), quite fun to read.

The blog series is not complete, and I am fascinated to see what comes next.


The writers start with the observation that we have seen a startling increase in obesity rates from 1980 to the present day as seen in the animation below from Our World in Data.

As of 1980, around 11% of the population of the Americas was obese. In 2016, this went up to a staggering 28%. What has caused this?

The team at Slime Mold Time Mold (SMTM) reviews research and data spanning the last 150 years. From studies done on the BMI of Civil War veterans to looking at rates of obesity in Macaques – they cover an incredibly wide range of material. Their thesis is that 1980 was an inflection point – after 1980 we see soaring rates of obesity across the industrialized world.

Strangely, this increase does not seem to be driven by a big change in the total number of calories consumed or in the way that we live our lives. Indeed, the diets in the early 20th century were more calorific and unhealthy than present-day diets.

Just think of all the big dinners on Downton Abbey.

Carbs, glorious carbs..

We eat fewer carbs today than we did a hundred years ago. While the research does show an increase in the total number of calories consumed, the data does not support the increase in obesity rates being driven by an increase in calorific consumption.

SMTM’s thesis is that environmental contamination is an important driver of the increase in obesity rates since 1980.

They look at a number of different culprits – Lithium, a group of chemicals called PFAs as well as the presence of antibiotics in livestock.


If indeed environmental contamination is a primary driver of the obesity crisis, we are looking at a public health scandal that will be bigger than smoking or lead in gasoline. The jury is still out, but SMTM’s work seems to point to a strong correlation.

It also makes me think, again, that we are very poor at thinking through the second-order consequences of our actions. I think we are tinkerers by nature and are biased to action. This has helped us make a lot of progress in a short time – but we are also continuously fixing (or abandoning) things that we have broken along the way.

Environmental contamination and obesity are just another of a variety of different examples of “progress” messing things up. Maybe the research will identify a smoking gun and the problem will be regulated away.

We are strikingly poor at figuring out how complex systems operate and how they may trigger feedback loops. Global warming, disinformation on social media, and many other modern ailments can be tracked to us not being able to think through the consequences of our actions.


So what is the solution? One way is a monastic retreat to the wilderness. But when even remote Alaska is contaminated with PFAS chemicals, retreat does not seem to be a viable option. Maybe we need to push regulatory bodies for stricter enforcement of laws and punitive measures for those who do not comply. But the revolving door between big government and big chem does not fill me with confidence that this is an avenue that holds much hope.

The one, teeny-tiny, ray of hope comes from us being able to deploy massive computational power to model and simulate the world a little better. Perhaps we could get better at figuring out how complex systems interact and that might help towards a more thoughtful and considered approach to change?


One can live in hope. Until then, I wait for my next SMTM fix.


Footnote

The work by SMTM is worth reading (and following) in full. It is a great example of just some “random people on the Internet” using open data to think through and attempt to answer difficult questions.

Reading SMTM reminded me of the early, pre-social media engagement-driven, Internet. I remember stumbling upon blogs such Naked Capitalism (Still going strong), The Epicurean Dealmaker (RIP), and Slate Star Codex (also RIP) and enjoying reading about topics that were outside my areas of work or of study.

The death of Google Reader and the rise of Social Media has made the Internet a much less surprising place.