What to do when your decentralized, scalable, performant blockchain turns out to be not so scalable, sort-of-centralized and not so performant?
Crypto’s selling point is robustness that is built on decentralization. No single points of failures should mean no downtime right? Right?
Turns out, crypto’s weaknesses are same as those of other, more mundane technologies. Bad code, bad actors and the fact that building scalable, distributed (and decentralized) systems is hard!
Solana, a Layer 1 blockchain, suffered a long outage over the weekend. This happened when the crypto markets are melting down..
Solana is supposed to be the answer to Ethereum’s performance and scalability issues. And yet, Solana has been plagued by performance issues and outages over the last few months.
This weekend’s issue was caused by “program cache exhaustion” due to “excessive duplicate transactions”. Solana developers released an emergency patch to resolve this issue and begged every validator to upgrade.
Where there is code, there are bugs..
Welcome to the brave new world, where the problems are the same as the ones in the old world. They just cost you a lot of funny money.
Roblox is one of the world’s biggest game platforms. With over fifty million daily users, it is a wildly popular platform to build and play games.
In October last year, they had an outage where the entire platform was down for over 72 hours. This was all over the news at the time..
Today, Roblox published a post mortem about the incident. It is fascinating reading for anyone interested in distributed systems, DevOps, and Engineering (link below). I will write up a more detailed note in a couple of days.
Summary – The outage was due to an issue in their service discovery infrastructure which is implemented in Consul – Roblox is deployed on-premise(!!) on 18,000 servers which run 170,000 service instances – These services rely on Consul (from HashiCorp) for service discovery and configuration – An upgrade to Consul and the resulting switch to the way services interact with Consul lead to a cascading set of failures resulting in the outage
Some Initial Thoughts – Distributed systems are hard, and the use of service-oriented architectures come with costs of coordination and service discovery – Microservice architectures do not reduce complexity, just move it up a layer of abstraction – The complexity of the modern software stack comes not just from your code, but also from your dependencies. – Leader election is one of the hardest problems in Computer Science 🙂
As engineers and designers, we need to focus on building products that have empathy and forgiveness for their users.
Software is eating the world, but as it optimizes for engagement and retention, it leaves behind confused and exhausted users.
Companies raise millions of dollars at billion-dollar valuations. With those valuations comes a drive to add new features. With the move to SaaS for everything, user interfaces and modes of interaction seem to change overnight.
Perhaps we could take inspiration from the consumer packaged goods industry.
As a new father, I have changed diapers in various circumstances. In the dark, in the park, trying to mitigate a full-on meltdown and sometimes just trying to stem an avalanche of 💩.
And yet, the diaper works as intended. Forgiveness is built into the design. I can operate it one-handed if I have to, and it gives some protection even when not used correctly. I can be confident that the design won’t change dramatically in the next iteration.
So, dear UX designer, next time you fire up Figma, think of the humble diaper, and a poor sleep-deprived dad dealing with a poop 🌋 at 3am.
Think of the mistakes a user may make and design your application to forgive them and not punish them when they make those mistakes when addled, distracted, or simply exhausted.
Benedict Evans, as usual, provides an insightful view of the “Digital Transformation” story. While crypto, machine learning, NFTs, and drones may generate the most headlines, we are also in the midst of a generational shift in how we do business. This shift is happening in boring corners of the B2B Enterprise software market but will have an impact bigger than some of the other, more alluring, technology trends.
Companies like UiPath (process automation) have successfully targeted the dull areas of enterprise software that are ripe for automation and streamlining. The software in “software eats the world” may include headline-grabbing items such as machine learning and distributed ledgers. But, much more significant changes are being brought about by the adoption of SaaS applications and workflows. Twenty years ago, you couldn’t add a new software application without going through procurement and IT. Now, all you need is a corporate credit card.
Demographic changes and shocks such as Covid are only accelerating the technology megatrends for which “Digital Transformation” is a catch-all term.
The concept of “generational shift” also works on multiple levels. Prolonged and painful migration projects can last for a generation (or longer). But we also have an entire generation of programmers and systems administrators who are now retiring, and you can’t find the talent to keep workhorse systems going.
I am thinking about the second-order consequences of this shift to a software-first world. There is going to be more efficiency, more competition, and a chance for aggressive upstarts to ride the technology wave and displace (rather than disrupt) less agile incumbents. But, we will also have a generational loss of knowledge that cannot really be replaced by software.
As every aspect of our economy is driven by software, we start seeing some of the characteristics of software show up. We have seen shortages of food, supply chain issues across industries as well as an incredible increase in ransomware attacks as overly optimized systems break under unexpected disruptions like the Covid pandemic, or insecure systems are targeted by malicious actors.
So, while I think this trend towards more Digital Transformation is good – in aggregate; there are also serious consequences that we may not being much attention to as we continue to be driven by software to optimize.
The team at Jeavio just completed our Campus hiring drive for 2021. We processed over 500 applications from talented, enthusiastic and hard working students looking to start their careers in 2022.
I looked through a number of resumes as part of our screening process. It was a surprising and (sometimes) frustrating process.
A consistent problem was students using the same (or very similar) single-page resume format. A good percentage of the resume also included personal details such as addresses etc. which are not really relevant at this stage of the recruitment process. I also have noted an explosion in folks including their CodeChef or HackerRank scores – again taking up crucial real estate that could have been used to try and differentiate themselves.
After many hours spent combing through hundreds of resumes – here are a few thoughts and tips for making an effective resume for someone just starting their career!
First impressions matter! Use a spelling and grammar checker to catch typos, misspelled words and poorly constructed sentences. Google Docs is free and does a fine job.
If you put a GitHub link in your resume, make sure it includes work you have done! Just a forked repo with no contributions is useless as a signal.
Your CodeChef or HackerRank score is not interesting. Most resumes I looked at this year had these scores and they don’t make your resume stand out.
A fancy resume layout is less important than the ability to write clearly. Typos in a beautifully formatted resume are an immediate disqualification from me.
Think about what makes you different. Interesting experiences, hobbies and an online portfolio is way more interesting than simply stating your GPA or your school projects.
Create a personal website – it’s easy! Use GitHub pages, Medium or any of a number of free services. Use the website to highlight your interests, passions, work and what you are looking for as you start your career. Use video, photos, writing – anything to make you stand out from the crowd!
Tom MacWright wrote a much discussedarticle on the state of modern web development a couple of weeks ago.
He states that the current default of building a Web Application as a SPA using React, or something similar, in the frontend and an API on the backend is overkill.
This is both an opinionated survey on the current state of web development and an extremely contrarian take on current accepted wisdom on how to build modern web applications. It’s worth a read.
His premise is that we are adding additional levels of abstraction such as Virtual DOMs, Server Side Rendering, Bundle Splitting, etc. He even goes on to say that trying to generic, purely REST-ful APIs is a bit like tilting at windmills, since we end up having them tightly coupled with frontend code anyway.
I am sympathetic to this point of view — mainly because I am rapidly transiting to “grumpy old man” phase of my career, and I find modern web development workflow terrifying complex. I had to work on a React application a couple of years ago. It made me want to run back into the warm embrace of Java Swing development (oh beautiful Java Swing).
The pure beauty of Java Swing
But, I also want to note that software engineering is pretty fashion driven where we bounce around between too much and too little abstraction. Every generation has this battle — Java didn’t have enough features so we ended up with Java EE, Java EE was too bloated so we ended up with Spring, the Java language was too complex so we built Go. Go is missing Generics so we will go ahead and add them. The pendulum keeps swinging and we have a complexity crusade every 5–7 years or so. Unsurprisingly, it has been just about 5 years since React has become the defacto way of building web applications. Time for the backlash!
I am sure we are going to be back at folks writing pure HTML and FTP-ing to an Apache server day now 😉
I love playing video games. My favourite game series is called Mass Effectmade by a game studio called BioWare.
I am a huge fan of BioWare’s games and I was looking forward to trying out their new game Anthem.
Anthem came out in early 2019 to universally poor reviews. The game was half finished, had poor gameplay and did not respect the player’s time. I ended up giving it a miss.
This week, Kotaku’s Jason Schreir has an excellent article about what went wrong with Anthem. It turns out Anthem’s development was troubled, and the development team faced the same sort of problems (some self-inflicted) that I have seen a few times in my career. I suggest you take some time to read the excellent article linked above. It is a wonderfully researched and well-written piece.
So what went wrong? Schreir’s article talks about game studio politics, financial pressures and many other contributed factors. I want to focus on what went wrong with Anthem as a software project. Here are some of my thoughts on this cautionary tale that I think may apply to many software development teams.
Building a product for a market you don’t understand
BioWare is famous for making role-playing games (also know as RPGs). The core team for Anthem had made well-loved games such as Mass Effect — single player, immersive, story-driven experiences. They were asked, by their owner EA, to build a multi-player, online, loot-driven shooter such as Destiny.
These games made money through micro-transactions; charging players small amounts of money to buy in-game items. The Anthem team did not understand the fundamentals of the product they were being asked to create. Basic mechanisms such as loot drops (where players get rare items for completing missions) did not work well. YouTube is full of players ranting about Anthem’s lack of loot.
Not being clear about the scope of the product
The Anthem lead team were slow in making decisions about critical features. Flying: a fundamental gameplay feature was added and removed multiple times. The team was only aware that flying was in scope following a demo to an EA executive who decreed that the game must allow it. This is like building a desktop app and not being sure whether mouse input should be enabled. It makes things difficult for the design and development teams.
Being forced to use a technology stack by decree
Most modern games are built using game engines such as Unity or Unreal Engine. EA mandated that all its games, including Anthem, must be built using the in-house Frostbite engine. They wanted to cut costs and re-use resources by not having to pay a licensing fee. Apparently, the Frostbite engine, while used successfully in games like the Battlefield series, was poorly documented and not suited to a game like Anthem. The development team struggled to make it work, and productivity suffered.
The same team that had made the Mass Effect series using the Unreal engine.
Shipping before being ready
A YouTube reviewer I follow mentioned that the game played as if the developers hadn’t even bothered playtesting their own game. The article talks about unstable builds and unavailable test environments. There was no time left for a thorough quality assurance process. When the game did come out, players complained about interminable load times and repetitive missions. There were also hard crashes and un-usable inventory systems: a big problem for a looter shooter.
In conclusion..
Anthem is a failure and has resulted in Bioware losing credibility as a game studio. I think it failed because it was a poorly managed software project. It was a poorly thought out product released in a broken state by a confused and stressed development team. This pattern is not unique just to the video game industry. I am sure software engineers of a certain vintage have seen this play out in a variety of industries.
I can’t open a newspaper or visit my friendly local bookstore without coming across a think piece about why AI is a *BIG DEAL* and how it changes everything. The tone of most of the material that I have come across is aptly summed up in this classic xkcd panel.
Classic xkcd panel on AI
In January 2019, I read Kai-Fu Lee’s fantastic book “AI Super-Powers: China, Silicon Valley, and The New World Order.” Mr. Lee is a thoughtful, even-handed guide to what is going on in the field of Artificial Intelligence (specifically Machine Learning) and how it may impact our future. The book is also an eye-opening account of the Chinese startup eco-system — but perhaps more on that another day.
Early in the book, Mr. Lee talks about how the spread of AI is happening in four waves. These waves are:
Internet AI
Business AI
Perception AI
Autonomous AI
Let’s take quick a look at each of these waves.
Internet AI
We deal with Internet AI every time we shop online, scroll through our social media feeds or Google something. From AI Superpowers:
Internet AI is mainly about using AI algorithms as recommendation engines: systems that learn our personal preferences and then serve up content hand-picked for us.
Examples of Internet AI include online advertising optimization, personalized news feeds, and algorithmic content recommendation.
Business AI
Advances in machine learning have allowed businesses to take advantage of labeled, structured data that resides in data repositories and train algorithms to outperform humans on clearly defined optimization tasks. Some examples here include automated credit scoring, fraud detection, algorithmic trading, and supply chain optimization. While not the most exciting topic, in the short term, Business AI has the potential to have a significant impact in the way we work and more potently, what *types of work* make sense to automate.
Business AI is about optimising and generating value from structured data.
Business AI has the potential to make what were once stable professions like accountancy, insurance, and medicine obsolete in their current form. It also has the potential to generate vast and lucrative new opportunities. More on this later.
Perception AI
Perception AI is about the “Digitisation of the physical world.” It is about using real-world data captured from IoT devices, cameras, smartphones, and other devices to blur the lines between the online and offline worlds. We already see applications of facial recognition and machine translation technology enhance offline experiences such as shopping and travel as well as enrich experiences such as education.
Perception AI is about blurring the lines between the online and offline world
Augmented reality (AR) devices and applications increase merging of the offline and online world. Perception AI also has worrying implications around surveillance, privacy and data protection.
Autonomous AI
Autonomous AI represents the culmination of the three preceding waves of AI. What was once science fiction is slowly becoming mundane. Autonomous AI is about fusing the ability to optimize from extremely complex datasets and integrate them with powerful sensory abilities resulting in machines that can understand and shape the world around them.
Autonomous AI results in machines that can understand and shape the world around them.
We already see some limited applications of Autonomous AI in the fields of self-driving cars, automated factories and pollinators.
What does it all mean?
Ben Evans, a partner at the storied VC firm Andreessen Horowitz, talks a little about the implications of advances in AI in the November 2018 presentation “The End of the Beginning”. He says:
“Tech is building different kinds of businesses, and so will take different shares of that opportunity, but more importantly change what those industries look like.“
He says further that a combination of high internet penetration, changing consumer expectations and a general “unbundling” of supply chains are creating business models that in turn are enabled and accelerated by AI. The breaking apart of tightly coupled logistics supply chains is just one example of this phenomenon.
At my work with Jeavio’s portfolio companies, I can already see this in action. We support entrepreneurs who are working in diverse fields such as customer experience analytics, construction and high tech agriculture. In each of these various fields, we see applications of Business AI that have the potential to disrupt existing models and generate tremendous value.
In my previous career working in high-frequency algorithmic trading, I have seen technology disrupt financial markets. Advances in AI are now doing the same in a wide variety of fields.
While AI cannot by itself generate new business models, it is already a potent force multiplier, which when deployed effectively, can increase efficiency and help businesses capture more value. We may not worry about our Robot Overlords just yet; we should keep an eye on the disruption and opportunities presented by the four waves of AI.
A contributor on GitHub finds an abandoned, but popular JS library and commits code that targets a Bitcoin wallet made by a particular company. Hundreds of other libraries use this library making this vulnerability affect thousands of applications since it is a transitive dependency.
NPM (and npmjs.com) provide a valuable service in hosting JavaScript dependencies. By blindly upgrading to latest version of libraries, developers can open themselves to malicious attacks similar to those described below.
I would recommend developers understand how npm’s package lock mechanism works. This will ensure that your dependencies are reproducible and force the use of known and trusted modules instead of downloading the latest version.
This is not a problem just with the JavaScript eco-system. Python (via pip or conda) and Java (via maven & gradle) have similar issues. However my, totally subjective and un-scientific, observation is that JavaScript libraries tend to have way more dependencies (see the “left-pad” debacle for example)..
In 2004, I graduated from University with a degree in Computer Science and a graduate job at a large investment bank. In the years since, I have worked as a technologist in some of the world’s biggest banks and as of 2017, at a startup in London.
I have had many titles: business analyst, project manager, team lead; but I have always written code.
Earlier this week, I came across this thread on Hacker News. It is an “Ask HN” thread: “What habits make a programmer great?”. Go ahead and read it, it has some great advice.
It got me thinking about what I could add to that discussion. I am not a great programmer, but I get things done and I have had many years of mistakes to learn from.
So here are my 2p..
You will get frustrated, and it’s OK
Our field worships productivity. We hear about amazing applications built over a weekend. There is always someone who gets things done faster.
The truth is that programming is hard, sometimes lonely and often frustrating. Some of us work long hours, face challenging deadlines and difficult clients. Getting stuff done can be a long and arduous grind.
Getting angry and frustrated is perfectly normal. However, if not managed properly, it will lead to burnout and disillusionment.
Find out what coping strategies works best for you. When I feel like things are getting a bit too much for me, I go for a walk. Others grab a coffee and vent to a friend, some even talk to rubber ducks..
Be kind to yourself. Programming is mentally taxing, and your effectiveness depends on a number of factors that may be outside your control. Some days will suck, and you will feel like you got nothing done. It’s OK.
Know and love your tools
Our field moves fast, and every day there is a new tool, package manager or framework that will solve all your problems and make you a 10X programmer. This is a trap.
Take IDEs for example. I use IntellIJ IDEA for Java, Python, and JavaScript development. It is probably the most complicated application on my computer. It has taken me years to just get to the point where I can navigate the interface, understand the capabilities and am productive using the IDE. If I switched to something else, my productivity would plummet immediately.
You have already sunk hours into learning the tools you are using currently. Think of the return on investment (ROI) for all the time you have spent before moving on. Learn and love the tools that you use, and strive for max ROI.
Tools and frameworks come and go, but time moves only in one direction.
Don’t dive right in
Go on, fire up that IDE, you know you want to..
I am biased towards action. Given a choice between just getting started on doing something (anything..) or spending a few minutes thinking and planning, I tend to choose action over contemplation. I see this same tendency in many other programmers.
Diving right in feels good, feels productive, however I have learnt that this is merely an illusion. This approach has let me to dead ends, wasted time and frustration.
When faced with a feature to build or a problem to solve, I ask myself these questions:
What is the goal?
Will doing this move things (the product, project or company) forward?
How does this problem relate to what I know already? Do I know enough?
Once I have processed the problem, I work on outlining a solution. This can be a doodle, a user story or sometimes just a bunch of comments in an IDE. This exercise makes writing code the easy part of the whole process.
“Start projects by measuring the current state of the world. This goes against our engineering instincts to start fixing things. But when you measure the baseline, you will actually know whether you are fixing things.”
Everyone comes across a codebase that is so convoluted, so messy, so gnarly that the first instinct is to just want to bin it and start all over again. In some cases, this may be the right thing to do, for example if your client is willing to pay for a feature that would not be possible to implement given the current codebase.
However, in most cases looking at ways to clean up the legacy codebase by refactoring it is the more productive approach. Modern IDEs and code coverage tools make it easy to refactor a large codebase. This combined with a pragmatic approach to unit testing will take you a long way towards quickly and safely delivering value.
Refactor vs. rewrite is not a binary choice. It is a continuum that is heavily biased towards refactoring.
I have and have been a team member of numerous projects that were ground up rewrites. Most were abject failures. We spent weeks (sometime months) delivering functionality that already existed. By the time we got to adding value, the projects was behind schedule and over budget.
Don’t ignore the context
Then there was the grim reaper of Git, and then there was the koda of Kotlin, and then we decided to write everything in Haskell..
Nobody wants to write terrible code. The messy codebase you inherited is the result of battles wons and lessons learnt. It may give you a headache, but there is always something to learn.
It is worth spending some time to understand what problems the codebase is trying to solve and why it ended up the way it did. In most cases, it is the legacy codebase that is being actively used by your customers and paying your salary.
By ignoring the “why” and focusing on “how”, we are doomed to repeat the mistakes of the past. If the codebase is messy and broken, try to understand what caused that situation and try not to repeat the same mistakes or adopt the same patterns.
In conclusion
I have been part of more projects that have been qualified successes and unqualified disasters than those that have been unqualified successes. I am a mediocre programmer, but I keep striving to learn from my mistakes and make small improvements everyday.
