Artificial Intelligence – A Guide For Thinking Humans
Introduction
Melanie Mitchell’s book “Artificial Intelligence – A Guide for Thinking Humans” is a primer on AI, its history, its applications, and where the author sees it going.
Ms. Mitchell is a scientist and AI researcher who takes a refreshingly skeptical view of the capabilities of today’s machine learning systems. “Artificial Intelligence” has a few technical sections but is written for a general audience. I recommend it for those looking to put the recent advances in AI in the context of the field’s history.
Key Points
“Artificial Intelligence” takes us on a tour of AI – from the mid-20th century, when AI research started in earnest, to the present day. She explains, in straightforward prose, how the different approaches to AI work, including Deep Learning and Machine Learning, based approaches to Natural Language Processing.
Much of the book covers how modern ML-based approaches to image recognition and natural language processing work “under the hood.” The chapters on AlphaZero and the approaches to game-playing AI are also well-written. I enjoyed these more technical sections, but they could be skimmed for those desiring a broad overview of these systems.
This book puts advances in neural networks and Deep Learning in the context of historical approaches to AI. The author argues that while machine learning systems are progressing rapidly, their success is still limited to narrow domains. Moreover, AI systems lack common sense and can be easily fooled by adversarial examples.
Ms. Mitchell’s thesis is that despite advances in machine learning algorithms, the availability of huge amounts of data, and ever-increasing computing power, we remain quite far away from “general purpose Artificial Intelligence.”
She explains the role that metaphor, analogy, and abstraction play in helping us make sense of the world and how what seems trivial can be impossible for AI models to figure out. She also describes the importance of us learning by observing and being present in the environment. While AI can be trained via games and simulation, their lack of embodiment may be a significant hurdle towards building a general-purpose intelligence.
The book explores the ethical and societal implications of AI and its impact on the workforce and economy.
What Is Missing?
“Artificial Intelligence” was published in 2019 – a couple of years before the explosion in interest in Deep Learning triggered due to ChatGPT and other Large Language Models (LLMs). So, this book does not cover the Transformer models and Attention mechanisms that make LLMs so effective. However, these models also suffer from the same brittleness and sensitivity to adversarial training data that Ms. Mitchell describes in her book.
Ms. Mitchell has written a recent paper covering large language models and can be viewed as an extension of “Artificial Intelligence.”
Conclusion
AI will significantly impact my career and those of my peers. Software Engineering, Product Management, and People Management are all “Knowledge Work.” And this field will see significant disruption as ML and AI-based approaches start showing up.
It is easy to get carried away with the hype and excitement. Ms. Mitchell, in her book, proves to be a friendly and rational guide to this massive field. While this book may not cover the most recent advances in the field, it still is a great introduction and primer to Artificial Intelligence. Some parts of the book will make you work, but I still strongly recommend it to those looking for a broader understanding of the field.
Big Tech has let go of thousands of workers in the last couple of months. In addition to the end of the era of cheap money and a broader economic slowdown, this story may have another angle.
This is the impact of AI and the possible closing of the “Productivity Gap.”
The Productivity Gap is a phenomenon where workers’ output, especially in developing countries, has been growing slower than expected. The shift to cloud computing and SaaS business models in the mid-2010s led to an explosion in both the valuations of technology companies and increases in the productivity of individual engineers and teams. A small startup could spin up and scale a business faster than ever.
Fast forward to the mid-2020s, and suddenly cloud computing is a commodity. Innovative Frameworks from the last decade, like React, Spring, and others, are bloated and complex.
For the last few years, companies like Meta, Alphabet, and Microsoft could hedge their bets and grow their teams because they were less likely to become disrupted by a small startup. Hoarding talent and doing “acqui-hires” was a feasible strategy.
Explaining the Tech Layoffs
Now there is once more a disruptive technology on the horizon. Generative AI Models are making giant leaps – a small team of ML-native programmers could build something that could blow incumbent services out of the water.
Alphabet’s panic over OpenAI’s ChatGPT is a case in point. Suddenly it doesn’t make sense to hoard talent to work on a platform that is about to be irrelevant.
AI-enabled software and infrastructure could close the productivity gap and fuel the rise of disruptive startups.
The incumbents are then cutting costs and preparing themselves for the next round of disruption by making significant investments in AI.
It no longer makes sense to hoard programmers when the entire industry could undergo a paradigm shift similar to that brought about by Cloud Computing 15 years ago.
The brutal layoffs we have seen in the last three months could be the result.
AI is having a moment. The emergence of Generative AI models showcased by ChatGPT, DALL-E, and others has caused much excitement and angst.
Will the children on ChatGPT take our jobs?
Will code generation tools like Github Copilot built on top of Large Language Models make software engineers as redundant as Telegraph Operators?
As we navigate this brave new world of AI, prompt engineering, and breathless hype, it is worth looking at these AI models’ capabilities and how they function.
Models like the ones ChatGPT uses are trained on massive amounts of data to act as prediction machines.
I.e., they can predict that “Apple” is more likely than “Astronaut” to occur in a sentence starting with: “I ate an.. “.
The only thing these models know is what is in their training data.
For example, GitHub Copilot will generate better Python or Java code than Haskell.
Why? Because there is way less open-source code available in Haskell than in Python.
If you ask ChatGPT to create the plot of a science fiction film involving AI, it defaults to the most predictable template.
“Rogue AI is bent on world domination until a group of plucky misfit scientists and tough soldiers stops it.”
Not quite HAL9000 or Marvin the Paranoid Android.
Why? Because this is the most common science fiction film plot.
Cats and Hats
Generative AI may generate infinite variations of a cat wearing a hat, but it has yet to be Dr. Suess.
AI is not going to make knowledge work obsolete. But, the focus will shift from Knowledge to Creativity and Problem-Solving.
Their goal is to “Democratize AI.” They have done so by open-sourcing the Stable Diffusion text-to-image model and are working on releasing other models, including large language models.
This approach is in stark contrast to the one taken by OpenAI, Facebook, Google, etc. These companies have gated access to ML models like GPT-3 via APIs or invite-only programs. The reasoning is that these models could be used to generate hateful text and images and are generally too dangerous to be released to the ignorant masses.
In a recent interview, Emad Mostaque, the CEO of Stability.Ai and a fascinating thinker, talks about the inevitability of generative and large language models leaking out to the wild. He wants to focus on giving people a framework for the ethical use of AI while giving them the tools to build and train models for their specific uses.
Stability.Ai has struck a deal with Eros Interactive to get access to their massive library of Indian content. Can you imagine what could be trained using that data?
Congratulations to Stability.Ai. I am curious about what this more open (or perhaps reckless?) approach to ML will bring us.
Image generated by Stable Diffusion – Prompt: “A happy robot drinking champagne at a cocktail party at night, oil painting, muted, candid, high resolution, trending on artstation”
For some, matrix multiplication may trigger memories of tedious high school algebra exercises. Last week, this humble mathematical operation was also the topic of a significant breakthrough in machine learning.
Art generated by Stable Diffusion
Background – Matrix Multiplication
Matrix multiplication is the foundation on which many core computational operations are built. Graphic processing, machine learning, computer gaming, etc. – all rely on matrix multiplication. At any given point in time, there are millions of computers doing (probably) billions of matrix multiplication operations. Making this humble operation faster would result in significant computational and efficiency gains.
Why do we want faster matrix multiplication?
Multiplying two matrices involves doing a large number of multiplication and addition operations. For example, multiplying a 4X5 and a 5X5 matrix involves 100 multiplication operations using the traditional matrix multiplication method that has been around since the early nineteenth century. In 1969, a mathematician, Volker Strassen, came up with an ingenious method that reduced the number of operations required by about 10%. This was hailed as a groundbreaking discovery in the world of mathematics.
DeepMind Enters the Arena
This brings us to DeepMind’s paper last week, where they used the AlphaTensor deep learning model to discover a new algorithm for matrix multiplication that is faster by about 10 – 20% than the Strassen method. This is a *colossal deal*! We are seeing a machine learning model find new algorithms to solve material, real-world problems. We have already seen DeepMind make groundbreaking discoveries in computational biology with AlphaFold. We now see applications of its Deep Learning models (based on playing games) to foundational aspects of the modern world. Exciting times are ahead!
TikTok* has caused political controversies, made Meta change its Instagram platform to mimic it, and caused many a moral panic. All signs of success.
TikTok’s use of machine learning to present a never-ending stream of engaging content is an example of the successful application of machine learning at a gargantuan scale.
But, as the linked WSJ article shows, TikTok’s growth is driven by massive investments in technology and advertising.
ByteDance, which owns TikTok, lost more than $7 billion from its operations in 2021 on $61.4b in revenues
The company spent $27.4b on user acquisition and $14.6b on R&D
I believe that the value of applied machine learning technologies will accrue to those companies that can deploy vast resources to acquire data (in TikTok’s case – users who generate the data) and build massive data and ML infrastructure. I am sure we will see similar revenue and spending trends if we analyze Meta and Google’s results.
While Data Science and Machine Learning careers grab the limelight, making ML platforms more efficient and processing data much cheaper will be more lucrative in the long term.
If a company spends significant cash on ML and data infrastructure, it will always look for people to make things more efficient. Possible careers for the future:
Natural Language Processing or NLP is a catch-all term for making sense of unstructured text-like data. Google search recommendations, chatbots, and grammar checkers are all forms of NLP. This is a field with many years of research. But, for the last 5-7 years, machine learning has reigned supreme.
Five years ago, machine learning approaches to NLP were labor intensive. Success meant having access to large amounts of clean and labeled training data that would train ML models. A text summarization model would be pretty different from one that did sentiment analysis.
The development of large language models or LLMs has revolutionized this field. Models like GPT-3 are a general-purpose tools that can be used to do several different tasks with very little training.
To show GPT-3 in action, I built a tiny slack bot that asks some questions and uses GPT-3 to generate actions. The video below is a demo of the bot and also an explanation of how to prompt GPT-3 to do NLP tasks.
The morning after a big software release can be both terrifying and exhilarating.
Kudos to the Ethereum team for pulling off a massively complex transition from Proof of Work to Proof of Stake. This reduces the energy consumption of the Ethereum blockchain by 99.95% (and global energy consumption by 0.2%).
I am still not convinced by the utility of the crypto ecosystem, and I am sure there will be bumps along the way. The transition to Proof of Stake will further entrench the power of those holding significant capital. Proof of Work meant influence aggregated to those who could deploy significant computing power by spending vast amounts of money on GPU hardware. The switch to Proof of Stake will remove the hardware intermediation layer. The massive savings in energy and speedups in transaction processing make it a worthwhile change.
Philosophical arguments aside, as a software engineer, I can appreciate a job well done 👏🏾👏🏾👏🏾.
I have played video games for thirty years. With two kids and a busy job, I don’t get as much time to play as I would like, but I pick up my Xbox controller whenever I get a chance. Over the last few months, this has meant playing Elden Ring, a role-playing game where you explore and adventure in a fantastical realm called the Lands Between.
Elden Ring – Bandai Namco
I am terrible at Elden Ring, yet I have spent hours playing it over the last six months. I am in awe of the game’s scale, beauty, and challenge.
When not playing video games, I support software development teams. Over the last ten years, I have worked as an Engineering Manager and, more recently, as a CTO at early-stage startups.
Managing and supporting teams is hard. You have to balance competing priorities and make decisions under conditions of ambiguity. Periods of stability can be interrupted by crises. It’s not that different from playing a game like Elden Ring!
As I reflected on why I enjoyed playing some video games more than others, I realized there are parallels between excellent video game design and supporting high-performance teams.
In this post, I explore what makes a video game great and what lessons we can apply from video game design to become better managers.
What Makes a Good Video Game?
Video game design is a vast and evolving topic. However, there are three critical elements to a good video game.
Good vs. Bad Video Design
The Story: What is this video game about, and why should I bother playing it?
A good video game story makes players want to invest their time in learning more about the world and the story. Games like Elden Ring, Horizon Zero Dawn, and God of War have stories that push players to do all sorts of side quests and missions. Exploring the world helps fill out the story, and each task moves the character and the story forward.
Hades – Supergiant Games
The Setting: Where am I going to be spending my time?
Seasoned gamers are familiar with the “one more turn” phenomenon. You want to keep playing because the game world is so darn fun. Dungeons filled with loot (and traps), exciting side missions, and beautiful scenery make the game’s exploration and progression fun. Games like the Mass Effect series make you care not just about the characters but also the broader game world and lore.
Gameplay Loop:How do I play the game?
Elden Ring is brutally difficult, yet I keep returning to the game. The reason is that while challenging, the gameplay is fair and predictable. And I get a real sense of accomplishment after clearing a particularly tricky dungeon or boss encounter. Hades is another game that has wonderfully compelling gameplay. Great video games have a simple yet addictive core gameplay loop. These are the actions that a player is expected to perform most often to make progress in the game. These must be balanced to avoid tedium while presenting fun and challenging experience.
From Video Games to High-Performance Teams
What do video games have to teach us about supporting high-performance teams?
We will approach this by looking at the same attributes that we explored for successful video games:
Story → Vision
Setting → Workplace
Gameplay → The Day-to-Day Work
Vision: Why am I being asked to do this?
A compelling narrative is about selling a vision – what will the player gain at the end of this quest line, boss battle, or challenging project? An honest, well-articulated vision helps give direction to a team. In his viral talk, “Start with Why,” Simon Sinek talks in detail about this “inside out” approach. Having a vision contradictory or inconsistent with the day-to-day work could lead to frustration and a lack of trust. The narrative must be straightforward and backed up with action aligned with the company’s stated values.
Workplace: Where do I spend my time?
A leader must create a workplace that maximizes productivity while allowing creativity, serendipity, and exploration. This is true both for in-person and remote work. Encouraging (reasonable) risk-taking and exploration enable more engaged and motivated teams. A video game with a predictable and tired setting (post-apocalyptic zombie infestations, for example..) is boring. Similarly, an environment that is dull or unpleasant is a drag on motivation and productivity. Psychological safety is also essential. As any player of online games knows, dealing with abuse and cheating makes for a miserable experience. A workplace perceived as hostile and a leader unwilling to support and protect their team will lead to people walking out of the door.
The Day to Day: How I do my work…
A manager must focus on the “gameplay” loop for their team. What are the challenges that stop them from doing their work? For software engineering teams, this could be the ease of making changes, getting pull requests approved, and getting changes into production. I have rage-quit lots of games because “life is too short.” Online games where I keep getting taken out by snarky teenagers with fast twitch reflexes are a particular bug-bear. Elden Ring can also veer into frustrating territory until I realized I could avoid most difficult encounters until I was leveled up and ready. When supporting a team, you need to consider what can be done to remove obstacles for your team. It may mean picking the right battles and knowing when to compromise. Making the workday loop engaging for your team is a critical leadership skill.
Conclusion – Gaming and Learning
Video games are the dominant entertainment and artistic form of our time. Oscar Wilde opined, “Life Imitates Art far more often than Art Imitates Life.” I agree.
Video games have been around far longer than modern software engineering tools such as Agile, DevOps, and other current paradigms. The art of video game design has been refined through decades of experimentation and many, many failures.
Indeed, as managers, most of us will be supporting teams that grew up playing video games. As a medium, video games create interactive, compelling worlds where people enjoy spending their time.
Taking cues from how video games are designed could help us become more effective supporters and advocates for our teams.
Managing and supporting a team is a difficult job. A manager is often a coach, disciplinarian, a surrogate parent, and cheerleader – all rolled into one. I am always on the lookout for ways to be a better supporter of my teams. Over the last few days I discovered a fortunate intersection in my interests in sport and in management.
I am a fan of the Arsenal football club. Like many other Arsenal fans, I have been watching and enjoying the Amazon Prime show “All or Nothing: Arsenal,” which follows Arsenal through the 2021 – 2022 season. We get a close look at how Arsenal’s manager Mike Arteta works with his players and his management team and motivates them over a challenging 45-game season.
At 38 years old, Arteta is currently the youngest manager in the English Premier league. He has been at the helm since 2019. The Arsenal squad also has the youngest average age in the Premier League – this season, the first team averages just 25.2 years old.
Arteta’s reign has seen the club slump to 8th place in the 2019 and 2020 seasons before having a marked improvement in form to finish 5th in 2021.
Arteta comes across as an intense, detail-oriented and hands-on manager. I realized that Arteta’s approach to management was something I had come across before. It is strikingly similar to that described in Radical Candor by Kim Scott – one of my favorite books on building high-performance teams.
In this post, I will summarize the Radical Candor approach through the lens of Arteta’s unique take on people management.
What is Radical Candor?
Radical Candor is a book by Kim Scott published in 2017. It focuses on creating a culture of guidance, building an effective and cohesive team, and driving results collaboratively.
The book’s central thesis is that effective leadership requires direct, clear, truthful, and kind feedback, even when difficult. Scott believes getting to know each person in your team personally is essential to understanding their desires and motivations.
The book offers tactical and strategic advice to leaders on building high-performing teams in an open, healthy, and productive environment. I strongly recommend Radical Candor for those looking for an authentic and modern approach to people management.
We see Arteta speaking candidly and passionately with his players throughout the season. He is generous in his feedback when things go well. When things go poorly, Arteta is direct, passionate, and emotional. While he doesn’t mince words, he doesn’t humiliate his players in the dressing room or in front of the media.
Caring Personally while Challenging Directly
The 2X2 below shows “Radical Candor” as giving feedback by caring personally while challenging directly. It also covers some dysfunctional ways of giving feedback – obnoxious aggression, ruinous empathy, and manipulative insincerity.
From Radical Candor by Kim Scott
Ted Lasso aside, football managers are not known for their empathy. Indeed, the likes of Sir Alex Ferguson are revered for their ability to drive performance through aggression and intimidation. Ferguson’s proverbial “hairdryer treatment” would probably end up in the “Obnoxious Aggression” quadrant above.
While he is partial to the odd F-bomb, Arteta’s open displays of emotion and vulnerability inspire his players, as seen in this clip. At the end of a run of poor results in April at Crystal Palace and Brighton, we see a manager who cares about the results and is passionate about wanting to make things better. He calls out a lack of intensity from his players and gives specific feedback on the training pitch and in the dressing room.
This combination of caring personally and directly challenging poor performance is right out of the Radical Candor playbook.
Building Resilience Through Trust
The Radical Candor approach is built on a foundation of trust. Trust is difficult to gain and easy to lose. The key to building trust is to be transparent and authentic, clear and concise in communication, and consistent in your actions.
Arteta calls out his “non-negotiables” in explaining his management philosophy: respect, commitment, and passion. Throughout the show, we see Arteta embodying these values.
This results in significant friction with his star player Pierre-Emerick Aubameyang who does not meet Arteta’s high expectations around discipline and accountability. Aubameyang is the club captain and is a popular member of the squad.
Arteta ends up stripping Aubameyang from the captaincy of the team. This could have destabilized the team, but it seems to have the opposite effect. Arteta does not criticize Aubameyang, and his team is made aware of how important trust and accountability are to their manager. By showing consistency in his actions and clarity in his communication, Arteta builds trust and resilience, resulting in outstanding results on the pitch in the second half of the season.
Managing Rockstars and Superstars
In Radical Candor, Scott describes Rockstars as stable employees who are happy and effective in their roles. These are folks who are aware of their talents and limitations and can consistently perform at a high level. On the other hand, Superstars are on a steep career trajectory and can be change agents. They are ambitious and want new opportunities. A high-performing team usually has both rockstars and superstars.
Given his young team, Arteta works with plenty of players on steep growth trajectories. Bukayo Saka, Emile Smith Rowe, and Eddie Nketiah are all young and eager to learn and perform at the highest level. However, he also has players like Rob Holding and Mohammad Elneny. While experienced pros, they have specific roles and are not guaranteed a place in the starting lineup. Holding and Elneny are the rocks (and Rockstars) that provide a stabilizing influence in the dressing room and on the pitch while laying a foundation for the more flamboyant players up front.
As a manager, Arteta has to ensure that the players like Holding and Elneny feel valued and are ready to perform when called upon while the ambition and talents of the young Gooners are nurtured. You can see this come together towards the end of the season. Holding and Elneny perform well after being called into the starting eleven after injuries. He also gives the ambitious Nketiah an extended run. He repays his faith by scoring five goals in the last seven games. Arteta and his team need to understand each player’s mentality and ensure they feel motivated to perform when needed.
Conclusion
All or Nothing is entertainment and has been edited to push a narrative and maximize engagement. Mikel Arteta has come under intense criticism for being uncompromising and stubborn at times – especially with how he has managed high-profile players like Aubameyang and Mesut Ozil. But, the little glimpse we get in the documentary shows a young manager trying to build a successful team.
Plenty of books like Radical Candor have come out of Silicon Valley, and the content often reflects the author’s experience working in technology companies. The strength of a book, especially in the crowded management genre, is how applicable the message is across different domains.
Managing Arsenal presents quite different challenges from managing a software engineering team. However, I hope the lessons of Radical Candor and All or Nothing are valuable to managers looking to build and support a high-performance team.
