Context
Introduction
We approach product development through a foundational set of assumptions. Chief among these is the Assumption of Knowledge.
In this paradigm, we assume that the business or the customer inherently knows what they want, and the product developers inherently know how to build it. This mindset leads us to optimize large development groups where individuals and teams are siloed by expertise, tasked simply with delivering what the customer requires. When this system falters, we seek solutions in structure: we refine processes, establish rigid roles, and draw clear boundaries. We focus on ensuring people perform within the limits of their current abilities. If a business unit or a developer fails to meet expectations, the remedy is simple: provide more education or replace the individual.
Let us call this The Grown-Ups Company. Their primary metric of success is efficiency in delivery.
Conversely, a different set of assumptions has emerged in the modern landscape. These are the foundations of The Kids Company. In this environment, the starting point is an admission of uncertainty: the customer likely does not know exactly what they need, and the developers do not yet know how to solve the problem.
The defining trait of the Kids Company is intellectual humility. While the Grown-Ups Company views a lack of answers as a failure of the individual, the Kids Company accepts it as a baseline reality. Because they acknowledge this gap in knowledge, they organize their development differently. They prioritize learning—and, more importantly, they recognize the staggering cost of not learning.
In the Kids Company, learning is not an interruption to the work; learning is the work. When a developer encounters an unfamiliar codebase or a new technology, she doesn't see it as an obstacle to her "real" job—she sees mastering it as the job itself. The customer’s role is to learn alongside the developer, discovering through experimentation what truly provides value. In this world, complex jargon like "cognitive load" is set aside in favor of a simple truth: the learning process is only complete when the problem is solved.
This book belongs to a broader body of knowledge—a collection of modern product development philosophies that are built entirely upon this second assumption.
Large product development groups typically suffer from unnecessary organizational and technical complexity. Software teams feel the consequences more directly, with an ever-increasing number of often low-quality software products delivered under time pressure that they must support and maintain—not to mention coordination with management, stakeholders, and others in endless meetings. This book contains several suggestions and directions for understanding and addressing this very systemic problem of constant overwhelming pressure.
But, before discussing solutions to the problem of overwhelmed teams (perhaps a better term than “team cognitive load”), we first need to address the issue with using jargon and acronyms. It will become clear why.
Software development industry tends to (re)invent jargon and acronyms continuously. Although it is a necessary part of our professional language, jargon can also create an illusion of something being special, unique, or new. A significant advantage of such illusion is marketing. It becomes easier to get attention for a concept, solution, or methodology. The term “microservices” is an example. The architectural concept has become very popular over the last 10+ years, but what we define as microservices architecture is part of normal evolution and improvement in (distributed) software architecture and has existed for much longer. Hence, there is almost nothing new here. The harmful part of such "rebranding" is the ignorance towards the existing knowledge from the past; before the term "microservices" was coined.
A more significant issue with jargon is when it misrepresents a valuable practice. An example is the “CI server”. Continuous integration (CI) is an important practice that was popularized by XP and other lightweight approaches long ago. (Here is a definition: http://www.extremeprogramming.org/rules/integrateoften.html). The practice doesn’t involve a server, although having one in addition to the practice itself is helpful. Today, many assume they practice CI when they only run an automated pipeline—building code, running tests, and other activities on a (Jenkins) server, and, at the same time (ironically), they have long-lived branches. Due to this misrepresentation, too many teams are unfamiliar with the continuous integration practice, and it is also important for reducing cognitive load, which is why we explore it in this book.
This brings us to the term "team cognitive load", often shortened into just cognitive load, which, when introduced, immediately appealed to a wide audience since it gave a name to the direct experience of many overwhelmed software teams. Key advice in relation to cognitive load is that organizations should know how much a team can handle and avoid giving the team more responsibilities—this will stop the team’s cognitive load becoming too high.
Despite best intentions, this advice is more harmful than helpful in software product development. Misrepresenting cognitive load in our tech industry jargon causes severe issues in the product development system. Therefore, the purpose of this book is not to teach psychology but to rectify misunderstandings in software development and maintenance. As mentioned, we must address the real, widespread, and severe problem of overwhelmed teams. Ironically, we often face overwhelmed teams due to their limited responsibilities—precisely the opposite of the popular thinking about cognitive load.
We have worked with many well-functioning product development groups in different companies where this problem is so insignifcant that teams don't really discuss it. Rather, the teams in these better-functioning groups can handle even more (end-to-end) responsibilities, components, technologies, and code than overwhelmed teams elsewhere working on a subset of requirements to solve a customer problem. This seems counterintuitive and forces us to contemplate what is occurring. How can a team with much larger responsibility (working across multiple systems, large code bases, collaborating with other teams and parties) be noticably less overwhelmed than a team working on just a single relatively small component?
Interestingly, research in psychology and education related to cognitive load mainly focuses on methods to reduce cognitive load while achieving more—for example, learning faster and learning more. This, in a way, is the opposite of the notion that we should limit how much we can (and do) do. In the 19th century, Frederic Taylor proposed that, in pursuit of efficiency, we should limit responsibility and task diversity individual workers execute. This, not only made work indeed more efficient, but also much simpler for workers. There was barely any cognitive load involved to perform a task. As Frederic Talyor mentioned: "We just need slightly smarter horses." So, companies employ humans instead of horses. Now, of course, we understand that this is dehumanizing and doesn’t work when tasks are complex, such as those in software development. While nobody today argues that we should follow Taylorism, we still unconsciously, with all the best intentions, apply similar thinking when it comes to software teams. Perhaps it is because we still mistakenly seek to improve efficiency in software development.
As humans, we can write software and solve ever more complex problems because of our increasing ability to do more while constantly reducing cognitive load. No software engineer or scientist ever said: this problem is unsolvable since it surpasses human cognitive capacity. Instead, at worst they state it will take a long time before we solve it due to the amount of knowledge we still lack.
The art of programming is the art of organizing complexity, of mastering multitude and avoiding its bastard chaos as effectively as possible. - Dijkstra in Structured Programming - 1970
We, software developers, partition every problem into smaller ones and work on them individually, thereby avoiding maxing out our cognitive capacity. We constantly zoom in and out of our problem space regardless of its size, meaning that we avoid holding the entire problem in our working memory. Experienced coders utilize sophisticated partitioning techniques (we mention more of these later), such as implementing many design patterns, to organize and handle millions of lines of code. However, inexperienced engineers may try to complete tasks without these techniques and find themselves stuck, which becomes overwhelming. Resultantly, the senior engineers seem superhuman while the inexperienced engineers max out their cognitive load. This means that junior and senior developers could have the same maximum cognitive capacity but the difference in their ability to handle vast codebases could be huge.
One Caution to Avoid Misunderstandings: We do not recommend pushing or forcing teams or individuals to take on more responsibilities and larger scopes of work than they can handle. Instead, we don't assume they are unable to handle more because of cognitive load—they can deal with it, because they are human; they are built to deal with ever-increasing complexity by reducing cognitive load. Instead, we must ask: What is standing in the way of us continuously improving our capability to deal with larger chunks of complexity?
Two Organizational Design Patterns To Take Into Account
As we pivot from understanding the foundational misconceptions surrounding cognitive load in software product development, it becomes imperative to consider the environment in which these development teams operate. The organizational setting, far from being a backdrop, plays a pivotal role in shaping the cognitive load experienced by teams. It's a variable that can exacerbate or alleviate the pressures on a team's cognitive resources. While startups or smaller entities might enjoy the simplicity of a single-team focus, most organizations navigate the complexity of multiple teams. When not managed adeptly, this complexity becomes a source of unnecessary cognitive burden, detracting from the core objectives of innovation and efficient problem-solving; or just simply keeping complexity in check during continuous changes and maintenance.
Within this complex landscape, two dominant organizational design patterns emerge, each with a distinct approach to balancing autonomy, alignment, and ownership. These patterns are not just abstract models; they are critical in understanding how cognitive load is distributed and managed across teams in larger organizations.
On the one hand, there exist structures that prioritize high autonomy in constructing and maintaining technical solutions, necessitating alignment in broader customer problem-solving across different units or teams. This approach emphasizes the ownership of technical components, fostering a sense of responsibility and autonomy.
On the other hand, there exists a paradigm that flips the focus, granting teams autonomy in directly addressing customer needs while seeking alignment in building and developmental aspects across the organizational spectrum. This model tends to dilute the ownership of technical specifics and favors concentrating on delivering customer value.
Both patterns offer unique advantages and challenges in terms of managing team cognitive load, which we will explore further in this book. But first, let us define the two organizational design patterns in more depth.