Context
Introduction
Large product development groups typically suffer from unnecessary organizational and technical complexity. Software teams feel the consequences more directly, with an ever-increasing number of often low-quality applications/components delivered under time pressure that they must support and maintain—not to mention coordination with management, stakeholders, and others in endless meetings. This book contains several suggestions and directions for addressing this type of problem.
But, before discussing solutions to the problem of overwhelmed teams (perhaps a better term than “team cognitive load”), we first need to address the issue with using jargon and acronyms. It will become clear why.
Software development experts tend to (re)invent jargon and acronyms continuously. Although it is a necessary part of our professional language, jargon can also create an illusion of something being special, unique, or new. The term “microservices” is an example of this. This concept has become very popular over the last 10+ years, but most of what we define as microservices architecture is part of normal evolution and improvement in (distributed) software architecture or has existed for much longer.
A more significant issue with jargon is when it misrepresents a valuable practice. An example is the “CI server”. Continuous integration (CI) is an important practice that was popularized by XP and other lightweight approaches long ago. (Here is a definition: http://www.extremeprogramming.org/rules/integrateoften.html). The practice doesn’t involve a server, although having one in addition to the practice itself is helpful. Today, many assume they practice CI when they only run an automated pipeline—building code, running tests, and other activities on a (Jenkins) server, and, at the same time (ironically), they have long-lived branches. Due to this misrepresentation, too many teams are unfamiliar with the practice of continuous integration, and it is also important for reducing cognitive load, which is why we explore it in this book.
This brings us to the term team cognitive load, often shortened into just cognitive load, which, when introduced, immediately appealed to a wide audience since it gave a name to the direct experience of many overwhelmed software teams. Key advice in relation to cognitive load is that organizations should know how much a team can handle and avoid giving the team more responsibilities—this will stop the team’s cognitive load becoming too high.
Despite best intentions, this advice is more harmful than helpful in software product development. Misrepresenting cognitive load in jargon causes severe issues in the product development system. Therefore, the purpose of this book is not to teach psychology but to rectify misunderstandings in software development and maintenance. As mentioned, we must address the real, widespread, and severe problem of overwhelmed teams. Ironically, we often face overwhelmed teams due to their limited responsibilities—precisely the opposite of the popular thinking about cognitive load.
We have worked with many well-functioning product development groups in different companies where this problem does not seem to exist. Rather, the teams in these better-functioning groups can handle even more (end-to-end) responsibilities, components, technologies, and code than overwhelmed teams elsewhere working on a subset of requirements to solve a customer problem. This seems counterintuitive and forces us to contemplate what is occurring.
Interestingly, research in psychology and education related to cognitive load mainly focuses on methods to reduce cognitive load while achieving more—for example, learning faster and learning more. This, in a way, is the opposite of the notion that we should limit how much we can (and do) do. In the 19th century, Frederic Taylor proposed that, in pursuit of efficiency, we should limit responsibility and the amount of diverse tasks individual workers execute. Now, of course, we understand that this is dehumanizing and doesn’t work when tasks are complex, such as those in software development. While nobody today argues that we should follow Taylorism, we still unconsciously, with all the best intentions, apply similar thinking when it comes to software teams. Perhaps it is because we still mistakenly seek to improve efficiency in software development.
As humans, we can write software and solve highly complex problems because of our increasing ability to do more while constantly reducing cognitive load.
The art of programming is the art of organizing complexity, of mastering multitude and avoiding its bastard chaos as effectively as possible. - Dijkstra in Structured Programming - 1970
We partition every problem into smaller ones and work on them individually, thereby avoiding maxing out our cognitive capacity. We constantly zoom in and out of our problem space regardless of its size, meaning that we avoid holding the entire problem in our working memory. Experienced coders utilize sophisticated partitioning techniques (we mention more of these later), such as implementing many design patterns, to organize and handle millions of lines of code. However, inexperienced engineers may try to complete tasks without these techniques and find themselves stuck, which becomes overwhelming. Resultantly, the senior engineers seem superhuman while the inexperienced engineers max out their cognitive load. This means that junior and senior developers could have the same maximum cognitive capacity but the difference in their ability to handle vast codebases could be huge.
One Caution to Avoid Misunderstandings: We do not recommend pushing or forcing teams or individuals to take on more responsibilities and larger scopes of work than they can handle. Don't assume they are unable to handle more because of cognitive load—they can deal with it, because they are human; they are built to deal with ever-increasing complexity by reducing cognitive load. Instead, we must ask: What is standing in the way of us continuously improving our capability to deal with larger chunks of complexity?
Two Organizational Design Patterns To Take Into Account
As we pivot from understanding the foundational misconceptions surrounding cognitive load in software product development, it becomes imperative to consider the environment in which these development teams operate. The organizational setting, far from being a backdrop, plays a pivotal role in shaping the cognitive load experienced by teams. It's a variable that can exacerbate or alleviate the pressures on a team's cognitive resources. While startups or smaller entities might enjoy the simplicity of a single-team focus, most organizations navigate the complexity of multiple teams. When not managed adeptly, this complexity becomes a source of unnecessary cognitive burden, detracting from the core objectives of innovation and efficient problem-solving.
Within this complex landscape, two dominant organizational design patterns emerge, each with a distinct approach to balancing autonomy, alignment, and ownership. These patterns are not just abstract models; they are critical in understanding how cognitive load is distributed and managed across teams in larger organizations.
On the one hand, there exist structures that prioritize high autonomy in constructing and maintaining technical solutions, necessitating alignment in broader customer problem-solving across different units or teams. This approach emphasizes the ownership of technical components, fostering a sense of responsibility and autonomy.
On the other hand, there exists a paradigm that flips the focus, granting teams autonomy in directly addressing customer needs while seeking alignment in building and developmental aspects across the organizational spectrum. This model tends to dilute the ownership of technical specifics and favors concentrating on delivering customer value.
Both patterns offer unique advantages and challenges in terms of managing team cognitive load, which we will explore further in this book. But first, let us define the two organizational design patterns in more depth.