Technical Debt Accumulation
On a warm August day six months ago, I began a new journey as a doctoral researcher. As part of that journey, I have been diving into an open biomechanical modeling source software called OpenSim. Successfully compiling the code base for the first time was a non-trivial task and was a cause for major celebration. As a nearly 20 year old software project (the first public release was on August 20, 2007), it has expanded far beyond its intended purpose and accumulated significant technical debt along the way.
One day while waiting for an infamously long build, I began to wonder why compiling took so long when I had only changed one line of code in the project. I instantly remembered a software engineering lecture from 6 years ago during my semester in Växjö, Sweden. For one of our assignments, we were tasked with decoupling all the circular dependencies from a legacy hotel reservation system which I was told the university had actually used until recently. I distinctly remember seeing the dependency graph below (from the assignment document which I found again through this github repo ) and thinking, “there is no way that I will ever run into this in real life.”
Fixing the Mess
After many painstaking hours of using every refactoring technique I knew at the time, I untangled the spider web of code.
The longest loop of circular dependencies in that chain contains 20 links and even at that fairly pedestrian size, it is
still very difficult to mentally conceptualize untangling it.
While this task was only focused on decoupling circular dependencies between code components, the problem can also manifest
in library dependencies, especially if transitive dependencies are considered.
A Dependency Web in the Wild
Using CMake’s handy Graphviz feature
I created the following visualization of the OpenSim library dependencies.
As you can see, the resemblance to the first graph is striking. These cyclic dependencies make it impossible for the build system and compiler to properly know which files have changed, which results in significant portions of the code base being rebuilt unnecessarily each time.
Given this was the state of the library dependencies, I was eager to investigate the state of the code itself. I created some analysis code (which is available in this repository ) to determine all the links and strongly connected components within the code base.
In this image, all components with the same color are strongly connected. This means that they all link back to each other in a massive circle.
This image only contains a subset of the core code because including more becomes unwieldy. The full visualization consumes 6 pieces of A3 paper which I have proudly displayed on my office wall. For those unfamiliar with metric paper sizes, this is a graph which is around 1 foot tall and spans over 8 feet wide. I have also enlisted the help of a python script to calculate a full list of all the cycles for your perusal.
Untangling the Web
Although visually stunning, these graphs serve as more than a conversation starter for those gazing towards the back wall of my office. It is difficult to understand and fix a problem if you cannot quantify it. These graphs visually quantify the magnitude of the problems unmanaged technical debt creates.
This web of entangled code creates spaghetti code on a magnitude that is difficult for one mind to fathom. These complex interactions force a recompilation of the entire code base for a simple one line change.
Untangling this web would yield immediate tangible benefits for everyone working on the project. It would create faster and more reliable builds and code that is easier to understand and maintain. I have opened this github issue in hopes of generating momentum for solving this problem.
Although initial enthusiasm for a fix was strong, the issue now seems to rests in abeyance. I implore you if you have made it this far, to like the issue and engage in the discussion. Fixing an entangled web of technical debt will never be fun or exciting. In these troubled times, we must strive for a higher standard of quality, not more lines of code.