Basic laws of statistics imply that most developers work on average products, supporting usual business workflows, helping the proverbial mill run without too much excitement. Although the right choice for most such products should be to “Choose boring technology” (as Dan McKinley nicely explained), very few teams actually do that. This leads to overcomplicated software that is costly to develop, test and maintain. Over a long run, bad technology choices cause technical debt, require costly migration, and slow down delivery. Overcomplicated products obstruct change to businesses that they were actually supposed to help. In a sense, the unfortunate faith of an average software product is to start defeating its own purpose. The reasons are purely human, not technical. In this post, I’ll explore some common reasons why this happens, and propose how deliberately investing in side-projects is a good way to prevent technological overuse but still keep everyone happy.
Before we continue, I have a confession to make. In the early noughties I had a gig to build a content management system for a publisher. This was before decent web-based opensource CMS systems appeared, so companies still built their own. The money wasn’t particularly good, and the task wasn’t particularly exciting. The client effectively wanted a CRUD interface to a simple database. Still, that gig felt like a dream job, because I had a chance to play with Enterprise Java Beans. The whole thing was horribly over-engineered, with session beans and entity beans magically replicating across a pair of active-active servers with remoting interfaces and resilient architecture, although the system could have effectively worked on a single machine, and never needed immediate fail-over or any redundancy more than daily database backups. Whether the product was a business success is difficult to tell, but the client was happy, because we delivered “on time and budget”. The fact that both time and budget were horribly inflated due to our technical choices didn’t really matter, or more likely the client representatives probably weren’t even aware of that. After all, we were the technical experts. From a purely technical side, I’d consider this a huge failure because it would have been significantly faster to just build a bunch of CGI scripts in Perl that did the same thing. The client was an “enterprise”, so a simple Perl app sounded too pedestrian for them anyway, and they had the wish and the money to pay for Enterprise Beans. From a personal side, although I’m now embarrassed about the technical aspect of the story, this was one of the most valuable projects I was ever involved in. I learnt a ton about EJB at a time when it was a hot technology, so I was able to negotiate an excellent increase in my contractor rates for many more jobs to come. It took a good few years until the industry realised how overcomplicated Enterprise Java Beans are, and for something like Spring framework to gain a significant share of the market. When I tell this story at conferences, most people in the audience have something similar to offer. The buzzwords are different, but the result is the same.
For those of us in IT who wanted to build stuff, and not to pursue computer science as an academic field, one of the key motivational drivers is being able to express creativity in a useful way. But the average business problem is, by definition, ordinary and unexciting. By implication, most people can’t find creativity and enjoyment in the business side of their products, so they have to look for it in the technical aspects. After all, there’s no shortage of new libraries, widgets and gadgets to try out. It might be difficult to find interest in building yet another ugly website that talks to yet another stupid database to store orders and generate invoices. But doing it through a cluster of microservices that talk to a data lake via an event bus makes the whole exercise not just slightly more bearable, but actually exciting. It’s like Minecraft, but for adults, and you get a decent salary for it. But introducing unnecessary complexity into software products isn’t really what we should be paid for.
Another force to consider is an unspoken truth about the employer-employee relationship in the software market. It’s very rare to find serious loyalty from any side. Large organisations spent the last two decades outsourcing, firing and moving people in bulk. Smaller companies may need to find employees quickly to ramp up with the demand, but they don’t exactly have a high survival rate. Mid-size companies often prefer contractors to employees for tax purposes and to work around employment laws. They all need flexibility when hiring (and firing) people. As technology gets older, it becomes more difficult to find people who want to work on it. It’s a lot simpler to find people who want to work with the latest version of React or Angular than those who want to maintain a legacy ASP web site or, God forbid, build a new one. Having shiny new tech in a job offer makes it easier to attract employees. Don’t forget that people who know old technologies tend to be older, with higher life expenses and higher salary expectations. Shiny new tech allows companies to hire younger employees and pay them less.
Today’s employees, on the other hand, tend to switch jobs a lot more frequently than their parents did, looking for better pay or even just fleeing boredom. A huge proportion of people in the industry aren’t really employed, but work as short-term contractors and consultants. Job security today isn’t as much about holding on to a position as it is about being able to find a new role easily. Hands-on experience with shiny new tech makes it easy to find a new job quickly if the need arises.
Even if it’s an unconscious decision, many developers will jump at the opportunity to use some shiny new technology, and most companies will let them. The reason why such behaviour is not just tolerated, but often encouraged, is that it is good for employers as well. Chasing an occasional hi-tech squirrel makes people actually enjoy their job every once in a while, and this should not be understated. Giving people exciting work helps with employee retention, which reduces costs associated with hiring replacements, and avoids knowledge drain. Even if the product doesn’t need the latest technology, giving people a chance to play with it at their current job means that they don’t have to switch companies to stay up to date with new market trends.
Playing with new technology is also a relatively cheap way of educating employees on the job. The software landscape shifts every few years in dramatic ways. Twenty years ago Web exploded. Ten years ago it was mobile devices. Now it’s the push to managed cloud services. Staying too long on old technology makes products outdated and difficult to sell. When the winter comes, having internal knowledge about new technology can be incredibly useful. Instead of paying for expensive employee training programs, you can pretty much just leave developers on their own with a bit of coffee and they will improve their skills and knowledge. Bringing shiny new technology into a product before it is needed might damage the product, but it will reduce the cost of retraining employees or hiring external consultants to bring in expertise in the future.
Balancing all these forces is quite a challenge for many software companies. Without a clear strategy, new shiny technology will usually find its way into your products in ways you do not want. That’s why I suggest deliberately having side-products where developers can experiment without risking the main line of work.
Deliberate side products should start with a clear expectation that they will only have secondary business value, and may never be properly released. They should not be expected to run under the same operational, legal or regulatory constraints as your main product line, in order for people to skip over all the usual boring stuff. Aim for maximising interest and learning, don’t worry about proper testing, documentation or keeping good coding practices. Keep it low risk, perhaps even open source it to ensure that the company never expects to derive valuable IP, and don’t worry too much about support.
As the name suggests, deliberate side-products need guidance and structure, they aren’t just a bit of free time for employees to play with. A while ago, when Google’s 20% learning budget became the copy-paste template of the day, lots of organisations expected that they will get the next GMail or Google Maps by investing slow Friday afternoons. They quickly realised that you can’t just copy one tiny bit of some other company culture, and that people aren’t really productive on Friday afternoons anyway.
I prefer setting up a time box and focusing 100% on building a different product for a while, as an intensive learning exercise. A few days is probably too short, a few months is probably too long for such an effort. Products driven by market cycles occasionally have quiet periods, when people end up implementing low-priority features because there are temporarily no more “big-ticket” items. Instead of just adding more complexity to the primary product, which makes it difficult to use and maintain for questionable benefit (after all, there was a reason why those left-over features had lower priority), take a break and create something completely different and exciting.
For example, in 2015 we took a break from MindMup during the summer (it’s usually quiet then) and experimented with building a markdown based living documentation/testing tool. We ended up polishing and releasing it on GitHub, but it failed to get any serious traction. It did however allow us to experiment with a bunch of JavaScript packaging methods and models, including trying out webpack and NPM build tasks, and about six months later we significantly simplified project packaging and deployment structure for our primary product based on that new knowledge. During the summer of 2018, I spent a few days with two colleagues building Desole, a monitoring system using all the shiny new toys that AWS had to offer. It was never properly finished, but I learnt a ton about lots of cloud services that I ended up using for other things since. And I got a chance to experiment with AWS code pipelines for deployment, which we now use for MindMup.
Building internal developer products is often a nice choice. These can also have an added bonus of speeding up delivery for your main product. If such tools actually turn out to be useful, you can also generate good will and get some free marketing by releasing an open-sourced version. That’s how Bug Magnet and Appraise came to be, both solving a boring and time consuming task we frequently had on MindMup, and then getting polished enough for external use. Developing them allowed me to play with some exciting tech - Chrome extensions and Puppeteer test automation - which I was then able to use on the main product as well. Sometimes these things get a life of their own and actually attract a community, such as Claudia.js (It’s probably the most popular thing I ever published on GitHub, led to a lot of good marketing for our main product, and even someone else writing a book about it). Apart from being a lot of fun to write, working on Claudia allowed me to learn and try out the new JavaScript language features at the time, which found their way into MindMup a year later, once the browser support improved. It also led to a decent amount of conference invites where I could promote our main product.
Another good option is to build something completely unrelated in a different domain. Let people do something fun they always wanted to try. During another summer break, inspired by Snowden’s whistleblowing stories and The Man in the High Castle series on Amazon, we worked on a video game. It was called East Germany 2020, about an alternative history timeline where the Berlin Wall never came down and Stasi runs the equivalents of Facebook and Google to spy on the world. We worked on it using the latest HTML5 features for about a month and a half and built an initial playable version, then abandoned it forever. I kind of hoped we’d make an actual product out of it, but the version we built would only work in the most recent Chrome release, and truth be told it wasn’t much fun to play. Launching it as a proper product would take months of additional work, and require a decent marketing budget, so we just dropped it. It was fun to build, we made a ton of mistakes while experimenting with new technology, but the mistakes didn’t affect our main product. We kept the new knowledge, and ideas like SVG animations and HTML5 Flexbox layouts later made it into MindMup, and made the user experience significantly better.
Deliberately creating side-products is a great way to experiment and learn and to inject excitement into work when it becomes a bit boring, but it can be a hard sell to organisations under constant time pressure. Yes, taking a few weeks off to build something that has no immediate value costs, but so does injecting unnecessary complexity into a product, or introducing unvalidated libraries and components. Once bad technical decisions go into the main product, taking them out tends to cost a lot more than just throwing away a side-product and keeping the knowledge. And if anything good comes out of it, then you can improve your primary product cheaply.
Many thanks to Paul Shepheard for an inspiring weekend discussion that led to this post. Cover photo by Philipp Lublasser on Unsplash.
