Stop Wasting Your Beer Money
Audience Member: “Can you show me that list of logging providers?”
Presenter: “Sure” as he puts up his slide that has providers such as Splunk, Loggly, SumoLogic
Audience Member: “Are there non-hosted, open source alternatives?”
Presenter: “Yes there’s options like Elk/Logstash, but I don’t really recommend that. It’s more work that you realize and you’re better off spending that time elsewhere.”
Audience Member: “Sure. But I’m just not sure we can afford hosted solutions.”
Presenter: “They have a free tier. And the costs are generally tied to usage so the costs should scale with your systems needs.”
Audience Member: “Yeah. But we’re pretty early stage and we don’t want to spend our money on logs.”
Presenter: “Well I’d cut out the pizza and beer and instead pay for Splunk.”
- My paraphrasing of an exchange after a presentation by Matthew Boeckman, an acquaintance of ours.
Amen!
Why are engineers so bad at paying other engineers for their work? If we can afford spending $50 on pizza and beer for a late night of coding, then we can afford spending $50 a month for a logging service. Yet something is keeping us from just paying for that service. I believe it starts with us consistently undervaluing our own time, which leads to us undervaluing other engineers’ time. We can see this in the way we treat open source software (OSS) projects. We’re damn near abusive. From the lack of paying to the horrific displays of disrespect shown in the issues and pull requests, there’s a complete lack of respect for the engineering time put in to these projects. I believe this stems from an inability to appreciate our costs and specifically how expensive building and maintaining systems is. One problem is, to us, it can feel cheap and easy to build. Coding is what we do. We like it and often prefer doing it over anything else. I’d often rather write code than talk to other humans. Unfortunately, this causes us to overvalue our own abilities while undervaluing our costs.
Unfortunately, this causes us to overvalue our own abilities while undervaluing our costs.
Those of us that have been around long enough have made the buy vs build mistake many times. I keep choosing build because it feels as if it has the lowest barrier to entry. And I can be impatient. This is especially true for pieces that are not on the critical path.
It usually starts with something as innocent as:
PM: “Hey, we need a tracing system.”
BEAU: “No problem. That’s easy. I’ll throw something out there.”
Why spend the time to do a proper cost analysis? Why look for off the shelf solutions? Why do any of those things when I can just “hack it out” real quick. Months later after clustering my way through development as I’m desperately trying to finish off that last 20% and ship my system, I end up on a desperation and stress-driven walk about town with my dogs. Often I hit a moment of clarity where I do a mini-retrospective and realize that maybe it wasn’t so easy to build that thing. And it certainly wasn’t cheap. Dammit, I have to maintain this thing too. I let my impatience and bias get in the way again. Nobody pisses future Beau off quite like past Beau does.
Now, before committing to any project, I attempt to live by a single rule: Building it is a last resort. I’m going to start by looking for an existing service that “just works.” If that doesn’t exist, I’ll look for the existing system that I can just host. Thankfully, these days I have many options that make this easier. It’s not just the optimal decision; it’s the only way you’ll survive.
The Modern Development Ecosystem
The cloud. DevOps. Serverless. These are all movements and markets created to commoditize the common needs. They may not be the perfect solution. And yes, you may end up “locked in.” But I believe that’s a risk worth taking. It’s not as bad as it sounds. Tim O’Reilly has a quote that sums this up:
“Lock-in” comes because others depend on the benefit from your services, not because you’re completely in control.
We are locked-in because we benefit from this service. First off, this means that we’re leveraging the full value from this service. And, as a group of consumers, we have more leverage than we realize. Those providers are going to do what is necessary to continue to provide value that we benefit from. That is what drives their revenue. As O’Reilly points out, the provider actually has less control than you think. They’re going to build the system they believe benefits the largest portion of their market. They will focus on what we, a player in the market, value.
Competition is the other key piece of leverage. As strong as a provider like AWS is, there are plenty of competing cloud providers. And while competitors attempt to provide differentiated solutions to what they view as gaps in the market they also need to meet the basic needs. This is why we see so many common services across these providers. This is all for our benefit. We should take advantage of this leverage being provided to us. And yes, there will still be costs to move from one provider to another but I believe those costs are actually significantly less than the costs of going from on-premise to the cloud in the first place. Once you’re actually on the cloud you gain agility.
So what are these common systems being commoditized and how can we benefit? Think of the systems we build. How much of those systems are what we consider to be the core “business” logic and how much are the supporting pieces? When I look at my systems, they often have a small, but critical, section of core business logic. That core then surrounded by pieces like:
- Identity Services (User authentication and account management)
- User Permissions
- Security
- Processing (queues, crons, workflows)
- Messaging
- Storage (structured and unstructured, persistent and temporal)
- Logging, analytics, monitoring
And those are just the pieces in my code base. That doesn’t touch on much of the development life cycle (builds, testing, chaos, load testing) and infrastructure (hosting, scheduling, dns, networking).
Simon Wardley has a tweet that shows just how common it is to build the same piece of non-core software over and over:
All of these pieces mentioned and many more are either being commoditized or already have been. And once something like this has become a commodity, it becomes extremely difficult to justify paying for custom solutions. Unless we are a provider of one of these services and attempting to fill a smaller niche, I struggle to come up with a good business reason for not leveraging them. Why? Because they are cheap (no, really, they are) and building and/or hosting is expensive.
The Costs of Building
We often throw terms like opportunity and maintenance costs around but I’ve found that we rarely appreciate the true costs. Which is unfortunate as these costs can be the determining factor in the success of a product. Ideally, we create processes that we support with tooling that give us the data to more easily quantify these costs. But even if we’re not working within an environment like that we should do our best to have an intuition. And part of that intuition is learning that we often have it wrong. We tend to overvalue our ability to outperform the market even in quality. Not to mention cost.
The only thing engineers might value less than engineering time is operational costs.
Hosting and operating software is expensive. The only thing engineers might value less than engineering time is operational costs. This is especially common at companies that separate operations from engineering. By the way, those working in operations can have the same bias issues as engineers. They overvalue their ability while undervaluing what they cost. With operations this is often due to control. When something is critical we don’t like relying on somebody else to manage it. This is one of the big mental hurdles you must get over when using hosted services. It can take a while to get used to relying on others to resolve our issues, but over time we’ll realize how nice it is to let somebody else handle the 3 AM outages. We start to appreciate how much better they are at maintaining the system that is their focus. We also get economies of scale. It’s highly unlikely that our logging system will be able to keep pace with a dedicated business in terms of features and support. This is unless of course we decide to invest heavily into the logging system. And we probably should not make those investments, given that we’re not a logging company and that system isn’t likely to provide us differentiated value over our competition in our current market.
Also, engineers aren’t the only ones with opportunity costs. Using a hosted system frees our ops folks to focus their time back on our high priority systems. Or if we really embrace using hosted services it will allow us to cut down the number of operations staff that we hire and instead put those resources back into product development.
There are many other maintenance costs that we often don’t think about. How do these systems get built and deployed? Are they now part of our dev lifecycle? Even if they’re 3rd party we’ll likely need a process for handling updates. What about security and compliance? If our company needs to meet compliance requirements it’s much easier to use a provided solution that has already met that compliance than go through the audit process of custom or 3rd party software.
Of course, the inverse problem can be a strategic reason that we do need to use a custom solution. There may not be a hosted solution that meets our compliance needs. In those cases, building or self-hosting may be our only option. This happened to us a few times at Workiva. We ended up creating our own build system because we needed something that both met our increasingly rapid and robust release process as well as compliance. That said these days many hosted systems meet many of the different compliance requirements. AWS is a prime example: https://aws.amazon.com/compliance/services-in-scope/. Also that custom build system at Workiva? It was a maintenance nightmare and something they’re actively looking to replace with one of the many hosted solutions that now exist.
An Opening for Competition
If we are not taking advantage of these systems then we are leaving an opportunity for our competition. It’s obvious that if we are building and maintaining system A we are not building and maintaining system B. But we don’t always appreciate that in the moment. When we build systems, we want as much as our time focused on customer value as possible. Taking that even a step further, we really want our time focused on solutions that provide strategic value. Does this system provide us a competitive advantage over our competition? If it does then to quote Joel Spolsky:
If it’s a core business function — do it yourself, no matter what.
If not we probably shouldn’t do it if a viable, affordable alternative already exists.
This should help clear up why we’re often met with denials and frustration when pitching these non-direct customer value projects to our management. They then have to turn around and explain to the folks writing the checks why we should be spending time on non-direct customer value projects vs direct customer value projects.
We have to be able to justify the value in everything we build. If not we have given our competition an edge that may be enough to beat us. If our R&D expenses are too high, then we have given leverage to our competition. For example, if we’re a company in Fintech and we have 40% of our R&D budget being spent on non-direct customer value we will likely be undercut by a competitor.
Especially with SaaS companies, we must minimize our margins to give our customers the most value. Our costs cannot scale linearly with usage. We must build systems in an efficient and elastic way. This will allow us to cut our costs and thus give ourselves more opportunity for growth. Many of the cloud based services are built in this exact way. They are meant to scale with us.
Winning
Engineering is often our most expensive resource. This resource should only be used on the most critical pieces of our system that give us a competitive advantage. Which leads us to the first critical piece. We must take a holistic view of our business, organization and systems. This includes doing an analysis of that picture to determine which pieces are critical, valuable, competitive advantages, etc. Once we have these pieces categorized it will then be easier to make the build/buy decision. Of course the hard part will be the many, many arguments over what exactly is a competitive advantage to the business.
Those arguments will be much easier if we do analysis of our market. And if we didn’t do this before starting the company or project then we may be in for a world of hurt. But even if we did do this, we must be continually updating this analysis. Of course as we get into the market, we should have relationships with our customers to provide even better detail of our value. We should be leveraging our sales, marketing, customer support departments to gather more data to create an up to date perspective on our value and priorities. It’s leadership’s job then to ensure we have alignment throughout the organization on these values and priorities.
I hope most would agree with building processes around these concepts to ensure we provide optimal value. We can look around the industry to see more and more companies are switching their organizations to these processes. Which is all the more motivation for us to do the same and be even more efficient.
However I believe there’s and area that often gets left out. We must do analysis of the tech industry itself. We can’t make the optimal decision around build vs market provided unless we know what the market can provide. This requires us to not only invest time into researching the industry but we also need away to evaluate the market. Thankfully there’s a process I’ve found that not only helps here but helps us analyze the pieces mentioned previously (and many more things beyond that).
Mapping
I first found mapping via Simon Wardley. Since then, I’ve seen similar concepts presented by Tim O’Reilly. O’Reilly recently covered the values of maps, especially for charting the future of tech and the markets in his latest book WTF: What’s the Future and Why It’s Up To Us. The key to mapping is adding the dimension of time to our diagrams. If you follow Wardley’s process and map systems, industries, markets, etc you’ll end up with something like:
Here you can see that he’s broken his time axis into the categories: Genesis, Custom Built, Product (+rental), Commodity (+ utility), Evolution. When we break these systems down in this way we can more easily see what pieces we should build vs rely on the market to provide.
I can’t possibly do Wardley’s mapping justice in this post but I highly recommend Simon’s book (you can skip to the specifics of this diagram here). There is also a tool to create our own maps.
Even if we don’t do the analysis of our systems, business or the markets I hope we at least appreciate how expensive and powerful we are as an engineers. And at a minimum, we start from the mentality of building as a last resort. We should leverage our most expensive and powerful resources for the solutions that will differentiate us from the competition. For everything else, we should attempt to use the existing solutions. We shouldn’t be afraid to spend money on those resources. Ideally we’ll just pay for systems that allow us to get to market as fast as possible while scaling costs along with our usage and growth.
Please reach out: realkinetic.com @real_kinetic @lyddonb
