Productize your engineering organization’s internal tools
Welcome to the (Internal Tools) team
In a software company, the internal tools teams provide a buffer layer between 3rd party software and the application developers. This can be everything from developer onboarding tools that reduce time to first commit to more advanced platforms like Kubernetes and the associated open source software used with it. These tools, and the teams that manage them, are often viewed as cost centers. The things they develop or implement don’t directly make the company money. Instead they are put in place to be a force multiplier for the product teams. Anecdotally, it seems common to put them in an entirely different organization from the product teams they support. As such, these tools teams then tend to lack architectural oversight and product management input to drive a comprehensive experience across their systems.
There are a few negative downstream effects to this organizational structure.
Ownership Bias
Most internal tools teams need to abide by the same controls as the product teams they support. What this means is that they may inadvertently show bias by building tools to fit their own needs over the common product team needs. An example of this is how the organization may be allowed to interact with Kubernetes. While the tools team may prefer the command line utility, the common user may be more inclined to use a dashboard. The team’s bias here would mean that the dashboard is given lower priority, even though application developers prefer it over the command line utility.
Disjointed APIs
Over time, internal tools teams have created many systems that solve individual problems. While they may talk to each other, they are ultimately disjointed and discrete systems. These systems may be intelligent enough to extrapolate what it needs from your code (magic) or you interact with it via a DSL, a REST API, or maybe a UI, but the point is consistency in this area is hard without intentional design and oversight. The question to ask yourself is: could a product team from another company be dropped into your ecosystem and succeed? Or would they need a 10 year vet with domain knowledge to point them in the right direction?
Hammers Seeing Nails
A problem that often arises with a team full of engineers tasked with being a force multiplier is that they start building things. While many of their projects are the result of direct business needs, some of them are the result of having free time, solving a one-off need, or are simply a passion project. Additionally, tools teams have built numerous projects in-house as a result of an existing open source project missing what is deemed a critical feature. All too often that critical feature is not so critical as to require a full rewrite. These types of projects can be detrimental to the organization and the team itself as it expands the maintenance burden while at the same time often providing little value to the organization as a whole.
Platform Engineering
The solution for these issues involves treating these tools teams, instead, as a Platform Engineering organization. Rather than having security, “DevOps”, and operations all working in silos, they are instead put under an umbrella called Platform Engineering. This is done at the organizational level so the individual teams still exist but tools they create and implement are actually part of a single product or platform.
The goal is to create a cohesive way to interact with the internal platform of tools rather than making each engineer at the company learn many bespoke APIs or understand the magic. This could encompass everything from developer onboarding to the SLOs of your application. If there was a pendulum swing correction for SysAdmins that created the DevOps movement, this would be the swing back in that the average frontend engineer shouldn’t need to be Kubernetes experts. The product organization wants them working on products and shipping features.
The Investment
Buying into the Platform means dedicating resources to glue these teams together, but mostly this means you need a team at the highest level getting on the same page and understanding what each individual pillar owns and maintains. Beyond that they need to understand what the business requirements and priorities are so they can work together with each pillar to determine how the solution is going to integrate with the platform.
Listen
Product teams know what is broken or painful for them, and they will tell you. Listening to them or surveying them and relaying their pains and desires back to them is Product Management 101. This is a skill that should be honed by all engineers but is part of the product manager skillset. If your organization lacks this skill, get someone involved that can help. Building something without consulting a user or assuming what the user wants is going to end up costing the company lots of money. You don’t have to (and often should not) solve everyone’s desires, but you should at least listen to them as it may be a more widespread problem than you originally thought.
Platform Characteristics
Your platform can be anything from an idea to a homegrown single-pane-of-glass web app. The charter of the Platform team should be to maximize the value and throughput of the application development teams. The goal should be to minimize human intervention and shift the product feedback left for quicker iteration. Ideally the Platform should help guide against wrongdoing by suggesting best practices, while actually preventing wrongdoing at deployment time — think of these as guardrails for software delivery. An example of this would be providing a reusable way to construct an S3 bucket with your organization’s best practices, while at the same time enforcing the S3 bucket encryption at the account level. This enables teams to use S3 outside the scope of the best practices without breaking any controls put in place. This idea may be called an escape hatch or a golden path, where if you leave the path, you are outside the scope of what the Platform supports.
Looking at the above image, at the core of the Platform are the business “needs”. Many of these features are probably already being satisfied by the tools that exist today. The business “wants” are the features that support the application teams in meeting those business needs. Enforce the needs, support the wants.
Identifying your Customer
In order to be a product, you must define your target customer. Where the customer-facing product teams get the benefit of choosing their specific customer to deliver an MVP, the Platform engineering teams often need to support every existing and future foreseeable use-case. When it comes down to it, if your company has deemed a product important enough to have its own engineering team, you need to support it as a Platform team. Picking and choosing which features get implemented and which get shelved for future quarters are driven by the business needs, but chances are the majority of engineering teams that are using your product are not asking for those same things. This is ultimately where the first struggle lies for Platform teams: it’s hard to focus on the needs of the many when the needs of the few are just as important. By leveraging the aforementioned escape hatches you can reduce the number of (but maybe not eliminate) one-offs and special use cases, allowing the Platform team to focus on more impactful features.
Stitch then Fix
Creating a Platform does not require a wholesale rewrite of all the internal applications it is supposed to encapsulate, rather you can start by applying a thin veneer over the top of existing internal products that grows into a robust solution over time. Start by stitching internal tooling together and then fixing the individual pieces to enable faster delivery of an MVP. If the integration point of your Platform is a DSL, rather than implementing each consumer API first, allow for the ability to simply point to the existing file, i.e. Dockerfile / Helm Chart.
Shipping the MVP
MVPs are rough and unstable. Identify power users that are okay with being an early adopter and want to influence the Platform, there are usually more than you think. Also find large teams and small teams, simpler product areas and more complex areas, and identify how their experience is different. When you ship the MVP, treat these teams with respect and treat their issues as top priority. The important thing is to treat it like any other product and iterate.
Measuring Success
Success is not measured by adoption percentage on the Platform. It is measured by application development throughput, and that is the most important thing to remember. If the organization makes an investment into the Platform it needs to be a force multiplier for application teams. Develop ways to measure success, use DORA or other methods to measure it, but measure it nonetheless to know if what is being delivered is working.
The Payoff
The Platform will never be “done”. Like any product, it continues to go through iterations. New business requirements will arise and technologies will evolve enabling its next features. The customers can be migrated from the escape hatch to newly supported use cases, removing that overhead for them, and enabling them to deliver more quickly.
While “DevOps” or “dev tools” teams have become prevalent within software organizations, they are often treated quite differently from product development teams. These teams tend to be viewed as a cost center and lack architectural oversight and product management. This results in tooling that may not adequately address developers’ needs, solutions that are disjointed or lacking consistency, and a propensity to “overbuild” which increases maintenance burden while providing little to no value to the organization.
Instead, viewing these tooling teams through the lens of product allows us to build a cohesive platform that improves developer efficiency. The Platform forces internal tools teams to be conscious of one another, to avoid overlapping terminology, and to represent concepts uniformly across their systems. Instead of approaching your internal tools like a bundle of projects, think of them more like a single product that developers interact with. To be successful, this requires adopting product management practices such as identifying and understanding your customer, listening and observing the customer, shipping MVPs and iterating, and measuring success. These ideas are at the heart of Platform Engineering. The result is a cohesive internal platform that changes the tools team from being a cost center to being a strategic force multiplier.
Need help making the transition?
Real Kinetic has extensive experience helping teams develop mature, effective internal tools teams. If you have questions or need help getting started, we’d love to hear from you. These emails come directly to us, and we respond to every one.
