Why Google App Engine
Google App Engine Standard (GAE) was released in mid 2008. It was a much simpler, much more limited version of the GAE available today. Building applications took real commitment, but if you stuck it out and learned the nuances it was — is — nearly magical. The problem is that those nuances are not all obvious, plentiful, and the engineering practices to make the magic happen require discipline. This post will explore some of the good and some of the bad.
Let’s start with two huge pros. If you design your system to fit GAE’s model, Google will manage most of your operations for you. Scaling up/down, fault-tolerance, zero-downtime releases, traffic splitting for A/B testing or beta service releases, and host server management are all handled for you. That’s a lot of stuff, and a big claim. I have spent the last 5 years managing teams and consulting with clients to build out similar functionality on traditional VM based infrastructure; the amount of effort, time, and money that it will save you upfront is hard to articulate. You don’t get all of that for free though. You must design your application to match GAE’s expectations.
That is not the biggest advantage though. The biggest advantage is that you can stop talking about infrastructure and focus instead on your application and your business logic. That is probably what your customers are paying you for anyways. That means a more effective use of your engineering dollars. You’ll still need ongoing investment to make sure your developer and deployment workflow is streamlined, but that’s a much smaller surface area. This is a huge win — one that is hard to appreciate until you experience it.
Those are two big wins. However, like everything, there are always tradeoffs and costs. Now let’s dig into the nuance of my claims. Both of the wins above carry significant cost and require a willingness to embrace the costs in order to get the benefits. If you don’t fully embrace and buy into the costs then you’ll get the worst of both worlds: an application that doesn’t scale well, has performance problems, and teams focused on fighting infrastructure rather than building code that differentiates your products — all on a platform you have little control over.
Here’s the most important challenge: you must think stateless. Ok, done. If you design and build stateless systems your app can be scaled up and down, failed over, released, and rolled back far more easily. This is true with many systems, but particularly true of App Engine. This sounds like it should be easy, but it isn’t easy unless you carefully think it through. You’ll need to think about request patterns, data structures, caching strategies, and application startup times very carefully. If you don’t, you could hit contention, performance problems, and race conditions to name a few. There are mechanisms in GAE to help with these problems, such as Memcache, task queues, and datastore, but it’s not a free lunch. We’ll take a look at these next.
Autoscaling is super cool, unless the app can’t scale up efficiently. Having a 60 second app startup time can wreak havoc on system scaling. You might wind up spinning up way too many instances because your app is slow to load. Or, some lucky user will get to sit there waiting. App Engine provides mechanisms to help in these cases — idle instances. Idle instances are like a buffer. If your app starts getting more traffic, App Engine will use idle instances to serve that traffic and then add additional instances to preserve your specified idle instance count. This acts like headroom for your app. The faster your startup time, the fewer idle instances you’ll need. That translates to cost and performance.
Closely related is your request latency. If your requests are slow you’re going to need a lot more instances to handle load. Small, fast requests play better. Sadly this is tricky to accomplish. My general suggestion is to target something below 1 second, ideally well below. I’m sure some developers would be surprised at the number of applications we encounter that would call a 1 second response time extremely fast. Like your application startup time, request times directly impact both your cost and user experience. If you want fast requests, you need to design for it, particularly your data models.
Speaking of data models, they matter. Your data story impacts everything on top. Designing models to work well for a stateless application with (potentially) high degrees of parallelism is difficult. Models need to be designed to minimize the number of reads to serve your most frequent requests while minimizing the number of writes required when updating them. If multiple calls might concurrently write data, transaction boundaries will need carefully thought through. If you’re coming from an enterprise SQL world, you're probably accustomed to normalizing your database and wrapping an entire request in a huge transaction. Do that and you’ll be in for pain. Instead, you need to denormalize models and keep transactions tight.
Data modeling would take up an entire post. If you use Cloud SQL, you’ll be in a world similar to other environments. To really leverage App Engine, check out the Cloud Datastore. My epiphany when working with Datastore was this: think like an RDMS would think. If it needs a temporary table, you’ll need a “temporary” table that you maintain. If it would require a full table scan, you’ll be doing a full table scan. Design to make your common operations efficient. Leverage task queues to update background records.
App Engine’s Task Queues are amazingly useful for background or asynchronous work. They aren’t threads and really shouldn’t be used like threads. You should also be very attentive to ensure tasks are always idempotent. They can run multiple times and sometimes fail and retry at the very end of a request. Leverage task names to help with deduplication and be careful to avoid task-bombs (like a fork-bomb, but distributed). Also be careful of long-running tasks. Instead, leverage chaining (sequential workers) or forking (parallel workers) to break work up and reduce failure domains.
Memcache is not a database. It will occasionally be reset or your app could be migrated to a different data center where Memcache is not pre-loaded. Make sure your app can effectively operate without Memcache. We see apps build Memcache into their stack as a critical component a lot. It is dangerous.
Use services. Don’t just build a monolithic app (though it is OK to, up to a point). I generally like to split my “backend” (i.e. task queue) traffic up from my user traffic using services. If I have commonly hit endpoints, I like splitting them into a dedicated service as well. That allows me to ensure the instances are sized appropriately and optimize this service for cost and performance. Lastly, sticking your admin pages on a dedicated service can sometimes be helpful as well. Google provides some documentation on how to break your application into multiple modules and services.
A well designed, modular app can be split out to run off App Engine. It takes some thought and care, but if you use good internal APIs you will be able to pull portions off GAE. Once you’re ready to start splitting services out, you’ve got a reasonable path with App Engine Flex and GKE.
Those are the key areas where we see developers struggle. If you can learn to design for those limitations, you’ll be surprised at how capable the platform is. I love using App Engine to quickly prototype ideas. If you’re following a lean mindset, it is brilliant. You can quickly deploy a very capable backend and start iterating on your product. The cool thing is that if you start hitting success, you’ll be able to focus on your product instead of solving multi-regional deploys and failover.
Remember App Engine is fundamentally a Serverless platform. All of the above items are applicable when developing other serverless applications as well. You don’t have to go as fine-grained with App Engine, but you’ll get a lot of the same benefits.
Our team at Real Kinetic has extensive experience building serverless systems on and off Google App Engine. Contact us to learn more about working with us.
