Serverless and its tradeoffs

When I first heard of Serverless I thought it was a dumb idea. The implication that the servers had been magic-ed away angered me on a deep level. I was worried we had hit peak snake-oil.

On the one end you have bare metal. At this end of the spectrum, you have the greatest degree of flexibility, but also the greatest degree of operational responsibility. You can install and run exactly the version of Perl and insecure OpenSSL that your legacy HR depends on. But equally so you need to worry about operating and maintaining every aspect from the box up. When things like Spectre and Meltdown surface you’d better be ready to respond. This end of the spectrum is the right place to be if you believe your team/company can differentiate on operations. It’s unlikely you can Ops better than someone working for AWS in the general case, but you do know your particular software better than they do. If you can leverage that knowledge to drive costs down or availability up then this is a great place to be.

There is not a one-size-fits-all solution: it’s a spectrum of tradeoffs. You need to be aware of the tradeoffs at each point and choose a solution that fits your context. My team recently identified a number of discrete areas of a service that made sense to break out into their own smaller microservices. Though important, they were each fairly “boring” from an operational standpoint, so we chose AWS Lambda as the compute. We’ve been operating these now in production for a few months and I wanted to share what I’ve learnt through practice.

So what does this architecture look like? Running in AWS, your serverless application will be one Lambda (your code) fronted by one API Gateway (your interface). That’s it. Customers will make HTTP requests to your API Gateway. API Gateway will then apply the validation, throttling, and security that you’ve configured it to do. Valid requests will be forwarded to Lambda. At this point, your code will look very much like traditional service code: you’ll receive a request, which is a deserialisation of the HTTP request, you’ll do some work (maybe reading or writing to DynamoDB) and then you’ll return a response to be serialised back to HTTP.

You’ll be struck by how simple an API Gateway-fronted Lambda (let’s call it a Lambda Service) looks. The simplicity comes from the things that, at least you, do not have to worry about. You don’t manage a load balancer, or its TLS certificate, or a network, or an IP, or the monitoring of your hosts, or many other things. Somebody does, of course, but they are specialists in doing it. They will automate and build quality into all of those operations because their economies of scale will make doing so worthwhile.

Equally so, by keeping the choice limited and the interface simple, reliability goes up. Fewer moving parts makes the system (both the one you see and the one you don’t) easier to reason about, which makes availability easier to reason about. A Lambda Service is the Model-T Ford of services.

Cloudwatch and X-Ray continue to improve, and to date they’ve been enough for us, but I do wonder (perhaps in a system where 80% of your code is in Lambda Services) where their boundaries will lie.

It will take time to build up your intuition around how much to put into each Lambda Service. There are no Laws to follow when deciding this, and not a lot of best-practice to be found. Here’s my contribution:

The tradeoff you are making is around speed and impact of change versus coherency. If you have a lot of smaller Lambda Services then each one will be easier to update, and if you make a mistake the blast radius should be smaller. Equally so, more services (Lambda or not) means more complexity at the system level. Engineers will need to spend more time thinking about where the bug could be, or where their new feature should go, or exactly which features will be up or down in the case of an outage.

Given the dearth of best practices in this space, I decided to start with a heuristic to help me decide how large each Lambda Service should be. The heuristic was that each single controller (in the MVC-definition of the word) would become its own Lambda service. A service often only has multiple controllers and data types because the perceived cost of spinning up and operating a new service outweighs the perceived benefit of keeping components discrete. Serverless is a good opportunity to change that perception since its tradeoffs are different.

Reflecting back, I think this heuristic worked out fairly well. It doesn’t handle situations where a particular field on a table (perhaps a tracking ID) only makes sense in the context of the original service, and not in the isolation of the new Lambda service. Next time I’ll take more time up-front to think about where best to store the data that ties services together. Another architectural pattern might be needed here to handle this case.

Since Lambda is scaled elastically, you will experience a “cold-start” when the number of concurrent requests exceeds the number of available instances. This cold-start takes time as a new instance is provisioned. There’s a lot of magic under the hood that makes this very fast, but in the worst case it can take multiple seconds before its done.

Mine, and most people’s, intuition around Lambda cold-starts is bad. At first it seems like a deal-breaker: you can’t leave your customers waiting multiple seconds for an operation to complete that should return in milliseconds. The reality, though, is entirely different and the effects of cold-starts much more subtle.

Firstly, you will only receive the penalty of a cold-start when your load increases. Take any of your customer-dependent services and look at its load. Assuming you have a decent amount of traffic, you will find it remarkably consistent throughout the day. As long as traffic remains roughly steady you’re fine (instances stay warm for a while after they finish, to see if more traffic arrives). It’s often only when you get a sudden jump, perhaps due to a marketing email, that customers will be impacted by a cold-start.

Secondly, your web-site isn’t as fast as you think. Running some measurements of popular websites, I can see that many (highly-optimised) sites are difficult to interact with until at least the 2 second mark. Some elements may load faster, but until the DOM stops changing and my brain can process what it’s seeing 2 seconds seems about the soonest I can make use of it. Try it out yourself, and use that information to contextualise the cost of a cold-start.

Thirdly, cold-start times are a function, primarily, of the instance type. In the case of Lambda, the more memory the instance has the faster it starts (somewhat counter-intuitive, but for Lambda memory correlates to CPU). In practice we’ve found 1.5GB instances starting in a few hundred milliseconds.

If you let your intuition around cold-starts put you off Lambda Services entirely then you’re missing out on a significant opportunity. Run some experiments, collect some data, and then decide.

I’ve written a lot more than I originally intended, so I’m going to call it a day. There are things here I haven’t dug into, like testing, service creation automation, and reasoning about the system, that will probably end up in a Part 2.

In the meantime, this article hopefully adds something useful to the discussion around Serverless.

Related posts:

Have you ever seen a YouTube video you liked so much you wanted to keep it forever? With the way YouTube hands out copyright strikes like free candy, it’s best to save what you can, when you can. Any…

Local law enforcement officers are on the frontline of both public safety and community relations. America’s law enforcement officers put on their uniforms every day to help make our communities…

As much as I believe in you, I sometimes live my life like you barely exist at all. I find myself so overwhelmed with the world that I forget the solid foundation of your word. You know how heavy my…