Trials and Tribulations with serverless in AWS

So I needed to create a simple database backend with an api for a mobile app. It’s a small, simple database, with several classical join operations. Perfect for MySQL. It has an unusual characteristic: it’s only going to be used once or twice a month for a few days. It’s a problem management system for national fencing tournaments, used by officials to report and resolve problems in the tournament. I have the app working in Xamarin, and this seems like something trivial in AWS.

Because of the very low, sporadic use, this seems like serverless is ideal – only pay when we use it, and we don’t use it much. Probably fits in the free tier, but we’re willing to pay a bit if it doesn’t. Lambda is the obvious choice for the business logic, I’m using Swagger (OpenAPI) to define the interface, and AWS API Gateway to deploy it. It’s very easy to trigger the Lambda functions from the API operations. I’ve already figured out how to use Cognito for app login, and I’m using SNS for the messaging notifications to the app. All Amazon all the time, right?

So, AWS has serverless Aurora, which is a pretty straightforward MySQL system. Price is right, has the right level of scaling, and, again, serverless.

But you just try to make a Lambda function that calls Aurora Serverless. It’s AMAZINGLY difficult.

Aurora Serverless, unlike, say, DynamoDB (AWS’s NoSQL database) is all wrapped up in
Virtual Private Clouds. The only way to get access to the db is through a VPC. Okay, I’m a network guy. I’ve been around the block a few times: I was on the machine room floor when the Arpanet IMP was winched in to Carnegie Mellon. I worked at Xerox Parc in the glory days, and Bob Metcalf taught me about Ethernet. VPCs aren’t all that complicated – subnets, ACLs, yada, yada. Seems like a lot of work just to get at a serverless db. And it really is a LOT of work. The most surprising thing is that to get Lambda to talk to Aurora, you need a NAT. What? Yeah, two internal AWS services need a NAT between them. It’s that VPC thing. Yes, I do want the database secure; only Lambda can access it, plus my provisioning path (SSL from my desktop). But this is 2019. We don’t believe in walled gardens anymore, we use TLS, authentication and authorization. We don’t need really static IP addresses, DNS handles everything nicely. And we’re out of IPv4 addresses, all the mobile devices have IPv6 addresses and we should be doing all-IPv6 for new apps.

Not Amazon. It’s private address spaces, ACLs and dynamically managed public IPs. IPv4 only. SSL optional. And it’s complicated. I’ve poured through maybe 100 pages of documentation trying to make this all work. Used the forums, paid for support. Ran wizards. Followed tutorials.

Never got it to work. As close as I came was a stand alone DB on a simple VPC with a Bastion Host EC2 instance that let me connect MySQL workbench to the DB instance. When I tried to hook up to Lambda, it was start over, much more complex VPC, and then the NAT.

Amazon has two NAT thingies – a managed NAT gateway, and a DIY NAT “instance”. The problem with the NAT gateway is cost. It would cost hundreds of dollars a year to run that. The NAT instance is code on an EC2 server. A wizard makes the instance, and it runs on the smallest config. You can trigger cron jobs to turn it up and down.

But you need an elastic IP (a not quite so static IP), for reasons I don’t understand – everything else in AWS for this app is basically happy to use DNS and not need elastic IPs, even the website, hosted in an S3 bucket and Lambda, and running on my own domain. Not Aurora serverless connecting to Lambda – you need an NAT and a static IP. TLS is optional, and not as easy as it should be. Documentation is roughly a half a page. Never got it to even think it was working. Nothing talked to anything else. Couldn’t see the DB from Lambda, couldn’t SSH to it. Like it wasn’t there. No doubt something having to do with the VPC config, because Aurora wasn’t happy with the wizard created VPC.

So I backed it all out. Deleted all the EC2 instances, the databases and the VPCs. Redid the whole thing in DynamoDB. It’s not a great fit, but I can make it work NoSQL. And it’s trivial – create an IAM user with the right permissions and I’m off and running with the DynamoDB api. I had a simple Lambda function working in 30 minutes with zero prior experience with DynamoDB.

What’s up with that?