Creating an API with Amazon API Gateway and AWS Lambda

Programming
1 Comment

This is all about creating your first API in the Amazon cloud, serverless (= using someone else’s servers). In this example I’ll create:

  • An AWS Lambda function (in Python) that is the code that does the things I want to do when someone calls my API
  • An Amazon API Gateway that is the frontend for the Lambda function and that will be listening for the incoming API (HTTPS) requests.

In this post:

My initial goal is to create an API that returns “Hello!” when someone calls it.

If you want to follow the steps, the prerequisites are:

  • You should have an account in AWS
  • You should have relevant access (IAM role) to do the tasks we do; I’m using my testing root account here, but in the real life you should have some limited-access account to configure and run things especially in business use
  • You should have relevant understanding about the pricing of the components that are deployed here (see the product links above and later in this post)
  • Optional: If you want your API code to call other APIs or otherwise fetch data from somewhere, you will need a VPC configured with subnets and the relevant connectivity (like Internet Gateway or NAT Gateway or Direct Connect or Virtual Private Gateway, you know, something that implements the connectivity from your VPC to somewhere else)

I will work my way here just in the AWS console in the browser. When you progress in your cloud implementations, you will probably want to take advantage of AWS CloudFormation or some other kind of templates to create and maintain your API programmatically. My selected region is “Europe (Stockholm)” (eu-north-1), and all my links will use that, so be mindful of that if you get in trouble right away. Also note that the user interface can change at any time, so if you are reading this sometimes later, it isn’t maybe the same anymore. But here we go anyway.

Creating the API Gateway

I’ll start in the API Gateway console:

Because I didn’t have any APIs configured yet it presents me the API types. I’ll select HTTP API because that’s the simpliest one I think and it does everything I need right now. So let’s start with Build. It leads me to Step 1, Create an API:

I don’t have any Lambda function yet so I’ll just skip the Integrations part but I’ll enter the name for my API: My First API. Next: Step 2, Configure routes:

I am very much interested in the ANY method (for simplicity for now), but the Add route button is disabled because I don’t have anything that I could attach the routes yet. Next: Step 3, Define stages:

For now the stages are something I’m not interested in. Next: Step 4, Review and create:

Let’s go, Create, and here we have it:

It automatically generated an HTTPS URL for reaching my API. No hassle for creating and maintaining a TLS certificate myself! This is totally fine for my own use, but for wider and public use some custom domain name is probably better anyway, so check it out later as needed.

Let’s test the API right away from my home network:

markku@devel:~$ curl -i https://4gsv2dhgoj.execute-api.eu-north-1.amazonaws.com/ ; echo
HTTP/2 404
date: Fri, 09 Apr 2021 17:23:07 GMT
content-type: application/json
content-length: 23
apigw-requestid: dhsLTicYgi0EPFw=

{"message":"Not Found"}
markku@devel:~$

The -i option in curl causes it to show the response code and headers, along with the data. So, the API Gateway returns me a 404 with the error message in JSON format. Good so far as I don’t have anything yet to be served by the API.

Creating the Lambda function

Writing “lambda” in the search box on top of the screen and clicking the relevant link leads me to AWS Lambda console:

Create function:

I’m using Author from scratch, so I’ll enter My-API-Function as the function name, and select the latest available Python from the Runtime list.

Now, if you want to connect the function to your VPC (to be able to reach your EC2 instances or something like that from your function), you would open the Advanced settings and proceed there. But I don’t need that now.

Clicking Create function does what’s expected:

Double-clicking lambda_function.py in the Code source navigator shows the code. It’s short but looks complete Python, so let’s click the orange Test button:

If I wanted to test my function with various inputs, I could enter the input data there. The defaults are just fine, but an Event name is required, “test” is good enough, so Create. It brings us back to the Code source screen, so let’s click the Test button again, this time with actual results:

It looks like working. There is the response data: a Python dict with statusCode and body keys. The log below it shows the execution details. In this example the function was created with 128 MB of memory whereof 50 MB what actually used to run this. Outside the screenshot it also said that the Init Duration was 113 ms in this case.

Right, now let’s modify the source a bit because the response output was not exactly what I promised. So I’ll edit the source code (json.dumps('Hello!')). It says “Changes not deployed” so I’ll click Deploy, and it looks happy:

Clicking the Test button verifies it, the output text is changed (believe me, no screenshot this time).

So basically my Lambda function does not do any input checking at the moment, it just returns the dict when executed with whatever input. Fine for me! Let’s go back to the API Gateway.

Adding the integration in API Gateway

Before adding the integration I actually need to have a route configured. A “route” can mean something else as well but in this context it means a way of entry for the application or function. So after clicking My First API I’ll go to Routes:

Create is all I can do there:

The defaults are fine (ANY as the method and / as the path), so Create.

Then I’ll go to Integrations:

It advices to select a route, so I’ll select the ANY route I have:

Using the Create and attach an integration button leads me to a list where I can select my Lambda function:

Create, and it shows the integration:

I think this is the time to test the API again!

Testing the API

markku@devel:~$ curl -i https://4gsv2dhgoj.execute-api.eu-north-1.amazonaws.com/ ; echo
HTTP/2 200
date: Fri, 09 Apr 2021 17:57:34 GMT
content-type: text/plain; charset=utf-8
content-length: 8
apigw-requestid: dhxOMgW9gi0EPkg=

"Hello!"
markku@devel:~$

Wow, it works! Just like that. Status code is 200 as specified.

I small detail is that the hello text is surrounded by the quotes. That’s because the output is JSON, and the returned object is a text object, so it is quoted. In the real usage my function would return some useful data, like this:

markku@devel:~$ curl -i https://4gsv2dhgoj.execute-api.eu-north-1.amazonaws.com/ ; echo
HTTP/2 200
date: Fri, 09 Apr 2021 18:02:07 GMT
content-type: text/plain; charset=utf-8
content-length: 38
apigw-requestid: dhx46jOmAi0EMDg=

{"status": "OK", "rolling_version": 1}
markku@devel:~$

Whenever receiving some JSON in curl you may want to pipe the output to jq (if jq package is installed and available in your system). In this case the -i option must be removed (because its output is not JSON) and -s (silent) option is useful, like this:

markku@devel:~$ curl https://4gsv2dhgoj.execute-api.eu-north-1.amazonaws.com/ -s | jq .
{
  "status": "OK",
  "rolling_version": 1
}
markku@devel:~$

Logging

At this point I should check some logging. By default, the Lambda function logs in CloudWatch, so I’ll open the CloudWatch console and open the Log groups in the left:

There is a log group /aws/lambda/My-API-Function that looks interesting, so let’s open it:

There are several log streams already. Clicking the topmost (most recent) one:

Basically there are logs about every function invocation. My function did not output anything (just returned the data), so there are only the default START/END/REPORT entries.

Let’s add some output in the function:

pprint was imported in line 2 and then the event argument is pretty-printed.

Now let’s call the API again:

markku@devel:~$ curl https://4gsv2dhgoj.execute-api.eu-north-1.amazonaws.com/ -s | jq .
 {
   "status": "OK",
   "rolling_version": 1
 }
 markku@devel:~$

There was no change in the API response. But that was kind of expected: the return statement still returned just the status and rolling_version data.

Let’s look at the CloudWatch log group again. There is a new log stream (file) available, and it looks like this:

All the text output by the pprint() call was saved in the CloudWatch log stream. So those are all the contents of the event argument for the Lambda function. The indentations in the pprint() output were not preserved, so let’s use another way to inspect the data:

Calling the API gives us nice output (this time as a screenshot with colors):

Anyway, about the logging, instead of just printing to standard output you can use the logging module from the Python standard library, just like you maybe do in your other Python applications as well.

Handling input in the Lambda function

Let’s try a POST call to the API with some input data:

markku@devel:~$ curl https://4gsv2dhgoj.execute-api.eu-north-1.amazonaws.com/ -s -X POST -d '{"input_data": 111}' | jq .
{
  "status": "OK",
  "rolling_version": 1,
  "event_data": {
...
    "body": "eyJpbnB1dF9kYXRhIjogMTExfQ==",
    "isBase64Encoded": true
  }
}
markku@devel:~$

Looks like the body is there but it is Base64 encoded. We can handle it!

Testing:

markku@devel:~$ curl https://4gsv2dhgoj.execute-api.eu-north-1.amazonaws.com/ -s -X POST -d '{"input_data": 111}' | jq .
{
  "status": "OK",
  "rolling_version": 2,
  "request_body": "{\"input_data\": 111}"
}
markku@devel:~$

So there we have it, we can access the data that was sent to the API in the POST call. At this point the input data is still just a string, but a simple json.loads(body) can be used in the code (instead of body) to access the data as Python dict:

markku@devel:~$ curl https://4gsv2dhgoj.execute-api.eu-north-1.amazonaws.com/ -s -X POST -d '{"input_data": 111}' | jq .
{
  "status": "OK",
  "rolling_version": 2,
  "request_body": {
    "input_data": 111
  }
}
markku@devel:~$

Final touches

Now, if I test the API call again with plain GET, I’ll get an error:

markku@devel:~$ curl -i https://4gsv2dhgoj.execute-api.eu-north-1.amazonaws.com/ ; echo
HTTP/2 500
date: Fri, 09 Apr 2021 19:23:04 GMT
content-type: application/json
content-length: 35
apigw-requestid: dh9vygEUgi0EMDg=

{"message":"Internal Server Error"}
markku@devel:~$

The output message is from API Gateway. When the Lambda function does something unexpected (like crashes in an exception), API Gateway emits an error. The actual reason for the error can be found in the CloudWatch logs (check it yourself if wanted), and the reason here is trying to access event["body"] even though there is no body in the request (because it was a GET request).

In this case let’s just check the existence of the body key and adjust the variables accordingly:

GET works now:

markku@devel:~$ curl -i https://4gsv2dhgoj.execute-api.eu-north-1.amazonaws.com/ ; echo
HTTP/2 200
date: Fri, 09 Apr 2021 19:29:01 GMT
content-type: text/plain; charset=utf-8
content-length: 60
apigw-requestid: dh-nnjo9Ai0EM5Q=

{"status": "OK", "rolling_version": 2, "request_body": null}
markku@devel:~$

… Which now leads me to another finding: the content-type is said to be text/plain, but I’m trying to return proper JSON. So I’ll add a header in the response data:

Now it looks correct:

markku@devel:~$ curl -i https://4gsv2dhgoj.execute-api.eu-north-1.amazonaws.com/ ; echo
HTTP/2 200
date: Fri, 09 Apr 2021 19:32:19 GMT
content-type: application/json
content-length: 60
apigw-requestid: dh_Gdjoigi0EJbg=

{"status": "OK", "rolling_version": 2, "request_body": null}
markku@devel:~$

If you are further interested in the format that the API Gateway expects the Lambda function to return, see Handle Lambda errors in API Gateway document.

Happy API-ing in the cloud!

1 Comment

Add a Comment
  1. JK
    JK

    Excellent Post! Thanks!

    Reply

Leave a Reply