Debugging Tests with Distributed Tracing for LocalStack Apps – InApps 2022

LocalStack has an amazing community, and everybody loves the product. Not only does it save organizations money, but it also enables a highly efficient local development and test loop for developing cloud apps.

Since the local testing framework is provided by LocalStack, the need to set up multiple testing environments on the cloud disappears and distributed microservice applications can easily be developed, tested and deployed with green tests.

Not all the time, though! It’s a fact that tests fail. Some erroneous tests make developers tear their hair out looking for the root cause in log piles.

This is when Thundra comes to the rescue with Foresight for those who use LocalStack on their continuous integration (CI) pipeline. Foresight helps developers understand the root causes of failing and long-running tests easily and quickly. It gives rich insights about the passed, failed and aborted tests of the test suites.

Sample App

We’ve developed a small application that’ll spawn multiple AWS services on your local machine with LocalStack and monitor your distributed architecture and its tests with Thundra Foresight.

An AWS Lambda function, let’s call this Lambda #1, with an HTTP endpoint that takes a request, then writes to an Simple Queue Service (SQS) queue for further processing. Then, Lambda #1 writes the same request into a DynamoDB database for bookkeeping purposes. The SQS queue passes the incoming message to the processor Lambda, our Lambda #2.

Once the processing is successful, Lambda #2 notifies a Simple Notification Service (SNS) topic, which triggers our archive Lambda, Lambda #3. When Lambda #3 is triggered, it will update the request’s record on DynamoDB. It will also write the result of this whole request into a file and put it in an S3 bucket.

This might seem a lot, but all it does is pass the body of an HTTP request between different services and manipulate it along the way. This project is developed by Thundra’s software engineer Tolga Takır and the source code is available in our GitHub repository. In there, you’ll find the documentation on how to run this project on your machine.

After having all the requirements set up on our computer, we can simply run make start to spin up our application with LocalStack. This will take a couple of minutes, but once it’s completed, we’ll have our API running inside the LocalStack container.

At this point, we can run the following command to see if our API is up and running.

We’ve confirmed that our API is up and running. Let’s see what Thundra Foresight can do now.

How to Integrate Foresight into LocalStack

If you have the latest version of LocalStack, all you need to do is set THUNDRA_APIKEY as an environment variable, and you are good to go with application monitoring. To enable Foresight as well, THUNDRA_AGENT_TEST_PROJECT_ID environment variable must be set.

You can get your API key and project ID at https://foresight.thundra.io.

For this project, you can set them in the Makefile. If that’s done, let’s run the tests with make test. After some time, we should be able to see the results.

Looks like our test has failed. Let’s check Foresight, and then click to our test to see if we can see what went wrong.

When you are on the test detail page, you can click the Trace Map button on the top right. It will open a window and automatically redirect you to your invocations trace map. Here, we can see the whole flow, identical to what we’ve designed at the beginning of this post. We will also be able to tell where the error comes from. If we click the JUnit 5 node on our map, we’ll see the whole trace chart for our test.

On the right-hand side of the screen, we see the original assertion error and its method marked as red on the trace chart. Let’s click to that second red row.

This opened our debugging window again. Now, let’s zoom in on that and play the execution until the end.

We can see that our getResponse list doesn’t have the FINISHED state as expected.

Now, we have some idea where our error originated and what variables were present at the time of the execution of our test. We call this Time Travel Debugging. With this instrumentation method, Thundra Agent will give you the ability to save your traces and play them line by line to debug your application and your tests.

Now, let’s see how we can fix this.

Chaos Is the Answer

Well, usually developers write tests to ensure their application doesn’t break when they develop new features, fix bugs or change anything in their codebase. Generally, this will give you a good estimate of how robust your application is. Not all errors come from our inability to write bug-free code, however.

Purposefully injecting a bug, an exception or latency might seem counterintuitive, but it’s better than the unexpected. This practice of doing a controlled experiment on your distributed system is called chaos engineering. At Thundra, we value the chaos, do our best to be prepared for the unexpected, and we want our customers to be the same, if not more. By using Thundra Agent in your project, you’ll have the capability to engineer your own chaos experiment for your applications and your tests.

So, that’s what we did here. If you go to the following file, you’ll see that we’ve implemented a class for chaos injection and injected an error to our Lambda #3.

To fix the error, we can simply comment out these lines. Let’s run the tests again and see our tests become successful.

In addition to application-level chaos engineering with Thundra, we can also inject errors at the infrastructure level with LocalStack, for example, injecting Kinesis stream errors by configuring KINESIS_ERROR_PROBABILITY=0.8. Let’s not go into too much detail here. We’ll cover that in a separate blog post.

Summing up: LocalStack users can now have a granular view of their tests by plugging in Thundra Foresight with the demonstrated steps.

You can sign up for Thundra and integrate your open source project for “free forever and with full functionality.”

Photo by Stacey Gabrielle Koenitz Rozells from Pexels.