Cucumber + Junit5 + Localstack


Background


What are the options to test an Amazon lambda that interacts with Dynamo DB locally? To be honest, there are several options, such as using Mockito, using stubs, swapping Dynamo DB with Mongo DB, however none of these truly replicate the amazon lambda and dynamo db ecosystem.

Enter Localstack  (https://github.com/localstack/localstack), open source project that aims to emulate a local AWS cloud stack. With its docker version, we could spin up localstack and push our lambda and create our table in the dynamodb. To further increase its testing prowess, Localstack comes up with utils to integrate with Junit 4 and 5. Now we have the tools to create a BDD test using Cucumber, JUnit5 and Localstack as you can see below. Do note that due to the open source nature of localstack, it has several bugs that we manage to get around

Prereq:


  • Docker images: localstack/localstack:0.9.1
Note that the latest version of localstack docker image has bugs with the service port numbers
  • localstack lib: cloud.localstack:localstack-utils:0.1.15
Note that localstack 0.9.1 tested to work with 0.1.15
  • cucumber: cucumber-* 4.3.0
  • junit5: junit-jupiter-api:5.4.2
  • aws lib: aws-java-sdk-*:1.11.567
  • spring: spring-*:5.1.5.RELEASE
Spring is needed to provide cucumber with dependency injection for World object, alternatively you could use singleton class
  • shadowJar 5.0.0
ShadowJar is needed to explode all jars into a single uber jar during build time since localstack only works with an uber jar



Explanation, Tips and Tricks

  • Localstack only works with uber jar, creating an uber jar using shadow jar causes complication with jackson library, therefore we need to relocate the jackson libraries
  • Localstack always expect a json object as request and could not handle native variable or object, therefore the parameter to handleRequest() method in the class that implements RequestHandler will have to be a POJO
  • Localstack always default to region of "us-east-1", it doesn't care about credentials but something must exists and endpoint always be localhost with specific port for the service
  • Pay close attention on CucumberLocalstackTestRunner.java, it uses JUnit5 annotations, and invoke cucumber cli as an ordinary test, a workaround on using JUnit5 with cucumber until someone create cucumber-junit5 integration.
  • In addition, there is a fix using reflection to fix defect in localstack utils code which shall be explained below

Further Explanation

Cucumber with JUnit5
At the moment there is no cucumber junit5 integration library, therefore one way we can get away is to call cucumber cli in a JUnit5 ordinary test and check for failure:

@Testpublic void cucumberTest() {
  String[] argv = new String[]{"--plugin", "pretty", "--glue", "com.sevnis.localstackcucumberjunit5demo",      "src/test/resources/cucumber.feature"};  ClassLoader contextClassLoader = Thread.currentThread().getContextClassLoader();  byte result = Main.run(argv, contextClassLoader);  assertTrue(result == 0);}


LocalstackUtils Hack
LocalstackUtils try to obtain the service to port map from docker directly, however to some environment the docker exec command would return nothing and therefore ends up in empty map and eventually failure, to fix it we force check and update the port to map using java reflection

private static void extendLocalstack() throws Exception {
  Field serviceToPortMapField = LocalstackDocker.INSTANCE.getClass()
    .getDeclaredField("serviceToPortMap");  
    serviceToPortMapField.setAccessible(true); 
 
    Map<String, Integer> serviceToPortMap = (Map<String, Integer>) serviceToPortMapField
       .get(LocalstackDocker.INSTANCE);

  if (serviceToPortMap == null || serviceToPortMap.isEmpty()) {

    Field containerIdField = LocalstackDocker.INSTANCE.getLocalStackContainer().getClass()
       .getDeclaredField("containerId");    
    containerIdField.setAccessible(true);  
  
    String containerId = (String) containerIdField.get(LocalstackDocker.INSTANCE
       .getLocalStackContainer());

    String localStackPortConfig = new DockerExe().execute(Arrays.asList("exec", "-i", 
       containerId, "cat", "/opt/code/localstack/.venv/lib/python2.7/site-packages/" + 
       "localstack_client/config.py"));

    Map<String, Integer> ports = new RegexStream(
        Pattern.compile("'(\\w+)'\\Q: '{proto}://{host}:\\E(\\d+)'")
        .matcher(localStackPortConfig)).stream()
        .collect(Collectors.toMap(match -> match.group(1),            
            match -> Integer.parseInt(match.group(2))));

    serviceToPortMapField.set(LocalstackDocker.INSTANCE, 
    Collections.unmodifiableMap(ports));  
  }
}


Latest Localstack Docker 
Localstack Docker version 0.9.1 works with LocalstackUtils 0.1.15 however the runner only run with latest version and will check and download the latest localstack if it couldn't find one. In order to go around this issue do the following:
  • pull docker image for localstack/localstack:0.9.1 and then tag it to localstack/localstack:latest
  • set pullNewImage = false in the LocalstackDockerProperties
@LocalstackDockerProperties(pullNewImage = false, services = {"lambda", "dynamodb"})
public class CucumberLocalstackTestRunner {


References






Comments

Post a Comment

Popular posts from this blog

Image
Implementing gRPC Using Spring Boot and Gradle Background GRPC (gRPC) is a RPC framework pioneered by google later on made public and open source. GRPC has not been officially supported by Spring Framework, so far there are few community libraries that can work seamlessly with Spring Boot. In this article, I will use the lognet's grpc-spring-boot-starter. Project Structure Ideally, the proto files for gRPC is located in another directory, which will then be accessed by the gradle plugin to auto generate the required files.   Gradle Script The heart of gRPC enablement lies on the grpc boot starter, which includes few other tasks outlined as below. https://github.com/overtakerx/grpcbootdemo/blob/main/build.gradle Few important note: first you have to run gradlew compileJava for the auto generation to generate the files from the proto file, then you will need to rebuild the project for Intellij to recognise the auto-generated files. Full demo: https://github.com/overtakerx/grpcbootdem
Read more

Spring Boot 2: Parallelism with Spring WebFlux

Image
Spring Boot 2: Parallelism with Spring WebFlux Parallelism is rather unusual in Spring application, however when we encounter the need of HITLOL (HIgh Throughput LOw Latency), we need ways to make async calls. With Spring Boot 1, in high level definition, we can declare async on all the functions that are meant to be parallel and return Futures from there. Here is more detailed info:  https://medium.com/sipios/how-to-make-parallel-calls-in-java-springboot-application-and-how-to-test-them-dcc27318a0cf For Spring Boot 2, we shall employ Spring WebFlux libraries to accomplish parallelism. Mainly focused on functions: Mono.zipDelayError(), Mono.subscribeOn() and Mono.onErrorResume() The main function that drives the parallelism is zipDelayError(), the purpose of zipDelayError() as supposed to using zip() is to enable the other Monos to run to completion even if one Mono failed. AllDtos allDtos = Mono. zipDelayError (Arrays. asList (firstDto , secondDto , (Mono&l
Read more

Spring Boot Reactive API Part 2

Image
 Spring Boot Reactive: Improving CPU bound performance using Scheduler In part 1 , we have seen that Spring Boot Reactive doesn't really improve performance if the WAITING time is CPU bound / CPU intensive tasks. This article show a quick way on how we can improve the speed through specific configuration Setup Spring Boot's embedded Tomcat is reconfigured to only use 10 worker threads, this way we can clearly see the performance limitation and tweaks. Gatling is configured to simulate 30 concurrent requests with 30000 milliseconds request timeout. APIs internal process is calling java native code that calculate the value of PI over 1 million iterations Default / No scheduler The main reason why CPU bound tasks do not improve (or even degrade) with high parallelism is because of the costly overhead for context switching in the low level. In the scenario below, 10 main threads are all being used to calculate PI values and because there are less CPU than threads, a lot of context
Read more