Transactional Outbox Pattern

Transactional Outbox Pattern, Why? (part 1)





Background

In microservice and event driven architecture, transactional outbox pattern is crucial to maintain the state of data (or "aggregate" as per Domain Driven Design) in two or more persistent storages, especially when 2 Phase Commit is not available nor desired. You can read an excellent explanation of the pattern here. This article will be divided into two parts, part 1 will discuss why we need it, part 2 will discuss how we can implement it.

In my experience designing solutions with transaction outbox pattern, I have come across multiple similar questions challenging why do we have to do it in such a complex way, and why can't we just reverse the order, use try catch clause, database transaction, etc. Based on those frequent conversations, here is my attempt to explain why we can't solve the problem with ordering, try catch clause or transactions.

Suppose we have a system where we create a person record, which will be stored in postgresqlDB and published to a Kafka topic for other microservices to subscribe and act on it.


1. PostgresqlDB first then Kafka Topic

Summary: save the data to the postgresqlDB table first, catch any errors, if no error then publish the data to the kafka topic

Problem: publishing to kafka could fail right after successfully saving person A to the database, this will result in inconsistency of the data in the system whereby database contains person A but kafka topic doesn't




2. Kafka Topic first then PostgresqlDB

Summary: publish the data to kafka topic first, catch any errors, if no error then save the data to the postgresqlDB table

Problem: database save could fail right after successfully publishing to kafka topic, this will result in inconsistency of the data in the system whereby kafka topic contains person A but database doesn't 





3. PostgresqlDB transaction over Kafka Topic

Summary: start a database transaction, save the data to postgresqlDB first, publish the data to kafka topic, then commit the transaction

Problem: database transaction could fail (for example, network failure) right after successfully publishing to kafka topic, this will result in inconsistency of the data in the system whereby kafka topic contains person A but database doesn't 






Popular posts from this blog

Spring Boot 2: Parallelism with Spring WebFlux

Image
Spring Boot 2: Parallelism with Spring WebFlux Parallelism is rather unusual in Spring application, however when we encounter the need of HITLOL (HIgh Throughput LOw Latency), we need ways to make async calls. With Spring Boot 1, in high level definition, we can declare async on all the functions that are meant to be parallel and return Futures from there. Here is more detailed info:  https://medium.com/sipios/how-to-make-parallel-calls-in-java-springboot-application-and-how-to-test-them-dcc27318a0cf For Spring Boot 2, we shall employ Spring WebFlux libraries to accomplish parallelism. Mainly focused on functions: Mono.zipDelayError(), Mono.subscribeOn() and Mono.onErrorResume() The main function that drives the parallelism is zipDelayError(), the purpose of zipDelayError() as supposed to using zip() is to enable the other Monos to run to completion even if one Mono failed. AllDtos allDtos = Mono. zipDelayError (Arrays. asList (firstDto , secondDto , (Mono&l
Read more
Image
Implementing gRPC Using Spring Boot and Gradle Background GRPC (gRPC) is a RPC framework pioneered by google later on made public and open source. GRPC has not been officially supported by Spring Framework, so far there are few community libraries that can work seamlessly with Spring Boot. In this article, I will use the lognet's grpc-spring-boot-starter. Project Structure Ideally, the proto files for gRPC is located in another directory, which will then be accessed by the gradle plugin to auto generate the required files.   Gradle Script The heart of gRPC enablement lies on the grpc boot starter, which includes few other tasks outlined as below. https://github.com/overtakerx/grpcbootdemo/blob/main/build.gradle Few important note: first you have to run gradlew compileJava for the auto generation to generate the files from the proto file, then you will need to rebuild the project for Intellij to recognise the auto-generated files. Full demo: https://github.com/overtakerx/grpcbootdem
Read more

Cucumber + Junit5 + Localstack

Background What are the options to test an Amazon lambda that interacts with Dynamo DB locally? To be honest, there are several options, such as using Mockito, using stubs, swapping Dynamo DB with Mongo DB, however none of these truly replicate the amazon lambda and dynamo db ecosystem. Enter Localstack  ( https://github.com/localstack/localstack ), open source project that aims to emulate a local AWS cloud stack. With its docker version, we could spin up localstack and push our lambda and create our table in the dynamodb. To further increase its testing prowess, Localstack comes up with utils to integrate with Junit 4 and 5. Now we have the tools to create a BDD test using Cucumber, JUnit5 and Localstack as you can see below. Do note that due to the open source nature of localstack, it has several bugs that we manage to get around Prereq: Docker images: localstack/localstack:0.9.1 Note that the latest version of localstack docker image has bugs with the service port nu
Read more