FABIAN GOTZEN

Integration Testing made easy

7 February 2023

Integration Testing made easy

In this blogpost we will have a closer look at the Testcontainers Library. At first we will see a small introduction to the library and its benefits & drawbacks, before jumping into a complete project that uses the library to write easy and intuitive integration tests. We will set up a very basic Kotlin web application with Spring Boot 3 that integrates with a Kafka & Postgresql Testcontainer to produce and consume different Messages. To test that everything works as expected, we will spin up an integration test using the Testcontainers library.

lntroduction

We all know about the importance of writing good unit tests for our application, and while writing Unit tests for your applications is a good place to start, they are not enough to give yourself enough confidence in most situtations. After all, the interplay and interfaces between modules and systems are often were our dreams of understanding software engineering go to die 🙂

Let’s look at an typical example of this. Let’s see your API layer receives a Message from a Messaging system, maps it to a DTO, does some business related enrichments and passes it to the persistence layer. Now you want to check if your Entity was persisted correctly to your database.

For a long time, there were usually two approaches to solve this.

  1. Connect to a live system used for testing.
  2. Use In-Memory solutions like H2 and EmbeddedKafka.

Both of these approaches have several drawbacks that make them not ideal for writing your Integration tests against them. When you have to connect to a live system, you are automatically at the mercy of your infrastructure team if you work in Corporate IT, or you have to do it yourself in smaller shops. Especially in a bigger organization, this can take a lot of time to set up, needs a lot of coordination and let’s be honest, nobody wants to do it. While most organisations will probably have something in place for databases, when you want to introduce a new system like a Messaging System you wil have to go through it all again.

So naturally, in-memory solutions like H2 or EmbeddedKafka emerged. However, these approaches have a different set of issues, the most important one being: They are just not the same as running a live system. So you will encounter issues which are only relevant for your In-memory Database and you won’t be sure if they will also appear on your real databases on higher environments.

Fortunately, in the rise of containerized systems we now have a third option:

  1. Testcontainers

Instead of relying on live systems or In-Memory solutions, we spin up a container in isolation where our tests are run against and after the test execution is finished, are removed. It’s not as reliable as a live system and not as quick as In-Memory solutions, but hits the perfect sweet spot between the two for integration testing your applications. And it’s super easy to do, let’s have a look at an example.

Build a simple Consumer Application with Kotlin & Spring Boot

Let’s first have a look at the code to get a feel for what our application should do. It’s a simple Kotlin web application that consumes a Message from Kafka, transforms it and saves it to a Postgres database. For the sake of brevity, we just use a simple User class to achieve this. With Kotlin, we can use a concise data class to achieve this.

data class UserDto(val id: UUID, val name: String, val age: Int)

Here we have a simple data carrier called UserDto that has an ID, name & age.
Afterwards, we need a producer that sends our messages to Kafka

@Service
class UserProducer @Autowired constructor(private val kafkaTemplate: KafkaTemplate<String?, UserDto?>) {
    fun send(userDto: UserDto) {
        this.kafkaTemplate.send("userTopic", userDto)
    }
}

And a fitting consumer that consumes the message from the userTopic.

@KafkaListener(id = "myId", topics = ["userTopic"])
@Component
class UserConsumer(private val userService: UserService) {
    @Bean
    fun topic() = NewTopic("userTopic", 10, 1)

    @KafkaHandler
    fun listen(userDto: UserDto) {
        println(userDto)
        userService.save(userDto);
    }
}

For simplicity’s sake, let’s just produce a message on the start up of our application. Together with a simple docker compose file we can now start our application and produce a message to a Kafka cluster.

@SpringBootApplication
@EnableKafka
class TestcontainersApplication {
    @Bean
    fun runner(producer: UserProducer) =
        ApplicationRunner { producer.send(UserDto(UUID.randomUUID(), "testUser", 20)) }
}

fun main() {
    runApplication<TestcontainersApplication>()
}

After starting the application, we can check that everything works and our User is succesfully printed:

UserDto(id = d6c5b068 - b616 - 423 a -b619 - 7341111 d0ba1, name = testUser, age = 20)

Now lets have a look at the integration test for this. We start with an Abstract class were we set up the containers we need.

@SpringBootTest
@ActiveProfiles("test")
@ContextConfiguration(initializers = [AbstractIntegrationTest.Initializer::class])
abstract class AbstractIntegrationTest @Autowired constructor(
    val userRepository: UserRepository,
    val userProducer: UserProducer,
) {
    companion object {
        val kafkaContainer = KafkaContainer(DockerImageName.parse("confluentinc/cp-kafka:6.2.1"))
        val postgresContainer = PostgreSQLContainer(DockerImageName.parse("postgres:12"))
    }
    internal class Initializer : ApplicationContextInitializer<ConfigurableApplicationContext> {
        override fun initialize(configurableApplicationContext: ConfigurableApplicationContext) {
            kafkaContainer.start()
            postgresContainer.start()
            TestPropertyValues.of(
                "spring.kafka.bootstrap-servers=${kafkaContainer.bootstrapServers}",
                "spring.datasource.url=${postgresContainer.jdbcUrl}",
                "spring.datasource.username=${postgresContainer.username}",
                "spring.datasource.password=${postgresContainer.password}"
            ).applyTo(configurableApplicationContext.environment)
        }
    }
}

Let’s walk through it step by step. We use the @SpringBootTest annotation to spin up our application context and use a test profile where we set custom properties for our database and kafka cluster. In a companion object (which is roughly equivalent to a static Java block) we define the Kafka & Postgrescontainers and start them with their corresponding images.
Then we write an Initializer who dynamically sets the properties from the container to our application properties. And that’s already everything we need for our initial set up. Now we can write the integration test that extends from our AbstractIntegrationtest.

class UserConsumerIntegrationTest(@Autowired userRepository: UserRepository, @Autowired userProducer: UserProducer) : AbstractIntegrationTest(userRepository, userProducer) {  
    val LOG = Logger.getLogger(this.javaClass.name)  
    @org.junit.jupiter.api.Test  
    fun testConsumeUserMessage() {  
        // arrange  
        val userDto = UserDto(UUID.randomUUID(), "TestUser", 10);  
        // act  
        this.userProducer.send(userDto)  
        LOG.info("Sending message with user: " + userDto.name)  
        org.awaitility.kotlin.await untilNotNull { this.userRepository.findById(userDto.id).get() }  
        val persistedUser = this.userRepository.findById(userDto.id).get()  
        // assert  
        assertAll("Assert that User was successfully saved",  
            { Assertions.assertEquals(persistedUser.name, userDto.name) },  
            { Assertions.assertEquals(persistedUser.age, userDto.age) }  
        )  
    }

This tests now spins up a Kafka & Postgres container, produces a message and verifies that everything was successfully persisted to the database. I’m using the awaitility library to make the test more robust and prevent timing issues.

Conclusion

And this is already everything you need to write a basic integration test for your simple Consumer application. As always, you will find the whole source code in my github repository.

In the second part of the series, we will have a look at how to further improve upon this setup. Hope you enjoyed the read!