My journey through Micronaut has been pretty amazing and using coroutines in the mix have given me quite a lot of experience in moving on to master these concepts.
In this post, I'm going to show you how I created an all round integration test, that looks almost like a complete E2E (End-to-End) test using a real docker image with Redis installed, listeners and a rest client.
In the following diagram we can see what I'm testing now:
So, here, I need to test one "happy case". Our "happy case" consists on the following:
We make a request to an API with the complete payload of a ticket reservation request. This includes the concert day and optionally car parking voucher, meals and drinks.
Once received, the application will immediately generate a receipt reference using the database. We need this reference from the database. It is with this reference that the complete request will be later processed.
The payload of the ticket is published to Redis Pub Sub
The listener will asynchronously get this ticket back and post it to another internal API called the Ticket API.
Once the Ok - 200 is received, then we register that in an audit table.
With these 5 steps, we ensure that a limited, but necessary data information is stored in the database in a reactive way using coroutine repos, we use a cheap operation such as publish to a Redis to make sure that our request gets processed and we make sure that we reply back to client the fastest possible way. The client get's a reference as a response which can be used later to print out tickets, show entrance permission, valid car parking voucher etc. In reception the customer gets a printed paper with the reference number being notified that the ticket processing should be done in 5 minutes.
Now, how do we test all of this? Here is a drawing of the general concept:
For this coffee bite, you don't really need to know the details of the application. This is just to give you a summary of what we are going to look at. The odd gorilla like icons, if you don't know, are Kong gateways that provide many gateway controls and fine tuning. Among them, one of the most used is rate-limiting just to give you an idea.
We are going to focus on this piece of code that receives incoming Ticket Reservation requests:
suspend fun createTicket(ticketDto: @Valid TicketDto): ReceiptDto {
val save = receiptRepository.save(Receipt())
val receiptDto = save.toDto
pubSubCommands.publish("ticketsChannel", ticketDto.copy(reference = receiptDto.reference))
return receiptDto
}
fun getAll(): Flow<Receipt> = receiptRepository.findAll()And we will look at this one:
override fun message(key: String, ticketDto: TicketDto) {
val ticketDtoSingle: Single<TicketDto> =
client.retrieve(HttpRequest.POST(url, ticketDto), TicketDto::class.java).firstOrError()
val singleScheduler = SingleScheduler()
ticketDtoSingle.subscribeOn(singleScheduler).doOnSuccess {
GlobalScope.launch {
auditLogRepository.save(
AuditLog(
auditLogType = AuditLogType.TICKET,
payload = ticketDto.toString()
)
)
}
}.subscribe()
singleScheduler.start()
}You can find the original code on this commit hash: buy-odd-yucca-concert.
Since we now know what we want to test, we know that first we need to start at least two containers. These containers are one PostgreSQL to be able to persist the receipt and one Redis container in order to provide the pub-sub support we need:
class TestPostgresSQLContainer(imageName: String) : PostgreSQLContainer<TestPostgresSQLContainer>(imageName)
private const val POSTGRESQL_PORT = 5432
private const val REDIS_PORT = 6379
abstract class AbstractBuyOddYuccaConcertContainerTest {
companion object {
@Container
@JvmField
val postgreSQLContainer: TestPostgresSQLContainer = TestPostgresSQLContainer("postgres:14")
.withUsername("kong")
.withPassword("kong")
.withDatabaseName("yucca")
.withExposedPorts(POSTGRESQL_PORT)
.withCreateContainerCmdModifier { cmd ->
cmd.withHostConfig(
HostConfig().withPortBindings(
PortBinding(
bindPort(POSTGRESQL_PORT),
ExposedPort(POSTGRESQL_PORT)
)
)
)
}
@Container
@JvmField
val redis: GenericContainer<*> = GenericContainer(parse("redis:5.0.3-alpine"))
.withExposedPorts(REDIS_PORT)
.withCreateContainerCmdModifier { cmd ->
cmd.withHostConfig(
HostConfig().withPortBindings(
PortBinding(
bindPort(REDIS_PORT),
ExposedPort(REDIS_PORT)
)
)
)
}
val config = ClassicConfiguration()
init {
postgreSQLContainer.start()
redis.start()
config.setDataSource(
postgreSQLContainer.jdbcUrl,
postgreSQLContainer.username,
postgreSQLContainer.password
)
config.schemas = arrayOf("ticket")
Flyway(config).migrate()
}
}
}This way, I'm achieving the startup of the running containers I need in order to make this integration test a possibility. Using testcontainers for the integration tests is quite an easy task to do, but testing the whole flow in a coroutine context and the usage of suspend functions, brings a whole dynamic that makes this a bit more difficult to achieve.
When I talk about testing the whole round trip, I truly mean the 5 steps to test as mentioned above, but this means that there are quite a few asynchronous steps that are inherently difficult to test in an integration test. In my first attempts I tried using sleep and delay. The thought behind this was just to wait until, for example, that the listener get the ticket back from redis and then send it via the reactive http client to the other internal APIs. However this is something not achievable in an asynchronous architectural environment. This is because of many reasons. When we start a test with Jupiter 5, at least, it has to run inherently in a blocking way so to provide coroutine context Using runTest allows us to force this and make our test work. When implementing our test in its receiver, we are running everything in a coroutine context. This coroutine can start other coroutines under it. When we do that though, we cannot just use delay or sleep and wait for something else to happens. If we delay in our test, we are delaying the whole coroutine and the subcoroutines. If we use sleep then we are sleeping the whole Thread. This will stop not only the coroutine, but also the also the whole Thread. So the only way to bypass all of this and effectively wait for the pub-sub process to complete is to start another Thread and make the test there. But now we are facing the issue of having already made our test overtly complicated and have added enough code to defeat the purpose of the integration tests in the first place!
Since we cannot/should not delay or sleep for our tests, we can use an external process by actually making the test from a broader stand point. We can use the actual controller endpoint. On one side we have our own controller on our buy-oyc-api-service module and we also have an external endpoint on another buy-oyc-ticket-service module. Just like MockMvc in Spring we also have ways to implement internal rest calls in our integration test. And just like in some of those tests, we can also use Wiremock to create mocked rest endpoints By having these two, it should be easy to implement the full circle. This is what we'll now see. We start by looking at the header of the test class we are going to have a look on and slowly move downwards:
@ExperimentalCoroutinesApi
@MicronautTest
@Property(name = "buy.oyc.ticket.port", value = "7999")
class ReceiptTest @Inject constructor(
private val receiptRepository: ReceiptRepository,
private val receiptReactiveClient: ReceiptReactiveClient,
) : AbstractBuyOddYuccaConcertContainerTest() {The @ExperimentalCoroutinesAPI is here, because Intellj detects the usage of coroutines as experimental. Using this annotation, makes the code clear that we are consciously using it and plus Intellij will stop complaining about this. The annotation @Property allows for an application property to be changed during initialization. I couldn't find a way yet to do this in runtime. By setting the port to 7999, I'm now saying that the ticket API is listening at port 7999. In the actual implementation this will be covered by a Kong Gateway endpoint. The following are two injectable beans that we need to use in our tests. One is the ReceiptRepository, where all receipts are created and the other is receiptReactiveClient, which is used to send the data to our API.
private val jacksonMapper = jacksonObjectMapper()
.registerModule(JavaTimeModule())We also need create JSON's, serialize them and then deserialize them. The jacksonObjectMapper provided by the jackson-datatype-jsr310 and jackson-module-kotlin packages, does not provide a way to deserialize LocalDate objects out of the box. This is why we need to register JavaTimeModule().
private fun stubResponse(
url: String,
body: String,
status: Int = HttpStatus.OK.code
) {
wireMockServer.stubFor(
WireMock.post(url)
.willReturn(
WireMock.aResponse()
.withStatus(status)
.withHeader(
HttpHeaders.CONTENT_TYPE,
MediaType.APPLICATION_JSON
)
.withBody(body)
)
)
}Since we are using wiremock, it makes sense to create a generic method that subs a response. This is nothing more than a stub for a rest endpoint where we always want o return a given body in a JSON format.
Setting up our test is very important but first we have to start our wiremock service:
companion object {
private val wireMockServer = WireMockServer(WireMockConfiguration().port(7999))
@JvmStatic
@AfterAll
fun tearDown() {
wireMockServer.stop()
}
}Starting the wireMockServer, also means that we have to stop it once all our tests from our test class have finished running.
@BeforeEach
fun setUpEach() = runTest {
receiptRepository.deleteAll()
val ticketDto = TicketDto(name = "name", address = "address", birthDate = LocalDate.now())
stubResponse(
API_YUCCA_TICKET, jacksonMapper
.writeValueAsString(ticketDto), 200
)
wireMockServer.start()
}This is the stub configuration with a dummy ticket payload we are going to send. We are essentially sending a ticket for a customer, who will not attend any concert day, will not have a parking place reserved, won't have drink or meal. What in regards to testing this phase of the project, this is all we need for the moment. Once we have our stubbed service created, we can now focus on the client to our own controller test:
@Client("/api")
interface ReceiptReactiveClient {
@Post
fun add(@Body ticket: TicketDto): Single<LinkedHashMap<String, String>>
@Get(value = "/", produces = [MediaType.APPLICATION_JSON_STREAM])
fun findAll(): Flux<ReceiptDto>
}By using @Client, we are immediately telling Micronaut that we want a receiptReactiveClient bean. Using the other @Post and @Get annotation, we then declare the methods we want to use with these requests. Since this only works in combination with @Controller in the actual code from the test code, this elegant solution isn't appropriate to access external API's.
And finally we can implement our test:
@Test
@Transactional
fun `should find all with an empty list`() = runTest {
val (_, referenceSaved, createdDate) = receiptRepository.save(Receipt())
val findAll = receiptReactiveClient.findAll()
findAll.shouldNotBeNull()
findAll.subscribe {
it.reference shouldBe referenceSaved
NANOS.between(it.createdAt, createdDate) shouldBeLessThan 1000
}
val awaitFirstReceiptDto = findAll.awaitFirst()
awaitFirstReceiptDto.reference shouldBe referenceSaved
NANOS.between(awaitFirstReceiptDto.createdAt, createdDate) shouldBeLessThan 1000
val testTicketDto = TicketDto(
name = "name",
address = "address",
birthDate = LocalDate.now()
)
val add = receiptReactiveClient.add(testTicketDto)
val blockingGet = withContext(Dispatchers.IO) {
add.blockingGet()
}
blockingGet["second"].shouldBe("Saved successfully !")
val findAll2 = receiptReactiveClient.findAll()
findAll2.shouldNotBeNull()
findAll2.subscribe()
val awaitFirstReceiptDto2 = withContext(Dispatchers.IO) {
findAll2.toIterable()
}.toList()
awaitFirstReceiptDto2.shouldHaveSize(2)
withContext(Dispatchers.IO) {
sleep(1000)
}
auditLogRepository.findAll().toList().shouldHaveSize(1)
}The test, for the most part is quite self explanatory up until we get to the last bit:
withContext(Dispatchers.IO) {
sleep(1000)
}
auditLogRepository.findAll().toList().shouldHaveSize(1)Here, I am allowed to wait. This is a short description on what I am allowed to do. You know may ask, "how come we can wait now, when delay and sleep ultimately stop the main thread?". Well, we can still run coroutines in different contexts and if the context isn't related to the current thread, then these will work for the test. This means that the current thread isn't the same thread running the actual service and so sleep does work in this case. For this integration test, I settle 1 second as the maximum time a round trip should take also considering the asynchronous processes. If this takes longer than 1 second, purely in order to process one request and send the ticket to the internal API's, then the test should fail. Longer than this, it would be a performance issue.
What I've learned in doing this is that coroutines can be quite misleading, bu they do seem to represent a good reactive solution. By reactive, we really mean a lot of things, but mostly the ability to respond as fast as possible using asynchronous processes like coroutines, implementing the observer pattern or virtual threads as in the case of project Loom. Using Wiremock, for these tests, may sound outdated to you, but all I really wanted for this test was simply an endpoint that returns a dummy response, so that my tests can go through and get an audit record in the Audit table. How that dummy rest endpoint is implemented isn't really the topic here.
A few words about Kotlin so far. I still see it as a player in the IT world that seems to have as a core this idea of beautifying all majorly used programming languages. It is quite beautiful and easy to use, but I still question the fact that it seem to provide shortcuts for things that already exist in other languages. In the case of Java it is working as a wrapper and that is why we always need DSL's and extra maven plugins. In case of coroutines it seems like project Loom already existed for while. In case of Kotlin JS, though not the topic of this coffee bite, we also see the creation of yet another TS like language that seems to want to mix bytecodes of jars and the javascript world together in one big melting pot. It goes really far how much Kotlin is portrayed to be stretched out. Coming from a Java background I have seen and learned things in practice that I probably wouldn't understand that well had I started with Kotlin first. And this is because Kotlin allows for a lot of things to happen under the hood. Nothing against that, per se, but factually engineering wise and setting all the aesthetics discussions aside, I'm still looking for the major advantage of using Kotlin. The last thing we want as engineers is to jump into a sort of botox engineering where beauty is elevated on the top of real engineering challenges. But I stay curious and I keep on track on discovering new ideas. And yes... I have seen how "beautiful" reified, data classes, receivers, functional programming in kotlin, and the by, and the fact that "we don't have to think about extend and implement" anymore, and internal and external, and a bunch of classes in one file, and extension functions, and the companion objects, and no primitive types ... I can go on forever. I've seen all of that. But again, that all falls down under beauty to me. The only potential achievement I've seen so far is coroutines, but we have project Loom in development for years over what it seems to be exactly the same concept under different names. So, it is not that my passion over Kotlin is gone, I'm just questioning the idea that "Kotlin is superior to Java".