TestContainers in Scala: Use Integration Tests for building your services
TL;DR
Integration tests are frequently seen as the most expensive tests in most environments. The reason is that they usually require a higher level of preparation and procedures to make them appropriate for your particular infrastructure dependencies. In addition, the time invested to make them work properly on the developer’s continuous integration/development environment is not trivial. It requires time from your developers and especially your DevOps team. With this blog article, I want to give you an overview of Testcontainers for Scala. This library can help make your life easier when creating Integration tests and use them in a straightforward and consistent way. Another important benefit is that you can depend less on mocking infrastructure and dependencies because you can go directly with the same technologies you will use in production. This will result in less time invested in mocking preparations and more confidence in the results of your integration tests.
Introduction
Integration tests are a type of test that is usually needed when building new services. They enable you to be more confident about the service’s contracts and interactions with other infrastructure or other services in the environment. In particular, when using a microservices architecture, it is even more crucial to have these types of tests because the interactions must be validated in a more granular way, as well as in a consistent fashion between the developer’s environment and the continuous integration/deployment (CI/CD) environment. This is a challenge because integration testing is seen as an expensive investment when building services. This is because staying consistent between the developer’s environment and CI/CD – while also having the flexibility to change and add testing conditions quickly – has usually not been an easy task, requiring a lot of mocking and boilerplate coding.
To tackle these problems, in 2015, a new open-source Java library, testcontainers, was created that has grown in adoption and popularity in developer communities and companies. In this article, I aim to show you how to leverage testcontainers for your Scala projects, with a sample small application showing some techniques that can be employed to make the work reusable and, hopefully, and to clarify how to use basic and some not so basic testcontainers features.
Testcontainers: how to use it in Scala projects
As previously mentioned, Testcontainers started as a Java project, but over time, other languages have also received support, such as Go, Python, Rust, .NET, Node.js, etc. In the Scala case, a new library was created as a wrapper around the Java: testcontainers-scala.
Example Overview
For this article, I will be using and exploring a sample implementation project available at https://github.com/hwgonz/testcontainers-sample. The sample is a microservice that projects a retail product into a Kafka topic. This microservice exposes an http API to accept and alter a simple validation and projects a new product as an event in a Kafka topic. So, as you can see, we will be using three types of containers for our testing purposes: a Generic container for our microservice, a Kafka container, and a MySQL container. The MySQL container is included just to show some of the new features testcontainers offers, but for the context of this implementation, it is not being used by the microservice at all.
Some basic Testcontainers Features
To test our microservice example, we need 3 containers in place. The main one for the Retail Product Service, one for Kafka, and another for MySQL. One of the main features that testcontainers offers is a very diverse container. These official containers are usually maintained by partner companies that take care of them.
For Kafka, there’s com.dimafeng.testcontainers.KafkaContainer class.
For MySQL, there’s com.dimafeng.testcontainers.MySQLContainer class.
And finally, for our microservice, we can use the com.dimafeng.testcontainers.GenericContainer class.
For our example, these container definitions can be seen in this BaseContainers Trait:
package com.acme.containers
import com.dimafeng.testcontainers._
import com.dimafeng.testcontainers.scalatest._
import com.acme.containers.BaseContainers.{AppPort, WaitStrategyForAPI}
import org.scalatest.Suite
import org.testcontainers.containers.wait.strategy.{HttpWaitStrategy, LogMessageWaitStrategy, WaitAllStrategy, WaitStrategy}
import org.testcontainers.containers.{Network, GenericContainer => JavaGenericContainer}
import org.testcontainers.images.builder.ImageFromDockerfile
import org.testcontainers.lifecycle.Startable
import org.testcontainers.utility.DockerImageName
import java.nio.file.Path
import scala.jdk.CollectionConverters._
trait BaseContainers extends TestContainersForAll {
self: Suite =>
protected val sharedNetwork: Network = Network.newNetwork()
protected lazy val kafkaContainer: KafkaContainer = new KafkaContainer(DockerImageName.parse("confluentinc/cp-kafka:7.4.0")) {
container.withNetwork(sharedNetwork)
container.withNetworkAliases("kafka")
}
protected lazy val mySQLContainer: MySQLContainer = new MySQLContainer(
mysqlImageVersion = Some(DockerImageName.parse("mysql:8.0")),
) {
container.withNetwork(sharedNetwork)
container.withNetworkAliases("mysql")
container.withEnv("ENVIRONMENT", "local")
}
protected def baseAppContainer(
name: String,
jarName: String,
mainClass: String,
baseFolder: String,
port: Int = AppPort,
//waitStrategy: WaitStrategy = WaitStrategyForAPI,
envVars: Map[String, String] = Map.empty,
containerDependencies: List[Startable]
): GenericContainer = new GenericContainer({
val c = new JavaGenericContainer(
new ImageFromDockerfile()
.withFileFromPath(jarName, Path.of(s"$baseFolder/$jarName").toAbsolutePath)
.withDockerfileFromBuilder { builder =>
builder
.from("openjdk:11-jre-slim")
.copy(jarName, s"/$jarName")
.cmd("java", "-Xmx256m", "-cp", s"/$jarName", mainClass)
.build()
}
)
c.dependsOn(containerDependencies.asJavaCollection)
c.withEnv(envVars.asJava)
//c.withStartupAttempts(3)
c.withEnv("ENVIRONMENT", "local")
c.withEnv("APP_PORT", port.toString)
c.withEnv("KAFKA_BOOTSTRAP_SERVERS", "kafka:9092")
//c.setWaitStrategy(waitStrategy)
c.withExposedPorts(port)
c.withNetwork(sharedNetwork)
c.withNetworkAliases(name)
c
})
}
object BaseContainers {
val AppPort = 9000
val WaitStrategyForAPI: WaitStrategy = new HttpWaitStrategy().forPath("/health").forStatusCode(200)
val WaitStrategyForService: WaitStrategy = new WaitAllStrategy()
.withStrategy(WaitStrategyForAPI)
.withStrategy(new LogMessageWaitStrategy().withRegEx("^.*Resetting offset for partition.*$"))
}There are various testcontainers features used in this trait. So, let’s review them one by one.
The first thing you can notice is that BaseContainers extends the TestContainersForAll trait. This means all tests will reuse the same containers, starting them before running all the tests in the suite and stopping them after all the tests are run. This handy feature allows us to make our tests use fewer resources and, as such, the fastest running tests. However, for those exceptional cases where you need a particular container for each test, there’s the TestContainerForEach trait, which provides a container before starting a test and stops it after it is completed.
You can see more details about these traits at https://github.com/testcontainers/testcontainers-scala/blob/master/docs/src/main/tut/usage.md.
Another aspect you should consider is using a shared network for your containers. In our case, we defined a shared network that is used by all containers in our example.
package com.acme.containers
import com.dimafeng.testcontainers._
import com.dimafeng.testcontainers.scalatest._
import com.acme.containers.BaseContainers.{AppPort, WaitStrategyForAPI}
import org.scalatest.Suite
import org.testcontainers.containers.wait.strategy.{HttpWaitStrategy, LogMessageWaitStrategy, WaitAllStrategy, WaitStrategy}
import org.testcontainers.containers.{Network, GenericContainer => JavaGenericContainer}
import org.testcontainers.images.builder.ImageFromDockerfile
import org.testcontainers.lifecycle.Startable
import org.testcontainers.utility.DockerImageName
import java.nio.file.Path
import scala.jdk.CollectionConverters._
trait BaseContainers extends TestContainersForAll {
self: Suite =>
protected val sharedNetwork: Network = Network.newNetwork()
protected lazy val kafkaContainer: KafkaContainer = new KafkaContainer(DockerImageName.parse("confluentinc/cp-kafka:7.4.0")) {
container.withNetwork(sharedNetwork)
container.withNetworkAliases("kafka")
}
protected lazy val mySQLContainer: MySQLContainer = new MySQLContainer(
mysqlImageVersion = Some(DockerImageName.parse("mysql:8.0")),
) {
container.withNetwork(sharedNetwork)
container.withNetworkAliases("mysql")
container.withEnv("ENVIRONMENT", "local")
}
protected def baseAppContainer(
name: String,
jarName: String,
mainClass: String,
baseFolder: String,
port: Int = AppPort,
//waitStrategy: WaitStrategy = WaitStrategyForAPI,
envVars: Map[String, String] = Map.empty,
containerDependencies: List[Startable]
): GenericContainer = new GenericContainer({
val c = new JavaGenericContainer(
new ImageFromDockerfile()
.withFileFromPath(jarName, Path.of(s"$baseFolder/$jarName").toAbsolutePath)
.withDockerfileFromBuilder { builder =>
builder
.from("openjdk:11-jre-slim")
.copy(jarName, s"/$jarName")
.cmd("java", "-Xmx256m", "-cp", s"/$jarName", mainClass)
.build()
}
)
c.dependsOn(containerDependencies.asJavaCollection)
c.withEnv(envVars.asJava)
//c.withStartupAttempts(3)
c.withEnv("ENVIRONMENT", "local")
c.withEnv("APP_PORT", port.toString)
c.withEnv("KAFKA_BOOTSTRAP_SERVERS", "kafka:9092")
//c.setWaitStrategy(waitStrategy)
c.withExposedPorts(port)
c.withNetwork(sharedNetwork)
c.withNetworkAliases(name)
c
})
}
object BaseContainers {
val AppPort = 9000
val WaitStrategyForAPI: WaitStrategy = new HttpWaitStrategy().forPath("/health").forStatusCode(200)
val WaitStrategyForService: WaitStrategy = new WaitAllStrategy()
.withStrategy(WaitStrategyForAPI)
.withStrategy(new LogMessageWaitStrategy().withRegEx("^.*Resetting offset for partition.*$"))
}. Then, when defining each container, you should use a network alias for each of them to allow for a better name resolution in this network. You can use the .withNetworkAlias method to achieve this.
In the same tone, another critical aspect is support for environment variables.
package com.acme.containers
import com.dimafeng.testcontainers._
import com.dimafeng.testcontainers.scalatest._
import com.acme.containers.BaseContainers.{AppPort, WaitStrategyForAPI}
import org.scalatest.Suite
import org.testcontainers.containers.wait.strategy.{HttpWaitStrategy, LogMessageWaitStrategy, WaitAllStrategy, WaitStrategy}
import org.testcontainers.containers.{Network, GenericContainer => JavaGenericContainer}
import org.testcontainers.images.builder.ImageFromDockerfile
import org.testcontainers.lifecycle.Startable
import org.testcontainers.utility.DockerImageName
import java.nio.file.Path
import scala.jdk.CollectionConverters._
trait BaseContainers extends TestContainersForAll {
self: Suite =>
protected val sharedNetwork: Network = Network.newNetwork()
protected lazy val kafkaContainer: KafkaContainer = new KafkaContainer(DockerImageName.parse("confluentinc/cp-kafka:7.4.0")) {
container.withNetwork(sharedNetwork)
container.withNetworkAliases("kafka")
}
protected lazy val mySQLContainer: MySQLContainer = new MySQLContainer(
mysqlImageVersion = Some(DockerImageName.parse("mysql:8.0")),
) {
container.withNetwork(sharedNetwork)
container.withNetworkAliases("mysql")
container.withEnv("ENVIRONMENT", "local")
}
protected def baseAppContainer(
name: String,
jarName: String,
mainClass: String,
baseFolder: String,
port: Int = AppPort,
//waitStrategy: WaitStrategy = WaitStrategyForAPI,
envVars: Map[String, String] = Map.empty,
containerDependencies: List[Startable]
): GenericContainer = new GenericContainer({
val c = new JavaGenericContainer(
new ImageFromDockerfile()
.withFileFromPath(jarName, Path.of(s"$baseFolder/$jarName").toAbsolutePath)
.withDockerfileFromBuilder { builder =>
builder
.from("openjdk:11-jre-slim")
.copy(jarName, s"/$jarName")
.cmd("java", "-Xmx256m", "-cp", s"/$jarName", mainClass)
.build()
}
)
c.dependsOn(containerDependencies.asJavaCollection)
c.withEnv(envVars.asJava)
//c.withStartupAttempts(3)
c.withEnv("ENVIRONMENT", "local")
c.withEnv("APP_PORT", port.toString)
c.withEnv("KAFKA_BOOTSTRAP_SERVERS", "kafka:9092")
//c.setWaitStrategy(waitStrategy)
c.withExposedPorts(port)
c.withNetwork(sharedNetwork)
c.withNetworkAliases(name)
c
})
}
object BaseContainers {
val AppPort = 9000
val WaitStrategyForAPI: WaitStrategy = new HttpWaitStrategy().forPath("/health").forStatusCode(200)
val WaitStrategyForService: WaitStrategy = new WaitAllStrategy()
.withStrategy(WaitStrategyForAPI)
.withStrategy(new LogMessageWaitStrategy().withRegEx("^.*Resetting offset for partition.*$"))
}As you can see, we can define and pass environment variables to our containers when required. The same happens with ports. We can use the exposedPorts method to publish the container’s ports to the outside world, like in
package com.acme.containers
import com.dimafeng.testcontainers._
import com.dimafeng.testcontainers.scalatest._
import com.acme.containers.BaseContainers.{AppPort, WaitStrategyForAPI}
import org.scalatest.Suite
import org.testcontainers.containers.wait.strategy.{HttpWaitStrategy, LogMessageWaitStrategy, WaitAllStrategy, WaitStrategy}
import org.testcontainers.containers.{Network, GenericContainer => JavaGenericContainer}
import org.testcontainers.images.builder.ImageFromDockerfile
import org.testcontainers.lifecycle.Startable
import org.testcontainers.utility.DockerImageName
import java.nio.file.Path
import scala.jdk.CollectionConverters._
trait BaseContainers extends TestContainersForAll {
self: Suite =>
protected val sharedNetwork: Network = Network.newNetwork()
protected lazy val kafkaContainer: KafkaContainer = new KafkaContainer(DockerImageName.parse("confluentinc/cp-kafka:7.4.0")) {
container.withNetwork(sharedNetwork)
container.withNetworkAliases("kafka")
}
protected lazy val mySQLContainer: MySQLContainer = new MySQLContainer(
mysqlImageVersion = Some(DockerImageName.parse("mysql:8.0")),
) {
container.withNetwork(sharedNetwork)
container.withNetworkAliases("mysql")
container.withEnv("ENVIRONMENT", "local")
}
protected def baseAppContainer(
name: String,
jarName: String,
mainClass: String,
baseFolder: String,
port: Int = AppPort,
//waitStrategy: WaitStrategy = WaitStrategyForAPI,
envVars: Map[String, String] = Map.empty,
containerDependencies: List[Startable]
): GenericContainer = new GenericContainer({
val c = new JavaGenericContainer(
new ImageFromDockerfile()
.withFileFromPath(jarName, Path.of(s"$baseFolder/$jarName").toAbsolutePath)
.withDockerfileFromBuilder { builder =>
builder
.from("openjdk:11-jre-slim")
.copy(jarName, s"/$jarName")
.cmd("java", "-Xmx256m", "-cp", s"/$jarName", mainClass)
.build()
}
)
c.dependsOn(containerDependencies.asJavaCollection)
c.withEnv(envVars.asJava)
//c.withStartupAttempts(3)
c.withEnv("ENVIRONMENT", "local")
c.withEnv("APP_PORT", port.toString)
c.withEnv("KAFKA_BOOTSTRAP_SERVERS", "kafka:9092")
//c.setWaitStrategy(waitStrategy)
c.withExposedPorts(port)
c.withNetwork(sharedNetwork)
c.withNetworkAliases(name)
c
})
}
object BaseContainers {
val AppPort = 9000
val WaitStrategyForAPI: WaitStrategy = new HttpWaitStrategy().forPath("/health").forStatusCode(200)
val WaitStrategyForService: WaitStrategy = new WaitAllStrategy()
.withStrategy(WaitStrategyForAPI)
.withStrategy(new LogMessageWaitStrategy().withRegEx("^.*Resetting offset for partition.*$"))
}Some containers, like in the MySQL and Kafka cases, by default, expose their usual ports, such as 9092 for Kafka and 3306 for MySQL. Here, I have to make a short interruption to explain that, in this case, these ports are from the containers and container’s network perspective. From the host’s perspective, they see a randomly assigned port to prevent issues in the host network or services. So if, for some reason, you need to know this port number when running a test, it is possible to get it using the testcontainer-scala’s mappedPort method (which is a wrapper for the getMappedPort in the Java base library). This is particularly useful when you need to define an URL that should include this randomly generated port to use it in an http request to perform a test.
package com.acme.api.retailproduct
import cats.effect.{IO, Resource}
import cats.effect.unsafe.implicits.global
import com.acme.api.utils.Containers
import com.acme.containers.BaseContainers.AppPort
import com.acme.BaseSpec
import com.acme.api.response.failure.BadRequestErrorResponse
import com.acme.api.response.success.RetailProductAccepted
import com.acme.containers.helpers.KafkaHelper
import com.acme.event.RetailProductEvent
import com.acme.model.RetailProduct
import org.http4s.circe.CirceEntityCodec._
import com.acme.model.RetailProductCodecsCamelCase._
import com.acme.kafka.serdes.DomainSerDes.jsonDeserializer
import org.http4s.client.Client
import org.http4s._
import io.circe.literal._
import org.http4s.circe._
import java.util.UUID
trait RetailProductSpec {
_: BaseSpec with Containers with KafkaHelper =>
val httpClientResource: Resource[IO, Client[IO]]
private lazy val url = Uri.unsafeFromString(s"http://${retailProductServiceContainer.host}:${retailProductServiceContainer.mappedPort(AppPort)}/retailproduct")
def retailProductServiceTests(): Unit =
"Retail Product endpoint" must {
"properly emit event" in {
val retailProduct = RetailProduct(
id = UUID.fromString("ebc295e1-a678-40e5-88cc-1541bcc40545"),
name = "Test product",
description = "A test product"
)
val request = Request[IO](
method = Method.POST,
uri = url,
).withEntity(retailProduct)
val (response, status) = httpClientResource
.use(_.run(request).use { response =>
response.as[RetailProductAccepted].map(_ -> response.status)
})
.unsafeRunSync()
status mustBe Status.Accepted
//println(appContainer.container.getLogs)
val event = kafkaConsumer[RetailProductEvent]("retail-product-submitted")
.collectFirst {
case record if record.data.id == response.id => record
}
.compile
.lastOrError
.unsafeRunSync()
// Check that we successfully stored this Retail Product Event in Kafka
event.data mustBe retailProduct
}
"return error when handling wrong body" in {
val body =
json"""{
"id": "ABCEDFGHI",
"name": "Test product",
"description": "A test product"
}"""
val request = Request[IO](
method = Method.POST,
uri = url,
)
.withEntity(body)
val (_, status) = httpClientResource
.use(_.run(request).use { response =>
response.as[BadRequestErrorResponse].map(_ -> response.status)
})
.unsafeRunSync()
println(status)
status mustBe Status.BadRequest
}
}
}It’s important also to review the GenericContainer capabilities. A generic container is useful as a target for our testing purposes by allowing us to prepare all the context to host our service or application. In our example, we must load the jar file containing our service implementation with all its dependencies. We can specify where to find the jar file in our host filesystem. We can specify that we want a new docker image from a dockerfile and even dynamically build this image.
package com.acme.containers
import com.dimafeng.testcontainers._
import com.dimafeng.testcontainers.scalatest._
import com.acme.containers.BaseContainers.{AppPort, WaitStrategyForAPI}
import org.scalatest.Suite
import org.testcontainers.containers.wait.strategy.{HttpWaitStrategy, LogMessageWaitStrategy, WaitAllStrategy, WaitStrategy}
import org.testcontainers.containers.{Network, GenericContainer => JavaGenericContainer}
import org.testcontainers.images.builder.ImageFromDockerfile
import org.testcontainers.lifecycle.Startable
import org.testcontainers.utility.DockerImageName
import java.nio.file.Path
import scala.jdk.CollectionConverters._
trait BaseContainers extends TestContainersForAll {
self: Suite =>
protected val sharedNetwork: Network = Network.newNetwork()
protected lazy val kafkaContainer: KafkaContainer = new KafkaContainer(DockerImageName.parse("confluentinc/cp-kafka:7.4.0")) {
container.withNetwork(sharedNetwork)
container.withNetworkAliases("kafka")
}
protected lazy val mySQLContainer: MySQLContainer = new MySQLContainer(
mysqlImageVersion = Some(DockerImageName.parse("mysql:8.0")),
) {
container.withNetwork(sharedNetwork)
container.withNetworkAliases("mysql")
container.withEnv("ENVIRONMENT", "local")
}
protected def baseAppContainer(
name: String,
jarName: String,
mainClass: String,
baseFolder: String,
port: Int = AppPort,
//waitStrategy: WaitStrategy = WaitStrategyForAPI,
envVars: Map[String, String] = Map.empty,
containerDependencies: List[Startable]
): GenericContainer = new GenericContainer({
val c = new JavaGenericContainer(
new ImageFromDockerfile()
.withFileFromPath(jarName, Path.of(s"$baseFolder/$jarName").toAbsolutePath)
.withDockerfileFromBuilder { builder =>
builder
.from("openjdk:11-jre-slim")
.copy(jarName, s"/$jarName")
.cmd("java", "-Xmx256m", "-cp", s"/$jarName", mainClass)
.build()
}
)
c.dependsOn(containerDependencies.asJavaCollection)
c.withEnv(envVars.asJava)
//c.withStartupAttempts(3)
c.withEnv("ENVIRONMENT", "local")
c.withEnv("APP_PORT", port.toString)
c.withEnv("KAFKA_BOOTSTRAP_SERVERS", "kafka:9092")
//c.setWaitStrategy(waitStrategy)
c.withExposedPorts(port)
c.withNetwork(sharedNetwork)
c.withNetworkAliases(name)
c
})
}
object BaseContainers {
val AppPort = 9000
val WaitStrategyForAPI: WaitStrategy = new HttpWaitStrategy().forPath("/health").forStatusCode(200)
val WaitStrategyForService: WaitStrategy = new WaitAllStrategy()
.withStrategy(WaitStrategyForAPI)
.withStrategy(new LogMessageWaitStrategy().withRegEx("^.*Resetting offset for partition.*$"))
}It is critical for successful testing that all infrastructure prerequisites for our container are ready and healthy before starting up. For this reason, the dependsOn function is essential.
package com.acme.containers
import com.dimafeng.testcontainers._
import com.dimafeng.testcontainers.scalatest._
import com.acme.containers.BaseContainers.{AppPort, WaitStrategyForAPI}
import org.scalatest.Suite
import org.testcontainers.containers.wait.strategy.{HttpWaitStrategy, LogMessageWaitStrategy, WaitAllStrategy, WaitStrategy}
import org.testcontainers.containers.{Network, GenericContainer => JavaGenericContainer}
import org.testcontainers.images.builder.ImageFromDockerfile
import org.testcontainers.lifecycle.Startable
import org.testcontainers.utility.DockerImageName
import java.nio.file.Path
import scala.jdk.CollectionConverters._
trait BaseContainers extends TestContainersForAll {
self: Suite =>
protected val sharedNetwork: Network = Network.newNetwork()
protected lazy val kafkaContainer: KafkaContainer = new KafkaContainer(DockerImageName.parse("confluentinc/cp-kafka:7.4.0")) {
container.withNetwork(sharedNetwork)
container.withNetworkAliases("kafka")
}
protected lazy val mySQLContainer: MySQLContainer = new MySQLContainer(
mysqlImageVersion = Some(DockerImageName.parse("mysql:8.0")),
) {
container.withNetwork(sharedNetwork)
container.withNetworkAliases("mysql")
container.withEnv("ENVIRONMENT", "local")
}
protected def baseAppContainer(
name: String,
jarName: String,
mainClass: String,
baseFolder: String,
port: Int = AppPort,
//waitStrategy: WaitStrategy = WaitStrategyForAPI,
envVars: Map[String, String] = Map.empty,
containerDependencies: List[Startable]
): GenericContainer = new GenericContainer({
val c = new JavaGenericContainer(
new ImageFromDockerfile()
.withFileFromPath(jarName, Path.of(s"$baseFolder/$jarName").toAbsolutePath)
.withDockerfileFromBuilder { builder =>
builder
.from("openjdk:11-jre-slim")
.copy(jarName, s"/$jarName")
.cmd("java", "-Xmx256m", "-cp", s"/$jarName", mainClass)
.build()
}
)
c.dependsOn(containerDependencies.asJavaCollection)
c.withEnv(envVars.asJava)
//c.withStartupAttempts(3)
c.withEnv("ENVIRONMENT", "local")
c.withEnv("APP_PORT", port.toString)
c.withEnv("KAFKA_BOOTSTRAP_SERVERS", "kafka:9092")
//c.setWaitStrategy(waitStrategy)
c.withExposedPorts(port)
c.withNetwork(sharedNetwork)
c.withNetworkAliases(name)
c
})
}
object BaseContainers {
val AppPort = 9000
val WaitStrategyForAPI: WaitStrategy = new HttpWaitStrategy().forPath("/health").forStatusCode(200)
val WaitStrategyForService: WaitStrategy = new WaitAllStrategy()
.withStrategy(WaitStrategyForAPI)
.withStrategy(new LogMessageWaitStrategy().withRegEx("^.*Resetting offset for partition.*$"))
}This allows us to specify all container dependencies for our target container. In the next section, we will review some more advanced features provided by testcontainers, using Wait Strategies.
Not-so-basic features reviewed
In the previous section, we reviewed many essential features that are offered by default by testcontainers. In this section, I look at some of the more advanced features that could be useful when testing your service or application.
Containers Parallel Startup
By default, when you need to work with multiple containers in your tests, these are started sequentially, which can be enough in most cases. But let’s be honest, if we have multiple cores available in the host machine, why not take advantage of them and try to start them in parallel and reduce the startup overhead? That’s precisely what testcontainers’ Startables.deepStart function offers. In our example, we can take advantage of starting both MySQL and Kafka in parallel so that our target container would be able to use them sooner.
package com.acme.api.utils
import com.dimafeng.testcontainers._
import com.acme.containers.BaseContainers
import org.scalatest.Suite
import com.dimafeng.testcontainers.lifecycle.and
import org.testcontainers.lifecycle.Startables
trait Containers extends BaseContainers {
self: Suite =>
override type Containers = KafkaContainer and GenericContainer and MySQLContainer
protected lazy val retailProductServiceContainer: GenericContainer = baseAppContainer(
name = "retail-product-app",
jarName = "run.jar",
mainClass = "com.acme.service.RetailProductApp",
baseFolder = "retailProductService/target/scala-2.13",
envVars = Map(
"ENVIRONMENT" -> "local",
),
containerDependencies = List(kafkaContainer)
)
override def startContainers: Containers = {
Startables.deepStart(kafkaContainer, mySQLContainer).join()
retailProductServiceContainer.start()
kafkaContainer and retailProductServiceContainer and mySQLContainer
}
}Wait Strategies
Testcontainers offer a wide range of possibilities for your container to wait until other dependency containers are ready. By default, testcontainers strategy is to wait up to 60 seconds for the container’s mapped port to start listening. But in many cases, we would like to advertise that our container is ready and healthy once some conditions are fulfilled. In our example, we can combine multiple wait strategies so that the external world knows our target container is prepared.
package com.acme.containers
import com.dimafeng.testcontainers._
import com.dimafeng.testcontainers.scalatest._
import com.acme.containers.BaseContainers.{AppPort, WaitStrategyForAPI}
import org.scalatest.Suite
import org.testcontainers.containers.wait.strategy.{HttpWaitStrategy, LogMessageWaitStrategy, WaitAllStrategy, WaitStrategy}
import org.testcontainers.containers.{Network, GenericContainer => JavaGenericContainer}
import org.testcontainers.images.builder.ImageFromDockerfile
import org.testcontainers.lifecycle.Startable
import org.testcontainers.utility.DockerImageName
import java.nio.file.Path
import scala.jdk.CollectionConverters._
trait BaseContainers extends TestContainersForAll {
self: Suite =>
protected val sharedNetwork: Network = Network.newNetwork()
protected lazy val kafkaContainer: KafkaContainer = new KafkaContainer(DockerImageName.parse("confluentinc/cp-kafka:7.4.0")) {
container.withNetwork(sharedNetwork)
container.withNetworkAliases("kafka")
}
protected lazy val mySQLContainer: MySQLContainer = new MySQLContainer(
mysqlImageVersion = Some(DockerImageName.parse("mysql:8.0")),
) {
container.withNetwork(sharedNetwork)
container.withNetworkAliases("mysql")
container.withEnv("ENVIRONMENT", "local")
}
protected def baseAppContainer(
name: String,
jarName: String,
mainClass: String,
baseFolder: String,
port: Int = AppPort,
//waitStrategy: WaitStrategy = WaitStrategyForAPI,
envVars: Map[String, String] = Map.empty,
containerDependencies: List[Startable]
): GenericContainer = new GenericContainer({
val c = new JavaGenericContainer(
new ImageFromDockerfile()
.withFileFromPath(jarName, Path.of(s"$baseFolder/$jarName").toAbsolutePath)
.withDockerfileFromBuilder { builder =>
builder
.from("openjdk:11-jre-slim")
.copy(jarName, s"/$jarName")
.cmd("java", "-Xmx256m", "-cp", s"/$jarName", mainClass)
.build()
}
)
c.dependsOn(containerDependencies.asJavaCollection)
c.withEnv(envVars.asJava)
//c.withStartupAttempts(3)
c.withEnv("ENVIRONMENT", "local")
c.withEnv("APP_PORT", port.toString)
c.withEnv("KAFKA_BOOTSTRAP_SERVERS", "kafka:9092")
//c.setWaitStrategy(waitStrategy)
c.withExposedPorts(port)
c.withNetwork(sharedNetwork)
c.withNetworkAliases(name)
c
})
}
object BaseContainers {
val AppPort = 9000
val WaitStrategyForAPI: WaitStrategy = new HttpWaitStrategy().forPath("/health").forStatusCode(200)
val WaitStrategyForService: WaitStrategy = new WaitAllStrategy()
.withStrategy(WaitStrategyForAPI)
.withStrategy(new LogMessageWaitStrategy().withRegEx("^.*Resetting offset for partition.*$"))
}Here, we are saying that our service will be ready once its /health endpoint returns 200 as a response status code using the HttpWaitStrategy. And when a particular text message is present in the logs, the Kafka client is ready and connected with the Kafka server, using the LogMessageWaitStrategy.
I want to also call your attention to another excellent capability that testcontainers support, using Docker’s underlying health check feature. For this, a Wait.forHealthcheck() strategy can be used to take advantage of this handy Docker feature.
Customizing your container with advanced Docker features
For extreme cases when you need access to some more advanced features in Docker container customization that testcontainers’ API doesn’t directly expose, you could use the withCreateContainerCmdModifier function. All the modifiers that you specify here will be applied on top of the container definition that Testcontainers creates, like, for example, the container’s hostname or the container’s memory size, or any additional labels that you would like to add for your container. There are many possibilities here that you can review in this Docker Java API code.
Troubleshooting
Sometimes, you might need to go a little deeper to understand what’s going on with your target container under testing and see what’s happening behind the scenes while running some of your tests. For that, I want to mention a couple of helpers that could help you.
Firstly, testcontainers provides a getLogs function in the inner container Java object inside any container type. For example, you could use appContainer.container.getLogs to get the container’s logs at any moment, which could help you in your service troubleshooting.
In addition, testcontainers utilizes SLF4J for logging, logback being the most common. Inside your logback XML file, you could use:
<logger name="org.testcontainers" level="DEBUG"/>
To increase the logging level for your testcontainers and get more detailed information about what’s going on.
Conclusion
After this overview, I hope that I have given you some good reasons to provide testcontainers a chance when building your integration testing strategy for your projects. Not only considering all the flexibility, the variety of containers, and features that can make a developer’s life easier, it is also important to increase their productivity. We could summarize testcontainers main advantages like this:
- A wide range of official containers support and the possibility to define your container images and customize them as you wish.
- Helpful features allowing you to control your container’s behavior and lifecycle.
- Consistent experience when used locally on your developer machine and your CI/CD environment.
- Troubleshooting support for your target container.
- Independent of your Scala stack, you can use it with Cats, ZIO, Akka, or any stack that you prefer.
Links
- https://github.com/testcontainers/testcontainers-scala
- https://github.com/hwgonz/testcontainers-sample
- https://github.com/hwgonz/testcontainers-sample/blob/master/utils/src/it/scala/com/acme/containers/BaseContainers.scala
- https://github.com/testcontainers/testcontainers-scala/blob/master/docs/src/main/tut/usage.md
- https://github.com/hwgonz/testcontainers-sample/blob/master/utils/src/it/scala/com/acme/containers/BaseContainers.scala#L19
- https://github.com/hwgonz/testcontainers-sample/blob/master/utils/src/it/scala/com/acme/containers/BaseContainers.scala#L56
- https://github.com/hwgonz/testcontainers-sample/blob/master/utils/src/it/scala/com/acme/containers/BaseContainers.scala#L62
- https://github.com/hwgonz/testcontainers-sample/blob/master/retailProductService/src/it/scala/com/acme/api/retailproduct/RetailProductSpec.scala#L28
- https://github.com/hwgonz/testcontainers-sample/blob/master/utils/src/it/scala/com/acme/containers/BaseContainers.scala#L45
- https://github.com/hwgonz/testcontainers-sample/blob/master/utils/src/it/scala/com/acme/containers/BaseContainers.scala#L55
- https://github.com/hwgonz/testcontainers-sample/blob/master/retailProductService/src/it/scala/com/acme/api/utils/Containers.scala#L26
- https://github.com/hwgonz/testcontainers-sample/blob/master/utils/src/it/scala/com/acme/containers/BaseContainers.scala#L77
- https://github.com/docker-java/docker-java/blob/3.2.1/docker-java-api/src/main/java/com/github/dockerjava/api/command/CreateContainerCmd.java
Read also
Authors
I'm a software engineer with more than 20 years of experience in software development, infrastructure and business intelligence projects, in various industries (banking, telecommunications, retail, oil & gas).As a Services Practice Lead, I have experience in building and developing high impact teams, that have delivered quantifiable business value to involved customers, making these teams a key business capability for an organization. I'm also an enthusiast about leveraging functional programming with Scala and reactive technologies, learning and practicing a lot about the following technologies and patterns: ZIO, Akka, serverless, event sourcing and CQRS.
Popular Posts in category
Looking into Scala.js
Your first microservices using Scala and Lagom
Handling Split Brain scenarios with Akka
