You have probably encountered this problem while working with SBT and bigger projects. I’m talking about compilation times and test execution times, in other words, having to wait instead of working. Imagine working with a build tool that rebuilds only what is necessary, using a distributed cache, so if module A is built by one of your team members you won’t have to do it again. Or imagine being able to run builds of different parts of your project in parallel, or run tests only for the affected code that has been changed. Sounds promising right? That’s why, in this tutorial, I will be showing you what Bazel build is and how to set your project up in Scala.
Introduction to Bazel build
Bazel is a build tool from Google, which allows you to easily manage builds and tests in huge projects. This tool gives huge flexibility when it comes to the configuration and granularity of the basic build unit. It can be a set of packages, one package or even just one file. The basic build unit is called a target, the target is an instance of rules. A rule is a function that has a set of inputs and outputs; if the inputs do not change then the outputs stay the same. By having more targets (the disadvantage of this solution is having more build files to maintain) where not all of them depend on each other, more builds can run in parallel because Bazel build uses incremental builds, so it rebuilds only the part of the dependency graph that has been changed, as well as only running tests for the affected parts.
It can distribute, build and test actions across multiple machines, and then build and reuse previously done cached work, which makes your builds even more scalable.
Bazel can also print out a dependency graph, the results of which can be visualized on this pagewebgraphviz.com
So if your project takes a lot of time to build, and you don’t want to waste any more time, this tool is what you need. Speed up your compile times, speed up your tests, speed up your whole team’s work!
In this tutorial, we will be using Bazel version 1.0.0.
Project structure
We will be working on a project with this structure:
├── BUILD ├── WORKSPACE ├── bazeltest │ ├── BUILD │ └── src │ ├── main │ │ └── scala │ │ └── bazeltest │ │ └── Main.scala │ └── test │ └── scala │ └── bazeltest │ └── MainSpec.scala ├── dependencies.yaml └── othermodule ├── BUILD └── src ├── main │ └── scala │ └── othermodule │ └── Worker.scala └── test └── scala └── othermodule └── WorkerSpec.scala
So we have two modules called: bazeltest and othermodule. Bazeltest will depend on othermodule.
Workspace file setup
Each project has one WORKSPACE file, where we will define things like Scala version and dependencies. If in the project directory there is a subdirectory with a WORKSPACE file, then while doing our builds this subdirectory will be omitted. To make it work with Scala, then let’s take an already prepared boilerplate WORKSPACE file from: https://github.com/bazelbuild/rules_scala#getting-started
Be aware of the change in rules_scala_version. Rules_scala_version is commit’s sha. So if you want to use the newest version of the rules, check GitHub repository and copy-paste commit’s sha. We also have to add at the end of the file:
We have named it App, just one dependency to the bazeltest package.In deps, we list our dependencies, where our own modules or third party can be. Main_class is our entry point.
Our Main.scala file will use some external third party dependency such as joda date time, and Worker from the subpack package.In srcs we set our Main.scala file, but it could be a list of files, listed one by one or a matching path pattern for example:
glob(["src/main/scala/bazeltest/*.scala"])
( then we use glob ), could even be a package with all the subpackages, such as:
glob(["src/main/scala/bazeltest/**/*..scala"])
and in deps all the necessary dependencies, so for this example our own sub pack package plus third part joda date time. For now, it points to the 3rdparty folder which does not exist yet, this will be done at one of the next steps so don’t worry.Visibility is used to define which other targets can use this target as a dependency, in this example, we define a project folder containing the main BUILD file. Now the BUILD file for othermodule:
Here we have set up a visibility param to the bazeltest package. So only this package can read from this one. If other packages try to reach this, we will see an error.
(Language is always required and may be one of java, Scala, Scala/unmangled.This is important, if you define an invalid language then errors will occur.Replacements are used for internal targets instead of Maven ones.)
Save the system variable of this project path, for example (working on a Mac):export MY_PROJ_DIR=`pwd` We will need this in a minute.
Clone https://github.com/johnynek/bazel-deps and enter the bazel-deps folder.Ensure that this tool uses the same rules_scala commit sha. Open the WORKSPACE file inside the bazel-deps and look for this:
git_repository(
name = "io_bazel_rules_scala",
remote = "https://github.com/bazelbuild/rules_scala",
commit = "0f89c210ade8f4320017daf718a61de3c1ac4773" # HEAD as of 2019-10-17, update this as needed
)
Commit is of course what we need to change ( if it is different than in our WORKSPACE file in rules_scala_version ).
INFO: Analyzed target //:App (1 packages loaded, 2 targets configured).
INFO: Found 1 target...
INFO: From Linking external/com_google_protobuf/libprotobuf_lite.a [for host]:
/Library/Developer/CommandLineTools/usr/bin/libtool: file: bazel-out/host/bin/external/com_google_protobuf/_objs/protobuf_lite/io_win32.o has no symbols
INFO: From Linking external/com_google_protobuf/libprotobuf.a [for host]:
/Library/Developer/CommandLineTools/usr/bin/libtool: file: bazel-out/host/bin/external/com_google_protobuf/_objs/protobuf/error_listener.o has no symbols
INFO: From Building external/com_google_protobuf/libprotobuf_java.jar (122 source files, 1 source jar):
warning: -parameters is not supported for target value 1.7. Use 1.8 or later.
Target //:App up-to-date:
bazel-bin/App
bazel-bin/App.jar
INFO: Elapsed time: 52.246s, Critical Path: 23.22s
INFO: 194 processes: 189 darwin-sandbox, 5 worker.
INFO: Build completed successfully, 198 total actions
INFO: Build completed successfully, 198 total actions
IN MAIN now: 2030-10-11T11:26:07.533+01:00
Doing something
The first run takes some time because it has to download the dependencies, so don’t worry.
Unit tests
Now let’s write some simple unit tests:
package bazeltest
import org.scalatest._
class MainSpec extends FlatSpec with Matchers {
"status" should "return OK" in {
Main.status() shouldBe "OKi"
}
}
package othermodule
import org.scalatest._
class WorkerSpec extends FlatSpec with Matchers {
val worker = new Worker()
"do something" should "return 12345" in {
worker.doSomething() shouldBe 12345
}
"pureFunc" should "return ABC" in {
worker.pureFunc() shouldBe "ABC"
And run them:bazel test //bazeltest:test-main
INFO: Analyzed target //bazeltest:test-main (0 packages loaded, 0 targets configured).
INFO: Found 1 test target...
Target //bazeltest:test-main up-to-date:
bazel-bin/bazeltest/test-main
bazel-bin/bazeltest/test-main.jar
INFO: Elapsed time: 1.047s, Critical Path: 0.89s
INFO: 3 processes: 2 darwin-sandbox, 1 worker.
INFO: Build completed successfully, 4 total actions
//bazeltest:test-main PASSED in 0.5s
Executed 1 out of 1 test: 1 test passes.
INFO: Build completed successfully, 4 total actions
bazel test //othermodule:test-othermodule
INFO: Analyzed target //othermodule:test-othermodule (0 packages loaded, 0 targets configured).
INFO: Found 1 test target...
Target //othermodule:test-othermodule up-to-date:
bazel-bin/othermodule/test-othermodule
bazel-bin/othermodule/test-othermodule.jar
INFO: Elapsed time: 1.438s, Critical Path: 1.29s
INFO: 2 processes: 1 darwin-sandbox, 1 worker.
INFO: Build completed successfully, 3 total actions
//othermodule:test-othermodule PASSED in 0.6s
Executed 1 out of 1 test: 1 test passes.
INFO: Build completed successfully, 3 total actions
Try now to change the status method from Main, to return “OK” instead of “OKi”.Run the tests again:bazel test //bazeltest:test-main
INFO: Analyzed target //bazeltest:test-main (0 packages loaded, 0 targets configured).
INFO: Found 1 test target...
FAIL: //bazeltest:test-main (see /private/var/tmp/_bazel_maciejbak/16727409c9f0575889b09993f53ce424/execroot/__main__/bazel-out/darwin-fastbuild/testlogs/bazeltest/test-main/test.log)
Target //bazeltest:test-main up-to-date:
bazel-bin/bazeltest/test-main
bazel-bin/bazeltest/test-main.jar
INFO: Elapsed time: 1.114s, Critical Path: 0.96s
INFO: 3 processes: 2 darwin-sandbox, 1 worker.
INFO: Build completed, 1 test FAILED, 4 total actions
//bazeltest:test-main FAILED in 0.6s
/private/var/tmp/_bazel_maciejbak/16727409c9f0575889b09993f53ce424/execroot/__main__/bazel-out/darwin-fastbuild/testlogs/bazeltest/test-main/test.log
INFO: Build completed, 1 test FAILED, 4 total actions
bazel test //othermodule:test-othermodule
INFO: Analyzed target //othermodule:test-othermodule (0 packages loaded, 0 targets configured).
INFO: Found 1 test target...
Target //othermodule:test-othermodule up-to-date:
bazel-bin/othermodule/test-othermodule
bazel-bin/othermodule/test-othermodule.jar
INFO: Elapsed time: 0.150s, Critical Path: 0.00s
INFO: 0 processes.
INFO: Build completed successfully, 1 total action
//othermodule:test-othermodule (cached) PASSED in 0.6s
Executed 0 out of 1 test: 1 test passes.
INFO: Build completed successfully, 1 total action
Bazel build sees what has been changed, and runs tests only for the affected classes.So test results for othermodule are taken from the cache, and only the main tests run.The test failed because we didn’t change the results in the Spec file, so the change expected the result in the test to the Main.status() shouldBe “OK”.Run tests again:bazel test //bazeltest:test-main
INFO: Analyzed target //bazeltest:test-main (0 packages loaded, 0 targets configured).
INFO: Found 1 test target...
Target //bazeltest:test-main up-to-date:
bazel-bin/bazeltest/test-main
bazel-bin/bazeltest/test-main.jar
INFO: Elapsed time: 1.389s, Critical Path: 1.22s
INFO: 2 processes: 1 darwin-sandbox, 1 worker.
INFO: Build completed successfully, 3 total actions
//bazeltest:test-main PASSED in 0.6s
Executed 1 out of 1 test: 1 test passes.
INFO: Build completed successfully, 3 total actions
Dependency graph
We can easily visualize our dependency graph:In the command line run:bazel query --noimplicit_deps "deps(//:App)" --output graph
In this post, we showed what is bazel, when to use it, and how to make basic configuration. It can take some time to properly set up complex projects using bazel build, but I guarantee you, in the end, it will speed up the whole team’s work.
I am a backend developer since 2013 February. During my career, I worked in many different projects, requiring high traffic, NoSQL and SQL databases, microservices connecting each other via akka, Kafka, RabbitMQ. I am continually improving my skills, currently in functional programming with the ZIO library, Kafka, and Spark. Besides work, I love reading books, learning Spanish (as I live in Canarias islands since 2016 November ), and enjoy the sun!
As your application gains popularity and usage, ensuring it can handle the increased traffic and demand is essential. Scaling your application increases its capacity to handle more users and data. This guide offers tips and best practices for scaling your applications effectively. Understand Your Application’s Architecture. Before you can scale your application, it’s essential to […]
If you’re using Apache Kafka for your data streaming needs, you may face challenges or want to optimize your implementation. Our consulting services can guide you in improving your Kafka setup, addressing issues, implementing best practices, and utilizing new features. What is Apache Kafka? Firstly, companies use Apache Kafka as an open-source distributed event streaming […]
Fintech is one of the world’s most exciting and fastest-growing industries. The fintech industry is worth approximately $180 billion (Deloitte). The fast-growing space is projected to reach $174 billion in 2023. And is predicted to reach $188 billion by 2024. It offers a unique set of career opportunities for future fintech developers. If you’re thinking about beginning a […]
