How to set up Bazel build tool for your Scala project
You have probably encountered this problem while working with SBT and bigger projects. I’m talking about compilation times and test execution times, in other words, having to wait instead of working. Imagine working with a build tool that rebuilds only what is necessary, using a distributed cache, so if module A is built by one of your team members you won’t have to do it again. Or imagine being able to run builds of different parts of your project in parallel, or run tests only for the affected code that has been changed. Sounds promising right? That’s why, in this tutorial, I will be showing you what Bazel build is and how to set your project up in Scala.
Bazel is a build tool from Google, which allows you to easily manage builds and tests in huge projects. This tool gives huge flexibility when it comes to the configuration and granularity of the basic build unit. It can be a set of packages, one package or even just one file. The basic build unit is called a target, the target is an instance of rules. A rule is a function that has a set of inputs and outputs; if the inputs do not change then the outputs stay the same. By having more targets (the disadvantage of this solution is having more build files to maintain) where not all of them depend on each other, more builds can run in parallel because Bazel build uses incremental builds, so it rebuilds only the part of the dependency graph that has been changed, as well as only running tests for the affected parts.
It can distribute, build and test actions across multiple machines, and then build and reuse previously done cached work, which makes your builds even more scalable.
Bazel can also print out a dependency graph, the results of which can be visualized on this page webgraphviz.com
So if your project takes a lot of time to build, and you don’t want to waste any more time, this tool is what you need. Speed up your compile times, speed up your tests, speed up your whole team’s work!
In this tutorial, we will be using Bazel version 1.0.0.
├── BUILD
├── WORKSPACE
├── bazeltest
│ ├── BUILD
│ └── src
│ ├── main
│ │ └── scala
│ │ └── bazeltest
│ │ └── Main.scala
│ └── test
│ └── scala
│ └── bazeltest
│ └── MainSpec.scala
├── dependencies.yaml
└── othermodule
├── BUILD
└── src
├── main
│ └── scala
│ └── othermodule
│ └── Worker.scala
└── test
└── scala
└── othermodule
└── WorkerSpec.scala
So we have two modules called: bazeltest and othermodule.
Bazeltest will depend on othermodule.
To make it work with Scala, then let’s take an already prepared boilerplate WORKSPACE file from:
https://github.com/bazelbuild/rules_scala#getting-started
Be aware of the change in rules_scala_version. Rules_scala_version is commit’s sha. So if you want to use the newest version of the rules, check GitHub repository and copy-paste commit’s sha.
We also have to add at the end of the file:
This will be used by a third-party tool called bazel-deps, but we will come back to this at the next step.
So after the changes:
If you wish to set a specific Scala version, add code from: https://github.com/bazelbuild/rules_scala#getting-started
In this file, we will setup the Scala rules and everything else that is needed to compile the Scala project.
To write BUILD files we will use the following methods:
In the bazeltest package BUILD file:
Now the BUILD file for othermodule:
Open the dependencies.yaml file and put this there:
Save the system variable of this project path, for example (working on a Mac):
We will need this in a minute.
Clone https://github.com/johnynek/bazel-deps and enter the bazel-deps folder. Ensure that this tool uses the same rules_scala commit sha.
Open the WORKSPACE file inside the bazel-deps and look for this:
This will download dependencies into a 3rdparty folder into your project directory.
Introduction to Bazel build
Bazel is a build tool from Google, which allows you to easily manage builds and tests in huge projects. This tool gives huge flexibility when it comes to the configuration and granularity of the basic build unit. It can be a set of packages, one package or even just one file. The basic build unit is called a target, the target is an instance of rules. A rule is a function that has a set of inputs and outputs; if the inputs do not change then the outputs stay the same. By having more targets (the disadvantage of this solution is having more build files to maintain) where not all of them depend on each other, more builds can run in parallel because Bazel build uses incremental builds, so it rebuilds only the part of the dependency graph that has been changed, as well as only running tests for the affected parts.
It can distribute, build and test actions across multiple machines, and then build and reuse previously done cached work, which makes your builds even more scalable.
Bazel can also print out a dependency graph, the results of which can be visualized on this page webgraphviz.com
So if your project takes a lot of time to build, and you don’t want to waste any more time, this tool is what you need. Speed up your compile times, speed up your tests, speed up your whole team’s work!
In this tutorial, we will be using Bazel version 1.0.0.
Project structure
We will be working on a project with this structure:├── BUILD
├── WORKSPACE
├── bazeltest
│ ├── BUILD
│ └── src
│ ├── main
│ │ └── scala
│ │ └── bazeltest
│ │ └── Main.scala
│ └── test
│ └── scala
│ └── bazeltest
│ └── MainSpec.scala
├── dependencies.yaml
└── othermodule
├── BUILD
└── src
├── main
│ └── scala
│ └── othermodule
│ └── Worker.scala
└── test
└── scala
└── othermodule
└── WorkerSpec.scala
So we have two modules called: bazeltest and othermodule.
Bazeltest will depend on othermodule.
Workspace file setup
Each project has one WORKSPACE file, where we will define things like Scala version and dependencies. If in the project directory there is a subdirectory with a WORKSPACE file, then while doing our builds this subdirectory will be omitted.To make it work with Scala, then let’s take an already prepared boilerplate WORKSPACE file from:
https://github.com/bazelbuild/rules_scala#getting-started
Be aware of the change in rules_scala_version. Rules_scala_version is commit’s sha. So if you want to use the newest version of the rules, check GitHub repository and copy-paste commit’s sha.
We also have to add at the end of the file:
|
This will be used by a third-party tool called bazel-deps, but we will come back to this at the next step.
So after the changes:
|
If you wish to set a specific Scala version, add code from: https://github.com/bazelbuild/rules_scala#getting-started
scala_repositories((
"2.12.8",
{
"scala_compiler": "f34e9119f45abd41e85b9e121ba19dd9288b3b4af7f7047e86dc70236708d170",
"scala_library": "321fb55685635c931eba4bc0d7668349da3f2c09aee2de93a70566066ff25c28",
"scala_reflect": "4d6405395c4599ce04cea08ba082339e3e42135de9aae2923c9f5367e957315a"
}
))
In this file, we will setup the Scala rules and everything else that is needed to compile the Scala project.
BUILD files setup
To write BUILD files we will use the following methods:
- load – which loads the Bazel Scala rules, and extensions
- scala_binary – generates a Scala executable
- scala_library – generates a .jar file from Scala source files.
- scala_test – generates a Scala executable that runs unit test suites written using the scalatest library.
load("@io_bazel_rules_scala//scala:scala.bzl", "scala_binary")
scala_binary(
name = "App",
deps = [
"//bazeltest"
],
main_class = "bazeltest.Main"
)
In the bazeltest package BUILD file:
load("@io_bazel_rules_scala//scala:scala.bzl", "scala_library", "scala_test")
scala_library(
name = "bazeltest",
srcs = ["src/main/scala/bazeltest/Main.scala"],
deps = [
"//othermodule",
"//3rdparty/jvm/joda_time:joda_time"
],
visibility = ["//:__pkg__"]
)
scala_test(
name = "test-main",
srcs = ["src/test/scala/bazeltest/MainSpec.scala"],
deps = [":bazeltest"]
)
glob(["src/main/scala/bazeltest/*.scala"]) glob(["src/main/scala/bazeltest/**/*..scala"]) Now the BUILD file for othermodule:
load("@io_bazel_rules_scala//scala:scala.bzl", "scala_library", "scala_test")
scala_library(
name = "othermodule",
srcs = glob(["src/main/scala/othermodule/*.scala"]),
deps = [],
visibility = ["//bazeltest:__pkg__"]
)
scala_test(
name = "test-othermodule",
srcs = ["src/test/scala/othermodule/WorkerSpec.scala"],
deps = [":othermodule"]
)
Dependencies
We will use a third-party tool for this: https://github.com/johnynek/bazel-depsOpen the dependencies.yaml file and put this there:
options:
buildHeader: [
"load(\"@io_bazel_rules_scala//scala:scala_import.bzl\", \"scala_import\")",
"load(\"@io_bazel_rules_scala//scala:scala.bzl\", \"scala_library\", \"scala_binary\", \"scala_test\")"
]
languages: [ "java", "scala:2.12.8" ]
resolverType: "coursier"
resolvers:
- id: "mavencentral"
type: "default"
url: https://repo.maven.apache.org/maven2/
- id: "hmrc"
type: "default"
url: https://hmrc.bintray.com/releases
strictVisibility: true
transitivity: runtime_deps
versionConflictPolicy: highest
dependencies:
joda-time:
joda-time:
lang: java
version: "2.10.4"
com.typesafe.scala-logging:
scala-logging:
lang: scala
version: "3.9.0"
com.typesafe.akka:
akka-http:
lang: scala
version: "10.1.7"
org.scalatest:
scalatest:
lang: scala
version: "3.0.8"
replacements:
org.scala-lang:
scala-library:
lang: scala/unmangled
target: "@io_bazel_rules_scala_scala_library//:io_bazel_rules_scala_scala_library"
scala-reflect:
lang: scala/unmangled
target: "@io_bazel_rules_scala_scala_reflect//:io_bazel_rules_scala_scala_reflect"
scala-compiler:
lang: scala/unmangled
target: "@io_bazel_rules_scala_scala_compiler//:io_bazel_rules_scala_scala_compiler"
org.scala-lang.modules:
scala-parser-combinators:
lang: scala
target:
"@io_bazel_rules_scala_scala_parser_combinators//:io_bazel_rules_scala_scala_parser_combinators"
scala-xml:
lang: scala
target:
“@io_bazel_rules_scala_scala_xml//:io_bazel_rules_scala_scala_xml"Save the system variable of this project path, for example (working on a Mac):
export MY_PROJ_DIR=`pwd`We will need this in a minute.
Clone https://github.com/johnynek/bazel-deps and enter the bazel-deps folder. Ensure that this tool uses the same rules_scala commit sha.
Open the WORKSPACE file inside the bazel-deps and look for this:
git_repository(
name = "io_bazel_rules_scala",
remote = "https://github.com/bazelbuild/rules_scala",
commit = "0f89c210ade8f4320017daf718a61de3c1ac4773" # HEAD as of 2019-10-17, update this as needed
)bazel run //:parse generate -- --repo-root "$MY_PROJ_DIR" --sha-file 3rdparty/workspace.bzl
--deps dependencies.yamlThis will download dependencies into a 3rdparty folder into your project directory.
INFO: Analyzed target //:parse (0 packages loaded, 0 targets configured).
INFO: Found 1 target...
Target //src/scala/com/github/johnynek/bazel_deps:parseproject up-to-date:
bazel-bin/src/scala/com/github/johnynek/bazel_deps/parseproject
bazel-bin/src/scala/com/github/johnynek/bazel_deps/parseproject.jar
INFO: Elapsed time: 0.168s, Critical Path: 0.01s
INFO: 0 processes.
INFO: Build completed successfully, 1 total action
INFO: Build completed successfully, 1 total action
wrote 26 targets in 8 BUILD files
The first run
Before doing the first run, let’s implement our Main and Worker classes.package bazeltest
import othermodule.Worker
import org.joda.time.DateTime
object Main extends App {
println("IN MAIN now: "+DateTime.now().plusYears(11))
val worker = new Worker
worker.doSomething()
def status(): String = "OKi"
}
package othermodule
class Worker {
def doSomething() : Int = {
println("Doing something")
12345
}
def pureFunc(): String = "ABC"
}
bazel run //:App
INFO: Analyzed target //:App (1 packages loaded, 2 targets configured).
INFO: Found 1 target...
INFO: From Linking external/com_google_protobuf/libprotobuf_lite.a [for host]:
/Library/Developer/CommandLineTools/usr/bin/libtool: file: bazel-out/host/bin/external/com_google_protobuf/_objs/protobuf_lite/io_win32.o has no symbols
INFO: From Linking external/com_google_protobuf/libprotobuf.a [for host]:
/Library/Developer/CommandLineTools/usr/bin/libtool: file: bazel-out/host/bin/external/com_google_protobuf/_objs/protobuf/error_listener.o has no symbols
INFO: From Building external/com_google_protobuf/libprotobuf_java.jar (122 source files, 1 source jar):
warning: -parameters is not supported for target value 1.7. Use 1.8 or later.
Target //:App up-to-date:
bazel-bin/App
bazel-bin/App.jar
INFO: Elapsed time: 52.246s, Critical Path: 23.22s
INFO: 194 processes: 189 darwin-sandbox, 5 worker.
INFO: Build completed successfully, 198 total actions
INFO: Build completed successfully, 198 total actions
IN MAIN now: 2030-10-11T11:26:07.533+01:00
Doing something
Unit tests
Now let’s write some simple unit tests:package bazeltest
import org.scalatest._
class MainSpec extends FlatSpec with Matchers {
"status" should "return OK" in {
Main.status() shouldBe "OKi"
}
}
package othermodule
import org.scalatest._
class WorkerSpec extends FlatSpec with Matchers {
val worker = new Worker()
"do something" should "return 12345" in {
worker.doSomething() shouldBe 12345
}
"pureFunc" should "return ABC" in {
worker.pureFunc() shouldBe "ABC"
bazel test //bazeltest:test-main
INFO: Analyzed target //bazeltest:test-main (0 packages loaded, 0 targets configured).
INFO: Found 1 test target...
Target //bazeltest:test-main up-to-date:
bazel-bin/bazeltest/test-main
bazel-bin/bazeltest/test-main.jar
INFO: Elapsed time: 1.047s, Critical Path: 0.89s
INFO: 3 processes: 2 darwin-sandbox, 1 worker.
INFO: Build completed successfully, 4 total actions
//bazeltest:test-main PASSED in 0.5s
Executed 1 out of 1 test: 1 test passes.
INFO: Build completed successfully, 4 total actions
bazel test //othermodule:test-othermodule
INFO: Analyzed target //othermodule:test-othermodule (0 packages loaded, 0 targets configured).
INFO: Found 1 test target...
Target //othermodule:test-othermodule up-to-date:
bazel-bin/othermodule/test-othermodule
bazel-bin/othermodule/test-othermodule.jar
INFO: Elapsed time: 1.438s, Critical Path: 1.29s
INFO: 2 processes: 1 darwin-sandbox, 1 worker.
INFO: Build completed successfully, 3 total actions
//othermodule:test-othermodule PASSED in 0.6s
Executed 1 out of 1 test: 1 test passes.
INFO: Build completed successfully, 3 total actions
bazel test //bazeltest:test-main
INFO: Analyzed target //bazeltest:test-main (0 packages loaded, 0 targets configured).
INFO: Found 1 test target...
FAIL: //bazeltest:test-main (see /private/var/tmp/_bazel_maciejbak/16727409c9f0575889b09993f53ce424/execroot/__main__/bazel-out/darwin-fastbuild/testlogs/bazeltest/test-main/test.log)
Target //bazeltest:test-main up-to-date:
bazel-bin/bazeltest/test-main
bazel-bin/bazeltest/test-main.jar
INFO: Elapsed time: 1.114s, Critical Path: 0.96s
INFO: 3 processes: 2 darwin-sandbox, 1 worker.
INFO: Build completed, 1 test FAILED, 4 total actions
//bazeltest:test-main FAILED in 0.6s
/private/var/tmp/_bazel_maciejbak/16727409c9f0575889b09993f53ce424/execroot/__main__/bazel-out/darwin-fastbuild/testlogs/bazeltest/test-main/test.log
INFO: Build completed, 1 test FAILED, 4 total actions
bazel test //othermodule:test-othermodule
INFO: Analyzed target //othermodule:test-othermodule (0 packages loaded, 0 targets configured).
INFO: Found 1 test target...
Target //othermodule:test-othermodule up-to-date:
bazel-bin/othermodule/test-othermodule
bazel-bin/othermodule/test-othermodule.jar
INFO: Elapsed time: 0.150s, Critical Path: 0.00s
INFO: 0 processes.
INFO: Build completed successfully, 1 total action
//othermodule:test-othermodule (cached) PASSED in 0.6s
Executed 0 out of 1 test: 1 test passes.
INFO: Build completed successfully, 1 total action
bazel test //bazeltest:test-main
INFO: Analyzed target //bazeltest:test-main (0 packages loaded, 0 targets configured).
INFO: Found 1 test target...
Target //bazeltest:test-main up-to-date:
bazel-bin/bazeltest/test-main
bazel-bin/bazeltest/test-main.jar
INFO: Elapsed time: 1.389s, Critical Path: 1.22s
INFO: 2 processes: 1 darwin-sandbox, 1 worker.
INFO: Build completed successfully, 3 total actions
//bazeltest:test-main PASSED in 0.6s
Executed 1 out of 1 test: 1 test passes.
INFO: Build completed successfully, 3 total actions
Dependency graph
We can easily visualize our dependency graph: In the command line run:bazel query --noimplicit_deps "deps(//:App)" --output graph
digraph mygraph {
node [shape=box];
"//:App"
"//:App" -> "//bazeltest:bazeltest"
"//bazeltest:bazeltest"
"//bazeltest:bazeltest" -> "//bazeltest:src/main/scala/bazeltest/Main.scala"
"//bazeltest:bazeltest" -> "//3rdparty/jvm/joda_time:joda_time"
"//bazeltest:bazeltest" -> "//othermodule:othermodule"
"//othermodule:othermodule"
"//othermodule:othermodule" -> "//othermodule:src/main/scala/othermodule/Worker.scala"
"//othermodule:src/main/scala/othermodule/Worker.scala"
"//3rdparty/jvm/joda_time:joda_time"
"//3rdparty/jvm/joda_time:joda_time" -> "//external:jar/joda_time/joda_time"
"//external:jar/joda_time/joda_time"
"//external:jar/joda_time/joda_time" -> "@joda_time_joda_time//jar:jar"
"//bazeltest:src/main/scala/bazeltest/Main.scala"
"@joda_time_joda_time//jar:jar"
"@joda_time_joda_time//jar:jar" -> "@joda_time_joda_time//jar:joda_time_joda_time.jar\n@joda_time_joda_time//jar:joda_time_joda_time-sources.jar"
"@joda_time_joda_time//jar:joda_time_joda_time.jar\n@joda_time_joda_time//jar:joda_time_joda_time-sources.jar"
}
Loading: 12 packages loaded
Generate jar
bazel build //:App
INFO: Analyzed target //:App (0 packages loaded, 0 targets configured).
INFO: Found 1 target...
Target //:App up-to-date:
bazel-bin/App
bazel-bin/App.jar
INFO: Elapsed time: 0.085s, Critical Path: 0.00s
INFO: 0 processes.
INFO: Build completed successfully, 1 total action
Bazel build: Summary
In this post, we showed what is bazel, when to use it, and how to make basic configuration. It can take some time to properly set up complex projects using bazel build, but I guarantee you, in the end, it will speed up the whole team’s work.
Useful links
- Official Bazel build documentation https://docs.bazel.build/versions/1.0.0/bazel-overview.html
- Building Scala with Bazel build- Natan Silnitsky https://www.youtube.com/watch?v=K2Ytk0S4PF0
- Building Java Applications with Bazel https://www.baeldung.com/bazel-build-tool
See also
Authors
Maciej Bąk
I am a backend developer since 2013 February. During my career, I worked in many different projects, requiring high traffic, NoSQL and SQL databases, microservices connecting each other via akka, Kafka, RabbitMQ. I am continually improving my skills, currently in functional programming with the ZIO library, Kafka, and Spark. Besides work, I love reading books, learning Spanish (as I live in Canarias islands since 2016 November ), and enjoy the sun!
Popular Posts in category
Looking into Scala.js
Your first microservices using Scala and Lagom
Handling Split Brain scenarios with Akka
