How to set up Bazel build tool for your Scala project
Introduction to Bazel build
Bazel is a build tool from Google, which allows you to easily manage builds and tests in huge projects. This tool gives huge flexibility when it comes to the configuration and granularity of the basic build unit. It can be a set of packages, one package or even just one file. The basic build unit is called a target, the target is an instance of rules. A rule is a function that has a set of inputs and outputs; if the inputs do not change then the outputs stay the same. By having more targets (the disadvantage of this solution is having more build files to maintain) where not all of them depend on each other, more builds can run in parallel because Bazel build uses incremental builds, so it rebuilds only the part of the dependency graph that has been changed, as well as only running tests for the affected parts.
It can distribute, build and test actions across multiple machines, and then build and reuse previously done cached work, which makes your builds even more scalable.
Bazel can also print out a dependency graph, the results of which can be visualized on this page webgraphviz.com
So if your project takes a lot of time to build, and you don’t want to waste any more time, this tool is what you need. Speed up your compile times, speed up your tests, speed up your whole team’s work!
In this tutorial, we will be using Bazel version 1.0.0.
Project structure
We will be working on a project with this structure:├── BUILD
├── WORKSPACE
├── bazeltest
│ ├── BUILD
│ └── src
│ ├── main
│ │ └── scala
│ │ └── bazeltest
│ │ └── Main.scala
│ └── test
│ └── scala
│ └── bazeltest
│ └── MainSpec.scala
├── dependencies.yaml
└── othermodule
├── BUILD
└── src
├── main
│ └── scala
│ └── othermodule
│ └── Worker.scala
└── test
└── scala
└── othermodule
└── WorkerSpec.scala
So we have two modules called: bazeltest and othermodule.
Bazeltest will depend on othermodule.
Workspace file setup
Each project has one WORKSPACE file, where we will define things like Scala version and dependencies. If in the project directory there is a subdirectory with a WORKSPACE file, then while doing our builds this subdirectory will be omitted.To make it work with Scala, then let’s take an already prepared boilerplate WORKSPACE file from:
https://github.com/bazelbuild/rules_scala#getting-started
Be aware of the change in rules_scala_version. Rules_scala_version is commit’s sha. So if you want to use the newest version of the rules, check GitHub repository and copy-paste commit’s sha.
We also have to add at the end of the file:
|
This will be used by a third-party tool called bazel-deps, but we will come back to this at the next step.
So after the changes:
|
If you wish to set a specific Scala version, add code from:https://github.com/bazelbuild/rules_scala#getting-started
scala_repositories((
"2.12.8",
{
"scala_compiler": "f34e9119f45abd41e85b9e121ba19dd9288b3b4af7f7047e86dc70236708d170",
"scala_library": "321fb55685635c931eba4bc0d7668349da3f2c09aee2de93a70566066ff25c28",
"scala_reflect": "4d6405395c4599ce04cea08ba082339e3e42135de9aae2923c9f5367e957315a"
}
))
In this file, we will setup the Scala rules and everything else that is needed to compile the Scala project.
BUILD files setup
To write BUILD files we will use the following methods:
- load – which loads the Bazel Scala rules, and extensions
- scala_binary – generates a Scala executable
- scala_library – generates a .jar file from Scala source files.
- scala_test – generates a Scala executable that runs unit test suites written using the scalatest library.
load("@io_bazel_rules_scala//scala:scala.bzl", "scala_binary")
scala_binary(
name = "App",
deps = [
"//bazeltest"
],
main_class = "bazeltest.Main"
)
We have named it App, just one dependency to the bazeltest package.In deps, we list our dependencies, where our own modules or third party can be. Main_class is our entry point.In the bazeltest package BUILD file:
load("@io_bazel_rules_scala//scala:scala.bzl", "scala_library", "scala_test")
scala_library(
name = "bazeltest",
srcs = ["src/main/scala/bazeltest/Main.scala"],
deps = [
"//othermodule",
"//3rdparty/jvm/joda_time:joda_time"
],
visibility = ["//:__pkg__"]
)
scala_test(
name = "test-main",
srcs = ["src/test/scala/bazeltest/MainSpec.scala"],
deps = [":bazeltest"]
)
Our Main.scala file will use some external third party dependency such as joda date time, and Worker from the subpack package.In srcs we set our Main.scala file, but it could be a list of files, listed one by one or a matching path pattern for example: glob(["src/main/scala/bazeltest/*.scala"])
( then we use glob ), could even be a package with all the subpackages, such as:glob(["src/main/scala/bazeltest/**/*..scala"])
and in deps all the necessary dependencies, so for this example our own sub pack package plus third part joda date time. For now, it points to the 3rdparty folder which does not exist yet, this will be done at one of the next steps so don’t worry.Visibility is used to define which other targets can use this target as a dependency, in this example, we define a project folder containing the main BUILD file.Now the BUILD file for othermodule:
load("@io_bazel_rules_scala//scala:scala.bzl", "scala_library", "scala_test")
scala_library(
name = "othermodule",
srcs = glob(["src/main/scala/othermodule/*.scala"]),
deps = [],
visibility = ["//bazeltest:__pkg__"]
)
scala_test(
name = "test-othermodule",
srcs = ["src/test/scala/othermodule/WorkerSpec.scala"],
deps = [":othermodule"]
)
Here we have set up a visibility param to the bazeltest package. So only this package can read from this one. If other packages try to reach this, we will see an error. Dependencies
We will use a third-party tool for this: https://github.com/johnynek/bazel-depsOpen the dependencies.yaml file and put this there:
options:
buildHeader: [
"load(\"@io_bazel_rules_scala//scala:scala_import.bzl\", \"scala_import\")",
"load(\"@io_bazel_rules_scala//scala:scala.bzl\", \"scala_library\", \"scala_binary\", \"scala_test\")"
]
languages: [ "java", "scala:2.12.8" ]
resolverType: "coursier"
resolvers:
- id: "mavencentral"
type: "default"
url: https://repo.maven.apache.org/maven2/
- id: "hmrc"
type: "default"
url: https://hmrc.bintray.com/releases
strictVisibility: true
transitivity: runtime_deps
versionConflictPolicy: highest
dependencies:
joda-time:
joda-time:
lang: java
version: "2.10.4"
com.typesafe.scala-logging:
scala-logging:
lang: scala
version: "3.9.0"
com.typesafe.akka:
akka-http:
lang: scala
version: "10.1.7"
org.scalatest:
scalatest:
lang: scala
version: "3.0.8"
replacements:
org.scala-lang:
scala-library:
lang: scala/unmangled
target: "@io_bazel_rules_scala_scala_library//:io_bazel_rules_scala_scala_library"
scala-reflect:
lang: scala/unmangled
target: "@io_bazel_rules_scala_scala_reflect//:io_bazel_rules_scala_scala_reflect"
scala-compiler:
lang: scala/unmangled
target: "@io_bazel_rules_scala_scala_compiler//:io_bazel_rules_scala_scala_compiler"
org.scala-lang.modules:
scala-parser-combinators:
lang: scala
target:
"@io_bazel_rules_scala_scala_parser_combinators//:io_bazel_rules_scala_scala_parser_combinators"
scala-xml:
lang: scala
target:
“@io_bazel_rules_scala_scala_xml//:io_bazel_rules_scala_scala_xml"
(Language is always required and may be one of java, Scala, Scala/unmangled.This is important, if you define an invalid language then errors will occur.Replacements are used for internal targets instead of Maven ones.) Save the system variable of this project path, for example (working on a Mac):
export MY_PROJ_DIR=`pwd`
We will need this in a minute.
Clone https://github.com/johnynek/bazel-deps and enter the bazel-deps folder.Ensure that this tool uses the same rules_scala commit sha.
Open the WORKSPACE file inside the bazel-deps and look for this:
git_repository(
name = "io_bazel_rules_scala",
remote = "https://github.com/bazelbuild/rules_scala",
commit = "0f89c210ade8f4320017daf718a61de3c1ac4773" # HEAD as of 2019-10-17, update this as needed
)
Commit is of course what we need to change ( if it is different than in our WORKSPACE file in rules_scala_version ).bazel run //:parse generate -- --repo-root "$MY_PROJ_DIR" --sha-file 3rdparty/workspace.bzl
--deps dependencies.yaml
This will download dependencies into a 3rdparty folder into your project directory.
INFO: Analyzed target //:parse (0 packages loaded, 0 targets configured).
INFO: Found 1 target...
Target //src/scala/com/github/johnynek/bazel_deps:parseproject up-to-date:
bazel-bin/src/scala/com/github/johnynek/bazel_deps/parseproject
bazel-bin/src/scala/com/github/johnynek/bazel_deps/parseproject.jar
INFO: Elapsed time: 0.168s, Critical Path: 0.01s
INFO: 0 processes.
INFO: Build completed successfully, 1 total action
INFO: Build completed successfully, 1 total action
wrote 26 targets in 8 BUILD files
The first run
Before doing the first run, let’s implement our Main and Worker classes.package bazeltest
import othermodule.Worker
import org.joda.time.DateTime
object Main extends App {
println("IN MAIN now: "+DateTime.now().plusYears(11))
val worker = new Worker
worker.doSomething()
def status(): String = "OKi"
}
package othermodule
class Worker {
def doSomething() : Int = {
println("Doing something")
12345
}
def pureFunc(): String = "ABC"
}
bazel run //:App
INFO: Analyzed target //:App (1 packages loaded, 2 targets configured).
INFO: Found 1 target...
INFO: From Linking external/com_google_protobuf/libprotobuf_lite.a [for host]:
/Library/Developer/CommandLineTools/usr/bin/libtool: file: bazel-out/host/bin/external/com_google_protobuf/_objs/protobuf_lite/io_win32.o has no symbols
INFO: From Linking external/com_google_protobuf/libprotobuf.a [for host]:
/Library/Developer/CommandLineTools/usr/bin/libtool: file: bazel-out/host/bin/external/com_google_protobuf/_objs/protobuf/error_listener.o has no symbols
INFO: From Building external/com_google_protobuf/libprotobuf_java.jar (122 source files, 1 source jar):
warning: -parameters is not supported for target value 1.7. Use 1.8 or later.
Target //:App up-to-date:
bazel-bin/App
bazel-bin/App.jar
INFO: Elapsed time: 52.246s, Critical Path: 23.22s
INFO: 194 processes: 189 darwin-sandbox, 5 worker.
INFO: Build completed successfully, 198 total actions
INFO: Build completed successfully, 198 total actions
IN MAIN now: 2030-10-11T11:26:07.533+01:00
Doing something
The first run takes some time because it has to download the dependencies, so don’t worry.Unit tests
Now let’s write some simple unit tests:package bazeltest
import org.scalatest._
class MainSpec extends FlatSpec with Matchers {
"status" should "return OK" in {
Main.status() shouldBe "OKi"
}
}
package othermodule
import org.scalatest._
class WorkerSpec extends FlatSpec with Matchers {
val worker = new Worker()
"do something" should "return 12345" in {
worker.doSomething() shouldBe 12345
}
"pureFunc" should "return ABC" in {
worker.pureFunc() shouldBe "ABC"
And run them:
bazel test //bazeltest:test-main
INFO: Analyzed target //bazeltest:test-main (0 packages loaded, 0 targets configured).
INFO: Found 1 test target...
Target //bazeltest:test-main up-to-date:
bazel-bin/bazeltest/test-main
bazel-bin/bazeltest/test-main.jar
INFO: Elapsed time: 1.047s, Critical Path: 0.89s
INFO: 3 processes: 2 darwin-sandbox, 1 worker.
INFO: Build completed successfully, 4 total actions
//bazeltest:test-main PASSED in 0.5s
Executed 1 out of 1 test: 1 test passes.
INFO: Build completed successfully, 4 total actions
bazel test //othermodule:test-othermodule
INFO: Analyzed target //othermodule:test-othermodule (0 packages loaded, 0 targets configured).
INFO: Found 1 test target...
Target //othermodule:test-othermodule up-to-date:
bazel-bin/othermodule/test-othermodule
bazel-bin/othermodule/test-othermodule.jar
INFO: Elapsed time: 1.438s, Critical Path: 1.29s
INFO: 2 processes: 1 darwin-sandbox, 1 worker.
INFO: Build completed successfully, 3 total actions
//othermodule:test-othermodule PASSED in 0.6s
Executed 1 out of 1 test: 1 test passes.
INFO: Build completed successfully, 3 total actions
Try now to change the status method from Main, to return “OK” instead of “OKi”.Run the tests again:
bazel test //bazeltest:test-main
INFO: Analyzed target //bazeltest:test-main (0 packages loaded, 0 targets configured).
INFO: Found 1 test target...
FAIL: //bazeltest:test-main (see /private/var/tmp/_bazel_maciejbak/16727409c9f0575889b09993f53ce424/execroot/__main__/bazel-out/darwin-fastbuild/testlogs/bazeltest/test-main/test.log)
Target //bazeltest:test-main up-to-date:
bazel-bin/bazeltest/test-main
bazel-bin/bazeltest/test-main.jar
INFO: Elapsed time: 1.114s, Critical Path: 0.96s
INFO: 3 processes: 2 darwin-sandbox, 1 worker.
INFO: Build completed, 1 test FAILED, 4 total actions
//bazeltest:test-main FAILED in 0.6s
/private/var/tmp/_bazel_maciejbak/16727409c9f0575889b09993f53ce424/execroot/__main__/bazel-out/darwin-fastbuild/testlogs/bazeltest/test-main/test.log
INFO: Build completed, 1 test FAILED, 4 total actions
bazel test //othermodule:test-othermodule
INFO: Analyzed target //othermodule:test-othermodule (0 packages loaded, 0 targets configured).
INFO: Found 1 test target...
Target //othermodule:test-othermodule up-to-date:
bazel-bin/othermodule/test-othermodule
bazel-bin/othermodule/test-othermodule.jar
INFO: Elapsed time: 0.150s, Critical Path: 0.00s
INFO: 0 processes.
INFO: Build completed successfully, 1 total action
//othermodule:test-othermodule (cached) PASSED in 0.6s
Executed 0 out of 1 test: 1 test passes.
INFO: Build completed successfully, 1 total action
Bazel build sees what has been changed, and runs tests only for the affected classes.So test results for othermodule are taken from the cache, and only the main tests run.The test failed because we didn’t change the results in the Spec file, so the change expected the result in the test to the Main.status() shouldBe “OK”.Run tests again:bazel test //bazeltest:test-main
INFO: Analyzed target //bazeltest:test-main (0 packages loaded, 0 targets configured).
INFO: Found 1 test target...
Target //bazeltest:test-main up-to-date:
bazel-bin/bazeltest/test-main
bazel-bin/bazeltest/test-main.jar
INFO: Elapsed time: 1.389s, Critical Path: 1.22s
INFO: 2 processes: 1 darwin-sandbox, 1 worker.
INFO: Build completed successfully, 3 total actions
//bazeltest:test-main PASSED in 0.6s
Executed 1 out of 1 test: 1 test passes.
INFO: Build completed successfully, 3 total actions
Dependency graph
We can easily visualize our dependency graph:In the command line run:bazel query --noimplicit_deps "deps(//:App)" --output graph
digraph mygraph {
node [shape=box];
"//:App"
"//:App" -> "//bazeltest:bazeltest"
"//bazeltest:bazeltest"
"//bazeltest:bazeltest" -> "//bazeltest:src/main/scala/bazeltest/Main.scala"
"//bazeltest:bazeltest" -> "//3rdparty/jvm/joda_time:joda_time"
"//bazeltest:bazeltest" -> "//othermodule:othermodule"
"//othermodule:othermodule"
"//othermodule:othermodule" -> "//othermodule:src/main/scala/othermodule/Worker.scala"
"//othermodule:src/main/scala/othermodule/Worker.scala"
"//3rdparty/jvm/joda_time:joda_time"
"//3rdparty/jvm/joda_time:joda_time" -> "//external:jar/joda_time/joda_time"
"//external:jar/joda_time/joda_time"
"//external:jar/joda_time/joda_time" -> "@joda_time_joda_time//jar:jar"
"//bazeltest:src/main/scala/bazeltest/Main.scala"
"@joda_time_joda_time//jar:jar"
"@joda_time_joda_time//jar:jar" -> "@joda_time_joda_time//jar:joda_time_joda_time.jar\n@joda_time_joda_time//jar:joda_time_joda_time-sources.jar"
"@joda_time_joda_time//jar:joda_time_joda_time.jar\n@joda_time_joda_time//jar:joda_time_joda_time-sources.jar"
}
Loading: 12 packages loaded
Paste results to
webgraphviz.com
Generate jar
bazel build //:App
INFO: Analyzed target //:App (0 packages loaded, 0 targets configured).
INFO: Found 1 target...
Target //:App up-to-date:
bazel-bin/App
bazel-bin/App.jar
INFO: Elapsed time: 0.085s, Critical Path: 0.00s
INFO: 0 processes.
INFO: Build completed successfully, 1 total action
Bazel build: Summary
In this post, we showed what is bazel, when to use it, and how to make basic configuration. It can take some time to properly set up complex projects using bazel build, but I guarantee you, in the end, it will speed up the whole team’s work.
Useful links
- Official Bazel build documentation https://docs.bazel.build/versions/1.0.0/bazel-overview.html
- Building Scala with Bazel build- Natan Silnitsky https://www.youtube.com/watch?v=K2Ytk0S4PF0
- Building Java Applications with Bazel https://www.baeldung.com/bazel-build-tool
See also