Learning Scala macros with Each Library

In this post, we will discuss Each Library. Each allows Scala users to write imperative syntax, which is later on translated into scalaz monadic expressions. In a way, it’s adding syntactic sugar on top of ordinary Scala. Simplicity and the way it extends ordinary code caught my attention, so I’ve decided to dig deeper.

This post will answer some basic questions about Each – how to use it, why it works and how to use the same principles in your code. Hopefully, we will demystify Scala macros along the way.

How to use Each?

Each library allows you to write operations on values enclosed in Monads in a plain, imperative syntax. In a sense it’s similar to await from Python on C#, but without being limited to Futures. It helps you to eliminate as much boilerplate code as possible and focus on the business logic.

Although it might sound complex in theory it’s straightforward in practice. I’m sure you will get the idea after looking at these examples.

https://gist.github.com/anonymous/deaf1dd3ed34e991b3cb18c58c4d8519

https://gist.github.com/anonymous/5d938f6412cbb5d8a46d6394e272438b

https://gist.github.com/anonymous/c11f73042aac91115e7f636cfdf9ba97

You got the idea – import each, wrap your computation in monadic[TheMonadYouAreUsing] { ... } and access raw values with .each inside the code block. Neat.

You can find more examples and use cases in the project’s README. We will focus now on inspecting Each and figuring out how to use similar techniques in our code.

Why it works?

The short answer is “Scala magic”, but if you are a curious developer you probably won’t be satisfied with this answer.

The (slightly) longer answer won’t come as a surprise to a seasoned Scala Developer – combining implicits, pimp my object pattern and Scala macros.

The in-depth answer goes like this … The import that we used before import com.thoughtworks.each.Monadic._ brings us few things in scope. One of them is an implicit conversion

https://gist.github.com/anonymous/013a891b360026eef527d8d705ced435

which is able to transform any Monad (F[A], where F is monad and A is the underlying type) into a ‘Pimp’ class EachOps. The class looks simple enough

https://gist.github.com/anonymous/6f885b82bf048f3b8b25ac77d0c4cdc2

We see our familiar each method here, but you might be surprised that it’s undefined at this point. The secret here is that each isn’t part of the final code. It makes sure the types do match and serves as a marker. Later on the block passed into monadic will be rewritten and each will be substituted with real calls. We will see that in a second.

The real thing happens inside the monadic[X] call. It takes the type information and body you passed in and expands a macro

https://gist.github.com/anonymous/6bb6dc3a9ae59e7135cd5a2ff08f9395

Macro is the place where our example becomes much more convoluted. Each covers many cases that we didn’t mention here, so we will drop some details to keep it understandable. For the sake of simplicity, we will use the Future addition example from the first paragraph. The high level objective of this macro is to extract information about the code we need to transform and then build a new tree with ordinary scalaz inside.

We start of with our apply method in MacroImplementation.

https://gist.github.com/anonymous/927430ea5654213d0ce21225d5b43fea

Here c is the whitebox context provided by scala, monad is the monad we are using in the current block (Future in our example) and body is what we would like to transform. It will roughly correspond to our code

https://gist.github.com/anonymous/42305b59bd58c806b94584a9b771d9ba

Now, that we have all the raw data in place, we can extract type information using scala.reflect.api and build a custom MonadicTransformer class to transform the plain code into it’s final form.

Scala represents abstract syntax trees as convenient case classes, so we can use plain pattern matching to traverse and transform the code recursively. We will not analyze all the cases here, but we will get back to it in the next section.

The point of all those transformations it to simplify each expressions into plain scalaz calls like bind invocation

https://gist.github.com/anonymous/6fbcf876e72a435a630165787e02daa9

Above you can see reflection API in action. To explain the code in a few words Apply indicates that a function should be called, Select and TermName help us to choose what code to call exactly, Function allows us to create a new function, List … that’s an ordinary list to provide arguments. I encourage you to explore the documentation to learn more.

At the end of the expansion process we should be left with a code like this

https://gist.github.com/anonymous/e6abb2f05b47243ce82d803c7de2e043

It might look hard with all the $ and brackets, but you can see the pattern here – nested calls leading to a simple Int sum at the end. The tree that we have built can now be used as a subtraction for our original block. No each calls are needed.

That’s it. In the next section we will try to use the same pattern on our own.

Using on our own

We took a sneak peak under the hood to get some understanding what is going on, now let’s take a look how much we really did understand. We will try to implement a very simple macro on our own. We will call it with withLogging. This macro will accept a generic function and print the return type before proceeding, so you can debug your code more easily.

Note that in order for macros to work they need to be in a separate unit of compilation – they cannot be created in the same project/module where you use them. I decided to stick to the previous example and drop the code straight into the codebase of Each, but you might want to put it into a sbt subproject.

Let’s get started with some dummy code we want to log:

https://gist.github.com/anonymous/68b72936b4b86e221c0f0be8348d814a

As you can see we have our withLogging macro-based function and we pass an anonymous () => Int function into it. withLogging takes a function and returns a new one. This new function does a simple trick: calls the argument, saves the result, prints it and returns it back. To help us with debugging we will also print the line number. Fairly straightforward stuff.

The code is as simple as the description:

https://gist.github.com/anonymous/048fc539206bb877afe104d983de777c

We start of with defining the external interface. It’s worth pointing out that Scala macros are typesafe. You cannot (or shouldn’t) cheat the type system. If the generated code doesn’t type-check, compilation will fail. That’s a good thing.

https://gist.github.com/anonymous/8f011a8d2a221ede3687fdb5fb0f17e3

Let’s walk through this code quickly.

It’s based around the same principles as original Each. We have a method that take the Context of our macro transformations and a Tree of the code that we want to transform. The following code is building and expression and an AST. To do that we use helper classes provided to us by Scala. They might look intimidating at the beginning, but if you get a better understanding of what they do it becomes simple.

What’s cool about them is that they are ordinary case classes and all the usual tricks apply – easy instance creation (as above), destructing in pattern matching and so on.

• Function: creates a function, needs a list of arguments and a block with the body.
• Block: represents code between brackets {...}.
• ValDef: defines a value, think val foo = .... It requires a name and value. It can be given modifiers an explicit type.
• Apply: calls a given function (tree) with given arguments
• Ident and TermName: are used to lookup terms.
• Literal: a “hard-coded” value.

Quite neat, isn’t it? Not that complicated either. Unfortunately, a bit verbose.

With Scala 2.11.X we have a new more natural way of building ASTs – quasiquotes. They allow you to write code as a String with the q interpolator to add Trees into the mix. It’s easier to explain with an example:

https://gist.github.com/anonymous/f2d205dc46957ee9a8376dcc8957a471

That’s even better!

Summary

In this post we have learnt a bit about Each as a tool, how it’s built and how can we use the same tricks in our code. We have played a bit with macros and shown that there’s no magic in Scala. But, most importantly, we shown that digging into an unknown code might be a excellent way to learn.

Links

• https://github.com/ThoughtWorksInc/each
• https://docs.scala-lang.org/overviews/reflection/symbols-trees-types.html
• https://docs.scala-lang.org/overviews/quasiquotes/syntax-summary.html

Do you like this post? Want to stay updated? Follow us on Twitter or subscribe to our Feed.

See also

• def hello = macro world
• Domain specific types in a play framework project.
• Exploring Tagless Final pattern for extensive and readable Scala code
• Top 15 Scala Libraries for Data Science in 2021
• Extending Spark pipeline