15.10.2015 / By Tomasz Perek

Shapeless Futures

Futures today

Today, when reactive programming is so popular concept, and you can see ‘We’re reactive!’ claim on so many pages that have anything to do with Scala (and not only), Futures are something you have to deal with all the time.

Things like getting records from the database, be it good old SQL, mongo, or elasticsearch, dispatching HTTP requests, communication between actors – are all handled in an asynchronous way, and you’ll have to deal with Futures whether you want it or not.

You probably will want to, because they are as a matter of fact extremely useful and convenient concept. Asynchronous programming is a lot simpler thanks to them. You can easily wait on future completion, but as it’s discouraged practice in most cases, you would rather chain it or do something else with it before it comes back to you with the result.

Quite often you have to deal with many futures at once. In such cases for comprehension comes handy, and because Scala’s Future is actually a monad, you can do things like that:

https://gist.github.com/tomaszperek/c772cf36d9ee522d4099

In other words, you can chain Futures and perform some computations on their assumed results without waiting for them to complete. That’s neat and very useful. However, there’s a catch in the example above and that is that subsequent Futures would be executed only after the ones preceding them are complete. In other words, you won’t have three parallel executions, just one after another.

This makes sense however and is what you’d expect and require in most cases. Just imagine a sequence of calls to the database where every query relies on results obtained earlier. But there are also situations when results of Futures are completely independent and there’s nothing that would justify running them one after another. That’d be just a waste of time. Scala Future has a nice and handy method, sequence. Let’s see how we could use it:

https://gist.github.com/tomaszperek/380bfad8852c52234e46

We could also sum resutls right away like so:

https://gist.github.com/tomaszperek/eb5696e5054c260f8da5

The problem

That’s great, but more curious observer will notice that all this will work nice if Futures forming the sequence are of the same type – in the example above it’s Future[Int]. There are many situations where you have Futures of different types, and you’d like to execute them in parallel. Not a problem if the number is two:

https://gist.github.com/tomaszperek/07204dea85aa63cc758e

Things get a bit hairy however when the number is three:

https://gist.github.com/tomaszperek/ed675cb4e27c15b72669

Yet it’s probably no challenge for you to create a method for handling three like so:

https://gist.github.com/tomaszperek/a6863bba0b169a4bb08d

Yes… But something tells me you don’t want to take it one step further, to four, right? On the other hand, I bet you were in situations when it would be nice to have it because otherwise, it would make your Scala code look more like Clojure, when it would come to zip 5 or 6 Futures. Well, luckily, there’s a way to do it with any arbitrary number.

A word on shapeless

Before we’ll come up with the solution, let me tell a few words about shapeless. As the main page of the project states, shapeless is “a type class and dependent type based generic programming library for Scala”. That sounds a bit scary.

Well, you need to bend your runtime mind a little in order to get end to end understanding of how does it work, but luckily we don’t have to take too deep dive here. Shapeless is about making an attempt to solve things during compilation where they would normally be tackled in runtime.

It’s rumored that Miles Sabin, the guy behind shapeless, doesn’t even exist at runtime. Shapeless aims to give you an extreme level of confidence that if something compiles, it will work as expected. Let’s see some simple examples now.

Suppose you want to have a constraint in some function that would restrict lists passed in as an argument to be the same. Normally you would do something like this:

https://gist.github.com/tomaszperek/fbd658a7f6a5f3d62b46

With shapeless though you can use Sized :

https://gist.github.com/tomaszperek/57374c9a98b4b702e64a

if you then call the function with the following args:

https://gist.github.com/tomaszperek/717b5f74679bd383e0f2

it will work fine. However, this:

https://gist.github.com/tomaszperek/9fb552e5e5b801c3406a

will not even compile. The reason is that type of Sized("abc", "def") is Sized[Seq[String], _2], but Sized("ghi", "jkl", "mno") has type Sized[Seq[String, _3]] which makes the compiler not accept this code.

HLists and spray-like Directives

The most known part of shapeless, probably the one most widely used, is HList, a heterogeneous list. This is basically a generalization of tuples and allows constructs like:

https://gist.github.com/tomaszperek/c81b831aeeb4b05231a3

and then it allows operations that are not allowed on tuples, like

https://gist.github.com/tomaszperek/855f9f6ae98280792e79

Where would HLists be useful? Whenever you’d like to use tuples, but don’t know their sizes or miss some functionality on them. They have been found useful in spray routing classes, which every Scala programmer is for sure familiar with. Let’s try to mimic them using shapeless and then look closer at some aspects. First, some basic building blocks with obvious purpose:

https://gist.github.com/tomaszperek/22a5239d7991fbb22635

Now, lets define our own simplified Directive:

https://gist.github.com/tomaszperek/d5987363d1f36c5c591f

Two lines that I’d like the reader to concentrate on first are

https://gist.github.com/tomaszperek/22ce27d08c827b3b6d4d

and

https://gist.github.com/tomaszperek/771cd4bd53af4dd5cd9b

as they make use of the very important and omnipresent concept of shapeless, that is providing implicit evidences for types and relationships between them. Here, with implicit fp: FnToProduct.Aux[F, X => Route] we are providing evidence that function f: F is convertible to HList-like equivalent (the equivalent’s signature would be close to def foo[X <: HList](x: X): Route).

With the other one, implicit prepend: Prepend[X, Y], we are proving that two HLists can be merged into third (or more precisely that there exists HList type consisting of elements in which X prepends Y). We make use of these evidences and perform conversions making it possible to use regular functions along with hlist-like happly. The fact that Directives are parametrized with HLists allows for great flexibility without sacrificing type safety. So let’s define some concrete directives:

https://gist.github.com/tomaszperek/6e46db08a65be2038a87

Now, let’s see how we could use them to construct routes:

https://gist.github.com/tomaszperek/834e9fa1ead49ceb1d13

Does it work? Yes!

https://gist.github.com/tomaszperek/aa7b63eac3fc3c894207

Just look how easily we can combine two Directives and make the combination work with (String, String) => Route function, even though we have never defined apply(f: (String, String) => Route) (or even apply(f: String => Route)).

Back to the Futures

Now let’s try not to mimic anything but make something new. Let’s get back to our initial problem with futures. The intention must be clear now – we’d like to have a way of turning HList of Futures into Future of HList. Something that would work like Future.sequence, but would preserve the type of every compound and have no limitation on size. Let’s start with defining base trait:

https://gist.github.com/tomaszperek/0ea145d930cfe3700055

So – we’d like to be able for a given HList to figure out its corresponding IsHListOfFutures object and then have access to hsequence operation.

Now, let’s see how that would look like for the simplest case, HNil:

https://gist.github.com/tomaszperek/52671f1e92aaa0bc7baf

Well, that’s kinda obvious, isn’t it? But now let’s take another step and try to tackle more complicated HLists:

https://gist.github.com/tomaszperek/6132a958ccaf92dc2eae

We are here defining IsHListOfFutures in terms of another IsHListOfFutures, so doing a type-level recursion. At some point, we’ll reach an implicit conversion from HNIl to HNilIsListOfFutures, and then, everything will be known about the type of what hsequence has to return. Now let’s put everything into IsHListOfFutures object which we’ll be using for providing the pieces of evidence:

https://gist.github.com/tomaszperek/4aecc9839d766b43a051

and define our hsequence function:

https://gist.github.com/tomaszperek/0dac78f8f93221207fba

Once we have IsHListOfFutures defined correctly and are able to provide it as evidence, the implementation, as it turns out, is easy.

Let’s now look at the specs I have prepared and see them in action:

https://gist.github.com/tomaszperek/62242102ee9f4bdf2acf

So now, using this, you can combine even 22 Futures and perform one hsequence call on them. Nice, isn’t it? But some malcontents might ask if it really is necessary to operate on HLists, do we really have to construct these objects all the time we want to perform hsequence operation? After all, it doesn’t look like “built into Scala”. Let’s take it just one step further and see what can be done:

https://gist.github.com/tomaszperek/4499aa2f4ec1d7136e9d

A few words on what we have just defined: we are providing two additional implicits to the zip function – one, gen, will be responsible to transforming function arguments into HList. The other, tupler, will be responsible for enabling doing the same, but in opposite direction and on results coming from hsequence. Let’s see this in action:

https://gist.github.com/tomaszperek/7580af3074ae7f763304

Now that’s something that looks ‘native’, isn’t it?

And that’s that. Using shapeless we have managed to create a nice function to deal with the parallel execution of an arbitrary number of Futures. I hope you found this post interesting and useful. Thanks for your attention!

PS. There are other people’s attempts to provide a sequence method. Check out and compare:

Do you like this post? Want to stay updated? Follow us on Twitter or subscribe to our Feed.