Shapeless Futures
Futures today
Today, when reactive programming is so popular concept, and you can see ‘We’re reactive!’ claim on so many pages that have anything to do with Scala (and not only), Futures are something you have to deal with all the time.
Things like getting records from the database, be it good old SQL, mongo, or elasticsearch, dispatching HTTP requests, communication between actors – are all handled in an asynchronous way, and you’ll have to deal with Futures whether you want it or not.
You probably will want to, because they are as a matter of fact extremely useful and convenient concept. Asynchronous programming is a lot simpler thanks to them. You can easily wait on future completion, but as it’s discouraged practice in most cases, you would rather chain it or do something else with it before it comes back to you with the result.
Quite often you have to deal with many futures at once. In such cases for comprehension comes handy, and because Scala’s Future is actually a monad, you can do things like that:
In other words, you can chain Futures and perform some computations on their assumed results without waiting for them to complete. That’s neat and very useful. However, there’s a catch in the example above and that is that subsequent Futures would be executed only after the ones preceding them are complete. In other words, you won’t have three parallel executions, just one after another.
This makes sense however and is what you’d expect and require in most cases. Just imagine a sequence of calls to the database where every query relies on results obtained earlier. But there are also situations when results of Futures are completely independent and there’s nothing that would justify running them one after another. That’d be just a waste of time. Scala Future has a nice and handy method, sequence. Let’s see how we could use it:
We could also sum resutls right away like so:
The problem
That’s great, but more curious observer will notice that all this will work nice if Futures forming the sequence are of the same type – in the example above it’s Future[Int]. There are many situations where you have Futures of different types, and you’d like to execute them in parallel. Not a problem if the number is two:
Things get a bit hairy however when the number is three:
Yet it’s probably no challenge for you to create a method for handling three like so:
Yes… But something tells me you don’t want to take it one step further, to four, right? On the other hand, I bet you were in situations when it would be nice to have it because otherwise, it would make your Scala code look more like Clojure, when it would come to zip 5 or 6 Futures. Well, luckily, there’s a way to do it with any arbitrary number.
A word on shapeless
Before we’ll come up with the solution, let me tell a few words about shapeless. As the main page of the project states, shapeless is “a type class and dependent type based generic programming library for Scala”. That sounds a bit scary.
Well, you need to bend your runtime mind a little in order to get end to end understanding of how does it work, but luckily we don’t have to take too deep dive here. Shapeless is about making an attempt to solve things during compilation where they would normally be tackled in runtime.
It’s rumored that Miles Sabin, the guy behind shapeless, doesn’t even exist at runtime. Shapeless aims to give you an extreme level of confidence that if something compiles, it will work as expected. Let’s see some simple examples now.
Suppose you want to have a constraint in some function that would restrict lists passed in as an argument to be the same. Normally you would do something like this:
With shapeless though you can use Sized :
if you then call the function with the following args:
it will work fine. However, this:
will not even compile. The reason is that type of Sized("abc", "def") is Sized[Seq[String], _2], but Sized("ghi", "jkl", "mno") has type Sized[Seq[String, _3]] which makes the compiler not accept this code.
HLists and spray-like Directives
The most known part of shapeless, probably the one most widely used, is HList, a heterogeneous list. This is basically a generalization of tuples and allows constructs like:
and then it allows operations that are not allowed on tuples, like
Where would HLists be useful? Whenever you’d like to use tuples, but don’t know their sizes or miss some functionality on them. They have been found useful in spray routing classes, which every Scala programmer is for sure familiar with. Let’s try to mimic them using shapeless and then look closer at some aspects. First, some basic building blocks with obvious purpose:
Now, lets define our own simplified Directive:
Two lines that I’d like the reader to concentrate on first are
and
as they make use of the very important and omnipresent concept of shapeless, that is providing implicit evidences for types and relationships between them. Here, with implicit fp: FnToProduct.Aux[F, X => Route] we are providing evidence that function f: F is convertible to HList-like equivalent (the equivalent’s signature would be close to def foo[X <: HList](x: X): Route).
With the other one, implicit prepend: Prepend[X, Y], we are proving that two HLists can be merged into third (or more precisely that there exists HList type consisting of elements in which X prepends Y). We make use of these evidences and perform conversions making it possible to use regular functions along with hlist-like happly. The fact that Directives are parametrized with HLists allows for great flexibility without sacrificing type safety. So let’s define some concrete directives:
Now, let’s see how we could use them to construct routes:
Does it work? Yes!
Just look how easily we can combine two Directives and make the combination work with (String, String) => Route function, even though we have never defined apply(f: (String, String) => Route) (or even apply(f: String => Route)).
Back to the Futures
Now let’s try not to mimic anything but make something new. Let’s get back to our initial problem with futures. The intention must be clear now – we’d like to have a way of turning HList of Futures into Future of HList. Something that would work like Future.sequence, but would preserve the type of every compound and have no limitation on size. Let’s start with defining base trait:
So – we’d like to be able for a given HList to figure out its corresponding IsHListOfFutures object and then have access to hsequence operation.
Now, let’s see how that would look like for the simplest case, HNil:
Well, that’s kinda obvious, isn’t it? But now let’s take another step and try to tackle more complicated HLists:
We are here defining IsHListOfFutures in terms of another IsHListOfFutures, so doing a type-level recursion. At some point, we’ll reach an implicit conversion from HNIl to HNilIsListOfFutures, and then, everything will be known about the type of what hsequence has to return. Now let’s put everything into IsHListOfFutures object which we’ll be using for providing the pieces of evidence:
and define our hsequence function:
Once we have IsHListOfFutures defined correctly and are able to provide it as evidence, the implementation, as it turns out, is easy.
Let’s now look at the specs I have prepared and see them in action:
So now, using this, you can combine even 22 Futures and perform one hsequence call on them. Nice, isn’t it? But some malcontents might ask if it really is necessary to operate on HLists, do we really have to construct these objects all the time we want to perform hsequence operation? After all, it doesn’t look like “built into Scala”. Let’s take it just one step further and see what can be done:
A few words on what we have just defined: we are providing two additional implicits to the zip function – one, gen, will be responsible to transforming function arguments into HList. The other, tupler, will be responsible for enabling doing the same, but in opposite direction and on results coming from hsequence. Let’s see this in action:
Now that’s something that looks ‘native’, isn’t it?
And that’s that. Using shapeless we have managed to create a nice function to deal with the parallel execution of an arbitrary number of Futures. I hope you found this post interesting and useful. Thanks for your attention!
PS. There are other people’s attempts to provide a sequence method. Check out and compare:
- https://stackoverflow.com/questions/23453570/using-shapeless-to-convert-tuple-of-future-to-future-of-tuple-by-way-of-hlist/23455152#23455152
- https://gist.github.com/timcharper/df19933d06601f3ac834
Do you like this post? Want to stay updated? Follow us on Twitter or subscribe to our Feed.
See also
Authors
