Shapeless Futures
Futures today
Today, when reactive programming is so popular concept, and you can see ‘We’re reactive!’ claim on so many pages that have anything to do with Scala (and not only), Future
s are something you have to deal with all the time.
Things like getting records from the database, be it good old SQL, mongo, or elasticsearch, dispatching HTTP requests, communication between actors – are all handled in an asynchronous way, and you’ll have to deal with Future
s whether you want it or not.
You probably will want to, because they are as a matter of fact extremely useful and convenient concept. Asynchronous programming is a lot simpler thanks to them. You can easily wait on future completion, but as it’s discouraged practice in most cases, you would rather chain it or do something else with it before it comes back to you with the result.
Quite often you have to deal with many futures at once. In such cases for comprehension comes handy, and because Scala’s Future is actually a monad, you can do things like that:
https://gist.github.com/tomaszperek/c772cf36d9ee522d4099
In other words, you can chain Future
s and perform some computations on their assumed results without waiting for them to complete. That’s neat and very useful. However, there’s a catch in the example above and that is that subsequent Future
s would be executed only after the ones preceding them are complete. In other words, you won’t have three parallel executions, just one after another.
This makes sense however and is what you’d expect and require in most cases. Just imagine a sequence of calls to the database where every query relies on results obtained earlier. But there are also situations when results of Future
s are completely independent and there’s nothing that would justify running them one after another. That’d be just a waste of time. Scala Future
has a nice and handy method, sequence
. Let’s see how we could use it:
https://gist.github.com/tomaszperek/380bfad8852c52234e46
We could also sum resutls right away like so:
https://gist.github.com/tomaszperek/eb5696e5054c260f8da5
The problem
That’s great, but more curious observer will notice that all this will work nice if Future
s forming the sequence are of the same type – in the example above it’s Future[Int]
. There are many situations where you have Future
s of different types, and you’d like to execute them in parallel. Not a problem if the number is two:
https://gist.github.com/tomaszperek/07204dea85aa63cc758e
Things get a bit hairy however when the number is three:
https://gist.github.com/tomaszperek/ed675cb4e27c15b72669
Yet it’s probably no challenge for you to create a method for handling three like so:
https://gist.github.com/tomaszperek/a6863bba0b169a4bb08d
Yes… But something tells me you don’t want to take it one step further, to four, right? On the other hand, I bet you were in situations when it would be nice to have it because otherwise, it would make your Scala code look more like Clojure, when it would come to zip 5 or 6 Future
s. Well, luckily, there’s a way to do it with any arbitrary number.
A word on shapeless
Before we’ll come up with the solution, let me tell a few words about shapeless. As the main page of the project states, shapeless is “a type class and dependent type based generic programming library for Scala”. That sounds a bit scary.
Well, you need to bend your runtime mind a little in order to get end to end understanding of how does it work, but luckily we don’t have to take too deep dive here. Shapeless is about making an attempt to solve things during compilation where they would normally be tackled in runtime.
It’s rumored that Miles Sabin, the guy behind shapeless, doesn’t even exist at runtime. Shapeless aims to give you an extreme level of confidence that if something compiles, it will work as expected. Let’s see some simple examples now.
Suppose you want to have a constraint in some function that would restrict lists passed in as an argument to be the same. Normally you would do something like this:
https://gist.github.com/tomaszperek/fbd658a7f6a5f3d62b46
With shapeless though you can use Sized
:
https://gist.github.com/tomaszperek/57374c9a98b4b702e64a
if you then call the function with the following args:
https://gist.github.com/tomaszperek/717b5f74679bd383e0f2
it will work fine. However, this:
https://gist.github.com/tomaszperek/9fb552e5e5b801c3406a
will not even compile. The reason is that type of Sized("abc", "def")
is Sized[Seq[String], _2]
, but Sized("ghi", "jkl", "mno")
has type Sized[Seq[String, _3]]
which makes the compiler not accept this code.
HLists and spray-like Directives
The most known part of shapeless, probably the one most widely used, is HList
, a heterogeneous list. This is basically a generalization of tuples and allows constructs like:
https://gist.github.com/tomaszperek/c81b831aeeb4b05231a3
and then it allows operations that are not allowed on tuples, like
https://gist.github.com/tomaszperek/855f9f6ae98280792e79
Where would HList
s be useful? Whenever you’d like to use tuples, but don’t know their sizes or miss some functionality on them. They have been found useful in spray routing classes, which every Scala programmer is for sure familiar with. Let’s try to mimic them using shapeless and then look closer at some aspects. First, some basic building blocks with obvious purpose:
https://gist.github.com/tomaszperek/22a5239d7991fbb22635
Now, lets define our own simplified Directive
:
https://gist.github.com/tomaszperek/d5987363d1f36c5c591f
Two lines that I’d like the reader to concentrate on first are
https://gist.github.com/tomaszperek/22ce27d08c827b3b6d4d
and
https://gist.github.com/tomaszperek/771cd4bd53af4dd5cd9b
as they make use of the very important and omnipresent concept of shapeless, that is providing implicit evidences for types and relationships between them. Here, with implicit fp: FnToProduct.Aux[F, X => Route]
we are providing evidence that function f: F
is convertible to HList
-like equivalent (the equivalent’s signature would be close to def foo[X <: HList](x: X): Route
).
With the other one, implicit prepend: Prepend[X, Y]
, we are proving that two HList
s can be merged into third (or more precisely that there exists HList
type consisting of elements in which X
prepends Y
). We make use of these evidences and perform conversions making it possible to use regular functions along with hlist-like happly
. The fact that Directive
s are parametrized with HList
s allows for great flexibility without sacrificing type safety. So let’s define some concrete directives:
https://gist.github.com/tomaszperek/6e46db08a65be2038a87
Now, let’s see how we could use them to construct routes:
https://gist.github.com/tomaszperek/834e9fa1ead49ceb1d13
Does it work? Yes!
https://gist.github.com/tomaszperek/aa7b63eac3fc3c894207
Just look how easily we can combine two Directive
s and make the combination work with (String, String) => Route
function, even though we have never defined apply(f: (String, String) => Route)
(or even apply(f: String => Route)
).
Back to the Futures
Now let’s try not to mimic anything but make something new. Let’s get back to our initial problem with futures. The intention must be clear now – we’d like to have a way of turning HList
of Futures
into Future
of HList
. Something that would work like Future.sequence
, but would preserve the type of every compound and have no limitation on size. Let’s start with defining base trait:
https://gist.github.com/tomaszperek/0ea145d930cfe3700055
So – we’d like to be able for a given HList
to figure out its corresponding IsHListOfFutures
object and then have access to hsequence
operation.
Now, let’s see how that would look like for the simplest case, HNil:
https://gist.github.com/tomaszperek/52671f1e92aaa0bc7baf
Well, that’s kinda obvious, isn’t it? But now let’s take another step and try to tackle more complicated HList
s:
https://gist.github.com/tomaszperek/6132a958ccaf92dc2eae
We are here defining IsHListOfFutures
in terms of another IsHListOfFutures
, so doing a type-level recursion. At some point, we’ll reach an implicit conversion from HNIl
to HNilIsListOfFutures
, and then, everything will be known about the type of what hsequence
has to return. Now let’s put everything into IsHListOfFutures
object which we’ll be using for providing the pieces of evidence:
https://gist.github.com/tomaszperek/4aecc9839d766b43a051
and define our hsequence
function:
https://gist.github.com/tomaszperek/0dac78f8f93221207fba
Once we have IsHListOfFutures
defined correctly and are able to provide it as evidence, the implementation, as it turns out, is easy.
Let’s now look at the specs I have prepared and see them in action:
https://gist.github.com/tomaszperek/62242102ee9f4bdf2acf
So now, using this, you can combine even 22 Future
s and perform one hsequence
call on them. Nice, isn’t it? But some malcontents might ask if it really is necessary to operate on HList
s, do we really have to construct these objects all the time we want to perform hsequence
operation? After all, it doesn’t look like “built into Scala”. Let’s take it just one step further and see what can be done:
https://gist.github.com/tomaszperek/4499aa2f4ec1d7136e9d
A few words on what we have just defined: we are providing two additional implicits to the zip
function – one, gen
, will be responsible to transforming function arguments into HList
. The other, tupler
, will be responsible for enabling doing the same, but in opposite direction and on results coming from hsequence
. Let’s see this in action:
https://gist.github.com/tomaszperek/7580af3074ae7f763304
Now that’s something that looks ‘native’, isn’t it?
And that’s that. Using shapeless we have managed to create a nice function to deal with the parallel execution of an arbitrary number of Futures
. I hope you found this post interesting and useful. Thanks for your attention!
PS. There are other people’s attempts to provide a sequence method. Check out and compare:
- https://stackoverflow.com/questions/23453570/using-shapeless-to-convert-tuple-of-future-to-future-of-tuple-by-way-of-hlist/23455152#23455152
- https://gist.github.com/timcharper/df19933d06601f3ac834
Do you like this post? Want to stay updated? Follow us on Twitter or subscribe to our Feed.
See also