Kafka consulting

When old-school developer goes freestyle

Kafka consulting

Full source code is available on GitHub.

Introduction

This post is about creating relatively simple project with Freestyle, my adventures during this process and some afterthoughts.

Self-description from the project website: Freestyle is a library that enables the building of large-scale modular Scala applications and libraries on top of Free monads/applicatives. 

freestyle logo

I’m limited to single blog post, so I’ll make non-trivial-scale application instead of large-scale one. Precisely it is a module to perform registration and log in with credentials or GitHub’s OAuth 2.0 and http4s server that uses this module in HttpServices (endpoints) for issuing JWT.

This article describes my journey and feelings during it, so it is written in the first person and past tense.

Author context

I guess most blog posts are written by someone either excited or angry about something. This post is not such a case. I got resistant to hypes and I won’t write a post that is a gospel of [put any buzzword here] saving our codebases. On the other hand my previous unsuccessful experiences with free algebrashaven’t discouraged me.

Free Monads is a topic which I’ve tried to get into by writing non-trivial applications at least few times but all attempts ended when I thought: All this boilerplate and additional types in codebase are not worth the benefits. Accidental complexity grows too big.”.

I’ve judged that I prefer to stick to “traditional” style, which perhaps is not so easily testable, and does not allow to run a program against many interpreters (I’d love to hear from someone who has more than one for tests and one for production).

Here, please don’t get me wrong, when I began I was thinking that Interpreter pattern is a good thing, although too expensive, a trade off that doesn’t pay off. Have Freestyle changed my opinion about it? You can read the whole story or jump to conclusion.

How it unfolded

Before I dived deeper in solving somehow real problem that I face in my daily programmers life, the registration and log in, I had started with something really simple, namely arithmetic, just to get used to Freestyle, and ran over against first line of hurdles.

Arithmetic

The target was to write free programs for calculating mathematical formulas. Following quick start I’ve created such algebras:

https://gist.github.com/anonymous/1b4438bca4a2b4bf2b670991533087e4

Freestyle transforms BasicMath and HighMath traits to two separate free algebras. I’ve combined these algebras together in AllTheMath with help of Freestyles @module to obtain new free algebra with operations of both components.

Formulas

The next step was to write a program that uses given algebras or to write interpreters. I chose to write a simple program to evaluate math formulae first, here it is:

https://gist.github.com/anonymous/1a20ef1588cf8f7a4d67425110584d42

Macros

At glance: Freestyle uses macros to generate boilerplate code for you, it expands macros indicated by @free and @module annotations.

There were some negative consequences of using macros I faced. The first one was that my Intellij Idea indicated errors: cannot find symbol FS in @freealgebras definition.

In fact @free trait HighMath is expanded to trait HighMath[FF$15532[_]] extends _root_.freestyle.internal.EffectLike[FF$15532]and EffectLike contains type alias FS, thus sbt build was OK. I just had to live with “red margin” in the IDE.

The second problem seemed more severe, because it caused sbt build failure!

https://gist.github.com/anonymous/2cd5774dd35e4aa8a91e088b5731a94f

Fortunately, that were missing imports of freestyle._ and freestyle.implicits._. Imports had been repeatedly removed by my beloved Idea, until I disabled organize imports in the IDE.

The last problem I encountered can be painful for ones who like to use sbtREPL. It turns out that console task failed because of org.scalameta % paradise % 3.0.0-M9, which is required by current version of Freestyle. Unfortunately this made working with examples and learning a bit harder.

https://gist.github.com/anonymous/4ae449fc17b5e1704504a2fb6a7cd843

A very, very positive thing I’ve learned when facing problems with Freestyle was helpfulness of developers on the 47deg/freestyle Gitter channel. They are so open and kind! Thank you guys again!

Id interpreter

After fixing compile-time errors I wanted to get the ultimate reward of a programmer: feeling when the whole code finally works. So I’ve rushed with interpreter implementation. I haven’t been fancy about it, selection between monix.eval.Coeval and Id has been won by the simpler one.

https://gist.github.com/anonymous/5ff8cf1f4b799b27d95bced39cbb218d

Interpreting program

The finale of my example in the arithmetic domain is running the computation:

https://gist.github.com/anonymous/0532bb6cbbb474576747d1f0810d30c7

In the last line, I run the program Formulas.`(a+b)^2` in the AllTheMathalgebra, interpreted to Id monad. Imports, reported as unused by Idea, by the way, bring in scope my Id interpreters for BasicMath and HighMath and Freestyle machinery (including one that wires my handlers into AllTheMath).

One thing worth noting at this point is that I still can write programs that use only BasicMath and I don’t need anything related to HighMath to run them. Only lines of code that bound them together were in @module AllTheMath.

Type annotations in wrong places

I’ll come back to my formulae definition:def `(a+b)^2`[F[_]](a: Int, b: Int)(implicit A: AllTheMath[F]): FreeS[F, Int]Opinions may be different but I don’t really like, and find it redundant, to add type parameter and implicit parameter. The good news is that there are two ways of dealing with this.

The first one is placing methods inside module definition. I had to rename method because some non alphanumeric characters in names cause macro expansion failure.

https://gist.github.com/anonymous/e2a531cebf80c6229caa8fae3f1ca0a4

Ironically, it causes a dual problem: I cannot define a return type, because macro adds type parameter with a semi-random name to the trait it operates on.

The second one requires some boilerplate code, but it can pay off in the case of a big number of methods. The trick is to define a class to capture type, like here:

https://gist.github.com/anonymous/5e297a353cd5785802c2fcc8caa0afee

Irrelevant type annotations are gone, the relevant one is present.

Conclusion

That’s it for the simple case of arithmetic. I’ve run into some problems with not found symbols and had to find out how to place types annotations, but that just happens in the learning period. I’ve become resistant to IntelliJ Idea false positives and imports management. In the end amount of boilerplate was acceptable for me. After this prelude, I was ready to solve the original problem.

Log in

Log in the domain at glance:

  • allows to sign up – register an account
  • allows to log in – obtain JWT
  • sign up and log in are possible with email and password or GitHub

This tiny slice of real log in domain was already demanding enough for my goals. It turned out that I needed LoginDatabase and GitHubClient at least. I wanted to have logging effects under control to be able to test if the most important events were not left without a trace, that brought Log algebra in. The fourth and the last component of the free algebras set was JwtService.

You don’t really need to contemplate my algebras deeply, I write them down mostly to show the volume of what is required.

https://gist.github.com/anonymous/24dec1251821ff089ad4a6c7e542a3a5

Further, I’ve joined (somehow following Freestyle getting started guide) GitHubClient with Log to obtain GitHub and put LoginDatabasetogether with Log to get Persistence.

https://gist.github.com/anonymous/c2e3f2dbec2cb327717dfa10f41af7e8

Final programs required Deps, a module with all algebras.

https://gist.github.com/anonymous/c43a99fa1969abc178db751b4d483e35

At this point I could see the huge benefit of Freestyle: its macros derived a type for each @free trait method and required types coproducts for each @free and @module. Thus, it really did reduce the amount of boilerplate to great extent!

You can see that I’ve defined a method inside module GitHub leaving it without the return type annotation. For more advanced LoginPersistenceprograms I’ve really wanted to see return types. I’ve defined PersistencePrograms boilerplate class, but I find it a fair trade off.

https://gist.github.com/anonymous/9b02c83f78da88572a0d58492a38884a

LoginPersistence.scala

Log in programs

Let me share the implementation of two out of four programs: log in with a password and register with a password. I’ve used a helper (boilerplate) class to capture types again:

https://gist.github.com/anonymous/f2b03a57e8585f3187a270a25cbfd410

Programs.scala

For registering user I’ve chosen to use inner methods:

https://gist.github.com/anonymous/d0b0ff714707046da550812602b3d307

I nested for-comprehensions in login:

https://gist.github.com/anonymous/bed7ea66ca2521944f5c35ec883ecc78

I’ve found out that when using Freestyle sometimes I need to put a lot of type annotations, you can observe this in the case of both methods. To be honest I don’t know why, because in PersistencePrograms it works without them.

Besides that (and that I use a monad for logging in Scala the first time) these programs don’t look very different to ones I would write if I have committed to some particular monad or used tagless final style. In some time I’ve gained the skill to quite quickly deal with errors caused by too few type annotations.

Handler

Writing handlers is nothing more than implementing the interface in a particular monad. Because I’ve used http4s in this project I’ve chosen to interpret in fs2.Task. There no are significant differences to imperative style. Nice thing was, that I could very easily obtain handlers for fs2.Task when I had cats.Id handlers. That holds true every time when there is a natural transformation from one monad to another.

https://gist.github.com/anonymous/bfbd497fbff130cfa267a29e8f94e9f5
Links to implementations of interpreters for interested readers:

Wiring up

To run programs for compound-free algebra in target monad, an interpreter for it is required. User has to provide interpreters for each component algebra in implicit scope only because one of the super-powers of Freestyle is that it constructs compound interpreter from implicitly available interpreters for you. I’m really glad that I didn’t have to make it manually.

https://gist.github.com/anonymous/6aea17e3551198bef6b9cd522b2b14e3

Main.scala

Such code reminds me of my preferred style of (static) dependency injection in Play Framework.

The next example presents using implicitly available interpreters, creating my Programs instance and interpreting one of the programs to fs2.Task.

https://gist.github.com/anonymous/6f947e2db5d078749231ea9387be386b

LogInService.scala

The code above concludes the effort of using Free Monads with Freestyle. In the end, there is some boilerplate in production – factitious classes (Programs and PersistencePrograms) for sake of having return type annotations in public methods and extra type annotations in functions bodies.

Personally, I didn’t find that I needed a huge amount of extra code to have free algebras based application. Of course, I didn’t reach this satisfying outcome easily, but I was still in the early learning phase.

Testing

Free algebras way of writing programs yields two kinds of objects to test: interpreters and programs. I haven’t implemented tests for interpreters because I’ve assumed that it won’t differ from testing regular service implementations.

When it came to testing programs, I had a real problem in finding what is endorsed way of doing it. Ok, I appreciate you may now think: “this guy is really not-so-bright”, but I’ll dare to be honest: this lack of guidance, and problems with finding it, made me lose some time in unproductive doubts.

Finally, I’ve used advice from Gitter channel and have used mocked interpreters. This way I could easily inject failures when testing error paths. I’ve written these mock interpreters manually because Freestyle handlers methods are protected[this], thus one can’t mock them with scalamock.

Not finished specs for register/login programs can be found in ProgramsSpecs.scala.

There is another thing that worries me about testing when using Free Algebras: a problem with lack of hierarchy and scalability.

For example when testing Programs I cannot mock PersistencePrograms(class I’ve created to have return type annotations, not for dependency injection). It contains pure functions, so there is no reason to mock them, you can tell, but then I have to mock all their dependencies! I have to fit all the problem in my head at once, what turns out to be hard sometimes.

On the other hand, again, it’s a trade-off with a benefit: I couldn’t make a false assumption about how PersistencePrograms works and “mis-mock” it.
Short recap:

  • testing interpreters is not different
  • testing programs require to mock all (leaf) interpreters
  • one won’t make mistake in the middle levels
  • using scalamock is hard because Freestyle protects handlers methods

Other problems

Doobie trap!

Freestyle provides DoobieM, with def transact(c: ConnectionIO)method, for support of Doobie database access. Helper for transforming Transactor to DoobieM.Handler is in place.

Wiring Doobie in is easy. The trap is in testability. I’ve found ConnectionIO really untestable, I wasn’t able to match on queries or updates in my test handler and I don’t believe that H2 is a solution for tests when in production another database is used. I’ve decided to throw DoobieM code away and implement my own, higher-level, algebra for interacting with DB. If I had written tests for its interpreter, then I would provision some DB and place these tests in the integration suite.

IDE

I don’t want to speculate about reasons but that is just a fact: I have been restarting Idea, few times per day at least, when playing with Freestyle. I turned type-aware highlighting off but it didn’t help a lot. IDE still was very sluggish and I was really irritated sometimes. Whole system was affected. This really was a huge blow for my productivity, the biggest impediment I’d say.

Conclusion

I started the evaluation of Freestyle with mixed feelings about Free Algebrasprogramming, perhaps more skeptical than enthusiastic. Some people even doubt that it is feasible with current programming languages but macros-based approach that Freestyle brings to the table looks promising. There are some areas to improve, two most important for me I’ve pointed already: problems with types annotations and suffering IDE.

I’m happy with the code that I’ve written but the process was quite painful a few times. Although Nice Guys at Gitter eased the pain! I think the ultimate question is: would I use Freestyle in my next commercial project (let’s assume I make it alone)? Today the honest answer is: no, I would not. But I truly hope that it will change soon.

Cost-benefit-analysis significantly changed widely for me, Freestyle reduced boilerplate and accidental code tremendously! For sure I’ll keep an eye on Freestyle.

Links

  1. Freestyle
  2. Doobie
  3. http4s
  4. Project repo

Post Scriptum

Please take a look at the type signature for Deps free algebra after stripping all packages names:

https://gist.github.com/anonymous/b6af1bff112a83158018a6e67eae5064

Do you like this post? Want to stay updated? Follow us on Twitter and let us know in the comments!

Read more

Download e-book:

Scalac Case Study Book

Download now

Authors

Lech Głowiak

I'm JavaScript and frontend developer, after some time as Full-Stack I decided to focus more on the front part of projects because it's what I enjoy most.

Latest Blogposts

17.04.2024 / By  Michał Szajkowski

Mocking Libraries can be your doom

Test Automations

Test automation is great. Nowadays, it’s become a crucial part of basically any software development process. And at the unit test level it is often a necessity to mimic a foreign service or other dependencies you want to isolate from. So in such a case, using a mock library should be an obvious choice that […]

04.04.2024 / By  Aleksander Rainko

Scala 3 Data Transformation Library: ducktape 0.2.0.

Scala 3 Data Transformation Library: Ducktape 2.0

Introduction: Is ducktape still all duct tape under the hood? Or, why are macros so cool that I’m basically rewriting it for the third time? Before I go off talking about the insides of the library, let’s first touch base on what ducktape actually is, its Github page describes it as this: Automatic and customizable […]

28.03.2024 / By  Matylda Kamińska

Scalendar April 2024

scala conferences april 2024

Event-driven Newsletter Another month full of packed events, not only around Scala conferences in April 2024 but also Frontend Development, and Software Architecture—all set to give you a treasure trove of learning and networking opportunities. There’re online and real-world events that you can join in order to meet colleagues and experts from all over the […]

software product development

Need a successful project?

Estimate project