Tubi x Scala: Why it works?
If you want to succeed in a World of ever-changing technology, you always need to be a step ahead of your competitors. Constantly pushing the boundaries is crucial if your business is to provide flawless customer experience, and grow.
Today, we’re going to show you the perfect example of a company that wasn’t scared of scaling up machine experimentation with Scala.
And let us just take a sneak peek at that in one sentence:
“I love working with Scala” |
Alexandros Bantis, backend Scala Developer at Tubi
The outcome is rather impressive, so read on to find out more about it!
Tubi: what is it all about
Tubi is an American streaming service. Tubi’s programming includes both movies and television series from various American content providers.
In September 2019, Tubi announced that its users had streamed more than 132 million hours of content. It’s the largest free movie and TV streaming service in the US. It’s “powered by ad bidding”, which means that their users don’t pay to watch but need to see ads in return for streaming access.
Below you’ll find a screenshot from the Scylla Summit 2019, at which Tubi gave a presentation: the numbers aren’t too shabby. Having over a thousand ad partners, more than 200 content partners, over 15 thousand titles, and more than 20 million active monthly users (and bear in mind that Tubi only operates in the US for now!) speaks for itself.
ML algorithms identify the content that the users want to watch and provide them with some recommendations. The system takes less than a second to select, sort, and display content for each user out of thousands of titles on Tubi’s AVOD (ad-supported video on demand) service.
And users don’t have to spend ages looking for something to watch.
As you can probably guess, it’s a lot of data to work with and process on a daily basis. Getting through sufficient data to draw the right conclusions is nothing but a challenge.
It comes as no surprise that to leverage all of that data, Tubi had to adjust its technology and solutions to serve not only its purpose but also to provide the best customer experience possible.
And this is where Scala steps in, supporting the transformation within Tubi.
Tubi’s Scala journey
So, what’s the marriage between Tubi and Scala? Tubi is one of the biggest companies in the World that benefits from implementing Scala in their strategies.
Tubi’s backend Scala Developer at the time, Alexandros Bantis, explained the bumpy ride from a Node.js system to a fully distributed Scala engine.
There was that one event called Scylla Summit 2019 in San Francisco, and this is where Bantis provided some insights into the personalization service infrastructure that supports Tubi’s impressive database of titles and users.
“I think Scala is awesome”
Alexandros Bantis
Tubi before Scala
When Alex started working for Tubi, he encountered the following infrastructure: a homepage service written in Node.js, while analytics and experiment services used Apache Spark. A machine learning data pipeline, also based on Apache Spark, fed a Redis database backend.
Does it sound complicated? Definitely, especially when it comes to running A/B experiments. To test some new ideas out using A/B models, ML engineers were forced to write the Apache Spark model, write the data to Redis, and then update the code in the Homepage Service application. The work had to be done in cooperation with the backend engineering team, which meant extra time was needed for the implementation and may have caused delays. The engineers also had to update the configuration on the separate Node.js experiment engine, implement changes, and restart. After an experiment had finished, the process had to be repeated in order to come back to the starting point.
Of course, that created a massive challenge and was an obstacle for the whole company. Machine learning engineers had to collaborate tightly with backend engineers to run even a single A/B test. Nothing could have been implemented without the backend team and multiple levels of approval, cooperation, and deployment.
And you may be able to guess in such an institution as Tubi, the number of A/B tests to implement or even discuss was rather on high. The collaboration left a lot to be desired, especially when it came to projects being time-bound: multiple back-and-forth cooperation cycle times negatively affected the implementation of A/B tests.
There had to be a better way to do it, and there was. We’ll cover it later on.
Tubi’s challenge
That “collaboration” issue was just one of the reasons why changes were needed. The architecture, to put it simply, was rather obsolete and no longer served Tubi well, since the company had been growing dynamically.
And scaling did put a few challenges in Tubi’s way. The technological requirements that changed or newly presented themselves were as follows:
- lower latency (collaboration issues to be eliminated and shortened)
- better fault tolerance (including better observability and debugging tools)
- higher efficiency (throughput)
- reliable speed and performance
- better, more stable, and maintainable code
- business logic technical debt removal
- more advanced support
- better test coverage
Another challenge was in providing a great customer experience. The time taken to deliver a personalized page of video recommendations to a user was about 300 ms when Alexandros took the project over.
“My mission was to create a back-end application that would make recommendations and ML algorithms that have been run in Spark space, and get them in front of the user.”
Alexandros Bantos, Tubi | https://www.youtube.com/watch?v=ZnZ8oqz71bw
After a round of changes implemented with Scylla and Scala, Alexandros shortened that time to just 10 milliseconds.
Tubi after Scala: an expert commentary
One of the areas in which Tubi found Scala the most useful, according to Alexandros, was Domain Modeling. Scala is a very expressive language. It supports a lot of constructs that other popular languages don’t. This allows a Scala developer to express business rules with more precision while remaining concise. In other words, Scala lets you say more with less.
And this advanced expressiveness is checked for correctness by the compiler. That means, when the domain model evolves and the developer introduces changes, the compiler is there to help identify inconsistencies that the change might have introduced. These problems will be caught early during the development, not while the system runs in production. This greatly decreases the cost of evolving the system towards better supporting the business requirements. As Alexandros says, Scala allowed them to move from a “move fast and break things” approach to a “move fast with a stable infrastructure” one.
Another area that Tubi found Scala extremely useful (and rightfully so) was concurrent programming. It’s well established that applications built on JVM can take full advantage of the processing power of the hardware they’re running on. Multi-threading is at the base of JVM, and it’s common for developers to use it. Yet concurrent programming is not easy. Care has to be taken when accessing and mutating states from different threads, and getting it wrong can result in race conditions, deadlocks, or simply poor performance. Scala’s expressive and advanced type system again came into play here, allowing library designers to provide proper abstractions over these complicated operations. One of these libraries, used in Tubi, is Akka. It brings the actor model, which is well known from Erlang, into the Scala world. It provides abstractions that take the burden of managing multi-threaded code off the programmer and onto the library itself, reducing the number of future problems to a minimum, and yet allowing the application to take full advantage of multi-core processors.
Lastly, Scala is a multi-paradigm language, which means that it supports both object-oriented and functional programming. Tubi found it very useful to use the best of both worlds, combined with the actor model, in building their system. Object-orientation provided modularity, functional programming was used to build bullet-proof business models and rules, and of all this was used along with Akka, which provided seamless and easy concurrency. All of the above contributed to Tubi now having a performant, highly scalable, easily maintainable, and easily extensible system.
Tubi x Scala in 2020
How to “judge” whether a particular company invests in or is looking for new resources? By checking their career page, you can often quickly evaluate if a specific technology is a skill that the company is looking for.
And Scala, in Tubi’s case, is definitely one of these skills.
Tubi hasn’t stopped recruiting people with a specific, Scala-focused, mindset.
Here’s just one example from Tubi’s LinkedIn job offers: in this particular case, they’re looking for a Scala Engineer.
We are looking for a Scala Engineer to work on building low-latency services to serve machine learning models, work on sophisticated algorithms for ad delivery pacing and forecasting, improve the way we deploy and operate our services, and even contribute to open source projects.
Such job offers are also available on various other sources, including AngelList.
Their Scala requirements are rather on the strict side, with a requirement of at least 3 years experience working with Scala and full efficiency. What’s more, Tubi may require a degree from an accredited university.
However, from the job descriptions, it’s possible to evaluate how each Scala job role is going to look like. Scala Engineers at Tubi are responsible for contributing to building low-latency services, working with micro and distributed systems, and writing shipping quality code.
Working with Machine Learning Infrastructure is a must.
What else does Tubi do towards Scala appraisal & education? They use Github, for example, where Tubi willingly shares repositories written in Scala. Their Medium profile is also filled with insights that are precious for Scala wannabes, and not only.
To wrap-up
There are so many benefits resulting from changes related to Scala: more maintainable code, resolving collaborations and implementation of A/B testing issues, or lower latency, just to name a few. Such changes don’t happen overnight, obviously. Tubi is just one example of a company that has tackled Scala in a truly impressive way. If you want to join the club, contact us and let’s work this Scala magic together.
Read more
Authors
I am currently working in the IT industry, went from programmer to COO of Goyello ( Aspire Systems Poland now), then I was one of the founders of an IT recruiting agency called Try Catch, after that I became Head of Business Development and shareholder in Scalac Sp.z o.o. Would describe myself as an enterprise adorer and father of three.
