In this article, we compare Tapir with endpoints4s. We highlight the differences by providing examples and explanations for the most common features you would like to have in your REST API. Both libraries only require you to describe the communication protocol in Scala. Once the communication protocol is written, you need to wire it with a specific HTTP Server (such as Akka HTTP) and/or body parsing (e.g. Circe). At the end, the library produces clients, documentation, and servers for you with implementations of your choices.Read more
The Scalac Factor
Every company has its own culture. So do we. We really love to work together as a team. Scalac started as a remote-friendly company without managers, and we continue with those values in mind. Remote work is only growing inside the company, and even though the flat structure isn’t that flat anymore you can still call our CEO by name, and we’ve got more roles rather than complicated structures. Startup atmosphere – as they call it – but we prefer to call it the Scalac’s atmosphere because we believe that the fact that we work hard, have fun, and do the right thing is something unique and great about our company.
Why things changed at all? Because now we have exactly 124 people in our team while only 3 years ago there were only 70. So obviously, we had to reorganize how things are done.
We have a core team, whose members are responsible for different business areas. Of course, at the top of our team, we have a CEO, but we also have technical leaders for different teams: Development – Backend (mostly Scala), Frontend, QA, UX/UI designers, Business Development, Project Management, Human Resources, Finance, and Marketing.
As I’ve mentioned before, we are far away from a corporate structure, which we want to avoid as long as possible. This is crucial, but the most valuable thing about Scalac is the people – working as one big team of functional, passionate people to keep our quality bar very high.
5 things to join Scalac
Are you wondering how it is to be a Scalacer or thought about applying to Scalac? Here’s a quick guide to prepare you for joining our crew and to briefly show you what the core values we cultivate are.
1. Be sure that your code is functional
If so – that’s the first strong sign that we want you in our team!
Be ready to learn
If your code is not functional… yet – this section could be for you. Our candidates are often strong and experienced Java, PHP, Python, etc. developers, willing to switch to Scala. Their code in the technical task is well-structured, has all the goodies of OOP. However, the code isn’t very functional or reactive. In this scenario, we could call it “Java/PHP/Python in Scala”. Too many variables, as well as a lack of the proper usage of techniques available in Scala, such as Tail Recursion.
Don’t get us wrong. This doesn’t mean that we will reject great guys like this at the beginning. We encourage them to work hard and change the way they code by providing solid feedback and learning sources.
How to learn?
We are always happy to send them Scala Tips, which were prepared by our developers, or a list of good sources to study to help start properly with Scala. We also try to share our developers’ thoughts and opinions about a candidate’s code, so if you want to consult them during the recruitment process about anything – don’t hesitate to let us know – we will be happy to help.
A good and proven method that works best for learning functional code is to read the famous red book “Functional Programming in Scala” or complete the Martin Odersky Course. They have several exercises that force you and your brain to switch to a functional way. It might be a bit painful at first, but when you manage to break through the wall of pain, you will be enlightened and delighted. And in our experience, you probably won’t want to go back and return to nonfunctional code. But let’s see – we await your opinion! d
2. Be sure that you are close to our values
Scalac at its core is a flat organization. We love to cooperate. We have three strong values at the center of our culture:
- Work hard
- Have fun
- Do the right thing
We work hard on our projects and put our whole hearts into making them real and work properly.
But after all this hard work, we like to spend some time together during our online informal meetings, talking about random stuff. Or during our integration trips when we can be together, doing new activities like off-roading, paintball, kitesurfing, and do even more talking, dancing, and signing up to the morning hours ;)
Do the right thing
Doing the right thing is just as important as the previous two. We want to believe that everything we do – we do with good intentions and for a good purpose. Often we get involved in charity campaigns to help those in need. We choose good and responsible sources for our gifts or swag if you prefer. For our employees, the most important areas for ‘doing the right thing’ are the environment, children, and animals – so we try to help them whenever we can.
If you’re one of us – we trust you
It may sound like a cliche but when you think about it, you realize that the answer to most of your questions is you. You know what is “the right thing”. So just do it. If it doesn’t work – you can make mistakes, we all make them. That’s the way we learn. The main reason is that we strongly believe in another of our values: “People at Scalac are very smart and responsible”. When you are smart and responsible, you are able to make decisions about your own tasks, take ownership, and face up to the consequences. You do it because you know it’s needed. This goes back to our core value “Do the right thing”.
Everything around coding has to be done by someone and in most cases, you are the one who decides what to focus on.
3. Be ready to be an integral part of Scalac
We use the Scalac Handbook to show new employees how we work, who is who, how all the processes work. We want to be transparent and open to discussion at all times. In our Handbook, we have an unwritten chapter, where everyone can add something his or herself to improve our way of working.
Scalac Times and Meetings
In our internal communications, we try to be as transparent as possible. We have our Scalac Times newspaper, which comes out cyclically every month and contains the most important information about what has been happening in our projects, teams, events, etc during the last month. In this newspaper, every team has its own space and can give any information it wants. We have also set up cyclical meetings between the whole team and our CEO, at which we can ask about anything connected to our company.
A really important part of the way we function is knowledge sharing. We love to share our knowledge everywhere, all around the world in fact. We want to be an integral part of the Scala community and develop it together at different conferences, in our branches, or via taking part in open source contributions. Scalac organizes its own events such as Functional Tricity meetups, Scala Wave conferences, Scalac Summer Camps, and Pizza meetings, where we can also share our expert knowledge with each other.
4. Be open to working remotely
Scalac is a remote-friendly company. This means that apart from the values we have already outlined, remote work is another core aspect of our company. How do we work remotely?
How to work remotely
Our employees have already written great articles on this subject, you can find them here:
- Working remotely – Simple guide for people who want to start but are afraid by Michał Szulczewski
- A few business lessons on remote work that crisis can teach us – by our CEO Łukasz Kuczera
- How to manage remote employees by Monika Hinc, our Head of HR
- How to work from home like a boss also by Monika Hinc, our Head of HR
In these texts, our Scalacers list several challenges of remote working and share tips on how to handle them from different perspectives. Please read them and think for a moment if it’s really for you. Maybe you would prefer to mix remote working with working from the office in Gdańsk or in coworking spaces. This type of work is challenging and requires a lot of self-organization and perseverance. But if it suits you – it can be a great, valuable experience!
Despite the remote character of our work, as we mentioned before, we love to spend time together. We pay a lot of attention to our relationships and nurturing them. After work, we can talk on our #cafe or #wine channels or meet during company retreats.
Watch this video to have a little sneak peek on how it looks like.
These days, after the coronavirus lockdown, the world and ways of working are changing and market research shows that remote work has become more popular than even just one year ago. And we’re quite lucky to have 6 years of experience in this matter.
According to the ‘Remote Managers 2020 Report’, 87% of remote managers believe that remote work really is the future.
5. Be ready to choose your own direction
It’s important to know where you are heading. Of course, your plans may change a year from now but we like to work with people who know what they want to invest their time in. We know that it’s hard to be good in every field.
There are cases when somebody is a full-stack developer but being good at both ends of the software requires a tremendous amount of hard work. We really appreciate it and support it as much as we can.
We are experts in the field of Scala functional programming and we want to work with the best Scala hAkkers in the world.
It’s important to know where you want to be in the future and we as a company can help you develop your career path in various directions:
- Consultant – focused on cooperation between technology and business, explaining technical issues to the clients, proposing solutions, and ways to implement projects.
- Team player or mentor supporter – focused on teamwork, always willing to help, share your knowledge, caring for the development of both yourself and the team.
- Typical hacker – focused on the hard, complicated programming tasks to bring projects to an end with good quality.
- Any other path which you have a vision for, or maybe you have an idea you want to develop, something you’re able to show us and convince us it’s a great idea. If so, yes! Let’s do it together!
We like to work with people who are well aware of their choices and are fully committed to achieving their goals. We help them to do this by sponsoring their trips to the best conferences around the world, providing books, holding both internal and external training.
On our Careers page you might like to take a look at us, our teams, how we work, what is important to us, which positions are open, and a lot more information which we hope will convince you to join us.
As I mentioned before we do not have only Scala team, but it is our core – especially in the context of functional programming, so it was only to highlight where we’re coming from. We’re constantly changing and building new teams.
So, are you ready to join Scalac? Do you agree with our core values? Is there anything you would do differently? If so, we’re happy and open to talk to you about it!
Originally written in 01.2016 by Maciej Greń, updated in 09.2020 by Katarzyna Królikowska
We compared hours and hourly rates and the outcome might surprise you
Automation testing or manual testing? That is the question! Which one is better? This is something that gives many in the tester community sleepless nights. Especially during a project when we have to determine the testing strategy, even at its later stages. This is also, unfortunately, a question with no clear answer. Why? Because the reality is that these two ways of testing are inseparably connected with each other.
If you’re still wondering which is more profitable, effective, or better suited to your or your client’s project, let’s dive right in. Here we’ll be revealing the true value of automation vs manual testing, based on one of our clients’ applications – CloudAdmin.
Together with Pawel Gieniec (CEO & Founder CloudAdmin), we are creating and developing a web application – CloudAdmin. Cloud Admin is a cost-optimization platform, which helps save thousands of dollars by eliminating hidden cloud instance buying costs.
The project turned out to be perfect for introducing an automation architecture for modules that are always tested during regression testing (the process of running older tests to ensure that new updates to a piece of software haven’t introduced or re-introduced previously eradicated bugs). Our goal is to save time, which we will then be able to spend on manually testing new functionalities. The application is developing dynamically, so we’ll be needing more and more time to check all of the fresh new features.
One of the core functionalities is logging. Nothing surprising, this is probably the module most often automated.
We decided to use automated testing with Python and pytest.
And this is how it went:
We based our login tests on these 7 test cases below.
The first video is from PyCharm (the Python IDE). It shows how long it takes to run and finish all of the tests for the login module. As you can see, all of them passed. This means all of the cases have been tested correctly and have achieved the expected results, i.e. the functionality works properly.
Now take a look at the video below. It shows how long it takes to carry out manual testing for the same test cases as for the automated tests.
How test automation reduces the time
At first, you might think “Hmm, this is no big deal. There’s only an 18-second difference in favor of automation.”. Okay, you’re right, but let’s think more in the long-term and globally.
First of all, it’s not just an 18-second gain. It’s an almost 2-minute yield for one run! This will ultimately give us more time to spend on the project. How?
If you decide to implement automation architecture for sections that always needs to be covered by manual regression testing, you will gain all of this time for doing other things (manual testing for new features, application development, automation architecture development).
The login module is just a drop in the ocean. But let’s take a look further. Let’s compare working times by adding the next 3 modules: AWS Ri Planner, AWS RI Management, AWS Spend.
As you can see, the automation tests give us a 35% time yield relative to the manual tests (9:16 min vs 14:14 min). And again, this is a 14:14-minute gain, we have saved on regression tests – for only 4 modules.
How test automation increases productivity
As mentioned before, by dint of test automation, the testing team gains more time they can spend on doing other tasks. But this is not the only thing they get. Regression testing can be tiring. Imagine how tedious it would be to repeat the same action over and over again. This might sound trivial, but the time involved in repetitive, manual testing is wasted, especially over a longer course of time. With this in mind, think about how much more productive testers could be if they were able to devote more time to performance testing, security testing, or testing of new features.
Moreover, testers have to quickly and continuously execute a lot of time-consuming tests simultaneously, just to be sure an application is performing as expected at all times.
How long will setting up an automation architecture take?
It’s rare that everyone on your team has automation experience, so when it comes to resources, you will really have to gauge how much time it will take for your tool to make an impact. You will either need to invest in expanding your team’s automation knowledge or hire automation specialists.
Let’s go back to our CloudAdmin tests.
Considering the 4 sections previously mentioned (Login, RI Planner, RI Management, Spend), our QA Engineer (the automation tests were written by one person) devoted 80 hours to:
- choosing and configurating tools – 16 hours
- writing automation tests for every test case – 16 hours per module / 64 hours per 4 modules
It might seem like this is a long period of time, but don’t be fooled. Automation architecture is reusable. We can use it limitlessly. Maintenance and development of automated test cases is an ongoing process. Unfortunately, there’s no guarantee that spending eighty hours automating a set of tests means you won’t have to deal with them again. But it will still take definitely less time in general. Take a look at the graph below:
The turnaround time for automatic testing is definitively longer than for manual. However, the benefits of automation outweigh the time benchmark in terms of:
- Reduction in execution time
- Reduction in effort
These two metrics will provide an incredible boost towards higher efficiency and a better rate of reaction to potential bugs.
What about costs?
Last but not least, is something most crucial when establishing a contract with a client – costs. Test automation might be a catchy slogan during negotiations but it’s worth checking the hidden costs. You don’t want the client to think this is just another fashionable balloon, ready to burst at any time.
If you intend to adopt an automated testing process to meet the rising demand for bug-free releases and faster delivery cycles, it’s vital to assess whether the return on investment (ROI) is worth the changes. Before executing, or even considering building an automation strategy, you will want to calculate the net gain you will achieve from transitioning. Divide this by the net investment needed to transition (i.e., the tools and resources you use), and you will see your ROI for automated testing.
The maintenance cost of test automation is not usually trivial but if you keep enhancing the test automation framework, improve the application testability and overall testing processes, a high ROI is very much achievable.
To get the total cost, you will also want to take into account the hourly cost of the number of team members executing the tests.
Based on the average earnings for an Automation QA Engineer and a Manual QA Engineer (I rely on data from Glassdoor), we can make a labor cost comparison simulation based on the CloudAdmin case study (hours needed to perform each type of tests for this application).
As I mentioned before, it took 80 hours for our QA Engineer to develop and execute the automated tests for the 4 CloudAdmin sections. How many times might we manually test the same part of the app at the same time? 337 times (for the record: manual testing time -> total time from the module times table). So what’s better:
- spending $2,100 on performing the same set of regression, manual tests 337 times
- spending $2,700 on building an automation architecture – which once created is reusable?
We’ll let you answer that question ;)
Final battle: Manual Testing vs Automation Testing
To wrap up, is an automation testing a cure-all for time-consuming manual testing? Not really. Manual testing will always be important. There are scenarios that will always require manually executed test cases. This means you will still have to create, run, and maintain the tests. But there are still a lot of situations when you can successfully implement automation. This can be a cost-effective method for regression testing of software products that have a long maintenance life.
With a balanced approach to the testing process and a smart and effective amalgamation of both automation and manual testing, you can spend more time focusing on building new features to enhance your product quality – ultimately giving yourself the opportunity to stay ahead and be more competitive in the market.
Check out other testing-related articles
- REST ASSURED API TESTING TUTORIAL. HOW TO USE REST ASSURED WITH JAVA AND SCALA
- HOW WE MADE MANUAL TESTING OF 590 TIMEZONES IN 2 DAYS POSSIBLE
- TESTING FOR MOBILE APPS
- PERFORMANCE TESTING
Check out other business-related articles
In the past, internet use was primarily based on websites that provided content and graphics. Today, the world has moved forward, which is why we should distinguish not only websites but also applications created by programmers and developers: web, mobile, desktop.
Today we want to cover the topic of web applications in the context of testing them manually. Considering aspects – like the buzzwordy User Experience – that they bring, as well as their responsiveness. We’ll cover basics that you have to take into account at the very beginning of your testing journey. Also, we’ll show you some great examples from applications like Cloud Admin (one of Scalac’s clients).
Why web apps? Slack is a web app. Trello is a web app. Google Docs. Gmail. Even Twitter. Do you need more arguments?
Now let’s see what exactly you’ll learn in this blog post:
- Difference between web, mobile and desktop applications
- What are UX and UI?
- How to test
What is the difference between mobile/web/desktop applications?
A desktop application is a computer program that we create for a specific operating system, and its use requires installation on a given hardware. Although desktop applications may seem outdated, they work only when there is a high level of security of any stored data, preferably located on an internal server, and which only the employees of a specific company have access to.
On the other hand, mobile applications are designed for use on smartphones and tablets. They are created for the needs of specific operating systems (Android, iOS), and they also require installation.
Internet applications, otherwise known as web applications, stand out from the others. These are computer programs that operate on a server, communicating with the computer’s host via a computer network, using a web browser for this purpose. Unlike mobile and desktop applications, the web application does not require installation on a computer, although Internet access is required.
Why are web applications still important nowadays?
The most popular type of application is mobile apps. This is because users can use them anywhere, provided they have internet access and a smartphone. Consequently, a lot of people don’t need to buy computers. Phones have now become our pocket computers. Nevertheless, there are still users who prefer or have to use web applications.
For example, web applications are often preferred when it comes to:
- bank transfers
- buying tickets (train, plane, cinema)
- booking hotels
- watching videos (bigger screen)
- work (the largest number of users)
Another aspect worth mentioning is that the group that prefers web to mobile applications are elderly people. It’s easier for them because the app is more intuitive on a computer and they can zoom in to see all the contents of a website.
Another argument supporting web applications is that not all applications have mobile apps, so sometimes users don’t really have any choice.
Finally, as we have all probably experienced in the past – websites often don’t look great on mobile browsers and some websites simply recommend using it only on a computer browser.
What are UX/UI And How To Test Them in a Web Application?
First off, you should know the difference between UX and UI in web applications:
UI – User Interface
At the most basic level, At the most basic level, the user interface (UI) is the series of screens, pages, and visual elements—like buttons and icons—that enable a person to interact with a product or service.
UX – User Experience
User experience design is a concept that has many dimensions. It includes a whole range of different disciplines such as interaction design, information architecture, visual design, usability, and human-computer interaction.
UX is the process of designing (digital or physical) products that are useful, easy to use, and easy to interact with. It’s about enhancing the experience that people have with your product and making sure they find value in what you’re providing.
In general, UX answers the question: “how can we improve a given product in terms of usability services for the application user?”
Why UI/UX is so important in testing
If you start testing a web application, you should be prepared to step into the shoes of a user who is using the app for the very first time. And you need to pay attention to design on the website. It’s always a good idea to base your ideas on previous experience of other apps that are considered to be user-friendly – and use them as a benchmark for the app you’re testing.
The purpose of UX in a business environment is to discover the individual personal characteristics of target groups of given products or services). Based on the information collected, the product or service can be matched to the characteristics of average users in order to facilitate and improve communication, thereby making the product more intuitive and more pleasant to use. There are thousands of similar apps on the market – and we all know the general success of an application might come down to differences in user experience. As testers, our ultimate goal is to make the app as good as possible and help our clients achieve these business goals.
Testing Web Applications: Usability Tests
It’s always a good idea to carry out usability tests if you have the option. Preparing a usability test involves carefully creating a scenario or a realistic situation in which a person performs a list of tasks using the product, while observers verify the accuracy of the tests and take notes. Such a test scenario may consist of several test cases.
This way, you can check the entire scope of the functionality. One of the goals is to check how people perform assumed tasks. It’s surprising how different this can be from what we had assumed. Having this kind of information enables us to better test the application, and to adjust the design to the real needs and – more importantly – habits of our users
Testing Web Applications: A/B Tests
A/B testing consists of creating two variants of a site that differ in only one element and then analyzing which of the variants is more effective.
Which elements can be changed to check which version of the website is better?
- CTA buttons (different color, copy, size)
- Location of the buttons
- Size of the pictures
- Page headers
- Website links
- Different product card layouts
- Less complicated checkout
- Homepage layout
- Customer reviews
- Number of fields in forms
- And probably everything else that may come to your mind
A/B versions of the website can then be sent to the Business, Marketing, and QA teams. They can then choose a better, more intuitive, and more user-friendly version, and with the biggest endorsements. Or – in the case of Business or Marketing – whichever brings better traffic and converts better.
If your website has a lot of visitors, you can upload an alternative version of the website (there are also tools that will help with that). This will let you collect information about the number of clicks on particular elements – so you can clearly see which version/element of the website is more effective. You can also send a short survey about the new website design to users. If there’s one thing you should know at the beginning of your testing journey it’s that there’s no better source of knowledge about users than users themselves.
Examples of A/B tests:
Testing Web Applications: Web Responsive Design
This is all about designing a website so that its layout adapts automatically to the size of any browser window in which it is displayed. No matter if it’s a browser on a computer, smartphone, or tablet. A website created with this technique will be almost universal: it will display well on large or small screens
As a QA’s we should do a few UX/UI tests to find the differences between mobile/web apps
Examples of web responsive design
Example 1 – HireHelp (Scalac’s Internal Application)
full-screen mode on the computer:
Pay attention to the menu on the left, and the 2 buttons on the right. Now narrow the screen and note the changes.
- The logo of the app has changed to a hamburger menu
- The menu has only icons, no names
- The 2 buttons on the right are fitted to the smaller screen (change of position)
We can open the menu to show the names of the tabs:
Now the background is greyed and the menu is more visible. So the website still looks good and remains fully functional.
Example 2: Scalac.io website
full-screen mode on the computer:
Scalac website on iPhone 8:
The first difference you should notice is that on the computer the header font is bold (Best Scala hAkkers). The mobile screen would be too small to use a bold font – the headline would take all of the screens.
On the computer (web app) we have a bigger screen, so the browser can display all of the menu tabs. On the phone (in this case iPhone) the menu tabs are all hidden in the menu button (hamburger). When a user clicks on it, the menu will appear:
Let’s check some other components of the website.
On the computer, we can see 3 tables that are situated horizontally;
On the iPhone, we have these tables arranged vertically;
There is also an additional button to go back to the top of the website. This is especially helpful when the website is long because the user doesn’t have to scroll up manually which can be quite irritating.
Applications Lacking the “Web responsive design”
Now, let’s have a look at a website that’s lacking web responsive design. On the computer, our table looks good:
Unfortunately, on a mobile, some elements are partly invisible, and the table is hard to read. The user has to move from left to right if he or she wants to read something from the table. This section of the website is recommended only for browsers on a computer.
A good solution for this situation is to create a different app for mobile or implement web responsive design. But, as I pointed out earlier, not everyone needs to have mobile apps, sometimes owners of a product don’t really need a mobile version.
To sum up – our mission when testing a website on different devices is to check all of the components (buttons, text, images, tables, animations, etc.) in various resolutions and check if all of the features are working properly. It’s also good practice to deliver feedback to the Project Manager about any different approaches or changes that may be required to boost the usability of the elements, structure, and features.
Now you know how to test the visual part of web applications, let’s move on to the functionalities.
Testing Web Applications: Different Devices
If you don’t have any physical phones/tablets you can use, Xcode or Android studio can simulate phones (recommended more for mobile apps than websites) or use the web responsive design mode in your browser:
You can also use it inside browsers:
Google Chrome: click right mouse button -> choose “inspect” -> in the tools click the icon of the phone:
You can choose phones from the list, or type your custom screen resolution
Mozilla Firefox: open the hamburger menu -> web developer -> responsive design mode
Just as in Google Chrome, you can choose the kind of phone or type in your customized resolution. Additionally, In Firefox, there’s an option to simulate a phone with a slow internet connection.
Testing Web Applications: Login And Register Unit
Nowadays it’s hard to find an application without a user account. The reason is that it’s good for product owners because they can collect data on their users, and use it to prepare customized offers. It’s also good for users because they have their own settings and data saved. No need for setting up everything from scratch every time you launch the app.
Now I want to show you how to test the login and register unit. As an example, I’m going to use the Cloud Admin app https://www.cloudadmin.io/. (One of Scalac’s clients).
Open the register screen; https://www.cloudadmin.io/register
Pay attention to the structure:
– ‘email’ field (greyed example with email structure)
– ‘Password’ field (greyed example)
– ‘Sign up’ button
– ‘I agree terms’ checkbox
– ‘Register’ button
– ‘Connect with Flexential’ (external API) button
Test case 1 (sad path): Try to sign up without any data
– Click the “sign up” button keeping the fields empty. The app should give a response error, and the fields should be checked in red (this is important information for users, showing them which fields have been wrongly filled).
If it’s working correctly:
Test case 2 (sad path): Try to sign up with a bad email structure:
– In ‘email’ type: this_is_test
– In ‘password’ type: 1234
– In ‘confirm password’ type: 1234
– Click on the checkbox with terms
– Click the ‘sign up’ button
The app should respond to 3 errors:
- The Email field should be checked in red with the information; “Please enter a valid email address.”
- The Password field should be checked in red with the information; “Password must be at least 6 characters long.” Now we know – the password has to have 6 characters minimum
- The confirm password field should be checked in red because the password is too short.
All of the structures should work correctly
Test case 3 (sad path): Try to sign up with a good email structure, and different passwords in the fields
– In ‘email’ type: email@example.com
– In the ‘password’ type: 123456
– In ‘confirm password’ type: 678901
– Click on the checkbox with terms
– Click the ‘sign up’ button
The app should respond with the error message “passwords do not match” if it’s working correctly.
Test case 4 (sad path): Try to sign up without checking to agree to terms:
- In ‘email’ type: firstname.lastname@example.org
- In the ‘password’ type: 123456
- In ‘confirm password’ type: 123456
- Click the ‘sign up’ button
The app should respond with the error message “You have to agree to all terms to continue.” if it’s working correctly.
Test case 5 (sad path): Try to sign up with all of the correct data:
- In ‘email’ type: email@example.com
- In the ‘password’ type: 123456
- In ‘confirm password’ type: 123456
- Click on the checkbox with terms
- Click the ‘sign up’ button
The user should be logged in automatically after registering:
Test case 6 (sad path): Try to sign up with an email of an already existing user (create this user earlier)
– In ‘email’ type: firstname.lastname@example.org
– In the ‘password’ type: 123456
– In ‘confirm password’ type: 123456
– Click on the checkbox with terms
– Click the ‘sign up’ button
The app should respond with the error message “The email is already in use. Log in” if it is working correctly.
To sum up test cases:
I have shown you some tests for the register unit:
- empty data
- bad email structure
- bad passwords
- not selected terms
- happy path to register
- existing user
It seems obvious in points 1 and 6 that it won’t work but I have had cases in projects where these tests have actually passed. So please don’t ignore them. A good practice is to create test case documentation because the register and login features are repeatable in other applications, and you will then have a checklist ready to use in the future. There are sometimes more or fewer fields, but the structure is usually similar.
In the login unit there will be very similar tests because there are only 3 elements: email, password, login button So your homework could be to create some tests for the login ;)
Testing Web Applications: Test Fields With Data
In applications, we often have a lot of fields with no validations (users can type whatever kinds of characters they wish ). For example, in surveys or questionnaires. Let me show you some test cases to test these kinds of fields:
Test case 1:
Leave field empty
Test case 2:
Add only space keys
Test case 3:
Add small letters
Test case 4:
Add big letters
Test case 5:
Test case 5:
Type special characters
Test case 6:
Add a mix of letters, numbers and special characters
Test case 7:
Add an emoji icon
Test case 8:
test the upper limit string of the characters (for example app enable up to 80 characters)
– more than 80 characters
– less than 80 characters
– up to 80 characters
Test case 9:
test the minimum limit string of characters(for example the app requires minimum 4 characters)
– less than 4 characters
– 4 characters
– more than 4 characters
Just as in register/login, a good practice is also to create test cases for these tests so you can use them again in other projects
Testing Web Applications: Calendar feature
This functionality can be found in all kinds of web applications:
- schools: when you want to enroll
- employment: in HR apps
- Insurance: If you want to compare insurance offers
- And many more. It’s important to test this feature because if it doesn’t work correctly, users could have a problem with filling in information to submit papers for studies, fill in CV information in an employer app, buy insurance, etc
So let’s focus a little on this calendar feature:
After clicking on the start date the calendar opens. The user can also type in the date using a keyboard. Let’s go a bit deeper into this feature:
- There’s no option to change the year using a button. After clicking on the arrows, the app changes only the month, so we should report this because if a user wants to choose May 2017, he has to click the “back” arrow lots of times.
- In this calendar, the dates use the U.S. calendar structure: month/day/year
After choosing the start/end day there is an “x” button to clear all of the data. This is important because a user can type a new date using a keyboard after clearing everything. Don’t forget about this functionality in the calendar feature ;)
Let’s do some tests :)
Test case 1 (sad path): Choose a later start date than the end date
Choose from the calendar start date: 05/25/2020
Choose from the calendar end date: 05/15/2020
The application shouldn’t allow the user to set the start date later than the end date
Test case 2 (sad path): Try to type day more than allowed in the calendar (32 or more)
In start date type: 05/50/2020
In end date type: 05/60/2020
The application shouldn’t allow the user to set the date later than 31
Test case 2 (sad path): Try to type a month more than allowed in the calendar (13 or more)
In the start date type: 13/02/2020
In the end date type: 05/60/2020
The application shouldn’t allow the user to set the month more than 12
Test case 3 (sad path): Type year only with 2 digits
In the start date type: 01/01/01
In the end date type: 01/02/01
In the app, a year should be 4 digits minimum
Test case 4 (sad path): Try to choose the same start date as the end date
Choose from the calendar start date: 05/10/2020
Try to choose from the calendar end date: 05/10/2020
The application shouldn’t allow a user to choose the same date. The app suggests choosing as the ‘end date’ the next day instead of the same as the ‘start day’.
Test case 5 (happy path): Choose start/end date from the calendar where the end date is a later date
Choose from the calendar start date: 05/15/2020
Choose from the calendar end date: 05/20/2020
Test case 6 (happy path): type start/end date from the calendar where the end date is a later date
Type the start date: 05/15/2020
Typ the end date: 05/20/2020
Now you know how to test a calendar feature. But did you spot the bug? ;)
Look at the second, third, and fourth screen on the calendar. There is no “x” button to clear the date. It needs to be added. So report it :)
In this article, I wanted to share with you some information about web applications and illustrate some test cases you can use in your projects. These are the basics, but as you know, well-tested basics are the key to success.
If you’re interested in articles about testing you might also like:
Why do you need this Rest Assured API testing tutorial? In the world of testing – automation can be a real game-changer. That’s why it’s on everyone’s tongues, especially “automated tests” in the context of GUI tests written in python/java with SeleniumWebDriver. Or possibly with the increasingly popular Cypress.io. However, GUI tests sit right at the top of the basic pyramid when it comes to testing.
The ultimate guide to do great on your next technical interview (Questions, Answers & more)
Common stereotypes show that in the IT industry, effective communication skills are not strong points in the profile of a typical programmer. They need only to have solid technical skills to help them accomplish specific tasks and to create particular products. But it turns out that nothing could be more wrong.
In the long run, when it comes to matching a new person to a team and making the right employment decisions, soft skills (communication skills, teamwork, creativity, etc.) are often more important than technical skills.
According to specialists from a Linkedin team in the report “Global Talent Trends 2019”, bad hiring decisions and parting ways with employees are in 89% of cases caused by a lack of soft skills or deficiencies in both areas. Only 11% of partings were caused solely by problems with technical expertise.
In the opinion of Deloitte’s “Soft skills for business success”, in 2030 as many as 2/3s of job offers on the market will require candidates to have specific soft skills.
Why we care about it at Scalac
At Scalac, our employees have the perfect combination of soft skills along with robust technical expertise, because as a company, we try to be as professional as possible when consulting with our clients. We always make sure we advise our clients, explain why any given solution is the best in our opinion and the pros, cons, opportunities, and threats of our proposals.
As a Scala development company, and experts in our field, we have to be able to express our opinions and knowledge professionally so our clients can be 100% sure that their cooperation with us was the right choice.
In this article, we would like to share this practical knowledge with you to help you nail any upcoming interview.
After reading this article, you will know:
- How to prepare for a technical interview
- How to show and share your knowledge
- What a technical conversation looks like and why it is needed
- Where you can find answers to technical interview questions (reliable sources)
- All communication tips & tricks
Tech job interview – is it indispensable?
The primary aim of a technical interview is to check the most essential knowledge of the candidate. The interview usually takes about an hour. Scalac is a remote-friendly company, so in our case, we most often meet online. We try to look more widely at the candidate’s competences and ask about things that they were not able to show during the implementation of their technical task. Sometimes, the candidate has made a mistake, used an inefficient way of using a given technology, or simply recombined it with a complicated solution. The technical interview is the right time for the candidate to clarify any important issues – by answering questions or initiating their own clarification of the case in question. Interestingly, our experience shows that sometimes a candidate hasn’t done the technical recruitment task completely independently. During the interview, we can easily verify this and avoid any misunderstandings on both sides.
Point of view
Based on our experience, we like to regard this stage of the process as a joint meeting of experts and practitioners in our field. After all, we are all programming enthusiasts in the conversation, regardless of the side, we are on – interviewer or interviewee. The interview is also the time to ask about the perspectives of the other technical people in the company, know their points of view and experiences. Why are they working here? How are they working? What projects are they involved in? What technologies do they use? What tools do they use? It’s best to ask about these kinds of issues at the source, and the interview is an ideal opportunity.
Of course, this is also an excellent opportunity to check the candidate’s soft skills, such as communication, proactive approach, problem-solving, searching for solutions, flexibility, and much more. Our recruiters often say that they are looking for people who can solve problems and the challenges they face. This is the most important to them. The data provided by Linkedin also confirms this – the most desirable soft feature on the labor market is creativity, which, contrary to popular belief, is not only reserved for the creative professions. Recently, this has evolved into a feature that allows you to solve problems in an original, unconventional, and at the same time effective way. Having this kind of competence turns out to be crucial in the face of very dynamic tasks and a rapidly changing reality.
Based on the Linkedin data, the most appreciated – and therefore desirable – soft features on the labor market are creativity, persuasion, collaboration, adaptability, and time management.
In the era of technical development and automation, the demand for social skills will continue to increase until 2030, because it is a feature whose machines, even with using the artificial intelligence, are not able to take over and replace so quickly (McKinsey Institute, “Skills shift automation and the future of the workforce”)
How to prepare for a technical interview
Technical competence is practiced every day and improved continuously because programmers use their practical knowledge when doing projects and programming new functionalities as part of their professional duties. However, sharing knowledge with someone else, – especially not in any specific practical case but also theoretically – can be more complicated and require more communication and persuasive skills than it may seem.
What questions are asked in a technical interview? Scala technical interview questions – (Scala Programming Language interview)
As Scalac is a Scala development company the leading technology we use is, not surprisingly, Scala, which is why most of our technical recruitment is for Scala Developers. Below, we’ve also collected some valuable content on typical Scala technical questions.
There are plenty of sources on the web where you can find sample questions, so not wanting to discover America again, we have decided to combine our own questions with the best ones on the web, revised by our developers. Now you have all the best questions (with answers) in one place! Scala development company tested.
Sign up to get a PDF with Questions
How can I pass a technical interview? 6 Tips from our developers
Some time ago, our developers prepared a few rules to send to candidates before a technical interview to prepare them for what it will contain. So it will not be too stressful, below are some tips that you should consider if you want to be well prepared for the main points of an interview
- Before the technical interview, make sure you have your solution from the previous stage to hand, so you can open it if needed. Or just make sure you remember your solution. Sometimes technical recruiters will want to ask you about something you did in your task.
- Some concepts might be easier to explain with code than with words, so make sure that any app for presenting your code works fine on your device. It could be a simple “online text file” so you can share what you have in mind. This practical way of explaining is sometimes easier and shows your practical knowledge and experience off better.
- Sometimes, it could happen that a technical recruiter will ask you to do some “live-coding” during the meeting. This can be helpful to convey meaning. The idea is not to write compiling the code.
- Even if you have a lot of experience and knowledge when describing a topic, it is better to come up with an answer that covers the main points as briefly as possible. If you explain something for a long time and it turns out that this is not the best question for you, you will have less time for another question which could be better and more comfortable for you.
- It’s always good to spend some time refreshing your knowledge because sometimes it’s hard to recall how something works if you used it a year or more ago.
- From a practical point of view, before the call, remember to check whether your Internet connection, microphone, speakers, and camera are working correctly so you can talk without any disruption and delay during the meeting. To feel comfortable, you can always ask your recruiter about any “dress code” for the interview – not to be overdressed or too informal.
6 pieces of advice on communication from a professional interviewer
From a typical communication point of view, it is worth remembering the following issues so you feel by the end of the interview that both sides have the right information necessary to make further decisions regarding the recruitment process.
- If the interview is online – everybody knows that sometimes the connection quality can be lacking, freeze, or something – so do not hesitate to ask for clarifications or to rephrase anything – it’s perfectly fine!
- If anything wasn’t clear – feel free to ask for clarification, explanation, or to repeat it – this is completely natural, -we should always strive for the best understanding from both sides.
- Remember that the technical person is there to also answer your own questions about the company you are applying to. Don’t be afraid to ask – if everything goes well you will be working together in the same team anyway.
- Remember one basic rule – “I don’t know” is sometimes a good answer. It shows your maturity and openness – so do not hesitate to say it if necessary It is better to admit you don’t know than to try to figure out a response when you actually don’t have any idea about something.
- Make sure you know how much time you will need for the technical interview and make some time in case it needs to be prolonged. You can always ask the recruiter about the probable duration.
- It’s perfectly fine to ask your recruiter what the interview will be about and what it will consist of. Remember, you can tell the recruiter about your previous experience to let him be aware of your strengths. You can also mention which topics you feel the most comfortable talking about and which specific topics you feel you are expert enough to show off your best side.
How to answer tech questions to show them you know what you’re talking about
It sometimes happens that technical teams in IT companies may have direct contact with clients, as well as with their technical and business units at the stage of determining a project’s requirements. Scalac is a Scala development company and since 2013 when the company was founded – staying in touch with the client side is an integral part of our work. That is why sharing knowledge is essential, and we make sure to this when talking to the candidates.
Using The Feynman Technique to boost your communications skills
When it comes to preparing for the communication aspect of an interview, and showing off your knowledge, a good example to follow is the theory of Richard Feynman, an American Nobel laureate in physics. This technique
“involves using a simple analogy or metaphor to explain any issue. Although its main purpose is to identify questions that you can’t answer, thanks to which you find gaps in your knowledge, and you can get to know what you are learning.”
Feynman suggests that a perfect method to prepare and, assimilate your knowledge, and to realize any possible deficiencies, is to clarify an issue in simple words to reach the recipient, imagining it is a 5-year-old child. This simple way of communication is excellent in interviews especially on complex, technical topics, in which, at the same time, we have to demonstrate our understanding.
By explaining the issues, both in-depth and in simple language, we can show off our practical knowledge and the fact that we know what we are talking about. At the same time, do not try to hide certain areas of ignorance behind a veil of complicated, theoretical expressions and words. That’s why it is worth relying on examples and, if possible, share completed case studies from previous personal experience.
Isn’t simple language too simple for a tech talk?
Technical knowledge should of course be at an advanced level in the case of a typical professional interview, especially for regular and senior positions. But even in these cases, sticking to simple, direct sharing of your knowledge is important. Not least precisely because it happens sometimes that business clients are not technical people. Explaining to them project assumptions, technical offers, main challenges, risks, and normal activities in a programming project may require more of a practical, pictorial, and transparent approach to prove our expertise and knowledge of a given issue.
What you’ve learned – you know.
Remember that every experience you have had can be a lesson which you can always draw conclusions from and make good practices for the future. That’s why it’s worth bearing, in mind that there are no stupid questions.
As a well-known Polish proverb says “who asks will not go astray” – you can confidently ask questions during the recruitment process. Ask for any clarification if you do not understand something, or simply ask for a reminder of any issue already raised to ensure everything is transparent and understandable. Go ahead and feel free to ask about any typically technical aspects you can talk about with a technical recruiter, such as “what technologies are most often used in your projects?”, “What is the scope of the work in the project?”.
During the discovery interview with the HR team, any questions regarding work culture and company operations are also welcome. A recruiter who is interested in real dialogue with a candidate will be happy to answer any issues.
What can you learn from feedback?
It is also vital to make sure you understand well any technical feedback provided – regardless of whether it may cause a further stage or delay the end of the process. You can always get valuable information and suggestions in feedback, which you could use to improve your skills and be able to present yourself better in the next stages of the recruitment process. We should always get the best out of feedback and regard it as an opportunity for development in areas that may require further work, and which will allow us to spread our wings in the future.
To sum up
After taking into account the above points, getting acquainted with the research we have quoted, and observing the labor market, we can confidently say that a technical interview is necessary in the recruitment process. That is true from ’ the point of view of both sides of the recruitment process
From the employer’s perspective…
considering the predicted increase in the importance of soft skills in the labor market, it is right that companies should develop their recruitment processes in this direction to make the right recruitment decisions.
From the candidate’s point of view…
the technical interview allows you to present yourself at your best, to explain your thinking and defend your arguments. It is also an opportunity to obtain additional, valuable, and technical information from other developers, already working in the company you are applying to.
It is worth noting that currently, soft skills are gaining value. We, therefore, hope that some of the above tips showing how to prepare well for a technical interview will prove useful will give you an advantage in the labor market and will help reduce the stress that often accompanies the recruitment processes.
Thank you for reading the article!
I hope that after reading this article there will be no doubt that technical interviews make sense and you will need to prepare for them not only in terms of typical programming knowledge, but also in terms of communication skills.
I can do nothing more than to keep my fingers crossed for the success of each of your future recruitment interviews and hope that we will meet at some stage of the recruitment process at Scalac ;)
What are your experiences regarding technical interviews in the companies you have applied for? Do you think they were useful and valuable? Do you now understand how necessary soft skills are in your job?
As a tester I often get to test functionalities in which the approach to the subject plays a key role. In a previous article, I have already written about how to approach manual testing on mobile devices generally. But here I would like to talk about one of the most interesting functionalities I have had an opportunity to test. Namely the testing of advertisements in different time zones.
The assumption of the functionality was that a banner placed on an advertisement on a given day and time could be displayed anywhere in the world, making it possible to promote movie premieres or sales promotions. The creator of the advertisement would be able to set the day and time strictly in the “User time zone” but would also be able to enter the time zone manually.
Why 590 timezones?
Initially, the programmers wanted to use UTC to divide it into and cover 24 time zones. Then all the creator would have to do, is simply enter the country and the program would automatically add the time zone.
However, after initial tests, it turned out that this solution has one main disadvantage, namely, there is no provision for summer and wintertime, which change in virtually every country.
After consultations, they decided that the best solution would be to use the moment.js library, which is based on times in Wikipedia. With this solution, the user could enter any destination region into the field regardless of the current time. The most up-to-date solution was divided into 590 regions (TZ database).
What devices did we have to check?
This functionality was supposed to work on both desktop and Android devices from 5.1.1 version to the newest (it’s now 10.0) and for iOS from 9.3.5 to the newest (13). There were 13 versions of both systems to cover and 590 possibilities to check. I and a second tester had a maximum of just two days to plan and conduct the tests. So we had to come up with a manual testing approach that would give us sufficient test coverage but in this very limited time.
How to approach manual testing when you have 590 Time zones and 2 days
Finding common ground
We started by listing all 590 zones and finding common features, i.e. the same UTC. For example, times in Europe / Astrakhan, Europe / Samara, Europe / Saratov, Asia / Dubai, Asia / Muscat, Asia / Tbilisi, Asia / Yerevan all have UTC +04: 00 in summer and wintertime so there would only have to be one test for all of these regions on all of the devices. After these merges, according to the UTC, there were 54 zones to check. However, 54 zones is still a lot, so we had to find another elimination factor.
The next step was to take all of the 54 available zones and check which ones have high priority, which medium and which did not need to be tested at all.
High importance in our case meant that the timezone was important from a business point of view. For example, the zone in which a large part of Europe is located, i.e. +01: 00 in winter and +02: 00 in summer. Medium priority was given to those that were less significant for business or less used, such as the zones in which the Azores or Egypt are located in. But also parts of Australia, for example, because Australia has 5 different time zones and some were more important than others. Zones on smaller islands, practically uninhabited, such as the zone in which Fakaofo is located, i.e. an archipelago from a group of coral islands with just 265 inhabitants, were also considered not important from a business point of view.
After counting and analyzing the highest priority zones, we managed to reduce it to 25 must-haves and 11 nice-to-haves that would also be checked eventually if time allowed. You should also remember that some of these zones would need to be divided into summer and wintertime as well. So let’s keep that in mind.
How to change time zones on devices. Preparing for the manual testing
Each of the devices being tested had to have a different zone with each test. The largest available city was selected from each time zone, and the times were changed based on that.
Changing the time zone on a specific device is not difficult, you just need to know where to look. I will show how to do it on macOS, iPhone, and Android devices (in this example Sony Xperia).
First device: MacBook
To change the time zone and Date & Time we have to press the date in the upper right-hand corner. Then press “Open Date & Time Preferences” and then go through 4 steps:
1. Unlock the possibility of changes
2. Turn off automatic date updating
3. After going to the Time Zone tab, turn off the ability to automatically load locations
4. Choose the location around the city or by clicking on the map
Second device: iPhone
On the iPhone we have to go to Settings -> General -> Date & Time and disable the option “Set Automatically”. Now we can change the time zone by entering the city name and change the date and time.
It is important that when you want to test the application on iOS, you need to first download the applications on the phone (from Xcode). Then change the date/time zone because in the reverse order you will not be able to upload the application in the future date (on Android there is no such dependence).
Third device: Sony Xperia
On a Sony Xperia phone (there are no major differences on other Android devices) we go to Settings. Then we go to Date & Time and disable the automatic date and time zone settings.
After preparing the test environment and time zones, we could then start on the test cases. After going through the simplest cases of playing advertisements at a given date and time, we also had to check whether everything was working properly in any given summer and winter time.
A good way to do this was to check the wintertime by setting the date in December or November. Because in October the time changes around the world, but it is worth giving yourself a safety buffer. And then test summertime at the end of May.
It’s also worth checking testing zones at the end of the day. This is to check the disappearance of the banner at 23:59 and its appearance at 00:01 the next day. The disappearance of the banner and appearance in two weeks, only on certain days or vice versa was also checked.
Some of the more interesting issues that came up during the whole manual testing process were:
- when we changed the orientation, the banner did not appear.
- Another thing was that when a given advertisement was fired by the user, the hour was not read in real-time. For example the banner was to appear at 17:00 and the user started displaying the advertisement at 16:59, the banner did not appear.
Thanks for reading the whole article! As (I hope) I have shown you, manual testing of time zones can be quite simple and interesting. You just have to make sure you clarify the most appropriate approach from the beginning.
In this case, after the feature went into production, we didn’t have to test it any further. If, of course, the tests were supposed to be repeated regularly, it would be necessary to devise a strategy for writing automated tests. A little spoiler alert… We were able to devise the test approach and do the testing itself in just two days! It was a lot of hard work, but the outcomes were totally worth it.
Kubernetes is not a novelty anymore – but somehow, its potential is still far from being exploited. It started being adopted on a massive scale only recently, and is now in production at 78% of enterprises surveyed by The Cloud Native Computing Foundation (CNCF). Before that, however, Kubernetes didn’t really get the attention it deserves.
If you’re still sceptical about Kubernetes for some reason, it might be the right time to change your mind and join these change-makers, instead of struggling to understand what this platform is all about. To make it easier for you: Here’s what Kubernetes is, along with the reasons why the tech world is going crazy about it (and why you should, too).Read more
JVM creators designed it with automatic memory management in mind, which means programmers don’t need to worry about memory allocation and memory. Unused objects can be released automatically in a transparent way, which is really convenient, especially when you’re new to JVM. But even in general, there’s less code to write and it’s less error-prone than the traditional approach which requires you to do everything manually.Read more
At Scalac we always try to use appropriate technology to meet the requirements given by the business. We’re not afraid to change them once we realize they’re not fit for purpose. By making these changes we learn, we grow, we evolve. In this article, we want to share what we learned during the migration from Apache Spark + Apache Cassandra into Apache Druid.
Why did we decide to do it?
What you can expect after the change?
How did this change reduce costs and simplify our solutions and why did we fail to deliver expectations with the first architecture?
Let’s find out together!
The Law of Expectation
Defining expectations is a crucial part of every architecture, application, module or feature development. The expectations that were presented to us were as follows.
We need to process and deliver data to the UI in near real-time.
Since our project is based on Blockchain, we want to react to all of the changes as soon as they are made. This can be achieved only in real-time architecture. This is one of the most important decisions to be taken at the beginning of the project because it limits the range of technologies we can take with us.
We need to be able to display aggregated data with different time granularities.
To improve performance and readability on the UI, we want to return aggregated data. This should make displaying the data more responsive and lightweight. The solution should be flexible enough to allow us to configure multiple time-ranges because they are not yet defined and may depend on the performance or layer findings.
We need to introduce a retention policy.
Since the data after a specific time are meaningless for us, we do not want to keep them forever and we want to discard them after 48 hours.
We want to deploy our solution on the cloud.
The cloud is the way-to-go for most companies, especially at the beginning of a project. It allows us to scale at any time and pay for the resources we are using. Often just a small number of developers start a project, so there is no time to focus on machine management. The entire focus should be on something that brings value to the business.
We want to be able to display data about wallets and transactions.
Since this project is all about Blockchain, we want to display some information on different levels. The most detailed is at the transaction level, where you can see data exchange and payments between different buyers and sellers. Wallet groups multiply transactions for the same buyer or seller. This allows us to observe the changes in the Blockchain World.
We want to display TOP N for specific dimensions.
In order to display wallets that have the biggest participation in day-by-day payments in Blockchain, we need to be able to create tables that will contain wallets sorted by the amounts of data changes or payments during a certain time.
Let’s take Spark along with us!
Since the road for creating real-time architecture was a bit dark for us at the time, we decided to take Apache Spark along with us! Nowadays, Spark is advertised as a tool that can do everything with data. We thought it would work for us… but we couldn’t see what was coming. It was dark, remember?
Initially, we came up with the above architecture. There are a couple of modules that are important, but they are not crucial for further discussion.
- Collector – This was responsible for gathering the data from different APIs such as Tezos and Ethereum, unifying the format between these two and pushing these changes to Apache Kafka in the form of JSON. Additionally, since we were dealing with streaming, we needed to know where we were, so we would be able to resume from a specific place so not to read the same data twice. That’s why Collector saves the number of processed blocks to the database – Apache Cassandra in our case. We had it in place, we simply used it for a slightly different purpose.
- API – This was a regular API, simply exposing a couple of resources to return results from Cassandra to our UI. It is important to know that it was not only the window for the data in Cassandra, in this API, we could also create users, roles and write them back to the database.
It’s always important to ask yourself why you are choosing these technologies, what you want them to do for you. Here are the motivations for the tools that we decided to use:
Why did we decide to use Apache Kafka?
At the very beginning, we wanted to create a buffer in case Spark would be too slow to process these data, to not lose them. Additionally, we wanted to separate two different tools from each other. Direct integration between Data Source and Spark might be hard and deployment of any of these components could break the entire pipeline. With Apache Kafka between them, it’s unnoticeable. Since the entire solution was designed to work with Blockchain, we knew that we couldn’t lose any transactions. Thus Kafka was a natural choice in this place. Additionally, to fulfill also the requirement for different time-series to be handled. We knew that we could read the same data, from the same topic by multiple consumers. Apache Kafka is very popular, with good performance and documentation.
Why did we decide to use Spark Streaming?
Spark Streaming was a good choice in our opinion due to the fact that it has good integration with Apache Kafka, it supports Streaming (in the form of micro-batches), it has good integration with Apache Cassandra and aggregating by a specific window is easy. Spark Streaming stores the results of the computation in the internal state, so we could configure it to precompute TOP N wallets by the transactions there. Scaling in Spark is not a problem. Spark supports multiple modes for emitting events (OutputMode) and for writing operations (SaveMode). We thought that
SaveMode.Append would be a good option for us.
Why did we decide to use Apache Cassandra?
We decided to use Apache Cassandra mostly due to fast reads and a good integration with Spark. Additionally, it’s quite easy to scale Cassandra as well. Cassandra is also good if we want to store time-series data. SQL should not be too hard for this , in our case data should be already prepared. Cassandra requires more on the writing side than reading.
1. First bump. Stream Handling in Spark.
At the beginning, we didn’t realise that one of the problems would be that Spark can read/write only from/to one stream at a time. This meant that each time we wanted to add another time dimension we would need to spawn another Spark Job. This problem had an impact on the code because it had to be generic enough to handle multiple dimensions by its configuration. Adding more and more Jobs always increases the complexity of the entire solution.
2. Second bump. Cluster requires more and more resources.
Along with the generalization of Spark Job, we had to change our infrastructure. We were using Kubernetes for this, but the problem was that with Spark we could not assign less than 1 core even though Kubernetes allows us to assign 0.5 CPU. This led to poor core utilization of Kubernetes Cluster. We were forced multiple times to increase the resources within the Cluster to run new components. At some point, we were using almost 49 cores!
3. Third bump. Spark’s OutputMode.
We didn’t have to wait too long for the first problems. That was actually a good thing! We realized that
OutputMode.Append was not going to work for us. You may ask yourself: why? Glad you asked! We’re happy to explain.
We need to be able to ask Apache Cassandra for the TOP N results. In our case, it was the TOP 10 wallets by the number of transactions made by them.
Firstly, let’s think about the table structure:
CREATE TABLE IF NOT EXISTS wallet ( wallet_id TEXT, window TEXT, window_ts TIMESTAMP, amount DOUBLE, PRIMARY KEY ((wallet_id, window, window_ts), amount) ) WITH CLUSTERING ORDER BY (amount DESC);
Secondly, let’s think about the query:
SELECT * FROM wallets WHERE window = '10m’ AND window_ts = '2020-01-01 10:00:00.000' LIMIT 10;
Due to the definition of primary key +
WITH CLUSTERING ORDER BY, the data will already be sorted by
Thirdly, let’s insert some data into it:
INSERT INTO wallets (wallet_id, window, window_ts, amount) VALUES ('WALLET_A', '10m', '2020-01-01 10:00:00.000', 1000); INSERT INTO wallets (wallet_id, window, window_ts, amount) VALUES ('WALLET_B', '10m', '2020-01-01 10:00:00.000', 1020); INSERT INTO wallets (wallet_id, window, window_ts, amount) VALUES ('WALLET_C', '10m', '2020-01-01 10:00:00.000', 1050); INSERT INTO wallets (wallet_id, window, window_ts, amount) VALUES ('WALLET_D', '10m', '2020-01-01 10:00:00.000', 800); INSERT INTO wallets (wallet_id, window, window_ts, amount) VALUES ('WALLET_B', '1m', '2020-01-01 10:00:00.000', 500);
Now imagine that Spark is grouping the data for a 10-minute window and it’s going to emit an event once the internal state changes. Only the change itself will be emitted.
The new transaction is coming for the wallet:
2020-01-01 10:03:00.000 and amount:
After that, Spark is going to emit an event and ask Cassandra to insert:
INSERT INTO wallets (wallet_id, window, window_ts, amount) VALUES ('WALLET_A', '10m', '2020-01-01 10:00:00.000', 1500);
Normally, what we would expect is that the first row from the data we inserted would be updated, but it wasn’t because the ‘amount’ which changed, is part of the primary key. Why is this important? Because with insert operation instead of update we will have multiple rows for the same wallet. We cannot change the primary key, because we want to sort the data by the amount and so the circle closes.
Can OutputMode.Complete push us forward?
Since the road with
OutputMode.Append was closed for us, we decided to change the output mode to Complete. As the documentation says, the entire state from Spark will be periodically sent to Cassandra. This would, of course, put more pressure on Cassandra, but at the same time, we would ensure that we would insert no duplicates into the database. In the beginning, everything worked well, but after some time, we noticed that with each insert, Spark was putting in more and more rows.
Let’s turn back. Spark + Cassandra does not meet our expectations.
Why did we decide to turn back and change the current architecture? There were two main reasons.
Firstly, we investigated why Spark was adding only new rows and it turned out that Watermark which is configured in Spark does not discard the data when they should be. This means that Spark would collect the data indefinitely in its internal state. Obviously, this was something that we could not live with and that was one of the most important reasons to change Spark + Cassandra. Spark could not produce the results for one of the most important queries, so we had to turn back.
Secondly, we noticed that maintenance was getting more complicated with each time-series dimension and the configuration for resources for Spark Cluster was not flexible enough for us. This led to too high costs for the entire solution.
The New Beginning, with Apache Druid.
You may ask yourself why we decided to choose Apache Druid? As it turned out, when you are deep in the forest, a Druid is the only creature that can navigate you through it.
At first glance the change is not that significant, but when you read the motivation, you will see how much good this change did for us.
- Firstly, Apache Druid allows us to explore the data, transform them and filter what we do not want to store. It contains databases as well, so we can configure the data retention policy and we can use SQL to query against collected data. This means that Druid provides a complete solution for processing and storing the data, whereas in the previous architecture we had been using Apache Spark + Apache Cassandra. By using this tool, we could delete two components from our initial architecture and replace it with a single one, which reduced the complexity of the entire solution. Additionally, it gave us a single place to configure it all. As a trade-off, this solution is not as flexible as Apache Spark + ApacheCassandra may be. A lack of flexibility can be noticed mostly at the data transformation + data filtering more than in the storage. Even though Druid provides simple mechanisms to do this, it’s not going to be as flexible as writing code. Since we mostly aggregate by time, this is a price we can pay.
- Secondly, the configuration of Apache Druid on top of Kubernetes is much better. Druid contains multiple separated applications that handle different parts of the processing. This allows us to configure resources at multiple levels, so we can use the resources we have more efficiently.
- Thirdly, in Apache Druid, we can use the same data source multiple times to create different tables. Since we used Apache Druid mostly to fulfill the TOP N query, we re-used the same topic in Kafka twice. With and without rollup. Since the smallest time granularity that we have is 1 minute, this is how rollup was configured. This speeds up the query causes that the data are already pre-aggregated. This may have a huge impact when the rollup ratio is high. In cases when it’s pretty low, we should disable it.
- Fourthly, since there is a UI in Apache Druid, you do not need to write any code to make it work. This is different from what we had to do in Apache Spark. At the end of the configuration process in Apache Druid, we got JSON, which can be persisted and versioned somewhere in an external repository. Note that some of the configurations (such as retention policies) are versioned by Apache Druid automatically.
- Fifthly, Apache Cassandra is not the best database to store data about users, permissions, mostly website management data. This is due to the fact that Cassandra is made for fast reads, and operations of this kind can have a negative performance impact. To avoid this, these operations are not supported by design. This is why relational databases are better for this kind of data and we decided to introduce PostgreSQL for it.
What did Druid tell us?
Most likely, you don’t want to run Druid in local mode.
By default, Druid is configured to run only one task at a time (in local mode). This is problematic, when you have more data sources to be processed, essentially, creating more than one table in Druid. We had to change druid.indexer.runner.type to “remote” to make it work as expected. When you have only one task running and multiple pending, make sure that you have this option changed to ‘remote’. You can read about that here.
The historical Process needs to see segments.
We encountered an issue with the Historical Process, which did not notice the segments (files) created by Middle Manager. This happened because we did not configure proper storage for the segments. Druid works in a way that the segments need to be accessible via Historical Service and Middle Manager because only the ownership is transferred from Middle Manager to Historical Service. That is why we decided to use storage from the Google Cloud Platform. You can read more about deep storage here.
What resources did we burn?
Apache Spark + Apache Cassandra
As you can see, we managed to drastically decrease the number of cores used to run our infrastructure as well as memory and storage. The biggest reason why this was possible was that we had multiple Spark Jobs running to do the same task that the single Apache Druid can do. The resources needed for the initial architecture had to be multiplied by the number of jobs running, which in our case was 24.
To sum up what we learned during this venture.
Apache Spark is not a silver bullet.
Spark allows us to do many, many things with the data. Beginning with data reading from many formats, cleaning data, transforming them, aggregating them and finally writing to multiple formats. It has modules for machine learning and graph processing. But Spark does not solve all the problems! As you can see, in our case it was impossible to produce results that could fulfill the requirements of aTOP N query.
Additionally, you also need to keep in mind that the architecture of your code, the generalization, and the complexity of it will significantly affect the resources needed to process the data. Due to the fact that we had 24 jobs to be executed, we needed a huge amount of resources to make it work.
Apache Spark is made for big-data.
When you have many gigabytes of data to be processed, you could consider Spark. One rule says that you should consider using Spark when the biggest machine does not allow you to process the data on it. Keep in mind that processing data on a single machine will be always better than doing it on multiple machines (I’m not saying faster, but sometimes this can be also seen). Nowadays everyone wants to use Spark, even if there are just a couple of megabytes of data. Bear in mind, choosing the right tool for the right job is half of the success.
There is always a tradeoff.
There are no silver bullets in the Software Development World, Spark is not one of them as you have seen above. There is always a tradeoff.
You can drive faster, but you will burn more fuel. You can write your code faster, but the quality will be poorer. You can be more flexible, generalize your code better, but it will become harder to read, extend, debug. Finally, you can pick a database that allows you to read fast, but you will need to work more on the structure of this database and writing into it will be slower.
You need to find the right balance and see if you can cope with the drawbacks that a specific technology brings along with its positives.
Use the right tool for the right job.
Good research is the key to making sure that the appropriate technology has been chosen. You need to remember that sometimes it is better to reserve a bit more time for investigation, writing a Proof of Concept before diving hard into one or other technologies. You will definitely end up wasting time having to rewrite your solution. Obviously, this depends on where you spot the first obstacles. In our example, you can see that firstly our infrastructure was quite complicated and required a lot of resources. We managed to change it dramatically by using a better tool which fitted the purpose better.
How it looks now. A working solution.
Even though this article sounds a bit theoretical, the project we took this experience from is real and still in progress.
The Analytical Dashboard that we created to visualize the data from different blockchains are based on React with Hooks, Redux, Saga on the front side, and Node.js, Apache Druid, Apache Kafka on the back-end.
On the day of writing this article, we are planning to re-write our data source component from Node.js to Scala + ZIO. So you can expect some updates, either in the form of a blog post or an open-source project at GitHub. We have a couple of Blockchains integrated, and we are planning to add more. We are also planning to make the API public so that everyone can experiment and learn from it just as we did.
You can check our Analytical Dashboard out right now at https://www.flowblock.io
Check out also our websites on: