Quarkus vs Phoenix

A benchmark comparing two frameworks

At my job, I am evaluating some microservices platform options. A good way to test them is doing some benchmarks. Everybody likes benchmarks. So, why not?

Disclaimer: although I tried to make conditions fair and “cientific”, it is an amateur benchmark and I am making this for fun. Please, do not take it too seriously.

Why Quarkus?

It is the “supersonic subatomic Java” framework, sponsored by Red Hat. My work place is a Red Hat shop and we have many subscriptions from them. So, Quarkus is not only a good fit, but we already use it on some projects. It promises a better startup time and memory footprint. Tested version: 1.6.1.

Why Phoenix?

Well, personal preference. I like Elixir, it is a powerful and fun language. When you discover Erlang and what problems it was made to solve, you will find it amazing. WhatsApp, Discord and many others uses it. The framework of choice for the Elixir language is Phoenix. Achieving 2 million connections in a single machine is really impressive too. Tested version: 1.5.4.

The benchmark

I couldn’t find any real benchmarks on these technologies. There is this benchmark on Netty and Cowboy, which are the technologies behind these frameworks, but using a “sleep” to simulate load is not ok. These frameworks handle databases and connection pools differently. Acquiring a database connection from a pool can timeout and throw an exception. A “sleep” command won’t timeout. Serializing data to JSON also imposes memory and CPU usage.

Techempower does something different, but it uses good hardware and low load. Its focus is on how many requests per second a framework can handle, and this is not ok too. I need a framework that can handle lots of requests within a constrained hardware, like a Heroku dyno. So, speed isn’t a concern at this point.

In this benchmark, I decided to use my work development database and data. It’s an Oracle database and my test data is on a table with 5 columns and 6000 rows. Yes, it is a lot of JSON, so it can force some timeouts, CPU and memory usage. Although Oracle JDBC driver is well tested and supported, in Phoenix we must rely on jamdb_oracle package. It works well and it is maintained, but it’s not “official”. Oracle database driver is not a first class citizen on Quarkus either, and we can not compile it to a native image using GraalVM out of the box.

We are going to use the siege tool. Quarkus will run on OpenJDK 11 by Red Hat and Phoenix on Erlang OTP 22, both on a Red Hat Linux 8 with 4 cores and 8 GB of RAM. But, we are pretending to be inside a Heroku dyno or docker container, so the memory limit will be 512 MB. On Heroku or docker, any memory allocation beyond this will use disk swap, reducing performance. Keep that in mind!

Comparing Apples to Oranges

Before we start, we have to notice something. We can not compare these frameworks as they come out of the box. To compare apples to apples (and not to oranges) and make things fair, we need to configure both frameworks with the same connection pool size and timeout. In Phoenix, the default connection timeout and pool size are very low, so it can fail fast and use less memory. In Quarkus, the connection timeout is longer and pool size bigger, so it can serve more requests at the cost of memory. Well, that is what I think, but I might be wrong. For this test, we had to impose some limits to the JVM, as suggested by Heroku:

java -Xss512k -Xmx300m -jar <my-app>.jar

We will use siege to hit the REST endpoints with 25, 50, 100 e 250 concurrent connections, 5 times each:

siege -d1 -r5 -c250 http://<my-server>/<my-resource>

CPU

Winner: Quarkus

Both frameworks have a similar CPU usage profile and “warmup” peak. Quarkus and JVM peaked at 56% of CPU usage while Phoenix and Erlang VM peaked at 79% of CPU usage. OpenJDK uses less CPU over time and is the clear winner.

Memory

Winner: Phoenix (x2)

Phoenix peaked at 364 MB while Quarkus peaked at 807 MB. Here we can see the problem on using Java for microservices. Java refuses to stay within the 512 MB limit. Actually, it uses more than twice the memory Phoenix used. As stated above, at Heroku or docker with a 512 MB of RAM limit, any memory beyond that will use disk swap, reducing performance. In this benchmark, I could not simulate this, because I had memory available.

Why Phoenix won 2 times? I could not show it on the graph, but another thing to notice is that Erlang/Elixir gives back unused memory to the operating system. It peaked at 364 MB, but after the test, it was using only 152 MB. OpenJDK, on the other hand, never return unused memory back to the operating system, it peaked at 807 MB and stayed there. So, Quarkus losed twice here. If you run your own cloud, like me, this is an important and desired behaviour.

Speed

Winner: Quarkus

Quarkus took, at most, 34 secs to respond all requests. Phoenix took 60 secs. Not much to argue here. Quarkus and JVM are faster. And in this benchmark, they are almost twice as fast than Phoenix. Quarkus is the clear winner. If your microservice have any speed requirement, Quarkus is a better fit than Phoenix.

Timeouts

Winner: Draw

What? Why a draw? Well, while handling 250 simultaneous connections, Quarkus timed out 110 times and Phoenix timed out 155 times. Quarkus should be the winner. The problem is the difference is not so big. As stated above, Quarkus used more memory than it was available. If it was running at Heroku or docker, it would be using disk swap, having perfomance penalties and leading to more timeouts. They also start timing out at the same number of connections, indicating overload. I could not simulate Quarkus using disk swap, so it had this “advantage”. I will consider it a draw.

Conclusion

As I said above, I was looking for a framework that could handle lots of requests using less memory. If you are like me and need to deploy your microservices on premise hardware or on Heroku, memory usage comes first. And the ability to return unused memory back to the cluster is not only desirable, it is a must have. Requirements like speed and CPU usage comes after. Phoenix is the clear winner under this circuntances.

Quarkus, on the other hand, shines where speed is a requirement and memory consumption is acceptable. The cool thing about a microservices archtecture is that you can use the best tool for the job. So, use both frameworks. They are new, comparable and capable.