Hi I’m Chris Millon! I’ve been a Server Engineer on F1 Clash at Hutch for 2 years. I previously worked in the Data Domain, FinTech industries and finally landed here, in mobile gaming. Today, I'm here to chat about how we've been using KEDA to help with calculating our drop rates. Hope you find it useful!

We’ve recently adopted KEDA (aka Kubernetes Event-Driven Autoscaling) to tackle crate drop rate calculations. This technology felt more desirable than, for example, Azure Function(s) because it integrates seamlessly with our current infrastructure, on top of Kubernetes.

KEDA scales any deployment based on “Scalers” that can vary based on a SQL query, the length of a queue or any other custom metric.

The problem

In F1 Clash there’s around 2000 crates, each crate has to be opened 10000 times (our desired sample size), a quick benchmark revealed a 1 CPU pod performs this task for a single crate in ~11.76 seconds. If we were to run all those calculations on a single machine, this would take 2000 * 11.76 seconds = ~06:32 hours which is too long for our taste, what about vertically scaling? We’d quickly be limited by how many CPUs are available on a single node, so we need to distribute this task across multiple machines.

Another consideration is those need to be re-calculated upon crate updates or crate opening algorithm changes, when this isn’t happening we don’t want any “Crate drop rate calculation services” running so as to not waste resources.

The solution

The task can be broken down and sent into a queue, where each crate to be processed represents a single message, then gets picked up by available crate drop rate calculation services, but there’s none running right now?!

That’s when KEDA comes in and checks for any queue messages and kicks off scaling:

The amount of desired replicas can be set based on how many messages exist or spawn a single instance to let Kubernetes’ Horizontal Pod Autoscaling do the job; we opted for the latter.

There is no work left anymore, we’re back to where we started:

Conclusion

KEDA gives Kubernetes “Serverless functions” capabilities, the initial setup uses resources because it still needs an operator running, eventually the more you use it, the more it pays off. We were able to distribute our crate drop rate calculations across multiple nodes and scaling up or down based on the number of queue messages. This approach not only saved us time, but also ensured that we were not wasting any resources when there were no queue messages. Overall, KEDA has been a valuable addition to our infrastructure and we are looking forward to exploring more use cases for it in the future.

Look out for more tech blog posts coming soon! Or bookmark the tech blog here: hutch.io/blog/tech/

And for our latest tech job vacancies at Hutch, head here: hutch.io/careers.