Outbrain's CTR model is a key technology in our product that generates hundreds of millions of predictions per second. In order to achieve this, we developed our own high-speed ML framework called "Fwumious Wabbit" in Rust. However, using this framework posed a new challenge as our serving stack is written in Java. We faced the question of how to efficiently transfer predictions from Rust to Java services. After initially using a local TCP server on the same container as the Rust module and after a failed attempt of using JNI, we decided to try Foreign Functions and Memory API. This change resulted in a more than 20% CPU usage decrease and a 20% reduction in average Latency.
In this talk, I wish to share the journey we had and dive a bit into Foreign Functions and Memory API that was presented as part of Project Panama