To Drive its Growing Services, Gojek Looks to Flyte™
Gojek is an on-demand multi-service platform and digital payment technology group based in Jakarta, Indonesia. It has 20+ services ranging from food, transportation to payments and entertainment. In 2021, the Gojek ecosystem contributed 1.7% of the Indonesian GDP (which is in the world’s top 20 GDPs). That means scalability and reliability are of critical importance.
Machine Learning forms a big part of the services offered by Gojek. It’s used for several use cases, such as generating pickup points, voucher allocation, optimizing prices, etc. Back in 2017-18, ML pipelines were built using an abstraction on top of Clockwork and Airflow, which today has 1000+ DAGs. The abstraction was necessary to run pipelines in separate containers in order to handle the dependency management. However, the abstraction wasn’t a smooth ride due to the less than ideal development experience, unscalability of scheduler, missing DAGs, and infrastructure overhead.
After some reflection, the Gojek team understood that they wanted a platform that is scalable, Kubernetes-native, a data-aware orchestrator, and provides a first-class development experience, and Flyte™ fit their criteria.
Pradithya Aria Pura from the Data Science Platform team at Gojek presents a detailed walkthrough of the adoption of Flyte™ at Gojek and what they love about Flyte™:
Some noteworthy excerpts from the video:
“… we started evaluations of several tools in the market that’ll satisfy our requirements, and eventually, we decided to use Flyte™. We’re happy with our experience so far with Flyte™.”
“Workflow versioning is quite important. When productionizing a pipeline, there are only a few platforms that provide this kind of versioning, and to us, it’s critical where we want to roll back to a specific workflow version in case there’s a bug introduced in the pipeline.”
“Flytekit (Python SDK) provides a powerful and expressive way of building pipelines. So far, we love the dynamic workflow and flow control.”
“Flyte™ is fast and scalable. During our evaluation stage, we did stress testing to understand whether Flyte™ can satisfy our requirements, and it provided us with good results.”