To drive its growing services, Gojek looks to Flyte™
The company
Gojek is an on-demand, multiservice platform and digital payment technology group based in Jakarta, Indonesia. It has 20+ services ranging from food, transportation to payments and entertainment. In 2021, the Gojek ecosystem contributed 1.7% of the Indonesian GDP (which is in the world’s top 20 GDPs). That means scalability and reliability are of critical importance.
To offer its services consistently at scale, Gojek employs machine learning to handle complex tasks such as generating pickup points, matching riders and drivers, allocating vouchers and optimizing prices. Back in 2017, When the company started ML development in 2017, it started building ML pipelines using an abstraction layer on top of Clockwork and Airflow, which today hosts more than a thousand DAGs.
The challenge
The abstraction was necessary to run pipelines in separate containers to handle dependency management. However, abstraction wasn’t a smooth ride: Restricting orchestration to running containers made them difficult to test on local machines and presented a formidable learning curve to data scientists inexperienced in building containers. What’s more, the scheduler didn’t always scale well, sometimes missed DAGs and imposed significant infrastructure overhead.
Bottom line: The platform “probably wouldn’t get us through the next five years,” said Pradithya Aria Pura from Gojek’s Data Science Platform team. The team wanted a scalable, data-aware orchestrator that was Kubernetes-native and provided a first-class development experience.
The solution
After comparing several leading tools in the market, the Gojek team chose Flyte, the orchestration engine that powers Union Cloud. “Flyte is fast and scalable, Pura said.
“During our evaluation stage, we did stress testing to understand whether Flyte can satisfy our requirements, and it provided us with good results.”
“Workflow versioning is quite important,” he continued. “When productionizing a pipeline, there are only a few platforms that provide this kind of versioning, and to us, it’s critical where we want to roll back to a specific workflow version in case there’s a bug introduced in the pipeline.”
“Flytekit (Python SDK) provides a powerful and expressive way of building pipelines. So far, we love the dynamic workflow and flow control.”
The results
Over the next six months, Pura said, Gojek will deprecate its legacy ML pipeline tool. Meanwhile, the company is growing its workflows with Flyte: “Optimistically, we will have around 100-200 new workflows added to Flyte per month,” Pura said.
Pura said Gojek is already seeing cost savings thanks to the switch. “From several pipelines we migrated to Flyte from our legacy tools, we see around 20% to 80% cost saving.”