Why Blackshark.ai chose Flyte™ to support earth’s ‘Digital Twin’
The company
Blackshark.ai is balancing the world on its shoulders. The Austrian startup uses AI models and satellite imagery to generate real-time, accurate, searchable and photorealistic digital 3D maps of the entire planet.
To get there, Blackshark extracts insights of the earth’s infrastructure from current satellite and aerial imagery using machine learning, and fills in missing attributes using AI for a photorealistic, geo-typical or asset-specific digital twin. (In one use case, Blackshark.ai enabled Microsoft’s Flight Simulator team to construct over 1.5 billion photorealistic 3D buildings, giving users an unprecedented immersive 3D flight experience.)
The work requires enormous amounts of data and staggering computing power. “Satellite imagery for the whole world consumes about 2.5 petabytes of data,” Flyte Python developer Maarten de Jong explained. “Processing that takes a lot of computation power, and from there on out we produce about 1.5 billion building footprints that are run on hundreds of machines in parallel.”
The challenge
From the start of Blackshark’s trajectory, the internal team was solely responsible for handling the infrastructure, code and resources to crunch all that data. Now scale presented fresh challenges: New models were being trained, people were doing explorations with novel techniques, and it all had to happen in parallel, which henceforth, posed challenges to Blackshark.
“At a certain point, your company grows from 30 to 100 employees, and there are few people who know how to work with the platform,” de Jong said. “It's just not scalable anymore.”
The solution
To get ahead of its growing data demands, Blackshark turned to Flyte, the orchestration platform that powers
Union Cloud. Today, Flyte runs the AI detection and content generation pieces of the Blackshark pipeline, effectively handling all the data the company can throw at it.
“The reason we’re mainly using Flyte is that it has cloud-native capabilities,” de Jong said. “We do everything in the cloud, and we don’t want to be limited to a single cloud provider, so having the ability to run everything through Kubernetes is amazing for us.”
The results
According to de Jong, Flyte has supported Blackshark’s linear growth in data volume and workflow runs. “We're dealing with a lot of data,” he said, “and we're mostly trying to optimize how we can avoid having our data volume increase too much by smart caching and task merging within Flyte.
That also means Flyte will support new clients with requests for additional calculations tailored to their specific projects.
“We're seeing market pull in that direction and are preparing for this,” de Jong said. “Having Flyte makes this really easy for us, since we can easily register different versions of workflows that include client-specific pre- or post-processing steps.”
And when it comes to the bottom line, Flyte “has made it much easier to avoid over-provisioning hardware,” de Jong said. “We especially like the ability to run additional clusters with worker nodes. We use the latter to integrate Flyte with a dedicated compute cluster for heavier machine learning workflows.”