06-28, 15:30–16:20 (Asia/Jerusalem), PyData
How we map continents at cm level accuracy from crowd sourced computer vision data using PySpark.
A tale of engineering challenges working with python at huge scale in production with a rapidly evolving development effort.
REM group in Mobileye is tasked with the challenge of creating and updating a high definition map at world scale with cm level accuracy of all road geometry and semantic elements to enable fully autonomous driving.
The map is constructed from crowd sourced anonymized data of millions of driving-assistance systems running computer vision processes in consumer vehicles.
This is the tale of the engineering challenges building a python based production solution running cutting edge algorithms efficiently on big data, while supporting a rapid pace development environment.
In this tale we will share how we addressed the need to:
- Build maps at huge scale in reasonable time and efficiency
- Enable 100+ developers to continuously evolve the technology at a fast pace, run their code on production loads, view and debug their results
We will discuss challenges of working with PySpark as our major computation engine, and some of the solutions we employed to adderess these challenges.
A peek into engineering in Mobileye REM group.
Hebrew
Target audience –R&D
I have a decade of experience as Algorithm & Software developer and team leader.
I'm working now in Mobileye on the amazing challenge of mapping the entire roads network of the world.
I worked on variety of fields including big data, process optimizations, and anomaly detection.
I love the challenge of optimizing anything in life and naturally thinking of the best way to do things.