2021-05-02, 11:30–11:55, General Track 2
At Bluevine we use Airflow to drive our all "offline" processing. In this talk, I'll present the challenges and opportunities we had by transitioning from servers running Python scripts with cron to a full blown Airflow setup.
In Bluevine, we were looking to upgrade our backend processing infrastructure from servers running Python scripts with Cron to a more scalable solution that allows for workflows (DAGs) and better observability of the application state. Airflow proved to be a valuable tool, though not without some sharp edges. Some of the points that I'll cover are:
- Supporting multiple Python versions
- Event driven DAGs
- Airflow Performance issues and how we circumvented them
- Building Airflow plugins to enhance observability
- Monitoring Airflow using Grafana
- CI for Airflow DAGs (super useful!)
- Patching Airflow scheduler