PyCon Israel 2024

It’s all about the db - our journey between databases to accommodate 100k events per second
09-16, 13:30–13:50 (Asia/Jerusalem), Main Hall (30)
Language: English

We built a python application that ingests 100k events per second. This lecture will share our lessons learned using different dbs for this job.


Our python based system needs to handle more than 100k events per second. During the last two years we have tried, failed and evolved using multiple databases such as nosql, timeseries db and OLAP columnar dbs.
This lecture will share the lessons learned for running such a system in a production environment


Expected experience level of participants

Intermediate

Target audience

Developers

I am Eyal Balla, a python cloud applications developer and manager for around a decade, I have deployed to production numerous times and have suffered from my mistakes and others and lived to tell.
Today in I am the data team lead in Cynerio, a cyber security company which helps hospitals be a more secure place