Senior Software Engineer at Singular. Playing a key role in Singular's Analytics Infrastructure team, with experience in both realtime and batch processing large-scale data pipelines.
Cutting the Right Corners: Handling High Cardinality by Understanding Your Data
Handling high cardinality with big data can be challenging. We improved our pipeline speed and stability by understanding which data matters more and creating a smart “Cardinality Protector” to reduce cardinality with minimal effect on the data.