PyCon Israel 2022

Rapid development of complex and varied algorithms requires tight and agile management. This talk will demonstrate the power of DAGs (Direct Acyclic Graphs) to enable a flexible algorithm development process with high quality and minimal risk at each stage in a continuous delivery ecosystem. I will show how Python concepts and libraries are used in tandem with DAG architecture to manage independent, atomic algorithmic units with unique characteristics — and at the same time to easily create complex flows through flexible combinational integrations of algorithmic building blocks.
Python-based DAG is central in creating a strong yet open and simple infrastructure for our SW architecture, that supports incremental deliveries for the development of new high-quality algorithmic solutions according to changing requirements.
The semiconductor industry is a dynamic field experiencing rapid growth, requiring the continuous delivery of algorithmic solutions to diverse challenges. While these solutions are inherently complex, they need to be developed quickly and agilely.
Our DAG-based SW architecture is perfect for meeting these needs.
DAG enables us to develop and enrich our application in both new algorithm units — independent Python packages with their unique parameters, tests etc. — and new algorithmic flows, while improving reuse and throughput without interfering with parallel development.
Moreover, DAG provides the flexibility to update any existing algorithmic unit with new requirements, according to changing customer challenges. This is optimally done through the combination of DAG and Python, which inherently enables algorithmic building blocks to act as independent packages and yet remain accessible to debugging through the applications that consume it. And since Python is object-based, we leverage it within the DAG architecture to enhance building blocks with classes, methods, attributes and more.
Each node in the graph activates an independent algorithmic unit and delivers all required inputs and parameters. DAG combines these units to generate different flows for different cases and requirements. Over time, each graph section can be developed, replaced or updated with minimal effect on other algorithmic units and flow.
We will learn that:
1. Python concepts and libraries (like Tickle and Dask) are the perfect drivers for a DAG-based SW architecture.
2. DAG as a SW architecture enables a flexible development process with high quality and minimal risk at each stage.
3. DAG as a SW architecture enables the development of a variety of different algorithms that are both independent and collaborative, thereby supporting reuse and saving TPT.