This workshop will build on the half-day workshop "Building Scalable and Maintainable Data Pipelines with Omnipy (Part 1 - Beginner level) we are holding before lunch.

In this second workshop, participants will learn how to develop various types of data flows in Omnipy, including integration with web services. They will make use of the powerful industry-developed Prefect orchestration engine to scale up the game and deploy high-throughput ETL flows using external compute resources.

The workshop is divided into three parts:

1. The first part will introduce the slogan "parse, don't validate" and show how these concepts are implemented in Omnipy. On this background, we will introduce the three types of data flows supported by Omnipy: linear, DAG, and function flows. We will also, through hands-on examples, show how to make use of various job modifiers to power up and customise predefined tasks and flows to construct more complex data flows.
2. The second part will focus on integrating data flows with web services through REST APIs. We will mainly focus on extracting data from data sources, but will also touch upon loading results onto data sinks. Hands-on examples will introduce tasks and flows that allow flattening of JSON data into relational tabular form for mapping, and then restructuring the results back to JSON.
3. The last part will introduce Omnipy's integration with S3-based cloud storage and the Prefect ETL orchestration library. As a hands-on exercise, the participant will scale up the data flow developed in the second part of the workshop by deploying it on an external compute infrastructure, potentially the Kubernetes-based NIRD Toolkit from SIGMA2 (if Prefect-integration in NIRD is finalised in time for the workshop).