Flyte Cookbook

Flytesnacks is intended to be a learn by example style cookbook. We cover all the various concepts in Flyte along with most of the maintained plugins. This cookbook is designed to get you running both locally, and on a Flyte cluster using Flyte Python SDK (flytekit)

All the examples in this book are executable locally and we recommend the best way to try out Flyte is to run these examples in your terminal or IDE. If you have access to a whole Flyte platform deployment, this cookbook can also be used as a fully functional workflow repo.

The tutorial is divided into 3 sections:

• Core Language Examples - This covers the basics of Flytekit, as well as working with a deployed Flyte platform.

• Case Studies - These are examples that showcase the power of Flyte.

• Plugins Examples - This section details flytekit extensibility

For more details on Flyte refer to the Homepage. For documentation on how to install, configure, administer Flyte, please refer to the platform documentation

Please report bugs, suggestions, or comments by creating a GitHub issue or just by opening a PR against this or any Flyte repo. It is our hope new contributors find the community welcoming!

Please join our Slack channel as well through this form.

Getting Started

• Tutorial: Hello World!
  • Prerequisites
  • Steps
• Getting Started with Flytekit [python]
  • Flyte Type-system
  • Mapping of Types between Python and Flyte - using Flytekit
• Basic Examples: Tasks, Workflows, Files etc
• Intermediate Examples: Custom Objects, Schema, Conditions, Dynamic, SubWorkflows etc
• Advanced Examples: Extend Flytekit, combine techniques etc
• Flyte cluster execution - setup & features
  • Build your Dockerfile
  • Serialize your workflows and tasks
    • In-container serialization
  • Register your Workflows and Tasks
  • Fast(er) iteration
  • Building Images
  • Some concepts available remote only

Case Studies

• Case Studies: Real world examples of using flytekit
  • PIMA Indians diabetes prediction using XGBoost
    • Why a Workflow?
    • Steps of the Pipeline
    • Takeaways
    • Walkthrough

Flytekit Plugins

• Flyte Plugins
  • What are Flytekit [python] only plugins?
  • What are Backend Plugins?
    • Native Backend Plugins
    • External Service Backend Plugins
• Executing K8s Pods
  • Installation
• Executing Spark Jobs natively on K8s Cluster
  • Pre-Requisites / Setup
  • How to build your Dockerfile for Spark on K8s
• Executing Jupyter Notebooks as tasks
  • Salient points
  • Installation
• Executing Hive Queries
  • Installation
  • No Need of a dockerfile
  • Configuring the backend to get hive working
• AWS Sagemaker Training
  • Installation
  • Builtin Algorithms
  • Training a custom model
  • Prerequisites
    • Creating a dockerfile for Sagemaker custom training [Required]
• Executing Distributed Pytorch training jobs on K8s
  • Installation
  • How to build your Dockerfile for Pytorch on K8s
• AWS Sagemaker distributed training using PyTorch
  • Installation
  • Creating a dockerfile for Sagemaker custom training [Required]

Additional Flyte Documentation

• Flyte Project
• Flytekit Python

Indices and tables

• Index

• Module Index

• Search Page