Contribute tutorials or integrations examples#
The first step to contributing a tutorial or integration example is to open a documentation issue to describe the example you want to write. The Flyte maintainers will help you figure out where your tutorial or integration example would best fit.
Prerequisites
Follow the Environment setup guide to get your development environment ready.
The tutorials and integrations examples live in the examples
directory of the flytesnacks repo, where each subdirectory contains a self-contained example project that covers a particular tutorial or integration. There are also subdirectories that contain the code included in the user guide via the rli
(remoteliteralinclude) directive.
examples
├── README.md
├── airflow_plugin
├── athena_plugin
├── aws_batch_plugin
├── basics
├── bigquery_agent
...
Adding an example script to an existing project#
If you’re adding a new example to an existing project, you can simply create a
new .py
file in the appropriate directory. For example, if you want to add a new
example in the examples/exploratory_data_analysis
project, simply do:
touch examples/exploratory_data_analysis/my_new_example.py
If you are creating a new integration or tutorial example, add the example to the README.md
file of the
example project as an entry in the auto-examples-toc
directive:
```{auto-examples-toc}
...
my_new_example
```
If you are creating a new user guide example, you can reference the code in the user guide documentation using the rli
(remoteliteralinclude) directive. You do not need to add the new example to the README.md
file of the example project.
Creating a new example project#
Important
If you’re creating a new tutorial or integration example that doesn’t fit into any of the existing subdirectories, you’ll need to set up a new example project.
In the flytesnacks
root directory, run the following command to create an example project:
./scripts/create-example-project.sh new_example_project
This will create a new directory under examples
:
examples/new_example_project
├── Dockerfile
├── README.md
├── new_example_project
│ ├── __init__.py
│ └── example.py
└── requirements.in
Creating Python examples#
Tutorial or integration examples#
If you are writing a tutorial or integration example, write your example Python script in percent format,
which allows you to interleave Python code and Markdown in the same file. Each
code cell should be delimited by # %%
, and each Markdown cell should be
delimited with # %% [markdown]
.
# %%
print("Hello World!")
# %% [markdown]
# This is a Markdown cell
# %%
print("This is another code cell")
Markdown cells have access to Sphinx directives through the
MyST Markdown format,
which is a flavor of Markdown that makes it easier to write documentation while
giving you the utilities of Sphinx. flytesnacks
uses the
myst-nb and
jupytext packages to interpret the
Python files as rst-compatible files.
Writing examples: explain what the code does#
Following the literate programming paradigm, make sure to
interleave explanations in the *.py
files containing the code example.
A Simple Example
Here’s a code snippet that defines a function that takes two positional arguments and one keyword argument:
def function(x, y, z=3):
return x + y * z
As you can see, function
adds the two first arguments and multiplies the sum with the third keyword
argument. Can you think of a better name for this function
?
Explanations don’t have to be this detailed for such a simple example, but you can imagine how this makes for a better reading experience for more complicated examples.
Creating examples in other formats#
Writing examples in .py
files is preferred since they are easily tested and
packaged, but flytesnacks
also supports examples written in .ipynb
and
.md
files in MyST Markdown format. This is useful in the following cases:
.ipynb
: When a.py
example needs a companion Jupyter notebook as a task, e.g. to illustrate the use ofNotebookTask
s, or when an example is intended to be run from a notebook..md
: When a piece of documentation doesn’t require testable or packaged flyte tasks/workflows, an example page can be written as a MyST Markdown file.
Note
If you want to add Markdown files to a user guide example project, add them to the flyte repository instead.
Writing a README#
The README.md
file needs to capture the what, why, and how of the example.
What is the tutorial or integration about? Its features, etc.
Why do we need this tutorial or integration? How is it going to benefit Flyte users?
Showcase the uniqueness of the tutorial or integration
Integration plugin installation steps
Finally, for tutorials and integrations only, write an auto-examples-toc
directive at the bottom of the file:
```{auto-examples-toc}
example_01
example_02
example_03
```
Where example_01
, example_02
, and example_03
are the Python module
names of the examples under the new_example_project
directory. These can also
be the names of the .ipynb
or .md
files (but without the file extension).
Tip
Refer to any subdirectory in the examples
directory
Test your code#
If the example code can be run locally, you can use pyflyte run
to test it:
pyflyte run <python_file>.py <task_or_workflow_name> --<arg1> <arg1_value> --<arg2> <arg2_value> ...
Testing on a cluster#
Install flytectl, the command line interface for flyte.
Note
Learn more about installation and configuration of Flytectl here.
Start a Flyte demo cluster with:
flytectl demo start
Testing the basics
project examples on a local demo cluster#
In this example, we’ll build the basics
project.
Change to the basics directory:
# from flytesnacks root directory
cd examples/basics
Build and push the container to the local Docker registry provided by the demo cluster:
docker build . --tag "localhost:30000/basics:v1" -f Dockerfile --push
Run a workflow in the local demo cluster by specifying the --image
flag
and passing the --remote
flag:
pyflyte run --remote \
--image localhost:30000/basics:v1 \
basics/hello_world.py hello_world_wf
Visit https://localhost:30081/console
to view the Flyte console, which consists
of the examples present in the flytesnacks/core
directory.
Updating dependencies#
If the project uses pinned dependencies in a requirements.in
file, run the pip-compile
command to create a new pinned requirements.txt
file:
pip-compile requirements.in --upgrade --verbose --resolver=backtracking
Rebuild the image#
If you’ve updated the source code or dependencies of the project, rebuild the image with:
docker build . --tag "localhost:30000/basics:v2" -f Dockerfile --push
Next, run the workflow again with the new image:
pyflyte run --remote \
--image localhost:30000/basics:v2 \
basics/hello_world.py hello_world_wf
Refer to this guide if the code itself is updated and requirements.txt is the same.
Pre-commit hooks#
We use pre-commit to automate linting and code formatting on every commit. Configured hooks include ruff to ensure newlines are added to the end of files, and there is proper spacing in files.
We run all those hooks in CI, but if you want to run them locally on every commit, run pre-commit install
after
installing the dev environment requirements. In case you want to disable pre-commit
hooks locally, run
pre-commit uninstall
. More info here.
Formatting#
We use ruff to autoformat code. They
are configured as git hooks in pre-commit
. Run make fmt
to format your code.
Spell-checking#
We use codespell to catch common misspellings. Run
make spellcheck
to spell-check the changes.
Update tutorials and integrations examples documentation pages#
The docs/conf.py
file in the flytesnacks repository contains the Sphinx configuration for building the flytesnacks
documentation.
At build time, the flytesnacks
Sphinx build system will convert the tutorials and integrations examples in the examples
directory into docs/flytesnacks/examples
and make them available in the documentation.
Important
The docs build system will convert the README.md
files in each tutorial and example integration project into an index.md
file, allowing you to reference the root page of each
example project in MyST Markdown format.
If you’ve created a new tutorial or integration example project, you’ll need to add its index
page
to the table of contents in docs/tutorials/index.md
or docs/integrations/index.md
and update the appropriate list-table
directive in the file to make sure the example shows up in the documentation:
```{list-table}
:header-rows: 0
:widths: 20 30
* - {doc}`Airflow agent </flytesnacks/examples/airflow_agent/index>`
- Run Airflow jobs in your workflows with the Airflow agent.
...
```
...
```{toc}
/flytesnacks/examples/bigquery_agent/index
```
Build the documentation locally#
Verify that the code and documentation look as expected:
Learn about the documentation tools here
Install the requirements by running
pip install -r docs-requirements.txt
.Run
make -C docs html
Tip
To run a fresh build, run
make -C docs clean html
.Open the HTML pages present in the
docs/_build
directory in the browser withopen docs/_build/index.html
Create a pull request#
Create the pull request in the flytesnacks repository, then ensure that the docs are rendered correctly by clicking on the documentation check: