Contributing to User Guide, Tutorials and Integrations#

Tags: Contribute, Basic

The examples documentation provides an easy way for the community to learn about the rich set of features that Flyte offers, and we are constantly improving them with your help!

Whether you’re a novice or experienced software engineer, data scientist, or machine learning practitioner, all contributions are welcome!

How to Contribute#

The Flyte documentation examples guides are broken up into three types:

User Guides: These are short, simple guides that demonstrate how to use a particular Flyte feature. These examples should be runnable locally.
Tutorials: These are longer, more advanced guides that use multiple Flyte features to solve real-world problems. Tutorials are generally more complex examples that may require extra setup or that can only run on larger clusters.
Integrations: These examples showcase how to use the Flyte plugins that integrate with the broader data and ML ecosystem.

The first step to contributing an example is to open up a documentation issue to articulate the kind of example you want to write. The Flyte maintainers will guide and help you figure out where your example would fit best.

Creating an Example#

Prerequisites

Follow the Environment Setup guide to get your development environment ready.

The flytesnacks repo examples live in the examples directory, where each subdirectory contains a self-contained example project that covers a particular feature, integration, or use case.

examples
├── README.md
├── airflow_plugin
├── athena_plugin
├── aws_batch_plugin
├── basics
├── bigquery_plugin
...

Adding an example script to an existing project#

If you’re adding a new example to an existing project, you can simply create a new .py file in the appropriate directory. For example, if you want to add a new example in the examples/basics project, simply do:

touch examples/basics/my_new_example.py

Once you’re done creating your example, add it to the README.md file of the example project as an entry in the auto-examples-toc directive:

```{auto-examples-toc}
...
my_new_example
```

Creating a new example project#

Important

If you’re creating a new example in the User Guide, Tutorials, or Integrations that doesn’t fit into any of the existing subdirectories, you’ll need to setup a new example project.

In the flytesnacks root directory, create one with:

./scripts/create-example-project new_example_project

This will create a new directory under examples:

examples/new_example_project
├── Dockerfile
├── README.md
├── new_example_project
│   ├── __init__.py
│   └── example.py
├── requirements.in
└── requirements.txt

Creating python examples#

Then, write your example python script in percent format, which allows you to interleave python code and markdown in the same file. Each code cell should be delimited by # %%, and each markdown cell should be delimited with # %% [markdown].

# %%
print("Hello World!")

# %% [markdown]
# This is a markdown cell

# %%
print("This is another code cell")

Markdown cells have access to sphinx directives through the myst markdown format, which is a flavor of markdown that makes it easier to write documentation while giving you the utilities of sphinx. flytesnacks uses the myst-nb and jupytext packages to interpret the python files as rst-compatible files.

Writing examples: explain what the code does#

Following the literate programming paradigm, make sure to interleave explanations in the *.py files containing the code example.

A Simple Example

Here’s a code snippet that defines a function that takes two positional arguments and one keyword argument:

def function(x, y, z=3):
    return x + y * z

As you can see, function adds the two first arguments and multiplies the sum with the third keyword argument. Can you think of a better name for this function?

Explanations don’t have to be this detailed for such a simple example, but you can imagine how this makes for a better reading experience for more complicated examples.

Creating examples in other formats#

Writing examples in .py files is preferred since they are easily tested and packaged, but flytesnacks also supports examples written in .ipynb and .md files in myst markdown format. This is useful in the following cases:

.ipynb: When a .py example needs a companion jupyter notebook as a task, e.g. to illustrate the use of NotebookTasks, or when an example is intended to be run from a notebook.
.md: When a piece of documentation doesn’t require testable or packaged flyte tasks/workflows, an example page can be written as a myst markdown file.

Writing a README#

The README.md file needs to capture the what, why, and how of the example.

What is the integration about? Its features, etc.
Why do we need this integration? How is it going to benefit the Flyte users?
Showcase the uniqueness of the integration
How to install the plugin?

Finally, write a auto-examples-toc directive at the bottom of the file:

```{auto-examples-toc}
example_01
example_02
example_03
```

Where example_01, example_02, and example_03 are the python module names of the examples under the new_example_project directory. These can also be the names of the .ipynb or .md files (but without the file extension).

Tip

Refer to any subdirectory in the examples directory

Test your code#

If the example code can be run locally, just use python <my_file>.py to run it.

Testing on a cluster#

Install flytectl, the commandline interface for flyte.

Note

Learn more about installation and configuration of Flytectl here.

Start a Flyte demo cluster with:

flytectl demo start

Testing the `basics` project examples on a local demo cluster#

In this example, we’ll build the basics project:

# from flytesnacks root directory
cd examples/basics

Build the container:

docker build . --tag "basics:v1" -f Dockerfile

Package the examples by running:

pyflyte --pkgs basics package --image basics:v1 -f

flytectl register files \
   -p flytesnacks \
   -d development \
   --archive flyte-package.tgz \
   --version v1

Visit https://localhost:30081/console to view the Flyte console, which consists of the examples present in the flytesnacks/core directory.

Updating dependencies#

Prerequisites

Install pip-tools in your development environment with:

pip install pip-tools

If you’ve updated the dependencies of the project, update the requirements.txt file by running:

pip-compile requirements.in --upgrade --verbose --resolver=backtracking

Rebuild the image#

If you’ve updated the source code or dependencies of the project, and rebuild the image with:

docker build . --tag "basics:v2" -f core/Dockerfile
pyflyte --pkgs basics package --image basics:v2 -f
flytectl register files \
    -p flytesnacks \
    -d development \
    --archive flyte-package.tgz \
    --version v2

Refer to this guide if the code in itself is updated and requirements.txt is the same.

Pre-commit hooks#

We use pre-commit to automate linting and code formatting on every commit. Configured hooks include black, isort, flake8 and linters to ensure newlines are added to the end of files, and there is proper spacing in files.

We run all those hooks in CI, but if you want to run them locally on every commit, run pre-commit install after installing the dev environment requirements. In case you want to disable pre-commit hooks locally, run pre-commit uninstall. More info here.

Formatting#

We use black and isort to autoformat code. They are configured as git hooks in pre-commit. Run make fmt to format your code.

Spell-checking#

We use codespell to catch common misspellings. Run make spellcheck to spell-check the changes.

Update Documentation Pages#

The docs/conf.py contains the sphinx configuration for building the flytesnacks documentation.

At build-time, the flytesnacks sphinx build system will convert all of the projects in the examples directory into docs/auto_examples, and will be available in the documentation.

Important

The docs build system will convert the README.md files in each example project into a index.md file, so you can reference the root page of each example project, e.g., in myst markdown format, you can write a table-of-content directive like so:

```{toc}
auto_examples/basics/index
```

If you’ve created a new example project, you’ll need to add the index page in the table of contents in docs/index.md to make sure the project shows up in the documentation. Additonally, you’ll need to update the appropriate list-table directive in docs/userguide.md, docs/tutorials.md, or docs/integrations.md so that it shows up in the respective section of the documentation.

Build the documentation locally#

Verify that the code and documentation look as expected:

Learn about the documentation tools here
Install the requirements by running pip install -r docs-requirements.txt.
Run make -C docs html

Tip

To run a fresh build, run make -C docs clean html.
Open the HTML pages present in the docs/_build directory in the browser with open docs/_build/index.html

Create a pull request#

Create the pull request, then ensure that the docs are rendered correctly by clicking on the documentation check.

You can refer to this PR for the exact changes required.

Contributing to User Guide, Tutorials and Integrations#

How to Contribute#

Creating an Example#

Adding an example script to an existing project#

Creating a new example project#

Creating python examples#

Writing examples: explain what the code does#

Creating examples in other formats#

Writing a README#

Test your code#

Testing on a cluster#

Testing the basics project examples on a local demo cluster#

Updating dependencies#

Rebuild the image#

Pre-commit hooks#

Formatting#

Spell-checking#

Update Documentation Pages#

Build the documentation locally#

Create a pull request#

Testing the `basics` project examples on a local demo cluster#