Example Contribution Guide#

The examples documentation provides an easy way for the community to learn about the rich set of features that Flyte offers, and we are constantly improving them with your help!

Whether you’re a novice or experienced software engineer, data scientist, or machine learning practitioner, all contributions are welcome!

How to Contribute#

The Flyte documentation examples guides are broken up into three types:

  1. User Guides: These are short, simple guides that demonstrate how to use a particular Flyte feature. These examples should be runnable locally.

  2. Tutorials: These are longer, more advanced guides that use multiple Flyte features to solve real-world problems. Tutorials are generally more complex examples that may require extra setup or that can only run on larger clusters.

  3. Integrations: These examples showcase how to use the Flyte plugins that integrate with the broader data and ML ecosystem.

The first step to contributing an example is to open up a documentation issue to articulate the kind of example you want to write. The Flyte maintainers will guide and help you figure out where your example would fit best.

Creating an Example#

Prerequisites

Follow the Environment Setup guide to get your development environment ready.

The flytesnacks repo examples live in the cookbook directory, and are organized as follows:

cookbook
├── core          # User Guide Basics features
├── deployment    # User Guide Production Config guides
├── larger_apps   # User Guide Building Large Apps
├── remote_access # User Guide Remote Access guides
├── testing       # User Guide Testing guides
├── case_studies  # Tutorials live here
└── integrations  # Integrations live here

Important

If you’re creating a new example in integrations or case_studies that doesn’takes fit into any of the existing subdirectories, you’ll need to setup a new example directory.

Create a new directory with mkdir {integrations, case_studies}/path/to/new/example_dir

Each example directory should contain:

  • Dockerfile

  • Makefile

  • README.rst

  • __init__.py

  • requirements.in

  • sandbox.config

You can copy one of the existing examples and modify it to your needs.

Create your example by going to the appropriate example directory and creating a .py file with a descriptive name for your example. Be sure to follow the percent format for delimiting code and markdown in your script.

Note

flytesnacks uses sphinx gallery to convert the python files to .rst format so that the examples can be rendered in the documentation.

Write a README#

The README.md file needs to capture the what, why, and how of the example.

  • What is the integration about? Its features, etc.

  • Why do we need this integration? How is it going to benefit the Flyte users?

  • Showcase the uniqueness of the integration

  • How to install the plugin?

Tip

Refer to any subdirectory in the cookbook directory for examples

Explain What the Code Does#

Following the literate programming paradigm, make sure to interleave explanations in the *.py files containing the code example.

A Simple Example

Here’s a code snippet that defines a function that takes two positional arguments and one keyword argument:

def function(x, y, z=3):
    return x + y * z

As you can see, function adds the two first arguments and multiplies the sum with the third keyword argument. Can you think of a better name for this function?

Explanations don’t have to be this detailed for such a simple example, but you can imagine how this makes for a better reading experience for more complicated examples.

Add Run Commands (Optional)#

For each example, you can also create run commands, which specify how to run the example using pyflyte run or FlyteRemote. To do so, create a _run subdirectory in the same directory as the example script.

mkdir _run

Let’s suppose our example script is called my_example.py with the following contents:

from flytekit import task, workflow

@task
def example_task(input: str) -> str:
    return f"hello {input}"

@workflow
def wf(input: str) -> str:
    return example_task(input=input)

In the _run subdirectory, create a bash and python script with the same name, but with a run_ prefix:

touch _run/run_my_example.sh
touch _run/run_my_example.py

The run_my_example.sh.sh script should contain the terminal command that users need to invoke to run the script with pyflyte run, for example:

pyflyte run --remote my_example.py:wf --input there

Similarly, the run_my_example.py script should contain the python code that users need to invoke to run the script with FlyteRemote, for example:

from flytekit.configuration import Config
from flytekit.remote import FlyteRemote

from my_example import wf

remote = FlyteRemote(
    config=Config.auto(),
    default_project="flytesnacks",
    default_domain="development",
)

registered_workflow = remote.register_script(wf)

execution = remote.execute(registered_workflow, inputs={"input": "there"})
print(f"Execution successfully started: {execution.id.name}")

Finally, in the my_example.py example script, place the following custom sphinx directive:

.. run-example-cmds::

This will insert a dropdown in the rendered documentation page with instructions on how to run the example you just created:

Example run commands

Finally, to test the run commands in CI, add an ExampleTestCase entry to the test_example_suite function in cookbook/tests/run_cmds/test_run_examples.py. For example:

@pytest.mark.parametrize(
    "example_test_case",
    [
        ExampleTestCase("hello-world", "core/flyte_basics/hello_world.py", {"o0": "hello world"}),
        ExampleTestCase("task", "core/flyte_basics/task.py", {"o0": 16}),
        ExampleTestCase("basic-workflow", "core/flyte_basics/basic_workflow.py", {"o0": 102, "o1": "helloworld"}),

        # add a new example test case
        ExampleTestCase(
            id="my-example",
            script_rel_path="path/to/my_example.py",
            expected_output={"o0": "hello there"},
        )
    ],
    ids=lambda x: x.id
)
@pytest.mark.parametrize(
    "run_type", ["pyflyte_run", "flytekit_remote"], ids=lambda x: x.replace("_", "-")
)
def test_example_suite(flyte_remote: FlyteRemote, example_test_case: ExampleTestCase, run_type: str):
    ...

Test your code#

If the example code can be run locally, just use python <my file>.py to run it.

Testing on a Cluster#

Install flytectl, the commandline interface for flyte.

Note

Learn more about installation and configuration of Flytectl here.

Start a Flyte demo cluster with:

flytectl demo start

Testing core directory examples on sandbox#

Build Docker container:

flytectl demo exec -- docker build . --tag "core:v1" -f core/Dockerfile

Package the examples by running

pyflyte --pkgs core package --image core:v1 -f

Register the examples by running

Visit https://localhost:30081/console to view the Flyte console, which consists of the examples present in the flytesnacks/cookbook/core directory.

To fetch new dependencies and rebuild the image, run

flytectl demo exec -- docker build . --tag "core:v2" -f core/Dockerfile
pyflyte --pkgs core package --image core:v2 -f
flytectl register files --archive -p flytesnacks -d development --archive flyte-package.tgz --version v2

Refer to this guide if the code in itself is updated and requirements.txt is the same.

Pre-commit hooks#

We use pre-commit to automate linting and code formatting on every commit. Configured hooks include black, isort, flake8 and linters to ensure newlines are added to the end of files, and there is proper spacing in files.

We run all those hooks in CI, but if you want to run them locally on every commit, run pre-commit install after installing the dev environment requirements. In case you want to disable pre-commit hooks locally, run pre-commit uninstall. More info here.

Formatting#

We use black and isort to autoformat code. They are configured as git hooks in pre-commit. Run make fmt to format your code.

Spell-checking#

We use codespell to catch common misspellings. Run make spellcheck to spell-check the changes.

Update Documentation Pages#

The cookbook/docs/conf.py contains the sphinx configuration for building the flytesnacks documentation.

For example, if you added the snowflake directory to the integrations/external_services folder, you then need to:

  • Add the Python file names to the CUSTOM_FILE_SORT_ORDER list

  • Add ../integrations/external_services/snowflake to example_dirs

  • Add auto/integrations/external_services/snowflake to gallery_dirs

If you’ve created a new section in the examples guides, you need to update the table of contents and navigation panels in the appropriate rst file.

Note

You will need to update the entries in the .. toc:: directive and .. panels:: directive.

panel and TOC

Update CI Workflows#

To make sure your example is tested in CI/CD, add the name and path to .github/workflows/ghcr_push.yml if you’re adding an integration or a tutorial.

QA your Changes#

Verify that the code and documentation look as expected:

  • Learn about the documentation tools here

  • Install the requirements by running pip install -r docs-requirements.txt in the cookbook folder

  • Run make html in the docs folder

    Tip

    For implicit targets, run make -C docs html.

  • Open the HTML pages present in the docs/_build directory in the browser

Create a Pull request#

Create the pull request, then ensure that the docs are rendered correctly by clicking on the documentation check.

Docs link in a PR

You can refer to this PR for the exact changes required.