Flytekit Contribution Guide

First off, thank you for thinking about contributing! Below you’ll find instructions that will hopefully guide you through how to fix, improve, and extend Flytekit.

Please also take some time to read through the design guides, which describe the various parts of Flytekit and should make contributing easier.

📜 Quick Background

The first version of the flytekit library was written circa 2017, before mypy typing was mainstream, and targeted Python 2. That legacy code will be fully deprecated and removed in 2022 but because there are still users of flytekit that rely on that legacy api, you’ll see 2 separate and distinct code paths within this repo. Users and contributors should ignore the legacy sections. Below is a listing of the most important packages that comprise the new API:

  • flytekit/core This holds all the core functionality of the new API.

  • flytekit/types We bundle some special types like FlyteFile, FlyteSchema etc by default here.

  • flytekit/extend This is the future home of extension points, and currently serves as the raw documentation for extensions.

  • flytekit/extras This contains code that we want bundled with flytekit but not everyone may find useful (for example AWS and GCP specific logic).

  • flytekit/remote This implements the interface to interact with the Flyte service. Think of the code here as the Python-object version of Console.

  • flytekit/testing is the future home for testing functionality like mock etc, and currently serves as documentation. All test extensions should be imported from here.

  • flytekit/models Protobuf generated Python code is not terribly user-friendly, so we improve upon those flyteidl classes here.

  • plugins is the source of all plugins

  • flytekit/bin/entrypoint.py The run time entrypoint for flytekit. When a task kicks off, this is where the click command goes.

  • flytekit/clis This is the home for the clis.

  • flytekit/configuration This holds all the configuration objects, but dependency on configuration should be carefully considered as it makes compiled Flyte tasks and workflows less portable (i.e. if you run pyflyte package can someone else use those serialized objects).

Most of the other folders are for legacy Flytekit, support for which will be dropped in early 2022. For the most part, please ignore the following folders:

  • flytekit/plugins

  • flytekit/common (the translator.py file is an exception)

  • flytekit/engines

  • flytekit/interfaces

  • flytekit/sdk

  • flytekit/type_engines

Please also see the design overview section for more in-depth information.

💻 Contribute Code

Setup (Do Once)

We recommend using a virtual environment to develop Flytekit. Inside the top level Flytekit repo folder, run:

virtualenv ~/.virtualenvs/flytekit
source ~/.virtualenvs/flytekit/bin/activate
make setup
pip install -e .
pip install gsutil awscli

Install shellcheck for linting shell scripts.

Note

It’s important to maintain separate virtualenvs for flytekit development and flytekit use. The reason is that installing a Python library in editable mode will link it to your source code. That is, the behavior will change as you work on the code, check out different branches, etc.

This will install flytekit dependencies and also install flytekit itself in editable mode. This basically links your virtual Python’s site-packages with your local repo folder, allowing your local changes to take effect when the same Python interpreter runs import flytekit.

Plugin Development

As discussed in the design component, Flytekit plugins currently live in this flytekit repo, but under a different top level folder plugins. In the future, this will be separated out into a different repo. These plugins follow a microlib structure, which will persist even if we move repos.

source ~/.virtualenvs/flytekit/bin/activate
cd plugins
pip install -e .

This should install all the plugins in editable mode as well.

Iteration

Make

Some helpful make commands

$ make
  setup        Install requirements
  fmt          Format code with black and isort
  lint         Run linters
  test         Run tests
  requirements Compile requirements

Testing

Three levels of testing are available.

Unit Testing

Running unit tests:

source ~/.virtualenvs/flytekit/bin/activate
make test
Cookbook Testing

Please see the cookbook and the generated docs for more information. This example repo can be cloned and run on a local Flyte cluster, or just in your IDE or other Python environment.

Follow the setup instructions for the cookbook and then override it with the version of Flytekit you’re interested in testing by running something like:

pip install https://github.com/flyteorg/flytekit/archive/a32ab82bef4d9ff53c2b7b4e69ff11f1e93858ea.zip#egg=flytekit
# Or for a plugin
pip install https://github.com/flyteorg/flytekit/archive/e128f66dda48bbfc6076d240d39e4221d6af2d2b.zip#subdirectory=plugins/pod&egg=flytekitplugins-pod

Change the actual link to be from your fork if you are using a fork.

End-to-end Testing

The Flyte developer experience team has put together an end-to-end testing framework that will spin up a K8s cluster, install Flyte onto it, and run through a series of workflows. Please contact us if you reach this stage and would like more information on this.

Pre-commit hooks

We use pre-commit to automate linting and code formatting on every commit. Configured hooks include black, isort, and flake8 and also linters to check for the validity of YAML files and ensuring that newlines are added to the end of files.

We run all those hooks in CI, but if you want to run them locally on every commit, run pre-commit install after installing the dev environment requirements. In case you want to disable pre-commit hooks locally, for example, while you’re iterating on some feature, run pre-commit uninstall. More info in https://pre-commit.com/.

Formatting

We use black and isort to autoformat code. In fact, they have been configured as git hooks in pre-commit. Run the following commands to execute the formatters.

source ~/.virtualenvs/flytekit/bin/activate
make fmt

Spell-checking

We use codespell to catch spelling mistakes in both code and documentation. Run the following commands to spell-check changes.

source ~/.virtualenvs/flytekit/bin/activate
make spellcheck

📃 Contribute to Documentation

  1. Install requirements by running make doc-requirements.txt in the root of the repo

  2. Make the required changes

  3. Verify if the documentation looks as expected by running make html in the docs directory

  4. Open HTML pages present in the docs/build directory in the browser

  5. After creating the pull request, check if the docs are rendered correctly by clicking on the documentation check

    Doc link in PR

📝 Releases and Project Management

Currently, Flytekit and all its plugins share one common version. To release, contact a member of the Flytekit repo maintainers or committers, and request a release. We will create a GitHub release off of master, which will automatically publish a Pypi package.