Contributor Guide

First off, thank you for thinking about contributing! Below you’ll find instructions that will hopefully guide you through how to fix, improve, and extend flytekit.

Please also take some time to read through the design guides, which describes the various parts of flytekit and should make contributing easier.

Setup (Do Once)

We recommend using a virtual environment to develop flytekit. Inside the top level flytekit repo folder

virtualenv ~/.virtualenvs/flytekit
source ~/.virtualenvs/flytekit/bin/activate
make setup
pip install -e .
pip install gsutil awscli

Install shellcheck for linting shell scripts.

Note

It’s important to maintain separate virtualenvs for flytekit development and flytekit use. The reason is that installing a Python library in editable mode will link it to your source code. That is, the behavior will change as you work on the code, check out different branches, etc.

This will install flytekit dependencies and also install flytekit itself in editable mode. This basically links your virtual Python’s site-packages with your local repo folder, allowing your local changes to take effect when the same Python interpreter runs import flytekit

Plugins

As discussed more in the design component, flytekit plugins currently live in this flytekit repo as well, but under a different top level folder, plugins. In the future, this will be separated out into a different repo. These plugins follow a microlib structure, which will persist even if we move repos.

source ~/.virtualenvs/flytekit/bin/activate
cd plugins
pip install -e .

This should install all the plugins in editable mode as well.

Iteration

Make

Some helpful make commands

$ make
  setup        Install requirements
  fmt          Format code with black and isort
  lint         Run linters
  test         Run tests
  requirements Compile requirements

Testing

Three levels of testing are available to help you

Unit Testing

Running unit tests

source ~/.virtualenvs/flytekit/bin/activate
make test

Cookbook Testing

Please see the cookbook and the generated docs for more information. This example repo can be cloned and run on a local Flyte cluster, or just in your IDE or other Python environment.

Follow the set up instructions for the cookbook and then override it with the version of flytekit you’re interested in testing by running something like

pip install https://github.com/flyteorg/flytekit/archive/a32ab82bef4d9ff53c2b7b4e69ff11f1e93858ea.zip#egg=flytekit
# Or for a plugin
pip install https://github.com/flyteorg/flytekit/archive/e128f66dda48bbfc6076d240d39e4221d6af2d2b.zip#subdirectory=plugins/pod&egg=flytekitplugins-pod

Change the actual link to be from your fork if you’re using a fork.

End-to-end Testing

The Flyte developer experience team has put together an end-to-end testing framework that will spin up a K8s cluster, install Flyte onto it, and run through a series of workflows. Please contact us if you reach this stage and would like more information on this.

Formatting

We use black and isort to autoformat code. Run the following command to execute the formatters

source ~/.virtualenvs/flytekit/bin/activate
make fmt

Releases and Project Management

Issues

Please submit issues to the main Flyte repo.

Releasing

Currently, flytekit and all its plugins share one common version. To release, contact a member of the flytekit repo maintainers or committers, and request a release. We will create a GitHub release off of master, which will automatically publish a Pypi package.