Flytekit Contribution Guide#
First off, thank you for thinking about contributing! Below youโll find instructions that will hopefully guide you through how to fix, improve, and extend Flytekit.
Please also take some time to read through the design guides, which describe the various parts of Flytekit and should make contributing easier.
๐ Background#
Below is a listing of the most important packages that comprise the flytekit SDK:
flytekit/core
This holds all the core functionality of the new API.flytekit/types
We bundle some special types likeFlyteFile, FlyteSchema etc
by default here.flytekit/extend
This is the future home of extension points, and currently serves as the raw documentation for extensions.flytekit/extras
This contains code that we want bundled with Flytekit but not everyone may find useful (for example AWS and GCP specific logic).flytekit/remote
This implements the interface to interact with the Flyte service. Think of the code here as the Python-object version of Console.flytekit/testing
is the future home for testing functionality likemock
etc, and currently serves as documentation. All test extensions should be imported from here.flytekit/models
Protobuf generated Python code is not terribly user-friendly, so we improve upon thoseflyteidl
classes here.plugins
is the source of all pluginsflytekit/bin/entrypoint.py
The run time entrypoint for flytekit. When a task kicks off, this is where the click command goes.flytekit/clis
This is the home for the CLIs.flytekit/configuration
This holds all the configuration objects, but dependency on configuration should be carefully considered as it makes compiled Flyte tasks and workflows less portable (i.e. if you runpyflyte package
can someone else use those serialized objects).
Please also see the design overview section for more in-depth information.
๐ป Contribute Code#
Setup (Do Once)#
We recommend using a virtual environment to develop Flytekit. Inside the top level Flytekit repo folder, run:
virtualenv ~/.virtualenvs/flytekit
source ~/.virtualenvs/flytekit/bin/activate
make setup
This will install Flytekit dependencies and Flytekit in editable mode. This links your virtual Pythonโs site-packages
with your local repo folder, allowing your local changes to take effect when the same Python interpreter runs import flytekit
.
Plugin Development#
As discussed in the design component, Flytekit plugins currently live in this Flytekit repo, but under a different top level folder plugins
.
In the future, this will be separated out into a different repo. These plugins follow a microlib structure, which will persist even if we move repos.
source ~/.virtualenvs/flytekit/bin/activate
cd plugins
pip install -e .
This should install all the plugins in editable mode as well.
Iteration#
Make#
Some helpful make commands
$ make
setup Install requirements
fmt Format code with ruff
lint Run linters
test Run tests
requirements Compile requirements
Testing#
Three levels of testing are available.
Unit Testing#
Running unit tests:
source ~/.virtualenvs/flytekit/bin/activate
make test
Cookbook Testing#
Please see the cookbook and the generated docs for more information. This example repo can be cloned and run on a local Flyte cluster, or just in your IDE or other Python environment.
Follow the setup instructions for the cookbook and then override it with the version of Flytekit youโre interested in testing by running something like:
pip install https://github.com/flyteorg/flytekit/archive/a32ab82bef4d9ff53c2b7b4e69ff11f1e93858ea.zip#egg=flytekit
# Or for a plugin
pip install https://github.com/flyteorg/flytekit/archive/e128f66dda48bbfc6076d240d39e4221d6af2d2b.zip#subdirectory=plugins/pod&egg=flytekitplugins-pod
Change the actual link to be from your fork if you are using a fork.
End-to-end Testing#
The Flyte developer experience team has put together an end-to-end testing framework that will spin up a K8s cluster, install Flyte onto it, and run through a series of workflows. Please contact us if you reach this stage and would like more information on this.
Pre-commit hooks#
We use pre-commit to automate linting and code formatting on every commit. Configured hooks include ruff and also linters to check for the validity of YAML files and ensuring that newlines are added to the end of files.
We run all those hooks in CI, but if you want to run them locally on every commit, run pre-commit install after installing the dev environment requirements. In case you want to disable pre-commit hooks locally, for example, while youโre iterating on some feature, run pre-commit uninstall. More info in https://pre-commit.com/.
Formatting#
We use ruff to autoformat code. In fact, they have been configured as git hooks in pre-commit. Run the following commands to execute the formatters.
source ~/.virtualenvs/flytekit/bin/activate
make fmt
Spell-checking#
We use codespell to catch spelling mistakes in both code and documentation. Run the following commands to spell-check changes.
source ~/.virtualenvs/flytekit/bin/activate
make spellcheck
๐ Contribute to Documentation#
Install requirements by running
make doc-requirements.txt
in the root of the repoMake the required changes
Verify if the documentation looks as expected by running
make html
in the docs directoryOpen HTML pages present in the
docs/build
directory in the browserAfter creating the pull request, check if the docs are rendered correctly by clicking on the documentation check
๐ Releases and Project Management#
Currently, Flytekit and all its plugins share one common version. To release, contact a member of the Flytekit repo maintainers or committers, and request a release. We will create a GitHub release off of master, which will automatically publish a Pypi package.