Flytekit Contribution Guide¶
First off, thank you for thinking about contributing! Below you’ll find instructions that will hopefully guide you through how to fix, improve, and extend Flytekit.
Please also take some time to read through the design guides, which describe the various parts of Flytekit and should make contributing easier.
📜 Quick Background¶
The first version of the flytekit library was written circa 2017, before mypy typing was mainstream, and targeted Python 2. That legacy code will be fully deprecated and removed in 2022 but because there are still users of flytekit that rely on that legacy api, you’ll see 2 separate and distinct code paths within this repo. Users and contributors should ignore the legacy sections. Below is a listing of the most important packages that comprise the new API:
flytekit/coreThis holds all the core functionality of the new API.
flytekit/typesWe bundle some special types like
FlyteFile, FlyteSchema etcby default here.
flytekit/extendThis is the future home of extension points, and currently serves as the raw documentation for extensions.
flytekit/extrasThis contains code that we want bundled with flytekit but not everyone may find useful (for example AWS and GCP specific logic).
flytekit/remoteThis implements the interface to interact with the Flyte service. Think of the code here as the Python-object version of Console.
flytekit/testingis the future home for testing functionality like
mocketc, and currently serves as documentation. All test extensions should be imported from here.
flytekit/modelsProtobuf generated Python code is not terribly user-friendly, so we improve upon those
pluginsis the source of all plugins
flytekit/bin/entrypoint.pyThe run time entrypoint for flytekit. When a task kicks off, this is where the click command goes.
flytekit/clisThis is the home for the clis.
flytekit/configurationThis holds all the configuration objects, but dependency on configuration should be carefully considered as it makes compiled Flyte tasks and workflows less portable (i.e. if you run
pyflyte packagecan someone else use those serialized objects).
Most of the other folders are for legacy Flytekit, support for which will be dropped in early 2022. For the most part, please ignore the following folders:
translator.pyfile is an exception)
Please also see the design overview section for more in-depth information.
💻 Contribute Code¶
Setup (Do Once)¶
We recommend using a virtual environment to develop Flytekit. Inside the top level Flytekit repo folder, run:
virtualenv ~/.virtualenvs/flytekit source ~/.virtualenvs/flytekit/bin/activate make setup pip install -e . pip install gsutil awscli
Install shellcheck for linting shell scripts.
It’s important to maintain separate virtualenvs for flytekit development and flytekit use. The reason is that installing a Python library in editable mode will link it to your source code. That is, the behavior will change as you work on the code, check out different branches, etc.
This will install flytekit dependencies and also install flytekit itself in editable mode. This basically links your virtual Python’s
site-packages with your local repo folder, allowing your local changes to take effect when the same Python interpreter runs
As discussed in the design component, Flytekit plugins currently live in this flytekit repo, but under a different top level folder
In the future, this will be separated out into a different repo. These plugins follow a microlib structure, which will persist even if we move repos.
source ~/.virtualenvs/flytekit/bin/activate cd plugins pip install -e .
This should install all the plugins in editable mode as well.
Some helpful make commands
$ make setup Install requirements fmt Format code with black and isort lint Run linters test Run tests requirements Compile requirements
Three levels of testing are available.
Running unit tests:
source ~/.virtualenvs/flytekit/bin/activate make test
Follow the setup instructions for the cookbook and then override it with the version of Flytekit you’re interested in testing by running something like:
pip install https://github.com/flyteorg/flytekit/archive/a32ab82bef4d9ff53c2b7b4e69ff11f1e93858ea.zip#egg=flytekit # Or for a plugin pip install https://github.com/flyteorg/flytekit/archive/e128f66dda48bbfc6076d240d39e4221d6af2d2b.zip#subdirectory=plugins/pod&egg=flytekitplugins-pod
Change the actual link to be from your fork if you are using a fork.
The Flyte developer experience team has put together an end-to-end testing framework that will spin up a K8s cluster, install Flyte onto it, and run through a series of workflows. Please contact us if you reach this stage and would like more information on this.
We use pre-commit to automate linting and code formatting on every commit. Configured hooks include black, isort, and flake8 and also linters to check for the validity of YAML files and ensuring that newlines are added to the end of files.
We run all those hooks in CI, but if you want to run them locally on every commit, run pre-commit install after installing the dev environment requirements. In case you want to disable pre-commit hooks locally, for example, while you’re iterating on some feature, run pre-commit uninstall. More info in https://pre-commit.com/.
source ~/.virtualenvs/flytekit/bin/activate make fmt
We use codespell to catch spelling mistakes in both code and documentation. Run the following commands to spell-check changes.
source ~/.virtualenvs/flytekit/bin/activate make spellcheck
📃 Contribute to Documentation¶
Install requirements by running
make doc-requirements.txtin the root of the repo
Make the required changes
Verify if the documentation looks as expected by running
make htmlin the docs directory
Open HTML pages present in the
docs/builddirectory in the browser
After creating the pull request, check if the docs are rendered correctly by clicking on the documentation check
📝 Releases and Project Management¶
Currently, Flytekit and all its plugins share one common version. To release, contact a member of the Flytekit repo maintainers or committers, and request a release. We will create a GitHub release off of master, which will automatically publish a Pypi package.