How do I enable backend plugins?¶
Flyte Backend plugins are awesome, but are not required to extend Flyte! You can always write a flytekit-only plugins. Refer to When & How to Extend Flyte.
Flyte has a unique capability of adding backend plugins. Backend plugins enable Flyte platform to add new capabilities. This has several advantages,
Advanced introspection capabilities - ways to improve logging etc
Service oriented architecture - ability to bugfix, deploy plugins without releasing new libraries and forcing all users to update their libraries
Better management of the system communication - For example in case of aborts, Flyte can guarantee cleanup of the remote resources
Reduced cost overhead, for many plugins which launch jobs on a remote service or cluster, the plugins are essentially just polling. This has a huge compute cost in traditional architectures like Airflow etc. Flyte on the other hand, can run these operations in its own control plane.
Potential to create drastically new interfaces, that work across multiple languages and platforms.
Ok, How do I enable the backend plugins?¶
To enable a backend plugin you have to add the
ID of the plugin to the enabled plugins list. The
enabled-plugins is available under the
tasks > task-plugins section of FlytePropeller’s configuration.
The plugin configuration structure is defined here. An example of the config follows,
tasks: task-plugins: enabled-plugins: - container - sidecar - k8s-array default-for-task-types: container: container sidecar: sidecar container_array: k8s-array
How do I find the
ID of the backend plugin?¶
This is a little tricky and sadly at the moment you have to look at the source code of the plugin to figure out the
ID. In the case of Spark, for example, the value of
ID is used here, defined as spark.