How do I enable backend plugins?

Tip

Flyte Backend plugins are awesome, but are not required to extend Flyte! You can always write a flytekit-only plugins. Refer to When & How to Extend Flyte.

Flyte has a unique capability of adding backend plugins. Backend plugins enable Flyte platform to add new capabilities. This has several advantages,

  1. Advanced introspection capabilities - ways to improve logging etc

  2. Service oriented architecture - ability to bugfix, deploy plugins without releasing new libraries and forcing all users to update their libraries

  3. Better management of the system communication - For example in case of aborts, Flyte can guarantee cleanup of the remote resources

  4. Reduced cost overhead, for many plugins which launch jobs on a remote service or cluster, the plugins are essentially just polling. This has a huge compute cost in traditional architectures like Airflow etc. Flyte on the other hand, can run these operations in its own control plane.

  5. Potential to create drastically new interfaces, that work across multiple languages and platforms.

Ok, How do I enable the backend plugins?

To enable a backend plugin you have to add the ID of the plugin to the enabled plugins list. The enabled-plugins is available under the tasks > task-plugins section of FlytePropeller’s configuration. The plugin configuration structure is defined here. An example of the config follows,

tasks:
  task-plugins:
    enabled-plugins:
      - container
      - sidecar
      - k8s-array
    default-for-task-types:
      container: container
      sidecar: sidecar
      container_array: k8s-array

How do I find the ID of the backend plugin?

This is a little tricky and sadly at the moment you have to look at the source code of the plugin to figure out the ID. In the case of Spark, for example, the value of ID is used here, defined as spark.

Enable a specific Backend Plugin in your own Kustomize generator

Flyte uses Kustomize to generate the the deployment configuration and it can be leveraged to kustomize your own deployment.

We will soon be supporting helm or a better deployment model - See issue #299.