Databricks agent example#
Flyte can be integrated with the Databricks service, enabling you to submit Spark jobs to the Databricks platform.
Installation#
The Databricks agent comes bundled with the Spark plugin. To install the Spark plugin, run the following command:
pip install flytekitplugins-spark
Example usage#
For a usage example, see Databricks agent example usage.
Local testing#
To test the Databricks agent locally, create a class for the agent task that inherits from AsyncAgentExecutorMixin. This mixin can handle asynchronous tasks and allows flytekit to mimic FlytePropeller’s behavior in calling the agent. For more information, see “Testing agents locally”.
Note
In some cases, you will need to store credentials in your local environment when testing locally.
Flyte deployment configuration#
Note
If you are using a managed deployment of Flyte, you will need to contact your deployment administrator to configure agents in your deployment.
To enable the Databricks agent in your Flyte deployment, see the Databricks agent setup guide.