Databricks

Tags: Spark, Integration, DistributedComputing, Data, Advanced

Flyte backend can be connected with Databricks service. Once enabled it can allow you to submit a spark job to Databricks platform. This section will provide how to use the Databricks Plugin using flytekit python.

Installation

The flytekit Databricks plugin is bundled into its Spark plugin, so to use, simply run the following:

pip install flytekitplugins-spark

How to Build Your Dockerfile for Spark on Databricks

Using Spark on Databricks is extremely easy and provides full versioning using the custom-built Spark container. The built container can also execute regular Spark tasks. For Spark, the image must use a base image built by Databricks and the workflow code must copy to /databricks/driver

FROM databricksruntime/standard:11.3-LTS
LABEL org.opencontainers.image.source=https://github.com/flyteorg/flytesnacks
# To build this dockerfile, run "make docker_build".

ENV VENV /opt/venv
ENV LANG C.UTF-8
ENV LC_ALL C.UTF-8
ENV PYTHONPATH /databricks/driver
ENV PATH="/databricks/python3/bin:$PATH"
USER 0

RUN sudo apt-get update && sudo apt-get install -y make build-essential libssl-dev git

# Install custom package
RUN /databricks/python3/bin/pip install awscli
WORKDIR /opt
RUN curl https://sdk.cloud.google.com > install.sh
RUN bash /opt/install.sh --install-dir=/opt

# Install Python dependencies
COPY databricks/requirements.txt /databricks/driver/requirements.txt
RUN /databricks/python3/bin/pip install -r /databricks/driver/requirements.txt

WORKDIR /databricks/driver
# Copy the makefile targets to expose on the container. This makes it easier to register.
# Delete this after we update CI
COPY databricks/in_container.mk /databricks/driver/Makefile

# Delete this after we update CI to not serialize inside the container
COPY databricks/sandbox.config /databricks/driver

# Copy the actual code
COPY databricks/ /databricks/driver/databricks/

# This tag is supplied by the build script and will be used to determine the version
# when registering tasks, workflows, and launch plans
ARG tag
ENV FLYTE_INTERNAL_IMAGE $tag

Configuring the backend to get Databricks plugin working

1. Make sure to add “databricks” in tasks.task-plugins.enabled-plugin in enabled_plugins.yaml

2. Add Databricks access token to Flytepropeller. here to see more detail to create Databricks access token.

kubectl edit secret -n flyte flyte-propeller-auth

Configuration will be like below

apiVersion: v1
data:
  FLYTE_DATABRICKS_API_TOKEN: <ACCESS_TOKEN>
kind: Secret
metadata:
  annotations:
    meta.helm.sh/release-name: flyte
    meta.helm.sh/release-namespace: flyte
...

Writing a PySpark Task

Writing a PySpark Task