Snowflake Plugin Setup#
This guide gives an overview of how to set up Snowflake in your Flyte deployment.
Add Flyte chart repo to Helm
helm repo add flyteorg https://flyteorg.github.io/flyte
Setup the cluster
Start the sandbox cluster
flytectl sandbox start
Generate Flytectl sandbox config
flytectl config init
Create a file named
values-override.yaml
and add the following config to it:
configmap:
enabled_plugins:
# -- Tasks specific configuration [structure](https://pkg.go.dev/github.com/flyteorg/flytepropeller/pkg/controller/nodes/task/config#GetConfig)
tasks:
# -- Plugins configuration, [structure](https://pkg.go.dev/github.com/flyteorg/flytepropeller/pkg/controller/nodes/task/config#TaskPluginConfig)
task-plugins:
# -- [Enabled Plugins](https://pkg.go.dev/github.com/flyteorg/flyteplugins/go/tasks/config#Config). Enable sagemaker*, athena if you install the backend
# plugins
enabled-plugins:
- container
- sidecar
- k8s-array
- snowflake
default-for-task-types:
container: container
sidecar: sidecar
container_array: k8s-array
snowflake: snowflake
Create a trial Snowflake account and follow the docs for creating an API key.
Add snowflake JWT token to FlytePropeller.
Note
Refer to the Snowflake docs to understand setting up the Snowflake JWT token.
kubectl edit secret -n flyte flyte-secret-auth
The configuration will look as follows:
apiVersion: v1
data:
FLYTE_SNOWFLAKE_CLIENT_TOKEN: <JWT_TOKEN>
client_secret: Zm9vYmFy
kind: Secret
metadata:
annotations:
meta.helm.sh/release-name: flyte
meta.helm.sh/release-namespace: flyte
...
Replace <JWT_TOKEN>
with your JWT token.
Upgrade the Flyte Helm release.
helm upgrade -n flyte -f https://raw.githubusercontent.com/flyteorg/flyte/master/charts/flyte-core/values-sandbox.yaml -f values-override.yaml flyteorg/flyte-core
Register the Snowflake plugin example.
flytectl register files https://github.com/flyteorg/flytesnacks/releases/download/v0.2.226/snacks-cookbook-external_services-snowflake.tar.gz --archive -p flytesnacks -d development
Launch an execution
Navigate to Flyte Console’s UI (e.g. sandbox) and find the workflow.
Click on Launch to open up the launch form.
Submit the form.
Retrieve an execution form in the form of a yaml file:
flytectl get launchplan --config ~/.flyte/flytectl.yaml --project flytesnacks --domain development snowflake.workflows.example.snowflake_wf --latest --execFile exec_spec.yaml --config ~/.flyte/flytectl.yaml
Launch! 🚀
flytectl --config ~/.flyte/flytectl.yaml create execution -p <project> -d <domain> --execFile ~/exec_spec.yaml