Snowflake agent¶
This guide provides an overview of how to set up the Snowflake agent in your Flyte deployment.
Set up the key pair authentication in Snowflake. For more details, see the Snowflake key-pair authentication and key-pair rotation guide.
Create a secret with the group “snowflake” and the key “private_key”. For more details, see “Using Secrets in a Task”.
kubectl create secret generic snowflake-private-key --namespace=flytesnacks-development --from-file=your_private_key_above
Specify agent configuration¶
Edit the relevant YAML file to specify the agent.
kubectl edit configmap flyte-sandbox-config -n flyte
tasks:
task-plugins:
enabled-plugins:
- container
- sidecar
- k8s-array
- agent-service
default-for-task-types:
- container: container
- container_array: k8s-array
- snowflake: agent-service
plugins:
agent-service:
supportedTaskTypes:
- snowflake
Create a file named values-override.yaml
and add the following configuration to it.
configmap:
enabled_plugins:
# -- Tasks specific configuration [structure](https://pkg.go.dev/github.com/flyteorg/flytepropeller/pkg/controller/nodes/task/config#GetConfig)
tasks:
# -- Plugins configuration, [structure](https://pkg.go.dev/github.com/flyteorg/flytepropeller/pkg/controller/nodes/task/config#TaskPluginConfig)
task-plugins:
# -- [Enabled Plugins](https://pkg.go.dev/github.com/flyteorg/flyteplugins/go/tasks/config#Config). Enable sagemaker*, athena if you install the backend
enabled-plugins:
- container
- sidecar
- k8s-array
- agent-service
default-for-task-types:
container: container
sidecar: sidecar
container_array: k8s-array
snowflake: agent-service
plugins:
agent-service:
supportedTaskTypes:
- snowflake
Ensure that the propeller has the correct service account for BigQuery.
Upgrade the Flyte Helm release¶
helm upgrade <RELEASE_NAME> flyteorg/flyte-binary -n <YOUR_NAMESPACE> --values <YOUR_YAML_FILE>
Replace <RELEASE_NAME>
with the name of your release (e.g., flyte-backend
),
<YOUR_NAMESPACE>
with the name of your namespace (e.g., flyte
),
and <YOUR_YAML_FILE>
with the name of your YAML file.
helm upgrade <RELEASE_NAME> flyte/flyte-core -n <YOUR_NAMESPACE> --values values-override.yaml
Replace <RELEASE_NAME>
with the name of your release (e.g., flyte
)
and <YOUR_NAMESPACE>
with the name of your namespace (e.g., flyte
).
For Snowflake agent on the Flyte cluster, see Snowflake agent.