Google BigQuery Plugin

This guide provides an overview of setting up BigQuery in your Flyte deployment. Please note that the BigQuery plugin requires Flyte deployment in the GCP cloud; it is not compatible with demo/AWS/Azure.

Set up the GCP Flyte cluster

• Ensure you have a functional Flyte cluster running in GCP.

• Create a service account for BigQuery. For more details, refer to: https://cloud.google.com/bigquery/docs/quickstarts/quickstart-client-libraries.

• Verify that you have the correct kubeconfig and have selected the appropriate Kubernetes context.

• Confirm that you have the correct Flytectl configuration at ~/.flyte/config.yaml.

Specify plugin configuration

Flyte binaryFlyte core

Edit the relevant YAML file to specify the plugin.

tasks:
  task-plugins:
    enabled-plugins:
      - container
      - sidecar
      - k8s-array
      - bigquery
    default-for-task-types:
      - container: container
      - container_array: k8s-array
      - bigquery_query_job_task: bigquery

Ensure that the propeller has the correct service account for BigQuery.

Upgrade the Flyte Helm release

Flyte binaryFlyte core

helm upgrade <RELEASE_NAME> flyteorg/flyte-binary -n <YOUR_NAMESPACE> --values <YOUR_YAML_FILE>

Replace <RELEASE_NAME> with the name of your release (e.g., flyte-backend), <YOUR_NAMESPACE> with the name of your namespace (e.g., flyte), and <YOUR_YAML_FILE> with the name of your YAML file.