Google Bigquery Plugin Setup#

This guide gives an overview of how to set up BigQuery in your Flyte deployment. BigQuery plugin needs Flyte deployment in GCP cloud; sandbox/AWS/Azure wouldn’t work.

  1. Setup the GCP Flyte cluster

  1. Create a file named values-override.yaml and add the following config to it. Please make sure that the propeller has the correct service account for BigQuery.

    # -- Tasks specific configuration [structure](
      # -- Plugins configuration, [structure](
        # -- [Enabled Plugins]( Enable sagemaker*, athena if you install the backend
        # plugins
          - container
          - sidecar
          - k8s-array
          - bigquery
          container: container
          sidecar: sidecar
          container_array: k8s-array
          bigquery_query_job_task: bigquery
  1. Upgrade the Flyte Helm release.

helm upgrade -n flyte -f values-override.yaml flyteorg/flyte-core
  1. Register the BigQuery plugin example.

flytectl register files --archive -p flytesnacks -d development
  1. Launch an execution

  • Navigate to the Flyte Console’s UI (e.g. sandbox) and find the relevant workflow

  • Click on Launch to open up a launch form

  • Submit the form to launch an execution

  • Retrieve an execution form in the form of a YAML file:

    flytectl get launchplan --config ~/.flyte/flytectl.yaml --project flytesnacks --domain <TODO:>  --latest --execFile exec_spec.yaml
  • Launch! 🚀

    flytectl --config ~/.flyte/flytectl.yaml create execution -p <project> -d <domain> --execFile ~/exec_spec.yaml