Google BigQuery agent¶

This guide provides an overview of setting up BigQuery agent in your Flyte deployment. Please note that the BigQuery agent requires Flyte deployment in the GCP cloud; it is not compatible with demo/AWS/Azure.

Set up the GCP Flyte cluster¶

Ensure you have a functional Flyte cluster running in GCP.
Create a service account for BigQuery. For more details, refer to: https://cloud.google.com/bigquery/docs/quickstarts/quickstart-client-libraries.
Verify that you have the correct kubeconfig and have selected the appropriate Kubernetes context.
Confirm that you have the correct Flytectl configuration at ~/.flyte/config.yaml.

Specify agent configuration¶

Edit the relevant YAML file to specify the agent.

kubectl edit configmap flyte-sandbox-config -n flyte

tasks:
  task-plugins:
    enabled-plugins:
      - container
      - sidecar
      - k8s-array
      - agent-service
    default-for-task-types:
      - container: container
      - container_array: k8s-array
      - bigquery_query_job_task: agent-service

plugins:
  agent-service:
    supportedTaskTypes:
    - bigquery_query_job_task

Create a file named values-override.yaml and add the following configuration to it.

configmap:
  enabled_plugins:
    # -- Tasks specific configuration [structure](https://pkg.go.dev/github.com/flyteorg/flytepropeller/pkg/controller/nodes/task/config#GetConfig)
    tasks:
      # -- Plugins configuration, [structure](https://pkg.go.dev/github.com/flyteorg/flytepropeller/pkg/controller/nodes/task/config#TaskPluginConfig)
      task-plugins:
        # -- [Enabled Plugins](https://pkg.go.dev/github.com/flyteorg/flyteplugins/go/tasks/config#Config). Enable sagemaker*, athena if you install the backend
        enabled-plugins:
          - container
          - sidecar
          - k8s-array
          - agent-service
        default-for-task-types:
          container: container
          sidecar: sidecar
          container_array: k8s-array
          bigquery_query_job_task: agent-service
    plugins:
      agent-service:
        supportedTaskTypes:
        - bigquery_query_job_task

Ensure that the propeller has the correct service account for BigQuery.

Upgrade the Flyte Helm release¶

helm upgrade <RELEASE_NAME> flyteorg/flyte-binary -n <YOUR_NAMESPACE> --values <YOUR_YAML_FILE>

Replace <RELEASE_NAME> with the name of your release (e.g., flyte-backend), <YOUR_NAMESPACE> with the name of your namespace (e.g., flyte), and <YOUR_YAML_FILE> with the name of your YAML file.

helm upgrade <RELEASE_NAME> flyte/flyte-core -n <YOUR_NAMESPACE> --values values-override.yaml

Replace <RELEASE_NAME> with the name of your release (e.g., flyte)

and <YOUR_NAMESPACE> with the name of your namespace (e.g., flyte).

For BigQuery agent on the Flyte cluster, see BigQuery agent.