Google BigQuery Plugin

This guide provides an overview of setting up BigQuery in your Flyte deployment. Please note that the BigQuery plugin requires Flyte deployment in the GCP cloud; it is not compatible with demo/AWS/Azure.

Set up the GCP Flyte cluster

  • Ensure you have a functional Flyte cluster running in GCP <https://docs.flyte.org/en/latest/deployment/gcp/index.html#deployment-gcp>__.
  • Create a service account for BigQuery. For more details, refer to GCP docs.
  • Verify that you have the correct kubeconfig and have selected the appropriate Kubernetes context.
  • Confirm that you have the correct Flytectl configuration at ~/.flyte/config.yaml.

Specify plugin configuration

flyte-binary

Edit the relevant YAML file to specify the plugin.


      tasks:
        task-plugins:
          enabled-plugins:
            - container
            - sidecar
            - bigquery
          default-for-task-types:
            - container: container
            - bigquery_query_job_task: bigquery

flyte-core

Create a file named values-override.yaml and add the following configuration to it.

        configmap:
          enabled_plugins:
            # -- Tasks specific configuration [structure](https://pkg.go.dev/github.com/flyteorg/flytepropeller/pkg/controller/nodes/task/config#GetConfig)
            tasks:
              # -- Plugins configuration, [structure](https://pkg.go.dev/github.com/flyteorg/flytepropeller/pkg/controller/nodes/task/config#TaskPluginConfig)
              task-plugins:
                # -- [Enabled Plugins](https://pkg.go.dev/github.com/flyteorg/flyteplugins/go/tasks/config#Config). Enable sagemaker*, athena if you install the backend
                enabled-plugins:
                  - container
                  - sidecar
                  - k8s-array
                  - bigquery
                default-for-task-types:
                  container: container
                  sidecar: sidecar
                  container_array: k8s-array
                  bigquery_query_job_task: bigquery

Ensure that flytepropeller has the correct service account for BigQuery.

Upgrade the Flyte Helm release

Replace <RELEASE_NAME> with the name of your release (e.g., flyte-backend), <YOUR_NAMESPACE> with the name of your namespace (e.g., flyte), <HELM_CHART> with either flyte-binary or flyte-core and <YOUR_YAML_FILE> with the name of your YAML file.