[Feature] Support running SQL models on Google Cloud Dataproc Serverless

[gddezero]

Is this your first time submitting a feature request?

• I have read the expectations for open source contributors
• I have searched the existing issues, and I could not find an existing issue for this feature
• I am requesting a straightforward extension of existing dbt functionality, rather than a Big Idea better suited to a discussion

Describe the feature

Context

Google Cloud Dataproc Serverless lets you run Spark workloads without requiring you to provision and manage your own Dataproc cluster. Use the Google Cloud console, Google Cloud CLI, or Dataproc API to submit a batch workload to the Dataproc Serverless service. The service will run the workload on a managed compute infrastructure, autoscaling resources as needed.

Dataproc Serverless is widely used for GCP customers to build data pipelines. A typical use case is submitting Spark SQL jobs to Dataproc Serverless to transform data and build data warehouse.

Current Status

dbt only supports running Python models on Dataproc Serverless as a companion service of BigQuery
https://docs.getdbt.com/docs/core/connect-data-platform/bigquery-setup#running-python-models-on-dataproc

Request

Support running SQL models on Dataproc Serverless

Describe alternatives you've considered

No response

Who will this benefit?

Customers using Google Cloud

Are you interested in contributing this feature?

No response

Anything else?

No response

[amychen1776]

Hello @gddezero Could you provide more context about why you prefer Datapoc for SQL rather than directly on BQ?

[amychen1776]

At this time, we will not be prioritizing this work (and it wouldn't be done on the Bigquery adapter if it was) so I'm closing this issue for now.

[gddezero]

@amychen1776 With Serverless Spark, user does not need to deploy Spark cluster and thrift server. It will greatly reduce the infrastructure management efforts. I should create this feature request in dbt-spark rather than dbt-bigquery.

[amychen1776]

@gddezero that would make the most sense :) For dbt-spark, it will require supporting a new way of connection since we do expect a thrift server/ODBC driver/HTTP

[gddezero]

@amychen1776 Thanks for your advice. I created a the feature request in dbt-spark: dbt-labs/dbt-spark#1131