Web Analytics Stack

[ThaliaBarrera]

Integrating Google Analytics and Google Search Console.

Extract web traffic and search query data from Google Analytics and Google Search Console, transform it using dbt for insights into user behavior and search performance, and store in a data warehouse like BigQuery.

How to get started:

• Check out the README and the Contribution instructions in this repo.
• Check one of the existing quickstarts and observe their structure.
• (Optional, but highly recommended) Implement one of the quickstarts by following the instructions in their README. I recommend starting with E-commerce analytics in BigQuery

[1cadumagalhaes]

Hey, I would like to work on this one

[ThaliaBarrera]

Thanks @1cadumagalhaes, I have assigned it to you!

[ThaliaBarrera]

Hi @1cadumagalhaes, this message is just to remind you that according to the Hackathon rules you have 5 days to submit progress on the issue. I’ll give you a couple more days to submit a PR, otherwise I’ll have to mark the issue as “To be Claimed” or reassign it.

Thank you for understanding! Let me know if you have any questions.

[1cadumagalhaes]

Hey @ThaliaBarrera , thanks
I'll submit them today, I got a little lost with the deadline but it's okay, thanks for the extra days

[ThaliaBarrera]

Thanks for the heads up @1cadumagalhaes! If you have questions, don’t hesitate to ask.

[1cadumagalhaes]

I'm finishing this one, but I got into an issue with the Google Analytics resource on Terraform (airbytehq/terraform-provider-airbyte#42) and I think I'll have to leave only the UI configuration for this source

[1cadumagalhaes]

Also I had some troubles making this source work at all.

One thing I noticed is that the documentation of the source in the UI points to the wrong API on the Google Cloud API console, it sends to Analytics Reporting and Analytics API but it should lead to Google Analytics Data API.

This last one is the only necessary, and without it the connection doesn't work

[1cadumagalhaes]

There are some limitations with Google Search Console as well, at least for a quickstart.
For some reason it isn't possible to use a Service Account to use the API without having a Google Workspace account, which isn't free (and I don't have one to test). So the only option is to use an access token with client id and etc, but since Airbyte doesn't provide a native way of requesting user permission this means I would need to develop an application to:

1. Use the client id and secret to request user permission using a consent screen (which isn't good to use in a local environment but there must be a way)
2. Get the access and refresh token

And then with these 4 values proceed to set up the source and connection

[ThaliaBarrera]

Thanks for pointing all these issues out @1cadumagalhaes! I’ll send these concerns to the team. And sorry about the time you spent trying to make this work.

In the meantime, would you like to work on another quickstart? You can choose from the ones available. If you have a stack you’re familiar with I can also create an issue for you to work on.