New Source: YouGov BrandIndex

[darian-heede]

Tell us about the new connector you’d like to have

• We need a source connector for the YouGov BrandIndex API.
• We need a source connector for v1 of the BrandIndex API.

Describe the context around this new connector

• The data is used for data analytics and reporting purposes.
• We would like to run syncs on a daily basis.
• We need all endpoints listed here, the most important one being the /v1/analyses/execute endpoint.

Describe the alternative you are considering or using

Manual Excel file downloads and integrations.

Are you willing to submit a PR?

Yes.

[darian-heede]

@sherifnada as per our conversation on slack, here's an exemplary custom schema we would need to pass as post data to the analyses/execute endpoint:

{
  "data": {
    "id": "some-analysis",
    "queries": [
      {
        "entity": {
          "region": "us",
          "sector_id": 1
        },
        "filters": [
          {
            "expression": "bixdemo_age2 > 1"
          }
        ],
        "id": "some-simple-query",
        "metrics_score_types": {
          "buzz": "net_score"
        },
        "period": {
          "end_date": {
            "days": 5
          },
          "start_date": {
            "days": 8
          }
        }
      },
      {
        "confidence_bound": {
          "percent": 95,
          "type": "upper"
        },
        "entity": {
          "brand_id": 1007,
          "region": "us",
          "sector_id": 1
        },
        "filters": [],
        "id": "some-complex-query",
        "metrics_score_types": {
          "buzz": "net_score",
          "wom": "negatives_neutrals"
        },
        "moving_average": 3,
        "period": {
          "end_date": {
            "days": 5
          },
          "start_date": {
            "date": "2018-12-25"
          }
        },
        "resample": {
          "size": 2,
          "type": "calendar_day"
        },
        "scoring": "total"
      }
    ],
    "title": "My Analysis"
  },
  "meta": {
    "version": "v1"
  }
}

So far, I just add the custom schema as string to the config.json. This works fine for now.

The request response, however, is a whole other thing. Here's the catalog to give you a feeling of what we're working with. The time series are contained in an array, and the metrics for the dates are contained in their own arrays. This is normalization hell.

Do you have any suggestions on how to go about this?

[sherifnada]

@darian-heede as long as the connector can programmatically "discover" this schema given the inputs in config.json the normalization module should be able to handle it no?

Some alternatives:

1. turn normalization off and use the raw JSON tables output by Airbyte.
2. split the subfields into substreams and each element in the array as a separate record in the new "child" streams
3. restructure the data in the source connector to make it conform to a reasonable schema in some other way

I don't love 2 and 3 because if tomorrow we decide it's actually better to structure the data in a different way, then it would require a potentially backwards breaking change to the schema but it seems like those are our options.

My preference would be:

1. use built in airbyte normalization
2. use JSON
3. transform in one of the ways above

If you can wave a magic wand here what would your preference be?

[darian-heede]

Thank you for your input @sherifnada. The issue here is with the raw JSON itself that is output by Airbyte or rather the YouGov API. Here's an example of the raw connector output. Unfortunately, the basic normalization doesn't really help, as the main query data isn't normalized at all (tested on postgres).

My magic wand preference would be for the normalization to work with the given catalog. I'll check if a catalog directly based on the custom schema will help.

Can you go into more detail for 2. and 3.?

[sherifnada]

@darian-heede normalization operates based on the output discovered catalog. So for the custom schema to be normalized, it would need to be described as accurately as possible by the catalog output in discover. Is that possible to do?

[darian-heede]

@sherifnada using a configured catalog based on the response message indeed helps with the normalization step. I used genSON to create the catalog from the message, which worked well. The only missing step is the unnesting of the arrays containing most dimensions and metrics. This can be done in further transformations, which is ok for my use case.

Nevertheless, the custom schema changes the workflow compared to other source connectors somewhat. As the configured catalog needs to be generated after a first message is fetched using the custom schema.

Anyway, I'm fine to continue for now.

[karinakuz]

Sent request about sandbox account (https://business.yougov.com/thank-you-contact?marketo=contact)

[karinakuz]

There is the problem with account:
You have not been authorized to use BrandIndex 2020.
If the problem persists, please contact BrandIndex for further assistance here. Please provide the following request tracker to your client services https://business.yougov.com/thank-you-contact?marketo=contact representative: request_id=1565930640

Sent letter to the support https://business.yougov.com/thank-you-contact.

[karinakuz]

Sent request to https://business.yougov.com/thank-you-contact?marketo=contact with request_id=1565930640