Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

remove pageToken params at first #4

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

remove pageToken params at first #4

wants to merge 1 commit into from

Conversation

hideto0710
Copy link

Description of change

fix #3

Manual QA steps

tap-youtube-analytics --config config.json --catalog catalog.json > state.json

config

{
  "client_id": "YOUR_CLIENT_ID",
  "client_secret": "YOUR_CLIENT_SECRET",
  "refresh_token": "YOUR_REFRESH_TOKEN",
  "channel_ids": "UCD8HOxPs4Xvsm8H0ZxXGiBw, UC7fk0CB07ly8oSl0aqKkqFg",
  "start_date": "2019-01-01T00:00:00Z",
  "user_agent": "tap-youtube-analytics <[email protected]>"
}

catalog

catalog.json 
{
  "streams": [
    {
      "tap_stream_id": "videos",
      "key_properties": [
        "id"
      ],
      "schema": {
        "selected": true,
        "properties": {
          "kind": {
            "type": [
              "null",
              "string"
            ]
          },
          "etag": {
            "type": [
              "null",
              "string"
            ]
          },
          "id": {
            "type": [
              "null",
              "string"
            ]
          },
          "published_at": {
            "format": "date-time",
            "type": [
              "null",
              "string"
            ]
          }
        },
        "type": "object",
        "additionalProperties": false
      },
      "stream": "videos",
      "metadata": [
        {
          "breadcrumb": [],
          "selected": true,
          "metadata": {
            "table-key-properties": [
              "id"
            ],
            "forced-replication-method": "INCREMENTAL",
            "valid-replication-keys": [
              "published_at"
            ],
            "inclusion": "available"
          }
        },
        {
          "breadcrumb": [
            "properties",
            "kind"
          ],
          "metadata": {
            "inclusion": "available"
          }
        },
        {
          "breadcrumb": [
            "properties",
            "etag"
          ],
          "metadata": {
            "inclusion": "available"
          }
        },
        {
          "breadcrumb": [
            "properties",
            "id"
          ],
          "metadata": {
            "inclusion": "automatic"
          }
        },
        {
          "breadcrumb": [
            "properties",
            "published_at"
          ],
          "metadata": {
            "inclusion": "available"
          }
        }
      ]
    }
  ]
}

logs

console 
INFO Authorized, token expires = 2021-09-11 23:41:05.502365
INFO last/currently syncing stream: None
INFO selected_streams: ['videos']
INFO STARTED Syncing: videos
INFO Endpoint: search_videos, URL: https://www.googleapis.com/youtube/v3/search?part=id,snippet&channelId=UCD8HOxPs4Xvsm8H0ZxXGiBw&order=date&type=video&maxResults=50
INFO METRIC: {"type": "timer", "metric": "http_request_duration", "value": 0.4383251667022705, "tags": {"endpoint": "search_videos", "http_status_code": 200, "status": "succeeded"}}
INFO Endpoint: search_videos, Page: 1, Results: 1-50 of Total: 168
INFO Endpoint: search_videos, URL: https://www.googleapis.com/youtube/v3/search?part=id,snippet&channelId=UCD8HOxPs4Xvsm8H0ZxXGiBw&order=date&type=video&maxResults=50&pageToken=CDIQAA
INFO METRIC: {"type": "timer", "metric": "http_request_duration", "value": 0.3607621192932129, "tags": {"endpoint": "search_videos", "http_status_code": 200, "status": "succeeded"}}
INFO Endpoint: search_videos, Page: 2, Results: 51-100 of Total: 168
INFO Endpoint: search_videos, URL: https://www.googleapis.com/youtube/v3/search?part=id,snippet&channelId=UCD8HOxPs4Xvsm8H0ZxXGiBw&order=date&type=video&maxResults=50&pageToken=CGQQAA
INFO METRIC: {"type": "timer", "metric": "http_request_duration", "value": 0.39394617080688477, "tags": {"endpoint": "search_videos", "http_status_code": 200, "status": "succeeded"}}
INFO Endpoint: search_videos, Page: 3, Results: 101-150 of Total: 168
INFO Endpoint: search_videos, URL: https://www.googleapis.com/youtube/v3/search?part=id,snippet&channelId=UCD8HOxPs4Xvsm8H0ZxXGiBw&order=date&type=video&maxResults=50&pageToken=CJYBEAA
INFO METRIC: {"type": "timer", "metric": "http_request_duration", "value": 0.37496185302734375, "tags": {"endpoint": "search_videos", "http_status_code": 200, "status": "succeeded"}}
INFO Endpoint: search_videos, Page: 4, Results: 151-168 of Total: 168
INFO Endpoint: videos, URL: https://www.googleapis.com/youtube/v3/videos?part=id,contentDetails,player,snippet,statistics,status&id=TEmJCdjmUAY,6gDvJVeVNUQ,Sa9skWBlaqg,3r_qaOUjJMM,t7k0I_vBsAc,PORT-p0ex3E,tftbi551s8Q,jVWqvDyIEO8,5Aj-KX-0oLs,YUv2KBae8UM,WKWM_Hl0Un4,xfCUh2lQZt8,OhPS98-tE8E,u94b9HTMOCA,WtAw4WCSyb4,83XRzRkR23s,s2u3aHEg2OU,CP3KuFQoStA,zx3dSrleZ0Y,77F8EyJiCdc,Iq0mTJkJeJ0,nLmZ_cF2Fx4,KtcEnGjp5ew,cBOkKLqOyic,kA-Z5RTNDj8,rQrAVtvT5-c,BtYkLTsgfDo,7nQZfMXI7O8,Re6VLOXvAtU,MMyiCvxH2uA,zswO4CI72ls,mGRdhC5uo8k,TBWKi41S1ho,To2kOOARDLE,ZhCDaLGIuG8,Du_gWOCd2L4,xog1ydy2TWA,SmSOr42eJWc,W0szWI6e87A,may0lirp8Ws,gBCAoDuiTeU,461T8F4tDR0,_gZhFcrH5Ak,rf8Fz7BYLS0,fTIxn3wtOmI,SacbLXskakg,f7MQ2SN4uRc,2ZVUiVeKEK0,jWqOHAJ6l1g,-VkATJm-Xj4&maxResults=50
INFO METRIC: {"type": "timer", "metric": "http_request_duration", "value": 0.4757251739501953, "tags": {"endpoint": "videos", "http_status_code": 200, "status": "succeeded"}}
INFO Endpoint: videos, Page: 1, Results: 1-50 of Total: 50
INFO Endpoint: videos, URL: https://www.googleapis.com/youtube/v3/videos?part=id,contentDetails,player,snippet,statistics,status&id=gHbi2xhtZ00,oiZ5kBb3QjQ,A5K_TUEMZ-Y,64GtYwziNKM,8rUpQYW_ScI,vr5aGseITWU,eVQTfw8Kn0o,ba-fLjK6pdU,v8UmJ536Pjw,DpoVAL-Esbk,ZJyCg6kko64,_ebv7C0d8-Y,sSXtEGwhrWM,damyrfRkdAQ,G6d5E3-WZp0,B9bAiRXJh4A,-WTep9tOv60,psEh21WvXN8,blqykZwvInY,o12B4hZoKR8,Yn-itakV0wU,EWl0t1G-RBs,i3iXG91P3Dg,vpxF9eQYATk,ucLwtAAp__o,PVlIP732UBM,lTyMcl5Jd20,4cxvYk957vs,y_o8XKNHQxY,KkjhNxcNCx0,C4xvRNSNnAk,3-Ew_r1F3OE,wAH7Tu9gMgM,dK-dMorenBA,19bPLWNHYaA,iFRN48wMV64,9jj5Zehy2B4,XK1BKrA2pug,kT9yFgmoCb8,gwY1WkK07Zo,QXFstgVAoms,hDubQ7XAJxw,2EEnjESCFRs,R5jjmvEmRFA,_gBOkmVMLsc,OsPHUvr4-gI,3qPuwYhCUPY,iVgwEia_4zY,Nc2j8kI5JsY,g5gHZVMeUe4&maxResults=50
INFO METRIC: {"type": "timer", "metric": "http_request_duration", "value": 0.39099693298339844, "tags": {"endpoint": "videos", "http_status_code": 200, "status": "succeeded"}}
INFO Endpoint: videos, Page: 1, Results: 1-50 of Total: 50
INFO Endpoint: videos, URL: https://www.googleapis.com/youtube/v3/videos?part=id,contentDetails,player,snippet,statistics,status&id=joaPReqWYhk,Px5ZGz7RSoU,VpYLtASXaSI,sxRRqlioqM4,mudkdXfsOrE,uH3uknlca7Y,zwFSBWW2ziE,NeYHMZWhDsI,0Lekjajh1Eg,41E9_28Rtew,qidKnGqd36U,pxMerU_bhV8,AvFUS8DK9Bo,IoUq81uUshk,3-tfHXi46Ls,oj1ALVMGbfQ,YZvaq2evbK8,cjrhbGeon5M,omZ_pnsNBhQ,vkzEDZIpD0Q,L7fdt8NZCCQ,r5VlNwVceTw,sm5gXVgdKVs,1JYG3e3PWLk,2eNzcvW6WQY,WVQ11nZUVRw,J0p3Eve3OC0,hYtL1Xl-Mcc,mU9Ff9DY2Rg,fTEiRl2cSPE,M3CbiYUXGMw,KQMY1kLdFjw,ZPd3pXIo-xc,OXPaD9WMq7U,VmRD5sBzCCg,yi_y9xLolSU,vH3ymT6sIJ4,LdrTi3rRtyw,-st2zEWwTMU,eGPGrC3RCqw,TURDTCX9unI,gBNZxso04Zo,K30DfNmiPsI,pDcI7KRbtyg,HWMpjXSLWSo,h0UamDe2PyQ,8Opf25y39dM,eZoTBLmrlz4,Bhi00DgsOVY,7nME5zoHU1g&maxResults=50
INFO METRIC: {"type": "timer", "metric": "http_request_duration", "value": 0.3703889846801758, "tags": {"endpoint": "videos", "http_status_code": 200, "status": "succeeded"}}
INFO Endpoint: videos, Page: 1, Results: 1-50 of Total: 50
INFO Endpoint: videos, URL: https://www.googleapis.com/youtube/v3/videos?part=id,contentDetails,player,snippet,statistics,status&id=FzMDdA-mhOk,4guFnIDSAf4,BEvFye-y_KQ,ybUCpY_h9K8,vy85yx6qh-U,IbmLzpbpE68,T5D-6gzUY_A,78wBk1-6Qnc,j4p28FPJ7PY,4ugtIV1fhM0,C0KQZUyfZ3Q,vwSAImRnLoA,G9VbKTpaff0,ix3WUx25xII,KhrPiFZ4Sm8,jW4q0w6CCIo,Mmlj3WUBHbc,R4PEMkDb418&maxResults=50
INFO METRIC: {"type": "timer", "metric": "http_request_duration", "value": 0.20641613006591797, "tags": {"endpoint": "videos", "http_status_code": 200, "status": "succeeded"}}
INFO Endpoint: videos, Page: 1, Results: 1-18 of Total: 18
INFO Endpoint: search_videos, URL: https://www.googleapis.com/youtube/v3/search?part=id,snippet&channelId=UC7fk0CB07ly8oSl0aqKkqFg&order=date&type=video&maxResults=50
INFO METRIC: {"type": "timer", "metric": "http_request_duration", "value": 0.35837888717651367, "tags": {"endpoint": "search_videos", "http_status_code": 200, "status": "succeeded"}}
INFO Endpoint: search_videos, Page: 1, Results: 1-50 of Total: 189
INFO Endpoint: search_videos, URL: https://www.googleapis.com/youtube/v3/search?part=id,snippet&channelId=UC7fk0CB07ly8oSl0aqKkqFg&order=date&type=video&maxResults=50&pageToken=CDIQAA
INFO METRIC: {"type": "timer", "metric": "http_request_duration", "value": 0.39958882331848145, "tags": {"endpoint": "search_videos", "http_status_code": 200, "status": "succeeded"}}
INFO Endpoint: search_videos, Page: 2, Results: 51-100 of Total: 189
INFO Endpoint: search_videos, URL: https://www.googleapis.com/youtube/v3/search?part=id,snippet&channelId=UC7fk0CB07ly8oSl0aqKkqFg&order=date&type=video&maxResults=50&pageToken=CGQQAA
INFO METRIC: {"type": "timer", "metric": "http_request_duration", "value": 0.47443413734436035, "tags": {"endpoint": "search_videos", "http_status_code": 200, "status": "succeeded"}}
INFO Endpoint: search_videos, Page: 3, Results: 101-150 of Total: 189
INFO Endpoint: search_videos, URL: https://www.googleapis.com/youtube/v3/search?part=id,snippet&channelId=UC7fk0CB07ly8oSl0aqKkqFg&order=date&type=video&maxResults=50&pageToken=CJYBEAA
INFO METRIC: {"type": "timer", "metric": "http_request_duration", "value": 0.4976370334625244, "tags": {"endpoint": "search_videos", "http_status_code": 200, "status": "succeeded"}}
INFO Endpoint: search_videos, Page: 4, Results: 151-185 of Total: 189
INFO Endpoint: videos, URL: https://www.googleapis.com/youtube/v3/videos?part=id,contentDetails,player,snippet,statistics,status&id=4H3Na-dhIY4,ylL8dKpG9QI,tJI7JKgLOnY,FNBIsexCXM0,qtHoQ71M9ok,4lr7Osefzoc,eGfyyU9qouA,Phlmweq_EcA,d50ydKyP8vA,exIS9Etx71s,VZuRwKIr34k,df7hQS1JfDM,z1W4c7ym49Y,tRCzC1TJUo8,pElhBw9EGt8,VH_lVKwgB-4,dYtnP9s3QXs,K8V_SP5iulw,uTO0KnDsVH0,_BcPYef-66g,ZyUvqo-cu_g,wV1AGU0rxpw,FGhIbI9AJr0,rv6WD8R3lTo,B1ErENTsf2s,c1cYfAw915A,Q8bHuJdgzqg,3kMSj4JuUqE,ctNrfRXsg4I,43NsdiPXeMQ,DdF3cl-IBlM,3LUoe_-zrv8,P817n2JNdfI,1BGPJ0tVPWs,rG0mRyPLQ6A,0nBTiAt2kbc,VmkbHpO1As0,pYl9ftYGYR8,mIkObWAwMnc,jHd7SPreuAQ,aCau4w0ZFJ0,h9isLVXCfEI,xrQP_QPj33I,LMs6wjPP3aI,Nu0vyXUrdvU,e7KI9_0zsck,4TYKG157gEQ,NdGs2gKFyMA,CiQQ8zJUWOk,8G3umZ9iMqw&maxResults=50
INFO METRIC: {"type": "timer", "metric": "http_request_duration", "value": 0.41202282905578613, "tags": {"endpoint": "videos", "http_status_code": 200, "status": "succeeded"}}
INFO Endpoint: videos, Page: 1, Results: 1-50 of Total: 50
INFO Endpoint: videos, URL: https://www.googleapis.com/youtube/v3/videos?part=id,contentDetails,player,snippet,statistics,status&id=_m2hQrFdiP4,bG-FMPrxNA0,475ryoXOIFk,8d5M0ZU0H1M,GzFln4dVQBQ,JTVA3yn2GyU,4RYpR073iWw,o6NRRekF9fI,36wK5Hg6ahA,YvHGsgjbLDE,c-86erW1RgQ,n2jR-hOP3Hg,8ak-tuqLPLw,KpzmKFDAVHE,ewfY1I2iOV4,f59gisLhZ2I,7LyiPKjUnPI,8X2JLwR2tEk,8IWHjP0_YRE,dENBslwgSTY,3MIDH2SnOgI,UdNpNN5YKKM,hAo6NGQlkOA,3DVDPgPEprg,_RsDFaGnYV8,THs2a9vUTsg,bd7brSeQ-nA,t6cnyE-tQA0,TDcuzEGHDSE,ClrzryIkpTA,eEYaw4Bgpig,soEBBr9mmfM,QAKS6rSbZT0,UgpasSMG6RY,DBdsciM5f_k,WCTnR6oS6oI,c9VYairGvn4,IxaSkrTJtiE,d5aO-Q6NPck,PFeITK7BgbE,pBdcdVmIwzE,XpDCule6Cyg,qZCGxpyAgSk,H4gO1FhRIjc,gsSYyGrtEAU,KPnGhtn1VVs,GIurudkLM58,EMet3PGXzoQ,s0GxtS29X9k,bC4nQRgTV_0&maxResults=50
INFO METRIC: {"type": "timer", "metric": "http_request_duration", "value": 0.4692659378051758, "tags": {"endpoint": "videos", "http_status_code": 200, "status": "succeeded"}}
INFO Endpoint: videos, Page: 1, Results: 1-50 of Total: 50
INFO Endpoint: videos, URL: https://www.googleapis.com/youtube/v3/videos?part=id,contentDetails,player,snippet,statistics,status&id=DZCLsGHS2jM,ngQAlMmTJPU,druH6vW_OJI,DYijjS3Twag,r6aaEdEWUXg,UiR0fCmW7OU,FQabDyuC_wQ,mx2ST27iK8U,WPu7jvZUeHI,TgNy5t8IaNk,437PWLz5JqM,AOoxcw4Kl4k,cjraJKf0yrk,HN_2wCEaR_A,dyngXZ_EkEM,35VBv-WfzB0,pNlA04uQ5J0,dm2VoQocFfg,24WeV1UsQwI,tmMoK1AxbZY,oItsMVr6s6o,-ORdtZrgzHk,AP_xo2HzqO0,NLyA5oCguSo,Kj731Cuu_zI,75HbTB7jIp0,ENQIWpSOzUk,NDdvFrVDJY0,VzFlfZl58RQ,3FL5_ZsM4Ow,FGe0f7lE-EI,6FM3bUOZ_ig,yEfU_Xtp6aw,TDVWuItyPJw,zaj1AZ9Er1M,zmW5WTBlTz8,hoqc8I8yvDo,cCgofxDRHig,q1gUzLTMWKs,soIbQLzS4v4,NDM79IC-uac,8E7yWfFQWzc,Lc1NPKY9JWs,PEbdjFkFOHQ,sKBUED-Sj2g,T9KX533ze2w,VNaZMkPtv7o,B6NbCs5pt20,s_UcjHthxc0,-UN5oxiKw5U&maxResults=50
INFO METRIC: {"type": "timer", "metric": "http_request_duration", "value": 0.4778571128845215, "tags": {"endpoint": "videos", "http_status_code": 200, "status": "succeeded"}}
INFO Endpoint: videos, Page: 1, Results: 1-50 of Total: 50
INFO Endpoint: videos, URL: https://www.googleapis.com/youtube/v3/videos?part=id,contentDetails,player,snippet,statistics,status&id=ao0fpioDJOE,2bqcwqRWv3Q,ujx-czBUd_Y,Vwr0eUhAGHY,o_TfemFjME0,3GUg1uGTIF0,jCAF2e7V6_w,tgPOIvx5Ngk,RH4gW-1_4Qc,Y8syqWkoQXc,L2dmOGbE5n8,GyPd77ttGt4,cC8fs4iyATg,M52TUm11A1E,ajxsmz5L35g,R3dY8Ez5pA0,D-oDqG82FIg,vmEb0UtwU5s,aIG9o-S6cN4,0yExE1kXkto,4X4l58ChYDA,vhk1cD71rM0,t-ES9KW_Ij8,G2JfFK4q06Q,To-FnfaBCBo,XgBeGLwf4VE&maxResults=50
INFO METRIC: {"type": "timer", "metric": "http_request_duration", "value": 0.17652606964111328, "tags": {"endpoint": "videos", "http_status_code": 200, "status": "succeeded"}}
INFO Endpoint: videos, Page: 1, Results: 1-26 of Total: 26
INFO Write state for stream: videos, value: 2021-09-10T16:56:11Z
INFO Stream: videos, Processed 344 records
INFO METRIC: {"type": "counter", "metric": "record_count", "value": 344, "tags": {"endpoint": "videos"}}
INFO FINISHED Syncing Stream: videos
INFO   Records Synced for Stream: 344

Risks

  • If pageToken is set outside of the get_paginated_data function, it will be overwritten.

Rollback steps

  • revert this branch

@cmerrick
Copy link
Contributor

Hi @hideto0710, thanks for your contribution!

In order for us to evaluate and accept your PR, we ask that you sign a contribution license agreement. It's all electronic and will take just minutes.

@cmerrick
Copy link
Contributor

You did it @hideto0710!

Thank you for signing the Singer Contribution License Agreement.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

pageToken parameter is kept when extracting multiple channels.
2 participants
@hideto0710 @cmerrick