-
Notifications
You must be signed in to change notification settings - Fork 28
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
tap-gitlab - sync_pipelines_extended is failing due to difference in schema #79
Comments
Hi @amalkumarCurve it looks like you're using version 0.9.15 of the tap, which is from the If so, can you try switching to version 2 or directly using code from the |
Hi @laurentS , Thanks for your response. I tried using version 2 i.e. (https://github.com/MeltanoLabs/tap-gitlab/releases/tag/v2.0.0-alpha4). However, getting a different issue now. i.e. Since in my config I disabled the flag
|
Indeed! I believe this line Line 163 in 7d285b8
should read as (note the not ):
if stream_name in OPTIN_STREAM_NAMES and not self.config.get( Can you try this out and let me know if it solves your problem? |
@laurentS Traceback (most recent call last): |
Related: I've found that the legacy version of this tap failed silently when access was denied on a number of stream types. I've started #78 which would give the new 2.x edition ability to ignore access denied issues when met. |
Error:
Traceback (most recent call last):
File "/Users/amalkumar/venv/bin/tap-gitlab", line 11, in
load_entry_point('tap-gitlab==0.9.15', 'console_scripts', 'tap-gitlab')()
File "/Users/amalkumar/venv/lib/python3.7/site-packages/tap_gitlab/init.py", line 959, in main
raise exc
File "/Users/amalkumar/venv/lib/python3.7/site-packages/tap_gitlab/init.py", line 956, in main
main_impl()
File "/Users/amalkumar/venv/lib/python3.7/site-packages/tap_gitlab/init.py", line 951, in main_impl
do_sync()
File "/Users/amalkumar/venv/lib/python3.7/site-packages/tap_gitlab/init.py", line 904, in do_sync
sync_group(gid, pids)
File "/Users/amalkumar/venv/lib/python3.7/site-packages/tap_gitlab/init.py", line 679, in sync_group
sync_project(pid)
File "/Users/amalkumar/venv/lib/python3.7/site-packages/tap_gitlab/init.py", line 834, in sync_project
sync_pipelines(data)
File "/Users/amalkumar/venv/lib/python3.7/site-packages/tap_gitlab/init.py", line 723, in sync_pipelines
sync_pipelines_extended(project, transformed_row)
File "/Users/amalkumar/venv/lib/python3.7/site-packages/tap_gitlab/init.py", line 744, in sync_pipelines_extended
transformed_row = transformer.transform(row, RESOURCES[entity]["schema"], mdata)
File "/Users/amalkumar/venv/lib/python3.7/site-packages/singer/transform.py", line 152, in transform
raise SchemaMismatch(self.errors)
singer.transform.SchemaMismatch: Errors during transform
user: data does not match {'type': 'object', 'properties': {'name': {'type': 'string'}, 'username': {'type': 'string'}, 'id': {'type': 'integer'}, 'state': {'type': 'string'}}}
committed_at: data does not match {'type': 'string', 'format': 'date-time'}
coverage: data does not match {'type': 'number'}
: data does not match {'type': 'object', 'properties': {'project_id': {'type': ['integer', 'null']}, 'id': {'type': ['integer', 'null']}, 'status': {'type': ['string', 'null']}, 'ref': {'type': ['string', 'null']}, 'sha': {'type': ['string', 'null']}, 'before_sha': {'type': ['string', 'null']}, 'tag': {'type': ['boolean', 'null']}, 'yaml_errors': {'type': ['string', 'null']}, 'user': {'type': 'object', 'properties': {'name': {'type': 'string'}, 'username': {'type': 'string'}, 'id': {'type': 'integer'}, 'state': {'type': 'string'}}}, 'created_at': {'anyOf': [{'type': 'string', 'format': 'date-time'}, {'type': 'null'}]}, 'updated_at': {'anyOf': [{'type': 'string', 'format': 'date-time'}, {'type': 'null'}]}, 'started_at': {'anyOf': [{'type': 'string', 'format': 'date-time'}, {'type': 'null'}]}, 'finished_at': {'anyOf': [{'type': 'string', 'format': 'date-time'}, {'type': 'null'}]}, 'committed_at': {'anyOf': [{'type': 'string', 'format': 'date-time'}, {'type': 'null'}]}, 'duration': {'anyOf': [{'type': 'integer'}, {'type': 'null'}]}, 'coverage': {'anyOf': [{'type': 'number'}, {'type': 'null'}]}, 'web_url': {'type': ['string', 'null']}}}
Errors during transform: [user: data does not match {'type': 'object', 'properties': {'name': {'type': 'string'}, 'username': {'type': 'string'}, 'id': {'type': 'integer'}, 'state': {'type': 'string'}}}, committed_at: data does not match {'type': 'string', 'format': 'date-time'}, coverage: data does not match {'type': 'number'}, : data does not match {'type': 'object', 'properties': {'project_id': {'type': ['integer', 'null']}, 'id': {'type': ['integer', 'null']}, 'status': {'type': ['string', 'null']}, 'ref': {'type': ['string', 'null']}, 'sha': {'type': ['string', 'null']}, 'before_sha': {'type': ['string', 'null']}, 'tag': {'type': ['boolean', 'null']}, 'yaml_errors': {'type': ['string', 'null']}, 'user': {'type': 'object', 'properties': {'name': {'type': 'string'}, 'username': {'type': 'string'}, 'id': {'type': 'integer'}, 'state': {'type': 'string'}}}, 'created_at': {'anyOf': [{'type': 'string', 'format': 'date-time'}, {'type': 'null'}]}, 'updated_at': {'anyOf': [{'type': 'string', 'format': 'date-time'}, {'type': 'null'}]}, 'started_at': {'anyOf': [{'type': 'string', 'format': 'date-time'}, {'type': 'null'}]}, 'finished_at': {'anyOf': [{'type': 'string', 'format': 'date-time'}, {'type': 'null'}]}, 'committed_at': {'anyOf': [{'type': 'string', 'format': 'date-time'}, {'type': 'null'}]}, 'duration': {'anyOf': [{'type': 'integer'}, {'type': 'null'}]}, 'coverage': {'anyOf': [{'type': 'number'}, {'type': 'null'}]}, 'web_url': {'type': ['string', 'null']}}}]
Steps to reproduce:
API version:
0.9.15
&0.10.0
Python: 3.7.3
Config:
{
"api_url": "https://",
"private_token": "",
"groups": "",
"projects": "",
"start_date": "",
"ultimate_license": true,
"fetch_merge_request_commits": true,
"fetch_pipelines_extended": true
}
Command:
tap-gitlab --config tap-gitlab-config
The text was updated successfully, but these errors were encountered: