Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

labelbox integration #2995

Merged
merged 58 commits into from
Dec 12, 2024
Merged
Changes from 1 commit
Commits
Show all changes
58 commits
Select commit Hold shift + click to select a range
ce6f182
wip labelbox integration
tyesayan Nov 21, 2024
8307576
changes:
tyesayan Nov 22, 2024
c24cc23
document labelbox converter api
tyesayan Nov 22, 2024
a92d216
change labelbox metadata in deeplake dataset info
tyesayan Nov 23, 2024
e42e2c3
add secret
artgish Nov 25, 2024
762ba0a
add retries for labelbox video download and mark to skip integration …
tyesayan Nov 25, 2024
58294a3
fix labelbox json parsing issues
tyesayan Nov 25, 2024
6289211
add labelbox groupping support
tyesayan Nov 25, 2024
00feb16
fix CI failures
tyesayan Nov 25, 2024
ebcc166
add creds and fail_on_error args to labelbox integration functions
tyesayan Nov 25, 2024
fb83641
Merge branch 'v3' into v3-labelbox-integration
tyesayan Nov 26, 2024
ae58f9b
reformat labelbox files
tyesayan Nov 26, 2024
d1e52b7
Merge branch 'v3-labelbox-integration' of https://github.com/activelo…
tyesayan Nov 26, 2024
5cd0dda
fix mypy
activesoull Nov 26, 2024
2baedcf
add cred_id arg for labelbox dataset create functions
tyesayan Nov 26, 2024
34b0761
Merge branch 'v3-labelbox-integration' of https://github.com/activelo…
tyesayan Nov 26, 2024
b49c3e4
fix typo
tyesayan Nov 26, 2024
40af0ac
fix typo
tyesayan Nov 26, 2024
e80e9da
fix remote urls upload to labelbox
Nov 27, 2024
cfeff84
update test_labebox project id
Nov 27, 2024
253a945
add url_presigner to handle delegated access video resources from sto…
tyesayan Nov 27, 2024
d2c8175
Merge branch 'v3-labelbox-integration' of https://github.com/activelo…
tyesayan Nov 27, 2024
6589ffd
faster frame fetching and some imporvements in labelbox integration
Nov 27, 2024
879f684
labelox integration perforamce improvement and fixes
tyesayan Nov 27, 2024
94f2aea
download videos before generating frames
tyesayan Dec 2, 2024
64aa4e7
fix mask generating issue
tyesayan Dec 2, 2024
d4d2510
fix av error exception
tyesayan Dec 2, 2024
3977c1a
implement custom interpolators for labelbox
tyesayan Dec 2, 2024
f420b76
bring back generating frames from stream for labelbox
tyesayan Dec 2, 2024
cafdfca
fix labelbox annotations interpolation
tyesayan Dec 2, 2024
9ae0cb5
fix labelbox samples count
tyesayan Dec 3, 2024
6ee3bbe
add fail_on_labelbox_project_export_error argument to skip labelbox e…
tyesayan Dec 3, 2024
3dca0a9
Merge branch 'v3' of https://github.com/activeloopai/deeplake into v3…
tyesayan Dec 4, 2024
e8b5e95
fix assertion failure in labelbox integration while parsing segments
tyesayan Dec 4, 2024
a143b4e
add metadata support in labelbox integration
tyesayan Dec 5, 2024
edf0e54
fix labelbox values interpolation
tyesayan Dec 5, 2024
669e26f
add duplication check for create_labelbox_annotation_project
tyesayan Dec 6, 2024
88b548b
fix duplicate label addition for checkboxes during interpolation
tyesayan Dec 7, 2024
60e2426
update connect to labelbox test
tyesayan Dec 8, 2024
f29fceb
reformat labelbox integration files
tyesayan Dec 8, 2024
f30b95b
fix labelbox_utils mypy errors
tyesayan Dec 8, 2024
e3cec14
update labelbox integration test
tyesayan Dec 10, 2024
e70faac
fix labelbox integration test
tyesayan Dec 10, 2024
b68d910
Merge branch 'v3' of https://github.com/activeloopai/deeplake into v3…
tyesayan Dec 10, 2024
1719be5
fix mypy lint error
tyesayan Dec 10, 2024
c15d447
Trigger Action
zaaram Dec 10, 2024
bb18a99
add labelbox azure utils test
tyesayan Dec 11, 2024
560d4b9
Merge branch 'v3-labelbox-integration' of https://github.com/activelo…
tyesayan Dec 11, 2024
d7a6ed2
cleanup labelbox debug classes
tyesayan Dec 11, 2024
b5b2f23
reformat labelbox ingestion files
tyesayan Dec 11, 2024
937b2d9
fixed darglint
activesoull Dec 11, 2024
06dde0c
Merge branch 'v3-labelbox-integration' of github.com:activeloopai/dee…
activesoull Dec 11, 2024
dd3557b
fixed darglint
activesoull Dec 11, 2024
519b357
use labelbox export_v2 instead of export
tyesayan Dec 11, 2024
7b38832
update error log for labelbox project export
tyesayan Dec 11, 2024
b13f582
skip test_connect_to_labelbox
tyesayan Dec 11, 2024
1a2acc0
reformat test_labelbox.py
tyesayan Dec 11, 2024
90839e0
skip test_labelbox_azure_utils
tyesayan Dec 11, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
bring back generating frames from stream for labelbox
tyesayan committed Dec 2, 2024

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
commit f420b765f8f89e6bdf49fef2c98542b32830ad81
20 changes: 0 additions & 20 deletions deeplake/integrations/labelbox/labelbox_.py
Original file line number Diff line number Diff line change
@@ -51,25 +51,25 @@
- Supports Video ontology from labelbox.
- The function first validates the project data before setting up converters.
"""
project_json = labelbox_get_project_json_with_id_(client, project_id, fail_on_error)

Check warning on line 54 in deeplake/integrations/labelbox/labelbox_.py

Codecov / codecov/patch

deeplake/integrations/labelbox/labelbox_.py#L54

Added line #L54 was not covered by tests

if len(project_json) == 0:
print("no data")
return None

Check warning on line 58 in deeplake/integrations/labelbox/labelbox_.py

Codecov / codecov/patch

deeplake/integrations/labelbox/labelbox_.py#L56-L58

Added lines #L56 - L58 were not covered by tests

ds_name = project_json[0]["projects"][project_id]["name"]
deeplake_dataset = deeplake_ds_loader(ds_name)

Check warning on line 61 in deeplake/integrations/labelbox/labelbox_.py

Codecov / codecov/patch

deeplake/integrations/labelbox/labelbox_.py#L60-L61

Added lines #L60 - L61 were not covered by tests

if not validate_project_data_(project_json, deeplake_dataset, project_id, "video"):
if fail_on_error:
raise Exception("Data validation failed")

Check warning on line 65 in deeplake/integrations/labelbox/labelbox_.py

Codecov / codecov/patch

deeplake/integrations/labelbox/labelbox_.py#L63-L65

Added lines #L63 - L65 were not covered by tests

ontology_id = project_json[0]["projects"][project_id]["project_details"][

Check warning on line 67 in deeplake/integrations/labelbox/labelbox_.py

Codecov / codecov/patch

deeplake/integrations/labelbox/labelbox_.py#L67

Added line #L67 was not covered by tests
"ontology_id"
]
ontology = client.get_ontology(ontology_id)

Check warning on line 70 in deeplake/integrations/labelbox/labelbox_.py

Codecov / codecov/patch

deeplake/integrations/labelbox/labelbox_.py#L70

Added line #L70 was not covered by tests

converters = {

Check warning on line 72 in deeplake/integrations/labelbox/labelbox_.py

Codecov / codecov/patch

deeplake/integrations/labelbox/labelbox_.py#L72

Added line #L72 was not covered by tests
"rectangle": bbox_converter_,
"radio": radio_converter_,
"checklist": checkbox_converter_,
@@ -78,7 +78,7 @@
"raster-segmentation": raster_segmentation_converter_,
"text": text_converter_,
}
return labelbox_video_converter(

Check warning on line 81 in deeplake/integrations/labelbox/labelbox_.py

Codecov / codecov/patch

deeplake/integrations/labelbox/labelbox_.py#L81

Added line #L81 was not covered by tests
ontology,
converters,
project_json,
@@ -131,7 +131,7 @@
Returns:
Dataset: Created Deeplake dataset containing processed video frames and metadata for Labelbox project
"""
ds = deeplake.empty(

Check warning on line 134 in deeplake/integrations/labelbox/labelbox_.py

Codecov / codecov/patch

deeplake/integrations/labelbox/labelbox_.py#L134

Added line #L134 was not covered by tests
deeplake_ds_path,
creds=deeplake_creds,
org_id=deeplake_org_id,
@@ -139,66 +139,66 @@
overwrite=overwrite,
)

data_filler["create_tensors"](ds)

Check warning on line 142 in deeplake/integrations/labelbox/labelbox_.py

Codecov / codecov/patch

deeplake/integrations/labelbox/labelbox_.py#L142

Added line #L142 was not covered by tests

for idx, video_path in enumerate(video_paths):
for frame_indexes, frames in frames_batch_generator_(video_path, batch_size=video_generator_batch_size):
data_filler["fill_data"](ds, [idx] * len(frames), frame_indexes, frames)

Check warning on line 146 in deeplake/integrations/labelbox/labelbox_.py

Codecov / codecov/patch

deeplake/integrations/labelbox/labelbox_.py#L144-L146

Added lines #L144 - L146 were not covered by tests

if lb_dataset_name is None:
lb_dataset_name = os.path.basename(deeplake_ds_path) + "_from_deeplake"

Check warning on line 149 in deeplake/integrations/labelbox/labelbox_.py

Codecov / codecov/patch

deeplake/integrations/labelbox/labelbox_.py#L148-L149

Added lines #L148 - L149 were not covered by tests

assets = video_paths

Check warning on line 151 in deeplake/integrations/labelbox/labelbox_.py

Codecov / codecov/patch

deeplake/integrations/labelbox/labelbox_.py#L151

Added line #L151 was not covered by tests

# validate paths
all_local = [os.path.exists(p) for p in video_paths]
if any(all_local) and not all(all_local):
raise Exception(f'video paths must be all local or all remote: {video_paths}')

Check warning on line 156 in deeplake/integrations/labelbox/labelbox_.py

Codecov / codecov/patch

deeplake/integrations/labelbox/labelbox_.py#L154-L156

Added lines #L154 - L156 were not covered by tests

if len(all_local):
if not all_local[0]:
assets = [{

Check warning on line 160 in deeplake/integrations/labelbox/labelbox_.py

Codecov / codecov/patch

deeplake/integrations/labelbox/labelbox_.py#L158-L160

Added lines #L158 - L160 were not covered by tests
"row_data": p,
"media_type": "VIDEO",
"metadata_fields": [],
"attachments": []
} for p in video_paths]

print('uploading videos to labelbox')
lb_ds = lb_client.create_dataset(name=lb_dataset_name)
task = lb_ds.create_data_rows(assets)
task.wait_till_done()

Check warning on line 170 in deeplake/integrations/labelbox/labelbox_.py

Codecov / codecov/patch

deeplake/integrations/labelbox/labelbox_.py#L167-L170

Added lines #L167 - L170 were not covered by tests

if task.errors:
raise Exception(f'failed to upload videos to labelbox: {task.errors}')

Check warning on line 173 in deeplake/integrations/labelbox/labelbox_.py

Codecov / codecov/patch

deeplake/integrations/labelbox/labelbox_.py#L172-L173

Added lines #L172 - L173 were not covered by tests

print('successfuly uploaded videos to labelbox')

Check warning on line 175 in deeplake/integrations/labelbox/labelbox_.py

Codecov / codecov/patch

deeplake/integrations/labelbox/labelbox_.py#L175

Added line #L175 was not covered by tests

# Create a new project
project = lb_client.create_project(

Check warning on line 178 in deeplake/integrations/labelbox/labelbox_.py

Codecov / codecov/patch

deeplake/integrations/labelbox/labelbox_.py#L178

Added line #L178 was not covered by tests
name=os.path.basename(deeplake_ds_path), media_type=lb.MediaType.Video
)

ds.info["labelbox_meta"] = {

Check warning on line 182 in deeplake/integrations/labelbox/labelbox_.py

Codecov / codecov/patch

deeplake/integrations/labelbox/labelbox_.py#L182

Added line #L182 was not covered by tests
"project_id": project.uid,
"type": "video",
"sources": video_paths,
}

task = project.create_batches_from_dataset(

Check warning on line 188 in deeplake/integrations/labelbox/labelbox_.py

Codecov / codecov/patch

deeplake/integrations/labelbox/labelbox_.py#L188

Added line #L188 was not covered by tests
name_prefix=lb_dataset_name, dataset_id=lb_ds.uid, priority=lb_batch_priority
)

if task.errors():
if fail_on_error:
raise Exception(f"Error creating batches: {task.errors()}")

Check warning on line 194 in deeplake/integrations/labelbox/labelbox_.py

Codecov / codecov/patch

deeplake/integrations/labelbox/labelbox_.py#L192-L194

Added lines #L192 - L194 were not covered by tests

if lb_ontology:
project.connect_ontology(lb_ontology)

Check warning on line 197 in deeplake/integrations/labelbox/labelbox_.py

Codecov / codecov/patch

deeplake/integrations/labelbox/labelbox_.py#L196-L197

Added lines #L196 - L197 were not covered by tests

ds.commit()

Check warning on line 199 in deeplake/integrations/labelbox/labelbox_.py

Codecov / codecov/patch

deeplake/integrations/labelbox/labelbox_.py#L199

Added line #L199 was not covered by tests

return ds

Check warning on line 201 in deeplake/integrations/labelbox/labelbox_.py

Codecov / codecov/patch

deeplake/integrations/labelbox/labelbox_.py#L201

Added line #L201 was not covered by tests


def create_dataset_for_video_annotation(
@@ -221,7 +221,7 @@
- create_tensors_default_: Creates default tensor structure
- fill_data_default_: Fills tensors with default processing
"""
return create_dataset_for_video_annotation_with_custom_data_filler(

Check warning on line 224 in deeplake/integrations/labelbox/labelbox_.py

Codecov / codecov/patch

deeplake/integrations/labelbox/labelbox_.py#L224

Added line #L224 was not covered by tests
deeplake_ds_path,
video_paths,
lb_client,
@@ -284,72 +284,52 @@
Notes:
- The function does not fetch the annotations from Labelbox, only the video frames. After creating the dataset, use the converter to apply annotations.
"""
ds = deeplake.empty(

Check warning on line 287 in deeplake/integrations/labelbox/labelbox_.py

Codecov / codecov/patch

deeplake/integrations/labelbox/labelbox_.py#L287

Added line #L287 was not covered by tests
deeplake_ds_path,
overwrite=overwrite,
creds=deeplake_creds,
org_id=deeplake_org_id,
token=deeplake_token,
)
data_filler["create_tensors"](ds)

Check warning on line 294 in deeplake/integrations/labelbox/labelbox_.py

Codecov / codecov/patch

deeplake/integrations/labelbox/labelbox_.py#L294

Added line #L294 was not covered by tests

proj = labelbox_get_project_json_with_id_(lb_client, project_id, fail_on_error)
if len(proj) == 0:
print("no data")
return ds

Check warning on line 299 in deeplake/integrations/labelbox/labelbox_.py

Codecov / codecov/patch

deeplake/integrations/labelbox/labelbox_.py#L296-L299

Added lines #L296 - L299 were not covered by tests

if not validate_project_creation_data_(proj, project_id, "video"):
if fail_on_error:
raise Exception("Data validation failed")

Check warning on line 303 in deeplake/integrations/labelbox/labelbox_.py

Codecov / codecov/patch

deeplake/integrations/labelbox/labelbox_.py#L301-L303

Added lines #L301 - L303 were not covered by tests

video_files = []

Check warning on line 305 in deeplake/integrations/labelbox/labelbox_.py

Codecov / codecov/patch

deeplake/integrations/labelbox/labelbox_.py#L305

Added line #L305 was not covered by tests

if url_presigner is None:
def default_presigner(url):
if lb_api_key is None:
return url, {}
return url, {"headers": {"Authorization": f"Bearer {lb_api_key}"}}
url_presigner = default_presigner

Check warning on line 312 in deeplake/integrations/labelbox/labelbox_.py

Codecov / codecov/patch

deeplake/integrations/labelbox/labelbox_.py#L307-L312

Added lines #L307 - L312 were not covered by tests

for idx, p in enumerate(proj):
video_url = p["data_row"]["row_data"]
header = None

tmp_path = None
if not os.path.exists(video_url):
if not is_remote_resource_public_(video_url):
video_url, header = url_presigner(video_url)

# temp solution for some cases when we can't download video directly
tmp_path = tempfile.NamedTemporaryFile(delete=True)
try:
download_file_(video_url, tmp_path.name, header=header)
header = None
video_url = tmp_path.name
except Exception as e:
tmp_path.close()
tmp_path = None
if fail_on_error:
raise Exception(f"An error occurred: {e} while downloading video from {video_url}")
print(f"An error occurred: {e} while downloading video from {video_url}")
continue

for frame_indexes, frames in frames_batch_generator_(video_url, header=header, batch_size=video_generator_batch_size):
data_filler["fill_data"](ds, [idx] * len(frames), frame_indexes, frames)

if tmp_path:
tmp_path.close()
video_files.append(external_url_from_video_project_(p))

Check warning on line 322 in deeplake/integrations/labelbox/labelbox_.py

Codecov / codecov/patch

deeplake/integrations/labelbox/labelbox_.py#L314-L322

Added lines #L314 - L322 were not covered by tests

ds.info["labelbox_meta"] = {

Check warning on line 324 in deeplake/integrations/labelbox/labelbox_.py

Codecov / codecov/patch

deeplake/integrations/labelbox/labelbox_.py#L324

Added line #L324 was not covered by tests
"project_id": project_id,
"type": "video",
"sources": video_files,
}

ds.commit()

Check warning on line 330 in deeplake/integrations/labelbox/labelbox_.py

Codecov / codecov/patch

deeplake/integrations/labelbox/labelbox_.py#L330

Added line #L330 was not covered by tests

return ds

Check warning on line 332 in deeplake/integrations/labelbox/labelbox_.py

Codecov / codecov/patch

deeplake/integrations/labelbox/labelbox_.py#L332

Added line #L332 was not covered by tests


def create_dataset_from_video_annotation_project(
@@ -372,7 +352,7 @@
- create_tensors_default_: Creates default tensor structure
- fill_data_default_: Fills tensors with default processing
"""
return create_dataset_from_video_annotation_project_with_custom_data_filler(

Check warning on line 355 in deeplake/integrations/labelbox/labelbox_.py

Codecov / codecov/patch

deeplake/integrations/labelbox/labelbox_.py#L355

Added line #L355 was not covered by tests
deeplake_ds_path,
project_id,
lb_client,
2 changes: 1 addition & 1 deletion deeplake/integrations/labelbox/labelbox_converter.py
Original file line number Diff line number Diff line change
@@ -12,24 +12,24 @@
context,
group_mapping=None,
):
self.labelbox_feature_id_to_type_mapping = dict()
self.regsistered_actions = dict()
self.label_mappings = dict()
self.values_cache = dict()
self.registered_interpolators = dict()

Check warning on line 19 in deeplake/integrations/labelbox/labelbox_converter.py

Codecov / codecov/patch

deeplake/integrations/labelbox/labelbox_converter.py#L15-L19

Added lines #L15 - L19 were not covered by tests

self.project = project
self.project_id = project_id
self.dataset = dataset

Check warning on line 23 in deeplake/integrations/labelbox/labelbox_converter.py

Codecov / codecov/patch

deeplake/integrations/labelbox/labelbox_converter.py#L21-L23

Added lines #L21 - L23 were not covered by tests

self.group_mapping = group_mapping if group_mapping is not None else dict()

Check warning on line 25 in deeplake/integrations/labelbox/labelbox_converter.py

Codecov / codecov/patch

deeplake/integrations/labelbox/labelbox_converter.py#L25

Added line #L25 was not covered by tests

self.labelbox_type_converters_ = converters

Check warning on line 27 in deeplake/integrations/labelbox/labelbox_converter.py

Codecov / codecov/patch

deeplake/integrations/labelbox/labelbox_converter.py#L27

Added line #L27 was not covered by tests

self.register_ontology_(ontology, context)

Check warning on line 29 in deeplake/integrations/labelbox/labelbox_converter.py

Codecov / codecov/patch

deeplake/integrations/labelbox/labelbox_converter.py#L29

Added line #L29 was not covered by tests

def register_feature_id_for_kind(self, kind, key, obj, tensor_name):
self.labelbox_feature_id_to_type_mapping[obj.feature_schema_id] = {

Check warning on line 32 in deeplake/integrations/labelbox/labelbox_converter.py

Codecov / codecov/patch

deeplake/integrations/labelbox/labelbox_converter.py#L32

Added line #L32 was not covered by tests
"kind": kind,
"key": key,
"name": obj.name,
@@ -37,88 +37,88 @@
}

def dataset_with_applied_annotations(self):
idx_offset = 0
print('total annotations projects count: ', len(self.project))

Check warning on line 41 in deeplake/integrations/labelbox/labelbox_converter.py

Codecov / codecov/patch

deeplake/integrations/labelbox/labelbox_converter.py#L40-L41

Added lines #L40 - L41 were not covered by tests

for p_idx, p in enumerate(self.yield_projects_(self.project, self.dataset)):
if "labels" not in p["projects"][self.project_id]:
print('no labels for project with index: ', p_idx)
continue
print('parsing annotations for project with index: ', p_idx)
for lbl_idx, labels in enumerate(p["projects"][self.project_id]["labels"]):
self.values_cache = dict()
if "frames" not in labels["annotations"]:
continue
frames = labels["annotations"]["frames"]
if not len(frames):
print(

Check warning on line 54 in deeplake/integrations/labelbox/labelbox_converter.py

Codecov / codecov/patch

deeplake/integrations/labelbox/labelbox_converter.py#L43-L54

Added lines #L43 - L54 were not covered by tests
"skip",
external_url_from_video_project_(p),
"with label idx",
lbl_idx,
"as it has no frames",
)
continue

Check warning on line 61 in deeplake/integrations/labelbox/labelbox_converter.py

Codecov / codecov/patch

deeplake/integrations/labelbox/labelbox_converter.py#L61

Added line #L61 was not covered by tests

assert len(frames) <= p["media_attributes"]["frame_count"]

Check warning on line 63 in deeplake/integrations/labelbox/labelbox_converter.py

Codecov / codecov/patch

deeplake/integrations/labelbox/labelbox_converter.py#L63

Added line #L63 was not covered by tests

print('parsing frames for label index: ', lbl_idx)
for i in tqdm.tqdm(range(p["media_attributes"]["frame_count"])):
if str(i + 1) not in frames:
continue
self.parse_frame_(frames[str(i + 1)], idx_offset + i)

Check warning on line 69 in deeplake/integrations/labelbox/labelbox_converter.py

Codecov / codecov/patch

deeplake/integrations/labelbox/labelbox_converter.py#L65-L69

Added lines #L65 - L69 were not covered by tests

if "segments" not in labels["annotations"]:
continue
segments = labels["annotations"]["segments"]

Check warning on line 73 in deeplake/integrations/labelbox/labelbox_converter.py

Codecov / codecov/patch

deeplake/integrations/labelbox/labelbox_converter.py#L71-L73

Added lines #L71 - L73 were not covered by tests
# the frames contain only the interpolated values
# iterate over segments and assign same value to all frames in the segment
self.parse_segments_(segments, frames, idx_offset)

Check warning on line 76 in deeplake/integrations/labelbox/labelbox_converter.py

Codecov / codecov/patch

deeplake/integrations/labelbox/labelbox_converter.py#L76

Added line #L76 was not covered by tests

self.apply_cached_values_(self.values_cache)

Check warning on line 78 in deeplake/integrations/labelbox/labelbox_converter.py

Codecov / codecov/patch

deeplake/integrations/labelbox/labelbox_converter.py#L78

Added line #L78 was not covered by tests

idx_offset += p["media_attributes"]["frame_count"]

Check warning on line 80 in deeplake/integrations/labelbox/labelbox_converter.py

Codecov / codecov/patch

deeplake/integrations/labelbox/labelbox_converter.py#L80

Added line #L80 was not covered by tests

return self.dataset

Check warning on line 82 in deeplake/integrations/labelbox/labelbox_converter.py

Codecov / codecov/patch

deeplake/integrations/labelbox/labelbox_converter.py#L82

Added line #L82 was not covered by tests

def register_tool_(self, tool, context):
if tool.tool.value not in self.labelbox_type_converters_:
print("skip tool:", tool.tool.value)
return

Check warning on line 87 in deeplake/integrations/labelbox/labelbox_converter.py

Codecov / codecov/patch

deeplake/integrations/labelbox/labelbox_converter.py#L85-L87

Added lines #L85 - L87 were not covered by tests

prefered_name = tool.name

Check warning on line 89 in deeplake/integrations/labelbox/labelbox_converter.py

Codecov / codecov/patch

deeplake/integrations/labelbox/labelbox_converter.py#L89

Added line #L89 was not covered by tests

if tool.tool.value in self.group_mapping:
prefered_name = self.group_mapping[tool.tool.value]

Check warning on line 92 in deeplake/integrations/labelbox/labelbox_converter.py

Codecov / codecov/patch

deeplake/integrations/labelbox/labelbox_converter.py#L91-L92

Added lines #L91 - L92 were not covered by tests
else:
prefered_name = tool.name

Check warning on line 94 in deeplake/integrations/labelbox/labelbox_converter.py

Codecov / codecov/patch

deeplake/integrations/labelbox/labelbox_converter.py#L94

Added line #L94 was not covered by tests

should_group_with_classifications = len(tool.classifications) > 0
tool_name = (

Check warning on line 97 in deeplake/integrations/labelbox/labelbox_converter.py

Codecov / codecov/patch

deeplake/integrations/labelbox/labelbox_converter.py#L96-L97

Added lines #L96 - L97 were not covered by tests
prefered_name + "/" + prefered_name
if should_group_with_classifications
else prefered_name
)

self.labelbox_type_converters_[tool.tool.value](

Check warning on line 103 in deeplake/integrations/labelbox/labelbox_converter.py

Codecov / codecov/patch

deeplake/integrations/labelbox/labelbox_converter.py#L103

Added line #L103 was not covered by tests
tool, self, tool_name, context, tool.tool.value in self.group_mapping
)

for classification in tool.classifications:
self.register_classification_(classification, context, parent=prefered_name)

Check warning on line 108 in deeplake/integrations/labelbox/labelbox_converter.py

Codecov / codecov/patch

deeplake/integrations/labelbox/labelbox_converter.py#L107-L108

Added lines #L107 - L108 were not covered by tests

def register_classification_(self, tool, context, parent=""):
if tool.class_type.value not in self.labelbox_type_converters_:
return

Check warning on line 112 in deeplake/integrations/labelbox/labelbox_converter.py

Codecov / codecov/patch

deeplake/integrations/labelbox/labelbox_converter.py#L111-L112

Added lines #L111 - L112 were not covered by tests

if tool.class_type.value in self.group_mapping:
prefered_name = (parent + "/" if parent else "") + self.group_mapping[

Check warning on line 115 in deeplake/integrations/labelbox/labelbox_converter.py

Codecov / codecov/patch

deeplake/integrations/labelbox/labelbox_converter.py#L114-L115

Added lines #L114 - L115 were not covered by tests
tool.class_type.value
]
else:
prefered_name = (parent + "/" if parent else "") + tool.name

Check warning on line 119 in deeplake/integrations/labelbox/labelbox_converter.py

Codecov / codecov/patch

deeplake/integrations/labelbox/labelbox_converter.py#L119

Added line #L119 was not covered by tests

self.labelbox_type_converters_[tool.class_type.value](

Check warning on line 121 in deeplake/integrations/labelbox/labelbox_converter.py

Codecov / codecov/patch

deeplake/integrations/labelbox/labelbox_converter.py#L121

Added line #L121 was not covered by tests
tool,
self,
prefered_name,
@@ -127,134 +127,134 @@
)

def register_ontology_(self, ontology, context):
for tool in ontology.tools():
self.register_tool_(tool, context)

Check warning on line 131 in deeplake/integrations/labelbox/labelbox_converter.py

Codecov / codecov/patch

deeplake/integrations/labelbox/labelbox_converter.py#L130-L131

Added lines #L130 - L131 were not covered by tests

for classification in ontology.classifications():
if classification.scope.value != "index":
print("skip global classification:", classification.name)
continue
self.register_classification_(classification, context)

Check warning on line 137 in deeplake/integrations/labelbox/labelbox_converter.py

Codecov / codecov/patch

deeplake/integrations/labelbox/labelbox_converter.py#L133-L137

Added lines #L133 - L137 were not covered by tests

def parse_frame_(self, frame, idx):
if "objects" in frame:
for _, obj in frame["objects"].items():
self.parse_object_(obj, idx)

Check warning on line 142 in deeplake/integrations/labelbox/labelbox_converter.py

Codecov / codecov/patch

deeplake/integrations/labelbox/labelbox_converter.py#L140-L142

Added lines #L140 - L142 were not covered by tests

if "classifications" in frame:
for obj in frame["classifications"]:
self.parse_classification_(obj, idx)

Check warning on line 146 in deeplake/integrations/labelbox/labelbox_converter.py

Codecov / codecov/patch

deeplake/integrations/labelbox/labelbox_converter.py#L144-L146

Added lines #L144 - L146 were not covered by tests

def parse_object_(self, obj, idx):
if obj["feature_schema_id"] not in self.regsistered_actions:
print("skip object:", obj["feature_schema_id"])
return

Check warning on line 151 in deeplake/integrations/labelbox/labelbox_converter.py

Codecov / codecov/patch

deeplake/integrations/labelbox/labelbox_converter.py#L149-L151

Added lines #L149 - L151 were not covered by tests

self.regsistered_actions[obj["feature_schema_id"]](idx, obj)

Check warning on line 153 in deeplake/integrations/labelbox/labelbox_converter.py

Codecov / codecov/patch

deeplake/integrations/labelbox/labelbox_converter.py#L153

Added line #L153 was not covered by tests

if "classifications" in obj:
for obj in obj["classifications"]:
self.parse_classification_(obj, idx)

Check warning on line 157 in deeplake/integrations/labelbox/labelbox_converter.py

Codecov / codecov/patch

deeplake/integrations/labelbox/labelbox_converter.py#L155-L157

Added lines #L155 - L157 were not covered by tests

def parse_classification_(self, obj, idx):
if obj["feature_schema_id"] not in self.regsistered_actions:
print("skip classification:", obj["feature_schema_id"])
return

Check warning on line 162 in deeplake/integrations/labelbox/labelbox_converter.py

Codecov / codecov/patch

deeplake/integrations/labelbox/labelbox_converter.py#L160-L162

Added lines #L160 - L162 were not covered by tests

self.regsistered_actions[obj["feature_schema_id"]](idx, obj)

Check warning on line 164 in deeplake/integrations/labelbox/labelbox_converter.py

Codecov / codecov/patch

deeplake/integrations/labelbox/labelbox_converter.py#L164

Added line #L164 was not covered by tests

if "classifications" in obj:
for obj in obj["classifications"]:
self.parse_classification_(obj, idx)

Check warning on line 168 in deeplake/integrations/labelbox/labelbox_converter.py

Codecov / codecov/patch

deeplake/integrations/labelbox/labelbox_converter.py#L166-L168

Added lines #L166 - L168 were not covered by tests

def find_object_with_feature_id_(self, frame, feature_id):
if isinstance(frame, list):
for f in frame:
if ret := self.find_object_with_feature_id_(f, feature_id):
return ret

Check warning on line 174 in deeplake/integrations/labelbox/labelbox_converter.py

Codecov / codecov/patch

deeplake/integrations/labelbox/labelbox_converter.py#L171-L174

Added lines #L171 - L174 were not covered by tests

if "objects" in frame:
if feature_id in frame["objects"]:
return frame["objects"][feature_id]
for _, obj in frame["objects"].items():
if ret := self.find_object_with_feature_id_(obj, feature_id):
return ret

Check warning on line 181 in deeplake/integrations/labelbox/labelbox_converter.py

Codecov / codecov/patch

deeplake/integrations/labelbox/labelbox_converter.py#L176-L181

Added lines #L176 - L181 were not covered by tests

if "classifications" in frame:
for obj in frame["classifications"]:
if ret := self.find_object_with_feature_id_(obj, feature_id):
return ret
k = self.labelbox_feature_id_to_type_mapping[obj["feature_schema_id"]][

Check warning on line 187 in deeplake/integrations/labelbox/labelbox_converter.py

Codecov / codecov/patch

deeplake/integrations/labelbox/labelbox_converter.py#L183-L187

Added lines #L183 - L187 were not covered by tests
"key"
]
if k in obj:
if ret := self.find_object_with_feature_id_(obj[k], feature_id):
return ret

Check warning on line 192 in deeplake/integrations/labelbox/labelbox_converter.py

Codecov / codecov/patch

deeplake/integrations/labelbox/labelbox_converter.py#L190-L192

Added lines #L190 - L192 were not covered by tests

if "feature_id" in frame and frame["feature_id"] == feature_id:
return frame

Check warning on line 195 in deeplake/integrations/labelbox/labelbox_converter.py

Codecov / codecov/patch

deeplake/integrations/labelbox/labelbox_converter.py#L194-L195

Added lines #L194 - L195 were not covered by tests

return None

Check warning on line 197 in deeplake/integrations/labelbox/labelbox_converter.py

Codecov / codecov/patch

deeplake/integrations/labelbox/labelbox_converter.py#L197

Added line #L197 was not covered by tests

def existing_sub_ranges_(self, frames, range):
sub_ranges = [(range[0], range[1])]
for i in range[0], range[1]:
if str(i) in frames:
continue
sub_ranges[-1] = (sub_ranges[-1][0], i)
sub_ranges.append((i, range[1]))
return sub_ranges

Check warning on line 206 in deeplake/integrations/labelbox/labelbox_converter.py

Codecov / codecov/patch

deeplake/integrations/labelbox/labelbox_converter.py#L200-L206

Added lines #L200 - L206 were not covered by tests


def parse_segments_(self, segments, frames, offset):
print('total segments count to parse:', len(segments))
for feature_id, ranges in segments.items():
print('parsing segments with feature id: ', feature_id)
for r in tqdm.tqdm(ranges):
sub_ranges = self.existing_sub_ranges_(frames, r)
for st, en in sub_ranges:
assert str(st) in frames
assert str(en) in frames

Check warning on line 217 in deeplake/integrations/labelbox/labelbox_converter.py

Codecov / codecov/patch

deeplake/integrations/labelbox/labelbox_converter.py#L210-L217

Added lines #L210 - L217 were not covered by tests

start = self.find_object_with_feature_id_(frames[str(st)], feature_id)
end = self.find_object_with_feature_id_(frames[str(en)], feature_id)

Check warning on line 220 in deeplake/integrations/labelbox/labelbox_converter.py

Codecov / codecov/patch

deeplake/integrations/labelbox/labelbox_converter.py#L219-L220

Added lines #L219 - L220 were not covered by tests

assert start
assert end

Check warning on line 223 in deeplake/integrations/labelbox/labelbox_converter.py

Codecov / codecov/patch

deeplake/integrations/labelbox/labelbox_converter.py#L222-L223

Added lines #L222 - L223 were not covered by tests

if start == end:
continue

Check warning on line 226 in deeplake/integrations/labelbox/labelbox_converter.py

Codecov / codecov/patch

deeplake/integrations/labelbox/labelbox_converter.py#L225-L226

Added lines #L225 - L226 were not covered by tests

assert start["feature_schema_id"] == end["feature_schema_id"]

Check warning on line 228 in deeplake/integrations/labelbox/labelbox_converter.py

Codecov / codecov/patch

deeplake/integrations/labelbox/labelbox_converter.py#L228

Added line #L228 was not covered by tests

for i in range(st + 1, en):
if str(i) in frames:
obj = self.find_object_with_feature_id_(frames[str(i)], feature_id)

Check warning on line 232 in deeplake/integrations/labelbox/labelbox_converter.py

Codecov / codecov/patch

deeplake/integrations/labelbox/labelbox_converter.py#L230-L232

Added lines #L230 - L232 were not covered by tests
else:
if st['feature_schema_id'] in self.registered_interpolators:
if start['feature_schema_id'] in self.registered_interpolators:
obj = self.registered_interpolators[start["feature_schema_id"]](start, end, (i - st) / (en - st))

Check warning on line 235 in deeplake/integrations/labelbox/labelbox_converter.py

Codecov / codecov/patch

deeplake/integrations/labelbox/labelbox_converter.py#L234-L235

Added lines #L234 - L235 were not covered by tests
else:
obj = end

Check warning on line 237 in deeplake/integrations/labelbox/labelbox_converter.py

Codecov / codecov/patch

deeplake/integrations/labelbox/labelbox_converter.py#L237

Added line #L237 was not covered by tests

self.regsistered_actions[obj["feature_schema_id"]](offset + i - 1, obj)

Check warning on line 239 in deeplake/integrations/labelbox/labelbox_converter.py

Codecov / codecov/patch

deeplake/integrations/labelbox/labelbox_converter.py#L239

Added line #L239 was not covered by tests


def apply_cached_values_(self, cache):
print('applying cached values')
for tensor_name, row_map in cache.items():
print('applying cached values for tensor: ', tensor_name)
max_val = max(row_map.keys())
values = []
for i in tqdm.tqdm(range(max_val + 1)):
if i in row_map:
values.append(row_map[i])

Check warning on line 250 in deeplake/integrations/labelbox/labelbox_converter.py

Codecov / codecov/patch

deeplake/integrations/labelbox/labelbox_converter.py#L243-L250

Added lines #L243 - L250 were not covered by tests
else:
values.append(None)

Check warning on line 252 in deeplake/integrations/labelbox/labelbox_converter.py

Codecov / codecov/patch

deeplake/integrations/labelbox/labelbox_converter.py#L252

Added line #L252 was not covered by tests

self.dataset[tensor_name].extend(values)

Check warning on line 254 in deeplake/integrations/labelbox/labelbox_converter.py

Codecov / codecov/patch

deeplake/integrations/labelbox/labelbox_converter.py#L254

Added line #L254 was not covered by tests

def yield_projects_(self, project_j, ds):
raise NotImplementedError("fixed_project_order_ is not implemented")

Check warning on line 257 in deeplake/integrations/labelbox/labelbox_converter.py

Codecov / codecov/patch

deeplake/integrations/labelbox/labelbox_converter.py#L257

Added line #L257 was not covered by tests


class labelbox_video_converter(labelbox_type_converter):
@@ -268,17 +268,17 @@
context,
group_mapping=None,
):
super().__init__(

Check warning on line 271 in deeplake/integrations/labelbox/labelbox_converter.py

Codecov / codecov/patch

deeplake/integrations/labelbox/labelbox_converter.py#L271

Added line #L271 was not covered by tests
ontology, converters, project, project_id, dataset, context, group_mapping
)

def yield_projects_(self, project_j, ds):
if "labelbox_meta" not in ds.info:
raise ValueError("No labelbox meta data in dataset")
info = ds.info["labelbox_meta"]
def sorter(p):
url = external_url_from_video_project_(p)
return info["sources"].index(url)
ordered_values = sorted(project_j, key=sorter)
for p in ordered_values:
yield p

Check warning on line 284 in deeplake/integrations/labelbox/labelbox_converter.py

Codecov / codecov/patch

deeplake/integrations/labelbox/labelbox_converter.py#L276-L284

Added lines #L276 - L284 were not covered by tests
11 changes: 0 additions & 11 deletions deeplake/integrations/labelbox/labelbox_utils.py
Original file line number Diff line number Diff line change
@@ -3,24 +3,13 @@
import labelbox as lb # type: ignore
import av
import requests
import urllib

def download_file_(url, path, header=None):
request = urllib.request.Request(
url,
headers=header
)
print(f"Downloading video from {url}")
with urllib.request.urlopen(request) as response, open(path, 'wb') as out_file:
while chunk := response.read(8192):
out_file.write(chunk)

def is_remote_resource_public_(url):
try:
response = requests.head(url, allow_redirects=True)
return response.status_code == 200
except requests.exceptions.RequestException as e:
return False

Check warning on line 12 in deeplake/integrations/labelbox/labelbox_utils.py

Codecov / codecov/patch

deeplake/integrations/labelbox/labelbox_utils.py#L8-L12

Added lines #L8 - L12 were not covered by tests

def frame_generator_(
video_path: str, header: dict, retries: int = 5
@@ -38,98 +27,98 @@
- frame_data (numpy.ndarray): The frame image data
"""

def get_video_container(current_retries):
try:
return av.open(video_path, options=header)
except Exception as e:
if current_retries > 0:
print(f"Failed opening video: {e}. Retrying...")
return get_video_container(current_retries - 1)

Check warning on line 36 in deeplake/integrations/labelbox/labelbox_utils.py

Codecov / codecov/patch

deeplake/integrations/labelbox/labelbox_utils.py#L30-L36

Added lines #L30 - L36 were not covered by tests
else:
raise e

Check warning on line 38 in deeplake/integrations/labelbox/labelbox_utils.py

Codecov / codecov/patch

deeplake/integrations/labelbox/labelbox_utils.py#L38

Added line #L38 was not covered by tests

try:
container = get_video_container(retries)
print(f"Start generating frames from {video_path}")
frame_num = 0
for frame in container.decode(video=0):
yield frame_num, frame.to_ndarray(format="rgb24")
frame_num += 1
except Exception as e:
print(f"Failed generating frames: {e}")

Check warning on line 48 in deeplake/integrations/labelbox/labelbox_utils.py

Codecov / codecov/patch

deeplake/integrations/labelbox/labelbox_utils.py#L40-L48

Added lines #L40 - L48 were not covered by tests

def frames_batch_generator_(video_path: str, header: dict=None, batch_size=100, retries: int = 5):
frames, indexes = [], []
for frame_num, frame in frame_generator_(video_path, header, retries):
frames.append(frame)
indexes.append(frame_num)
if len(frames) < batch_size:
continue
yield indexes, frames
frames, indexes = [], []

Check warning on line 58 in deeplake/integrations/labelbox/labelbox_utils.py

Codecov / codecov/patch

deeplake/integrations/labelbox/labelbox_utils.py#L51-L58

Added lines #L51 - L58 were not covered by tests

if len(frames):
yield indexes, frames

Check warning on line 61 in deeplake/integrations/labelbox/labelbox_utils.py

Codecov / codecov/patch

deeplake/integrations/labelbox/labelbox_utils.py#L60-L61

Added lines #L60 - L61 were not covered by tests

def external_url_from_video_project_(p):
if "external_id" in p["data_row"]:
return p["data_row"]["external_id"]
return p["data_row"]["row_data"]

Check warning on line 66 in deeplake/integrations/labelbox/labelbox_utils.py

Codecov / codecov/patch

deeplake/integrations/labelbox/labelbox_utils.py#L64-L66

Added lines #L64 - L66 were not covered by tests

def validate_video_project_data_impl_(project_j, deeplake_dataset, project_id):
if "labelbox_meta" not in deeplake_dataset.info:
return False
info = deeplake_dataset.info["labelbox_meta"]

Check warning on line 71 in deeplake/integrations/labelbox/labelbox_utils.py

Codecov / codecov/patch

deeplake/integrations/labelbox/labelbox_utils.py#L69-L71

Added lines #L69 - L71 were not covered by tests

if info["type"] != "video":
return False

Check warning on line 74 in deeplake/integrations/labelbox/labelbox_utils.py

Codecov / codecov/patch

deeplake/integrations/labelbox/labelbox_utils.py#L73-L74

Added lines #L73 - L74 were not covered by tests

if project_id != info["project_id"]:
return False

Check warning on line 77 in deeplake/integrations/labelbox/labelbox_utils.py

Codecov / codecov/patch

deeplake/integrations/labelbox/labelbox_utils.py#L76-L77

Added lines #L76 - L77 were not covered by tests

if len(project_j) != len(info["sources"]):
return False

Check warning on line 80 in deeplake/integrations/labelbox/labelbox_utils.py

Codecov / codecov/patch

deeplake/integrations/labelbox/labelbox_utils.py#L79-L80

Added lines #L79 - L80 were not covered by tests

if len(project_j) == 0:
return True

Check warning on line 83 in deeplake/integrations/labelbox/labelbox_utils.py

Codecov / codecov/patch

deeplake/integrations/labelbox/labelbox_utils.py#L82-L83

Added lines #L82 - L83 were not covered by tests

ontology_ids = set()

Check warning on line 85 in deeplake/integrations/labelbox/labelbox_utils.py

Codecov / codecov/patch

deeplake/integrations/labelbox/labelbox_utils.py#L85

Added line #L85 was not covered by tests

for p in project_j:
url = external_url_from_video_project_(p)
if url not in info["sources"]:
return False

Check warning on line 90 in deeplake/integrations/labelbox/labelbox_utils.py

Codecov / codecov/patch

deeplake/integrations/labelbox/labelbox_utils.py#L87-L90

Added lines #L87 - L90 were not covered by tests

ontology_ids.add(p["projects"][project_id]["project_details"]["ontology_id"])

Check warning on line 92 in deeplake/integrations/labelbox/labelbox_utils.py

Codecov / codecov/patch

deeplake/integrations/labelbox/labelbox_utils.py#L92

Added line #L92 was not covered by tests

if len(ontology_ids) != 1:
return False

Check warning on line 95 in deeplake/integrations/labelbox/labelbox_utils.py

Codecov / codecov/patch

deeplake/integrations/labelbox/labelbox_utils.py#L94-L95

Added lines #L94 - L95 were not covered by tests

return True

Check warning on line 97 in deeplake/integrations/labelbox/labelbox_utils.py

Codecov / codecov/patch

deeplake/integrations/labelbox/labelbox_utils.py#L97

Added line #L97 was not covered by tests


PROJECT_DATA_VALIDATION_MAP_ = {"video": validate_video_project_data_impl_}


def validate_project_data_(proj, ds, project_id, type):
if type not in PROJECT_DATA_VALIDATION_MAP_:
raise ValueError(f"Invalid project data type: {type}")
return PROJECT_DATA_VALIDATION_MAP_[type](proj, ds, project_id)

Check warning on line 106 in deeplake/integrations/labelbox/labelbox_utils.py

Codecov / codecov/patch

deeplake/integrations/labelbox/labelbox_utils.py#L104-L106

Added lines #L104 - L106 were not covered by tests


def validate_video_project_creation_data_impl_(project_j, project_id):
if len(project_j) == 0:
return True

Check warning on line 111 in deeplake/integrations/labelbox/labelbox_utils.py

Codecov / codecov/patch

deeplake/integrations/labelbox/labelbox_utils.py#L110-L111

Added lines #L110 - L111 were not covered by tests

for p in project_j:
for l in p["projects"][project_id]["labels"]:
if l["label_kind"] != "Video":
return False

Check warning on line 116 in deeplake/integrations/labelbox/labelbox_utils.py

Codecov / codecov/patch

deeplake/integrations/labelbox/labelbox_utils.py#L113-L116

Added lines #L113 - L116 were not covered by tests

if p["media_attributes"]["asset_type"] != "video":
return False

Check warning on line 119 in deeplake/integrations/labelbox/labelbox_utils.py

Codecov / codecov/patch

deeplake/integrations/labelbox/labelbox_utils.py#L118-L119

Added lines #L118 - L119 were not covered by tests

return True

Check warning on line 121 in deeplake/integrations/labelbox/labelbox_utils.py

Codecov / codecov/patch

deeplake/integrations/labelbox/labelbox_utils.py#L121

Added line #L121 was not covered by tests


PROJECT_DATA_CREATION_VALIDATION_MAP_ = {
@@ -138,15 +127,15 @@


def validate_project_creation_data_(proj, project_id, type):
if type not in PROJECT_DATA_CREATION_VALIDATION_MAP_:
raise ValueError(f"Invalid project creation data type: {type}")
return PROJECT_DATA_CREATION_VALIDATION_MAP_[type](proj, project_id)

Check warning on line 132 in deeplake/integrations/labelbox/labelbox_utils.py

Codecov / codecov/patch

deeplake/integrations/labelbox/labelbox_utils.py#L130-L132

Added lines #L130 - L132 were not covered by tests


def labelbox_get_project_json_with_id_(client, project_id, fail_on_error=False):
print('requesting project info from labelbox with id', project_id)

Check warning on line 136 in deeplake/integrations/labelbox/labelbox_utils.py

Codecov / codecov/patch

deeplake/integrations/labelbox/labelbox_utils.py#L136

Added line #L136 was not covered by tests
# Set the export params to include/exclude certain fields.
export_params = {

Check warning on line 138 in deeplake/integrations/labelbox/labelbox_utils.py

Codecov / codecov/patch

deeplake/integrations/labelbox/labelbox_utils.py#L138

Added line #L138 was not covered by tests
"attachments": False,
"metadata_fields": False,
"data_row_details": False,
@@ -159,52 +148,52 @@
}

# Note: Filters follow AND logic, so typically using one filter is sufficient.
filters = {

Check warning on line 151 in deeplake/integrations/labelbox/labelbox_utils.py

Codecov / codecov/patch

deeplake/integrations/labelbox/labelbox_utils.py#L151

Added line #L151 was not covered by tests
"last_activity_at": ["2000-01-01 00:00:00", "2050-01-01 00:00:00"],
"label_created_at": ["2000-01-01 00:00:00", "2050-01-01 00:00:00"],
}

project = client.get_project(project_id)
export_task = project.export(params=export_params, filters=filters)

Check warning on line 157 in deeplake/integrations/labelbox/labelbox_utils.py

Codecov / codecov/patch

deeplake/integrations/labelbox/labelbox_utils.py#L156-L157

Added lines #L156 - L157 were not covered by tests

export_task.wait_till_done()

Check warning on line 159 in deeplake/integrations/labelbox/labelbox_utils.py

Codecov / codecov/patch

deeplake/integrations/labelbox/labelbox_utils.py#L159

Added line #L159 was not covered by tests

# Provide results with JSON converter
# Returns streamed JSON output strings from export task results/errors, one by one

projects = []

Check warning on line 164 in deeplake/integrations/labelbox/labelbox_utils.py

Codecov / codecov/patch

deeplake/integrations/labelbox/labelbox_utils.py#L164

Added line #L164 was not covered by tests

# Callback used for JSON Converter
def json_stream_handler(output: lb.BufferedJsonConverterOutput):
projects.append(output.json)

Check warning on line 168 in deeplake/integrations/labelbox/labelbox_utils.py

Codecov / codecov/patch

deeplake/integrations/labelbox/labelbox_utils.py#L167-L168

Added lines #L167 - L168 were not covered by tests

def error_stream_handler(error):
if fail_on_error:
raise Exception(f"Error during export: {error}")
print(f"Error during export: {error}")

Check warning on line 173 in deeplake/integrations/labelbox/labelbox_utils.py

Codecov / codecov/patch

deeplake/integrations/labelbox/labelbox_utils.py#L170-L173

Added lines #L170 - L173 were not covered by tests

if export_task.has_errors():
export_task.get_buffered_stream(stream_type=lb.StreamType.ERRORS).start(

Check warning on line 176 in deeplake/integrations/labelbox/labelbox_utils.py

Codecov / codecov/patch

deeplake/integrations/labelbox/labelbox_utils.py#L175-L176

Added lines #L175 - L176 were not covered by tests
stream_handler=error_stream_handler
)

if export_task.has_result():
export_json = export_task.get_buffered_stream(

Check warning on line 181 in deeplake/integrations/labelbox/labelbox_utils.py

Codecov / codecov/patch

deeplake/integrations/labelbox/labelbox_utils.py#L180-L181

Added lines #L180 - L181 were not covered by tests
stream_type=lb.StreamType.RESULT
).start(stream_handler=json_stream_handler)

print('project info is ready for project with id', project_id)

Check warning on line 185 in deeplake/integrations/labelbox/labelbox_utils.py

Codecov / codecov/patch

deeplake/integrations/labelbox/labelbox_utils.py#L185

Added line #L185 was not covered by tests

return projects

Check warning on line 187 in deeplake/integrations/labelbox/labelbox_utils.py

Codecov / codecov/patch

deeplake/integrations/labelbox/labelbox_utils.py#L187

Added line #L187 was not covered by tests


def create_tensors_default_(ds):
ds.create_tensor("frames", htype="image", sample_compression="jpg")
ds.create_tensor("frame_idx", htype="generic", dtype="int32")
ds.create_tensor("video_idx", htype="generic", dtype="int32")

Check warning on line 193 in deeplake/integrations/labelbox/labelbox_utils.py

Codecov / codecov/patch

deeplake/integrations/labelbox/labelbox_utils.py#L191-L193

Added lines #L191 - L193 were not covered by tests


def fill_data_default_(ds, group_ids, indexes, frames):
ds["frames"].extend(frames)
ds["video_idx"].extend(group_ids)
ds["frame_idx"].extend(indexes)

Check warning on line 199 in deeplake/integrations/labelbox/labelbox_utils.py

Codecov / codecov/patch

deeplake/integrations/labelbox/labelbox_utils.py#L197-L199

Added lines #L197 - L199 were not covered by tests

Unchanged files with check annotations Beta

def bbox_converter_(obj, converter, tensor_name, context, generate_labels):
ds = context["ds"]
try:
ds.create_tensor(

Check warning on line 10 in deeplake/integrations/labelbox/v3_converters.py

Codecov / codecov/patch

deeplake/integrations/labelbox/v3_converters.py#L8-L10

Added lines #L8 - L10 were not covered by tests
tensor_name,
htype="bbox",
dtype="int32",
coords={"type": "pixel", "mode": "LTWH"},
)
except:
pass

Check warning on line 17 in deeplake/integrations/labelbox/v3_converters.py

Codecov / codecov/patch

deeplake/integrations/labelbox/v3_converters.py#L16-L17

Added lines #L16 - L17 were not covered by tests
if generate_labels:
print("bbox converter does not support generating labels")

Check warning on line 20 in deeplake/integrations/labelbox/v3_converters.py

Codecov / codecov/patch

deeplake/integrations/labelbox/v3_converters.py#L19-L20

Added lines #L19 - L20 were not covered by tests
converter.register_feature_id_for_kind("tool", "bounding_box", obj, tensor_name)

Check warning on line 22 in deeplake/integrations/labelbox/v3_converters.py

Codecov / codecov/patch

deeplake/integrations/labelbox/v3_converters.py#L22

Added line #L22 was not covered by tests
def bbox_converter(row, obj):
if tensor_name not in converter.values_cache:
converter.values_cache[tensor_name] = dict()
if row not in converter.values_cache[tensor_name]:
converter.values_cache[tensor_name][row] = []

Check warning on line 28 in deeplake/integrations/labelbox/v3_converters.py

Codecov / codecov/patch

deeplake/integrations/labelbox/v3_converters.py#L24-L28

Added lines #L24 - L28 were not covered by tests
converter.values_cache[tensor_name][row].append(

Check warning on line 30 in deeplake/integrations/labelbox/v3_converters.py

Codecov / codecov/patch

deeplake/integrations/labelbox/v3_converters.py#L30

Added line #L30 was not covered by tests
[
int(v)
for v in [
]
]
)
converter.regsistered_actions[obj.feature_schema_id] = bbox_converter

Check warning on line 41 in deeplake/integrations/labelbox/v3_converters.py

Codecov / codecov/patch

deeplake/integrations/labelbox/v3_converters.py#L41

Added line #L41 was not covered by tests
def interpolator(start, end, progress):
start_box = start['bounding_box']
end_box = end['bounding_box']
bbox = copy.deepcopy(start)
bbox['bounding_box'] = {

Check warning on line 48 in deeplake/integrations/labelbox/v3_converters.py

Codecov / codecov/patch

deeplake/integrations/labelbox/v3_converters.py#L44-L48

Added lines #L44 - L48 were not covered by tests
'top': start_box['top'] + (end_box['top'] - start_box['top']) * progress,
'left': start_box['left'] + (end_box['left'] - start_box['left']) * progress,
'width': start_box['width'] + (end_box['width'] - start_box['width']) * progress,
'height': start_box['height'] + (end_box['height'] - start_box['height']) * progress,
}
return bbox

Check warning on line 55 in deeplake/integrations/labelbox/v3_converters.py

Codecov / codecov/patch

deeplake/integrations/labelbox/v3_converters.py#L55

Added line #L55 was not covered by tests
converter.registered_interpolators[obj.feature_schema_id] = interpolator

Check warning on line 57 in deeplake/integrations/labelbox/v3_converters.py

Codecov / codecov/patch

deeplake/integrations/labelbox/v3_converters.py#L57

Added line #L57 was not covered by tests
def radio_converter_(obj, converter, tensor_name, context, generate_labels):
ds = context["ds"]

Check warning on line 61 in deeplake/integrations/labelbox/v3_converters.py

Codecov / codecov/patch

deeplake/integrations/labelbox/v3_converters.py#L61

Added line #L61 was not covered by tests
converter.label_mappings[tensor_name] = {

Check warning on line 63 in deeplake/integrations/labelbox/v3_converters.py

Codecov / codecov/patch

deeplake/integrations/labelbox/v3_converters.py#L63

Added line #L63 was not covered by tests
options.value: i for i, options in enumerate(obj.options)
}
if generate_labels:
print("radio converter does not support generating labels")

Check warning on line 68 in deeplake/integrations/labelbox/v3_converters.py

Codecov / codecov/patch

deeplake/integrations/labelbox/v3_converters.py#L67-L68

Added lines #L67 - L68 were not covered by tests
try:
ds.create_tensor(

Check warning on line 71 in deeplake/integrations/labelbox/v3_converters.py

Codecov / codecov/patch

deeplake/integrations/labelbox/v3_converters.py#L70-L71

Added lines #L70 - L71 were not covered by tests
tensor_name,
htype="class_label",
class_names=list(converter.label_mappings[tensor_name].keys()),
chunk_compression="lz4",
)
except:
pass

Check warning on line 78 in deeplake/integrations/labelbox/v3_converters.py

Codecov / codecov/patch

deeplake/integrations/labelbox/v3_converters.py#L77-L78

Added lines #L77 - L78 were not covered by tests
converter.register_feature_id_for_kind(

Check warning on line 80 in deeplake/integrations/labelbox/v3_converters.py

Codecov / codecov/patch

deeplake/integrations/labelbox/v3_converters.py#L80

Added line #L80 was not covered by tests
"annotation", "radio_answer", obj, tensor_name
)
def radio_converter(row, o):
if tensor_name not in converter.values_cache:
converter.values_cache[tensor_name] = dict()
if row not in converter.values_cache[tensor_name]:
converter.values_cache[tensor_name][row] = []
converter.values_cache[tensor_name][row] = [converter.label_mappings[tensor_name][o["value"]]]

Check warning on line 89 in deeplake/integrations/labelbox/v3_converters.py

Codecov / codecov/patch

deeplake/integrations/labelbox/v3_converters.py#L84-L89

Added lines #L84 - L89 were not covered by tests
for option in obj.options:
converter.regsistered_actions[option.feature_schema_id] = radio_converter

Check warning on line 92 in deeplake/integrations/labelbox/v3_converters.py

Codecov / codecov/patch

deeplake/integrations/labelbox/v3_converters.py#L91-L92

Added lines #L91 - L92 were not covered by tests
def radio_converter_nested(row, obj):
radio_converter(row, obj["radio_answer"])

Check warning on line 95 in deeplake/integrations/labelbox/v3_converters.py

Codecov / codecov/patch

deeplake/integrations/labelbox/v3_converters.py#L94-L95

Added lines #L94 - L95 were not covered by tests
converter.regsistered_actions[obj.feature_schema_id] = radio_converter_nested

Check warning on line 97 in deeplake/integrations/labelbox/v3_converters.py

Codecov / codecov/patch

deeplake/integrations/labelbox/v3_converters.py#L97

Added line #L97 was not covered by tests
def checkbox_converter_(obj, converter, tensor_name, context, generate_labels):
ds = context["ds"]

Check warning on line 101 in deeplake/integrations/labelbox/v3_converters.py

Codecov / codecov/patch

deeplake/integrations/labelbox/v3_converters.py#L101

Added line #L101 was not covered by tests
converter.label_mappings[tensor_name] = {

Check warning on line 103 in deeplake/integrations/labelbox/v3_converters.py

Codecov / codecov/patch

deeplake/integrations/labelbox/v3_converters.py#L103

Added line #L103 was not covered by tests
options.value: i for i, options in enumerate(obj.options)
}
if generate_labels:
print("checkbox converter does not support generating labels")

Check warning on line 108 in deeplake/integrations/labelbox/v3_converters.py

Codecov / codecov/patch

deeplake/integrations/labelbox/v3_converters.py#L107-L108

Added lines #L107 - L108 were not covered by tests
try:
ds.create_tensor(

Check warning on line 111 in deeplake/integrations/labelbox/v3_converters.py

Codecov / codecov/patch

deeplake/integrations/labelbox/v3_converters.py#L110-L111

Added lines #L110 - L111 were not covered by tests
tensor_name,
htype="class_label",
class_names=list(converter.label_mappings[tensor_name].keys()),
chunk_compression="lz4",
)
except:
pass

Check warning on line 118 in deeplake/integrations/labelbox/v3_converters.py

Codecov / codecov/patch

deeplake/integrations/labelbox/v3_converters.py#L117-L118

Added lines #L117 - L118 were not covered by tests
converter.register_feature_id_for_kind(

Check warning on line 120 in deeplake/integrations/labelbox/v3_converters.py

Codecov / codecov/patch

deeplake/integrations/labelbox/v3_converters.py#L120

Added line #L120 was not covered by tests
"annotation", "checklist_answers", obj, tensor_name
)
def checkbox_converter(row, obj):
if tensor_name not in converter.values_cache:
converter.values_cache[tensor_name] = dict()
if row not in converter.values_cache[tensor_name]:
converter.values_cache[tensor_name][row] = []

Check warning on line 128 in deeplake/integrations/labelbox/v3_converters.py

Codecov / codecov/patch

deeplake/integrations/labelbox/v3_converters.py#L124-L128

Added lines #L124 - L128 were not covered by tests
converter.values_cache[tensor_name][row].append(converter.label_mappings[tensor_name][obj["value"]])

Check warning on line 130 in deeplake/integrations/labelbox/v3_converters.py

Codecov / codecov/patch

deeplake/integrations/labelbox/v3_converters.py#L130

Added line #L130 was not covered by tests
for option in obj.options:
converter.regsistered_actions[option.feature_schema_id] = checkbox_converter

Check warning on line 133 in deeplake/integrations/labelbox/v3_converters.py

Codecov / codecov/patch

deeplake/integrations/labelbox/v3_converters.py#L132-L133

Added lines #L132 - L133 were not covered by tests
def checkbox_converter_nested(row, obj):
for o in obj["checklist_answers"]:
checkbox_converter(row, o)

Check warning on line 137 in deeplake/integrations/labelbox/v3_converters.py

Codecov / codecov/patch

deeplake/integrations/labelbox/v3_converters.py#L135-L137

Added lines #L135 - L137 were not covered by tests
converter.regsistered_actions[obj.feature_schema_id] = checkbox_converter_nested

Check warning on line 139 in deeplake/integrations/labelbox/v3_converters.py

Codecov / codecov/patch

deeplake/integrations/labelbox/v3_converters.py#L139

Added line #L139 was not covered by tests
def point_converter_(obj, converter, tensor_name, context, generate_labels):
ds = context["ds"]
try:
ds.create_tensor(tensor_name, htype="point", dtype="int32")
except:
pass

Check warning on line 147 in deeplake/integrations/labelbox/v3_converters.py

Codecov / codecov/patch

deeplake/integrations/labelbox/v3_converters.py#L143-L147

Added lines #L143 - L147 were not covered by tests
converter.register_feature_id_for_kind("annotation", "point", obj, tensor_name)

Check warning on line 149 in deeplake/integrations/labelbox/v3_converters.py

Codecov / codecov/patch

deeplake/integrations/labelbox/v3_converters.py#L149

Added line #L149 was not covered by tests
if generate_labels:
print("point converter does not support generating labels")

Check warning on line 152 in deeplake/integrations/labelbox/v3_converters.py

Codecov / codecov/patch

deeplake/integrations/labelbox/v3_converters.py#L151-L152

Added lines #L151 - L152 were not covered by tests
def point_converter(row, obj):
if tensor_name not in converter.values_cache:
converter.values_cache[tensor_name] = dict()
if row not in converter.values_cache[tensor_name]:
converter.values_cache[tensor_name][row] = []

Check warning on line 158 in deeplake/integrations/labelbox/v3_converters.py

Codecov / codecov/patch

deeplake/integrations/labelbox/v3_converters.py#L154-L158

Added lines #L154 - L158 were not covered by tests
converter.values_cache[tensor_name][row].append([int(obj["point"]["x"]), int(obj["point"]["y"])])

Check warning on line 160 in deeplake/integrations/labelbox/v3_converters.py

Codecov / codecov/patch

deeplake/integrations/labelbox/v3_converters.py#L160

Added line #L160 was not covered by tests
converter.regsistered_actions[obj.feature_schema_id] = point_converter

Check warning on line 162 in deeplake/integrations/labelbox/v3_converters.py

Codecov / codecov/patch

deeplake/integrations/labelbox/v3_converters.py#L162

Added line #L162 was not covered by tests
def interpolator(start, end, progress):
start_point = start['point']
end_point = end['point']
point = copy.deepcopy(start)
point['point'] = {

Check warning on line 168 in deeplake/integrations/labelbox/v3_converters.py

Codecov / codecov/patch

deeplake/integrations/labelbox/v3_converters.py#L164-L168

Added lines #L164 - L168 were not covered by tests
'x': start_point['x'] + (end_point['x'] - start_point['x']) * progress,
'y': start_point['y'] + (end_point['y'] - start_point['y']) * progress,
}
return point

Check warning on line 173 in deeplake/integrations/labelbox/v3_converters.py

Codecov / codecov/patch

deeplake/integrations/labelbox/v3_converters.py#L173

Added line #L173 was not covered by tests
converter.registered_interpolators[obj.feature_schema_id] = interpolator

Check warning on line 175 in deeplake/integrations/labelbox/v3_converters.py

Codecov / codecov/patch

deeplake/integrations/labelbox/v3_converters.py#L175

Added line #L175 was not covered by tests
def line_converter_(obj, converter, tensor_name, context, generate_labels):
ds = context["ds"]
try:
ds.create_tensor(tensor_name, htype="polygon", dtype="int32")
except:
pass

Check warning on line 183 in deeplake/integrations/labelbox/v3_converters.py

Codecov / codecov/patch

deeplake/integrations/labelbox/v3_converters.py#L179-L183

Added lines #L179 - L183 were not covered by tests
converter.register_feature_id_for_kind("annotation", "line", obj, tensor_name)

Check warning on line 185 in deeplake/integrations/labelbox/v3_converters.py

Codecov / codecov/patch

deeplake/integrations/labelbox/v3_converters.py#L185

Added line #L185 was not covered by tests
if generate_labels:
print("line converter does not support generating labels")

Check warning on line 188 in deeplake/integrations/labelbox/v3_converters.py

Codecov / codecov/patch

deeplake/integrations/labelbox/v3_converters.py#L187-L188

Added lines #L187 - L188 were not covered by tests
def polygon_converter(row, obj):
if tensor_name not in converter.values_cache:
converter.values_cache[tensor_name] = dict()
if row not in converter.values_cache[tensor_name]:
converter.values_cache[tensor_name][row] = []

Check warning on line 194 in deeplake/integrations/labelbox/v3_converters.py

Codecov / codecov/patch

deeplake/integrations/labelbox/v3_converters.py#L190-L194

Added lines #L190 - L194 were not covered by tests
converter.values_cache[tensor_name][row].append([[int(l["x"]), int(l["y"])] for l in obj["line"]])

Check warning on line 196 in deeplake/integrations/labelbox/v3_converters.py

Codecov / codecov/patch

deeplake/integrations/labelbox/v3_converters.py#L196

Added line #L196 was not covered by tests
converter.regsistered_actions[obj.feature_schema_id] = polygon_converter

Check warning on line 198 in deeplake/integrations/labelbox/v3_converters.py

Codecov / codecov/patch

deeplake/integrations/labelbox/v3_converters.py#L198

Added line #L198 was not covered by tests
def interpolator(start, end, progress):
start_line = start['line']
end_line = end['line']
line = copy.deepcopy(start)
line['line'] = [

Check warning on line 204 in deeplake/integrations/labelbox/v3_converters.py

Codecov / codecov/patch

deeplake/integrations/labelbox/v3_converters.py#L200-L204

Added lines #L200 - L204 were not covered by tests
[
start_line[i]['x'] + (end_line[i]['x'] - start_line[i]['x']) * progress,
start_line[i]['y'] + (end_line[i]['y'] - start_line[i]['y']) * progress,
for i in range(len(start_line))
]
return line

Check warning on line 212 in deeplake/integrations/labelbox/v3_converters.py

Codecov / codecov/patch

deeplake/integrations/labelbox/v3_converters.py#L212

Added line #L212 was not covered by tests
converter.registered_interpolators[obj.feature_schema_id] = interpolator

Check warning on line 214 in deeplake/integrations/labelbox/v3_converters.py

Codecov / codecov/patch

deeplake/integrations/labelbox/v3_converters.py#L214

Added line #L214 was not covered by tests
def raster_segmentation_converter_(
obj, converter, tensor_name, context, generate_labels
):
ds = context["ds"]
try:
ds.create_tensor(

Check warning on line 222 in deeplake/integrations/labelbox/v3_converters.py

Codecov / codecov/patch

deeplake/integrations/labelbox/v3_converters.py#L220-L222

Added lines #L220 - L222 were not covered by tests
tensor_name, htype="binary_mask", dtype="bool", sample_compression="lz4"
)
if generate_labels:
ds.create_tensor(

Check warning on line 226 in deeplake/integrations/labelbox/v3_converters.py

Codecov / codecov/patch

deeplake/integrations/labelbox/v3_converters.py#L225-L226

Added lines #L225 - L226 were not covered by tests
f"{tensor_name}_labels",
htype="class_label",
dtype="int32",
class_names=[],
chunk_compression="lz4",
)
converter.label_mappings[f"{tensor_name}_labels"] = dict()
except:
pass

Check warning on line 235 in deeplake/integrations/labelbox/v3_converters.py

Codecov / codecov/patch

deeplake/integrations/labelbox/v3_converters.py#L233-L235

Added lines #L233 - L235 were not covered by tests
converter.register_feature_id_for_kind(

Check warning on line 237 in deeplake/integrations/labelbox/v3_converters.py

Codecov / codecov/patch

deeplake/integrations/labelbox/v3_converters.py#L237

Added line #L237 was not covered by tests
"annotation", "raster-segmentation", obj, tensor_name
)
tool_name = obj.name

Check warning on line 241 in deeplake/integrations/labelbox/v3_converters.py

Codecov / codecov/patch

deeplake/integrations/labelbox/v3_converters.py#L241

Added line #L241 was not covered by tests
def mask_converter(row, obj):
try:
r = urllib.request.Request(

Check warning on line 245 in deeplake/integrations/labelbox/v3_converters.py

Codecov / codecov/patch

deeplake/integrations/labelbox/v3_converters.py#L243-L245

Added lines #L243 - L245 were not covered by tests
obj["mask"]["url"],
headers={"Authorization": f'Bearer {context["lb_api_key"]}'},
)
with urllib.request.urlopen(r) as response:
if generate_labels:
if (

Check warning on line 251 in deeplake/integrations/labelbox/v3_converters.py

Codecov / codecov/patch

deeplake/integrations/labelbox/v3_converters.py#L249-L251

Added lines #L249 - L251 were not covered by tests
tool_name
not in converter.label_mappings[f"{tensor_name}_labels"]
):
converter.label_mappings[f"{tensor_name}_labels"][tool_name] = (

Check warning on line 255 in deeplake/integrations/labelbox/v3_converters.py

Codecov / codecov/patch

deeplake/integrations/labelbox/v3_converters.py#L255

Added line #L255 was not covered by tests
len(converter.label_mappings[f"{tensor_name}_labels"])
)
ds[f"{tensor_name}_labels"].info.update(

Check warning on line 258 in deeplake/integrations/labelbox/v3_converters.py

Codecov / codecov/patch

deeplake/integrations/labelbox/v3_converters.py#L258

Added line #L258 was not covered by tests
class_names=list(
converter.label_mappings[f"{tensor_name}_labels"].keys()
)
)
val = []
try:
val = (

Check warning on line 265 in deeplake/integrations/labelbox/v3_converters.py

Codecov / codecov/patch

deeplake/integrations/labelbox/v3_converters.py#L263-L265

Added lines #L263 - L265 were not covered by tests
ds[f"{tensor_name}_labels"][row].numpy(aslist=True).tolist()
)
except (KeyError, IndexError):
pass

Check warning on line 269 in deeplake/integrations/labelbox/v3_converters.py

Codecov / codecov/patch

deeplake/integrations/labelbox/v3_converters.py#L268-L269

Added lines #L268 - L269 were not covered by tests
val.append(

Check warning on line 271 in deeplake/integrations/labelbox/v3_converters.py

Codecov / codecov/patch

deeplake/integrations/labelbox/v3_converters.py#L271

Added line #L271 was not covered by tests
converter.label_mappings[f"{tensor_name}_labels"][tool_name]
)
ds[f"{tensor_name}_labels"][row] = val

Check warning on line 274 in deeplake/integrations/labelbox/v3_converters.py

Codecov / codecov/patch

deeplake/integrations/labelbox/v3_converters.py#L274

Added line #L274 was not covered by tests
mask = np.array(Image.open(response)).astype(np.bool_)
mask = mask[..., np.newaxis]
try:
if generate_labels:
val = ds[tensor_name][row].numpy()
labels = ds[f"{tensor_name}_labels"].info['class_names']
if len(labels) != val.shape[-1]:
val = np.concatenate([ds[tensor_name][row].numpy(), np.zeros_like(mask)], axis=-1)
idx = labels.index(tool_name)
val[:,:,idx] = np.logical_or(val[:,:,idx], mask[:,:,0])

Check warning on line 285 in deeplake/integrations/labelbox/v3_converters.py

Codecov / codecov/patch

deeplake/integrations/labelbox/v3_converters.py#L276-L285

Added lines #L276 - L285 were not covered by tests
else:
val = np.logical_or(ds[tensor_name][row].numpy(), mask)
except (KeyError, IndexError):
val = mask

Check warning on line 289 in deeplake/integrations/labelbox/v3_converters.py

Codecov / codecov/patch

deeplake/integrations/labelbox/v3_converters.py#L287-L289

Added lines #L287 - L289 were not covered by tests
ds[tensor_name][row] = val
except Exception as e:
print(f"Error downloading mask: {e}")

Check warning on line 293 in deeplake/integrations/labelbox/v3_converters.py

Codecov / codecov/patch

deeplake/integrations/labelbox/v3_converters.py#L291-L293

Added lines #L291 - L293 were not covered by tests
converter.regsistered_actions[obj.feature_schema_id] = mask_converter

Check warning on line 295 in deeplake/integrations/labelbox/v3_converters.py

Codecov / codecov/patch

deeplake/integrations/labelbox/v3_converters.py#L295

Added line #L295 was not covered by tests
def text_converter_(obj, converter, tensor_name, context, generate_labels):
ds = context["ds"]
try:
ds.create_tensor(tensor_name, htype="text", dtype="str")
except:
pass

Check warning on line 303 in deeplake/integrations/labelbox/v3_converters.py

Codecov / codecov/patch

deeplake/integrations/labelbox/v3_converters.py#L299-L303

Added lines #L299 - L303 were not covered by tests
converter.register_feature_id_for_kind("annotation", "text", obj, tensor_name)

Check warning on line 305 in deeplake/integrations/labelbox/v3_converters.py

Codecov / codecov/patch

deeplake/integrations/labelbox/v3_converters.py#L305

Added line #L305 was not covered by tests
if generate_labels:
print("text converter does not support generating labels")

Check warning on line 308 in deeplake/integrations/labelbox/v3_converters.py

Codecov / codecov/patch

deeplake/integrations/labelbox/v3_converters.py#L307-L308

Added lines #L307 - L308 were not covered by tests
def text_converter(row, obj):
if tensor_name not in converter.values_cache:
converter.values_cache[tensor_name] = dict()
if row not in converter.values_cache[tensor_name]:
converter.values_cache[tensor_name][row] = []
converter.values_cache[tensor_name][row] = obj["text_answer"]["content"]

Check warning on line 315 in deeplake/integrations/labelbox/v3_converters.py

Codecov / codecov/patch

deeplake/integrations/labelbox/v3_converters.py#L310-L315

Added lines #L310 - L315 were not covered by tests
converter.regsistered_actions[obj.feature_schema_id] = text_converter

Check warning on line 317 in deeplake/integrations/labelbox/v3_converters.py

Codecov / codecov/patch

deeplake/integrations/labelbox/v3_converters.py#L317

Added line #L317 was not covered by tests