Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Community] Add IDOL, VITA to project #835

Open
wants to merge 1 commit into
base: dev-1.x
Choose a base branch
from

Conversation

pixeli99
Copy link
Collaborator

@pixeli99 pixeli99 commented Feb 9, 2023

No description provided.

@pixeli99
Copy link
Collaborator Author

There is a bug about data reading.

Under repair.

@SupreethN
Copy link

Hi,

Thank you for the fantastic work on developing Mask2Former, IDOL, and VITA using mmtracking. I can successfully run Mask2Former. However, I run into some issues running IDOL and VITA.

For IDOL, I get errors during the training stage. And I get the following error:

Traceback (most recent call last):
  File "tools/train.py", line 119, in <module>
    main()
  File "tools/train.py", line 115, in main
    runner.train()
  File "/home/supreeth/anaconda3/envs/mmtrack_vis_sota/lib/python3.8/site-packages/mmengine/runner/runner.py", line 1678, in train
    model = self.train_loop.run()  # type: ignore
  File "/home/supreeth/anaconda3/envs/mmtrack_vis_sota/lib/python3.8/site-packages/mmengine/runner/loops.py", line 264, in run
    self.run_iter(data_batch)
  File "/home/supreeth/anaconda3/envs/mmtrack_vis_sota/lib/python3.8/site-packages/mmengine/runner/loops.py", line 287, in run_iter
    outputs = self.runner.model.train_step(
  File "/home/supreeth/anaconda3/envs/mmtrack_vis_sota/lib/python3.8/site-packages/mmengine/model/base_model/base_model.py", line 114, in train_step
    losses = self._run_forward(data, mode='loss')  # type: ignore
  File "/home/supreeth/anaconda3/envs/mmtrack_vis_sota/lib/python3.8/site-packages/mmengine/model/base_model/base_model.py", line 314, in _run_forward
    results = self(**data, mode=mode)
  File "/home/supreeth/anaconda3/envs/mmtrack_vis_sota/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/supreeth/mmtracking_vis_sota/mmtrack/models/mot/base.py", line 106, in forward
    return self.loss(inputs, data_samples, **kwargs)
  File "/home/supreeth/mmtracking_vis_sota/projects/VIS_SOTA/IDOL/idol_src/idol.py", line 53, in loss
    losses = self.track_head.loss(x, data_samples)
  File "/home/supreeth/mmtracking_vis_sota/projects/VIS_SOTA/IDOL/idol_src/models/idol_query_track_head.py", line 488, in loss
    losses = self.loss_by_feat(*loss_inputs)
  File "/home/supreeth/mmtracking_vis_sota/projects/VIS_SOTA/IDOL/idol_src/models/idol_query_track_head.py", line 534, in loss_by_feat
    losses_mask, losses_dice, min_cost_qids = multi_apply(
  File "/home/supreeth/anaconda3/envs/mmtrack_vis_sota/lib/python3.8/site-packages/mmdet/models/utils/misc.py", line 219, in multi_apply
    return tuple(map(list, zip(*map_results)))
  File "/home/supreeth/mmtracking_vis_sota/projects/VIS_SOTA/IDOL/idol_src/models/idol_query_track_head.py", line 681, in loss_by_feat_single
    loss_dice = self.loss_dice(
  File "/home/supreeth/anaconda3/envs/mmtrack_vis_sota/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/supreeth/anaconda3/envs/mmtrack_vis_sota/lib/python3.8/site-packages/mmdet/models/losses/dice_loss.py", line 137, in forward
    loss = self.loss_weight * dice_loss(
  File "/home/supreeth/anaconda3/envs/mmtrack_vis_sota/lib/python3.8/site-packages/mmdet/models/losses/dice_loss.py", line 47, in dice_loss
    a = torch.sum(input * target, 1)
RuntimeError: The size of tensor a (8) must match the size of tensor b (600) at non-singleton dimension 0

Regarding VITA, I can train the model successfully. However, during evaluation, I get the following error:

Traceback (most recent call last):
  File "tools/test.py", line 88, in <module>
    main()
  File "tools/test.py", line 84, in main
    runner.test()
  File "/home/supreeth/anaconda3/envs/mmtrack_vis_sota/lib/python3.8/site-packages/mmengine/runner/runner.py", line 1724, in test
    metrics = self.test_loop.run()  # type: ignore
  File "/home/supreeth/anaconda3/envs/mmtrack_vis_sota/lib/python3.8/site-packages/mmengine/runner/loops.py", line 417, in run
    self.run_iter(idx, data_batch)
  File "/home/supreeth/anaconda3/envs/mmtrack_vis_sota/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 28, in decorate_context
    return func(*args, **kwargs)
  File "/home/supreeth/anaconda3/envs/mmtrack_vis_sota/lib/python3.8/site-packages/mmengine/runner/loops.py", line 436, in run_iter
    outputs = self.runner.model.test_step(data_batch)
  File "/home/supreeth/anaconda3/envs/mmtrack_vis_sota/lib/python3.8/site-packages/mmengine/model/base_model/base_model.py", line 145, in test_step
    return self._run_forward(data, mode='predict')  # type: ignore
  File "/home/supreeth/anaconda3/envs/mmtrack_vis_sota/lib/python3.8/site-packages/mmengine/model/base_model/base_model.py", line 314, in _run_forward
    results = self(**data, mode=mode)
  File "/home/supreeth/anaconda3/envs/mmtrack_vis_sota/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/supreeth/mmtracking_vis_sota/mmtrack/models/mot/base.py", line 108, in forward
    return self.predict(inputs, data_samples, **kwargs)
  File "/home/supreeth/mmtracking_vis_sota/projects/VIS_SOTA/VITA/vita_src/vita.py", line 106, in predict
    _frame_queries, _mask_features = self.seg_head.predict(feats)
  File "/home/supreeth/mmtracking_vis_sota/projects/VIS_SOTA/VITA/vita_src/models/vita_seg_head.py", line 217, in predict
    all_frame_queries, mask_features = self(x)
ValueError: too many values to unpack (expected 2)

Can you please help me with this @pixeli99?

Thanks!

@pixeli99
Copy link
Collaborator Author

pixeli99 commented Mar 1, 2023

I'm glad you are interested in this part of code, but unfortunately, the current code support for VITA is not complete, which will be repaired after the GenVIS reproduction is completed. If you need to do research, you can try the official repo first. The reproduction is expected to be completed in two weeks ✅ In addition, if you have any suggestions on the code or want to join me, I am also very welcome !😺

@SupreethN
Copy link

Thank you for the swift reply. That's great to know! I will checkout the official code repo once. I will let you know if I have any suggestions on the code!

@SupreethN
Copy link

Hi @pixeli99 ! Is there any update on the IDOL/ VITA code yet? I am looking forward to trying them! Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants