add initial code

.gitignore

@@ -0,0 +1,3 @@
+build/
+upn.egg-info/
+checkpoints/

LICENSE

@@ -0,0 +1,51 @@
+IDEA License 1.0
+
+This License Agreement (as may be amended in accordance with this License Agreement, “License”), between you, or your employer or other entity (if you are entering into this agreement on behalf of your employer or other entity) (“Licensee” or “you”) and the International Digital Economy Academy (“IDEA” or “we”) applies to your use of any computer program, algorithm, source code, object code, or software that is made available by IDEA under this License (“Software”) and any specifications, manuals, documentation, and other written information provided by IDEA related to the Software (“Documentation”). 
+
+By downloading the Software or by using the Software, you agree to the terms of this License. If you do not agree to this License, then you do not have any rights to use the Software or Documentation (collectively, the “Software Products”), and you must immediately cease using the Software Products. If you are agreeing to be bound by the terms of this License on behalf of your employer or other entity, you represent and warrant to IDEA that you have full legal authority to bind your employer or such entity to this License. If you do not have the requisite authority, you may not accept the License or access the Software Products on behalf of your employer or other entity.
+
+1. LICENSE GRANT
+
+a. You are granted a non-exclusive, worldwide, transferable, sublicensable, irrevocable, royalty free and limited license under IDEA’s copyright interests to use, reproduce, distribute, copy, create derivative works of, and make modifications to the Software solely for your non-commercial research purposes. 
+
+b. The grant of rights expressly set forth in this Section 1 (License Grant) are the complete grant of rights to you in the Software Products, and no other licenses are granted, whether by waiver, estoppel, implication, equity or otherwise. IDEA and its licensors reserve all rights not expressly granted by this License.
+
+c. If you intend to use the Software Products for any commercial purposes, you must request a license from IDEA, which IDEA may grant to you in its sole discretion.
+
+2. REDISTRIBUTION AND USE 
+
+a. If you distribute or make the Software Products, or any derivative works thereof, available to a third party, you shall provide a copy of this Agreement to such third party. 
+
+b. You must retain in all copies of the Software Products that you distribute the following attribution notice: "T  is licensed under the IDEA License 1.0, Copyright (c) IDEA. All Rights Reserved."
+
+d. Your use of the Software Products must comply with applicable laws and regulations (including trade compliance laws and regulations).
+
+e. You will not, and will not permit, assist or cause any third party to use, modify, copy, reproduce, create derivative works of, or distribute the Software Products (or any derivative works thereof, works incorporating the Software Products, or any data produced by the Software), in whole or in part, for in any manner that infringes, misappropriates, or otherwise violates any third-party rights.
+
+3. DISCLAIMER OF WARRANTY
+
+UNLESS REQUIRED BY APPLICABLE LAW, THE SOFTWARE PRODUCTS AND ANY OUTPUT AND RESULTS THEREFROM ARE PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING, WITHOUT LIMITATION, ANY WARRANTIES OF TITLE, NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. YOU ARE SOLELY RESPONSIBLE FOR DETERMINING THE APPROPRIATENESS OF USING OR REDISTRIBUTING THE SOFTWARE PRODUCTS AND ASSUME ANY RISKS ASSOCIATED WITH YOUR USE OF THE SOFTWARE PRODUCTS AND ANY OUTPUT AND RESULTS.
+
+4. LIMITATION OF LIABILITY
+
+IN NO EVENT WILL IDEA OR ITS AFFILIATES BE LIABLE UNDER ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, TORT, NEGLIGENCE, PRODUCTS LIABILITY, OR OTHERWISE, ARISING OUT OF THIS AGREEMENT, FOR ANY LOST PROFITS OR ANY INDIRECT, SPECIAL, CONSEQUENTIAL, INCIDENTAL, EXEMPLARY OR PUNITIVE DAMAGES, EVEN IF IDEA OR ITS AFFILIATES HAVE BEEN ADVISED OF THE POSSIBILITY OF ANY OF THE FOREGOING.
+
+5. INDEMNIFICATION
+
+You will indemnify, defend and hold harmless IDEA and our subsidiaries and affiliates, and each of our respective shareholders, directors, officers, employees, agents, successors, and assigns (collectively, the “IDEA Parties”) from and against any losses, liabilities, damages, fines, penalties, and expenses (including reasonable attorneys’ fees) incurred by any IDEA Party in connection with any claim, demand, allegation, lawsuit, proceeding, or investigation (collectively, “Claims”) arising out of or related to: (a) your access to or use of the Software Products (as well as any results or data generated from such access or use); (b) your violation of this License; or (c) your violation, misappropriation or infringement of any rights of another (including intellectual property or other proprietary rights and privacy rights). You will promptly notify the IDEA Parties of any such Claims, and cooperate with IDEA Parties in defending such Claims. You will also grant the IDEA Parties sole control of the defense or settlement, at IDEA’s sole option, of any Claims. This indemnity is in addition to, and not in lieu of, any other indemnities or remedies set forth in a written agreement between you and IDEA or the other IDEA Parties.
+
+6. TERMINATION; SURVIVAL
+
+a. This License will automatically terminate upon any breach by you of the terms of this License.
+
+b. If you institute litigation or other proceedings against IDEA or any entity (including a cross-claim or counterclaim in a lawsuit) alleging that the Software Products, or any portion of any of the foregoing, constitutes infringement of intellectual property or other rights owned or licensable by you, then any licenses granted to you under this Agreement shall terminate as of the date such litigation or claim is filed or instituted.
+
+c. The following sections survive termination of this License: 2 (Redistribution and use), 3 (Disclaimers of Warranty), 4 (Limitation of Liability), 5 (Indemnification), 6 (Termination; Survival), 7 (Trademarks) and 8 (Applicable Law; Dispute Resolution). 
+
+7. TRADEMARKS
+
+Licensee has not been granted any trademark license as part of this License and may not use any name or mark associated with IDEA without the prior written permission of IDEA, except to the extent necessary to make the reference required by the attribution notice of this Agreement.
+
+8. APPLICABLE LAW; DISPUTE RESOLUTION
+
+This License will be governed and construed under the laws of the People’s Republic of China without regard to conflicts of law provisions. The parties expressly agree that the United Nations Convention on Contracts for the International Sale of Goods will not apply. Any suit or proceeding arising out of or relating to this License will be brought in the courts, as applicable, in Shenzhen, Guangdong, and each party irrevocably submits to the jurisdiction and venue of such courts.

README.md

@@ -0,0 +1,78 @@
+
+<div align=center>
+  <img src="assets/upn_vis.jpg" width=600 >
+</div>
+
+
+
+<div align=center>
+
+![Static Badge](https://img.shields.io/badge/UPN-green) [![arXiv preprint](https://img.shields.io/badge/arxiv_2403.14610-blue%3Flog%3Darxiv)](https://arxiv.org/pdf/2403.14610.pdf)  [![Hits](https://hits.seeyoufarm.com/api/count/incr/badge.svg?url=https%3A%2F%2Fgithub.com%2FIDEA-Research%2FUniversal-Proposal-Network&count_bg=%2379C83D&title_bg=%23E56029&icon=askfm.svg&icon_color=%237DE587&title=VISTORS&edge_flat=true)](https://hits.seeyoufarm.com)
+</div>
+
+# 1. Introduction 📚
+**TL;DR: UPN is an object proposal model that can detect any object without any prompt input.**
+
+Universal Proposal Network (UPN) is a robust object proposal model designed as part of ChatRex to enable comprehensive and accurate object detection across diverse granularities and domains. Built upon T-Rex2, UPN is a DETR-based model with a dual-granularity prompt tuning strategy, combining fine-grained (e.g., part-level) and coarse-grained (e.g., instance-level) detection.
+
+<div align=center>
+  <img src="assets/upn_arch.jpg" width=800 >
+</div>
+
+# 2. Installation 🛠️
+```bash
+conda install -n upn python=3.9
+pip install torch==2.1.2 torchvision==0.16.2 --index-url https://download.pytorch.org/whl/cu121
+pip install -v- e .
+# install deformable attention
+cd upn/ops
+pip install -v -e .
+```
+
+To verify the installation, run the following command:
+```bash
+python tests/test_install.py
+```
+
+If the installation is successful, you will get two visualizations of the model's output in `tests` folder.
+
+
+# 3. Usage 🚀
+```python
+import torch
+from PIL import Image
+from tools.visualize import plot_boxes_to_image
+from upn import UPNWrapper
+
+ckpt_path = "checkpoints/upn_large.pth"
+test_image_path = "tests/test_image.jpeg"
+
+model = UPNWrapper(ckpt_path)
+# fine-grained prompt
+fine_grained_proposals = model.inference(
+    test_image_path, prompt_type="fine_grained_prompt"
+)
+fine_grained_filtered_proposals = model.filter(
+    fine_grained_proposals, min_score=0.3, nms_value=0.8
+)
+
+# coarse-grained prompt
+coarse_grained_proposals = model.inference(
+    test_image_path, prompt_type="coarse_grained_prompt"
+)
+coarse_grained_filtered_proposals = model.filter(
+    coarse_grained_proposals, min_score=0.3, nms_value=0.8
+)
+```
+
+# BibTeX 📚
+```
+@misc{jiang2024trex2,
+      title={T-Rex2: Towards Generic Object Detection via Text-Visual Prompt Synergy}, 
+      author={Qing Jiang and Feng Li and Zhaoyang Zeng and Tianhe Ren and Shilong Liu and Lei Zhang},
+      year={2024},
+      eprint={2403.14610},
+      archivePrefix={arXiv},
+      primaryClass={cs.CV}
+}
+```

configs/TSVDatasets

@@ -0,0 +1 @@
+TSVDatasets/

configs/upn_large.py

@@ -0,0 +1,73 @@
+transformer_cfg = dict(
+    type="DeformableTransformer",
+    num_queries=900,
+    encoder_cfg=dict(
+        type="UPNEncoder",
+        encoder_layer_cfg=dict(
+            type="DeformableTransformerEncoderLayer",
+            activation="relu",
+            d_model=256,
+            dropout=0.0,
+            d_ffn=2048,
+            n_heads=8,
+            n_levels=5,
+        ),
+        d_model=256,
+        num_layers=6,
+        use_checkpoint=False,
+        use_transformer_ckpt=False,
+    ),
+    decoder_cfg=dict(
+        type="UPNDecoder",
+        decoder_layer_cfg=dict(
+            type="DeformableTransformerDecoderLayer",
+            activation="relu",
+            d_model=256,
+            n_heads=8,
+            dropout=0.0,
+            d_ffn=2048,
+            n_levels=5,
+        ),
+        d_model=256,
+        return_intermediate=True,
+        num_layers=6,
+        rm_dec_query_scale=True,
+        use_detached_boxes_dec_out=False,
+    ),
+    learnable_tgt_init=True,
+    random_refpoints_xy=False,
+    num_feature_levels=5,
+    two_stage_bbox_embed_share=False,
+    two_stage_class_embed_share=False,
+    two_stage_keep_all_tokens=False,
+    two_stage_learn_wh=False,
+    two_stage_type="standard",
+    binary_query_selection=False,
+)
+
+vision_backbone = dict(
+    type="SwinWrapper",
+    backbone_cfg="swin_L_384_22k",
+    lr_backbone=1e-05,
+    dilation=False,
+    return_interm_indices=[0, 1, 2, 3],
+    backbone_freeze_keywords=None,
+    backbone_ckpt_path=None,
+    use_checkpoint=False,
+    position_embedding_cfg=dict(
+        type="PositionEmbeddingSineHW",
+        normalize=True,
+        num_pos_feats=128,
+        temperatureH=20,
+        temperatureW=20,
+    ),
+)
+
+model = dict(
+    type="UPN",
+    vision_backbone_cfg=vision_backbone,
+    transformer_cfg=transformer_cfg,
+    num_queries=900,
+    dec_pred_bbox_embed_share=True,
+    dec_pred_class_embed_share=True,
+)

requirements.txt

@@ -0,0 +1,10 @@
+mmengine==0.8.2
+numpy==1.22.0
+Pillow==10.1.0
+safetensors==0.3.1
+scipy==1.9.1
+setuptools==59.5.0
+termcolor==2.4.0
+timm==0.4.12
+transformers==4.25.1
+pycocotools==2.0.8