FSDagents

Full self-driving agents benchmark on closed-loop simulation

Agents:

• InterFuser
• [] VAD
• [] SparseDriving
• [] XLM

Visualization

TODOS

• architecture design with heads
  • maybe consider waypoint heads, heat map head, classification head etc
• add transformer-based sensor fusion benchmarks
• [] add inverse reinforcement learning benchmarks
• [] add LLM benchmark

2024-08:

• training pipeline works
• InterFuser uses focus view from front camera image. the resulting image size is smaller than other images, how to bundle this for stacking images before going into the model
  • focus view is padded to other shapes during pipeline
  • need remove paddings before extracting features
• address warnings during training, e.g., init_weights()
• L1 loss with mask for waypoint loss
• move pts to histogram into data preprocessor
• add TASK_UTILS registry
• Interfuser head/nect should rename to generic head/neck or TASKS_UTILS
• batch first for all data and model inputs. each module/head/neck can have their own batch_first definition to be compatible with called torch modules or control output shape
  • currently batch first is used to when needed by a built-in torch module such as GRU, do we assume batch_first everywhere?
• check model parameters: trainable/nontrainable. The original has 52935567 in total
  • here shows the use of dicts in the inputs for complexity analysis. get_complexity_info can be used but need support list of tensors as inputs instead of dictionaries.
  • small mismatch of model parameters compared with official models.
• [] how to control data type globally?
• [] data time is too long, 80-90% of total time
• [] add base planning module
• closed-loop evaluation code on carla sim environment based on carla leaderboard 2.0
• visualize multi-view/lidar data in data set
• visualization of prediction details during closed-loop simulation
• [] add goal_points to standard default dataset
• implement resnet-d variants to support interfuser -> add dependency on mmPretrain to reuse timm models
• [] interfuser seems not work with batch=1
• check where the pretrained model pth is downloaded
• weights mismatch after conversion.
• direct use cam2img instead of cam_instrinsics?
• vis.set_bev_image() reset the image shape to (480, 640, 3) when set the shape to (800, 800).
• add bev plot for instance trajectory
• add bev plot for multi-modal trajectory
• add map visualization
• input key in data should only contain img, pts. Other such as img_metas, pts_metas should be moved to data_sample metainfo.
• add multiview image names to visualization hook
• the label category number count seems always 0
• add nuscence dataset and examples
• add doc for visualization examples
• add ci tests