Xiuwei Xu | 许修为

I am a fourth year Ph.D student in the Department of Automation at Tsinghua University, advised by Prof. Jiwen Lu . In 2021, I obtained my B.Eng. in the Department of Automation, Tsinghua University.

I am interested in computer vision and robotics. My current research focuses on:

Email  /  CV  /  Google Scholar  /  Github

*Equal contribution,    ^†Project leader.

Preprint

We presented ESAM, an efficient framework that leverages vision foundation models for online, real-time, fine-grained, generalized and open-vocabulary 3D instance segmentation.

Our embodied agent efficiently explores the unknown environment to generate feasible plans with existing objects to accomplish abstract instructions. which can complete complex human instructions such as making breakfast, tidying bedrooms and cleaning bathrooms in house-level scenes.

We propose a TAsk Planing Agent (TaPA) in embodied tasks for grounded planning with physical scene constraint, where the agent generates executable plans according to the existed objects in the scene by aligning LLMs with the visual perception models.

Selected Publications

We propose a training-free object-goal navigation framework by leveraging LLM and VFMs. We construct an online hierarchical 3D scene graph and prompt LLM to exploit structure information contained in subgraphs for zero-shot decision making.

We propose a post-training quantization framework of large vision-language models (LVLMs). Our method compresses the memory by 2.78x and increase the generate speed by 1.44x on 13B LLaVA model without performance degradation on diverse multi-modal reasoning tasks.

We propose an effective and efficient 3D detector named DSPDet3D for detecting small objects. By scaling up the spatial resolution of feature maps and pruning uninformative scene representaions, DSPDet3D is able to capture detailed local geometric information while keeping low memory footprint and latency.

We propose a model and task-agnostic plug-and-play module, which converts offline 3D scene perception models (receive reconstructed point clouds) to online perception models (receive streaming RGB-D videos).

We propose a post-training quantization framework to compress diffusion models, which performs group-wise quantization to minimize rounding errors across time steps and selects generated contents in the optimal time steps for calibration.

We extend BR to BR++ by introducing differentiable label enhancement and label-assisted self-training. Our approach surpasses current weakly-supervised and semi-supervised methods by a large margin, and achieves comparable detection performance with some fully-supervised methods with less than 5% of the labeling labor.

We propose a hardware-algorithm co-optimizations method called MCUFormer to deploy vision transformers on microcontrollers with extremely limited memory, where we jointly design transformer architecture and construct the inference operator library to fit the memory resource constraint.

We propose a binary sparse convolutional network called BSC-Net for efficient point cloud analysis. With the presented shifted sparse convolution operation and efficient search method, we reduce the quantization error for sparse convolution without additional computation overhead.

We propose the extremely low-precision vision transformers in 2-4 bits, where a self-attention rank consistency loss and a group-wise quantization strategy are presented for quantization error minimization.

We propose a weakly-supervised approach for 3D object detection, which makes it possible to train a strong 3D detector with only annotations of object centers. We convert the weak annotations into virtual scenes with synthetic 3D shapes and apply domain adaptation to train a size-aware detector for real scenes.

Website Template