Xiuwei Xu | 许修为

I am a fifth year Ph.D student in the Department of Automation at Tsinghua University, advised by Prof. Jiwen Lu . In 2021, I obtained my B.Eng. in the Department of Automation, Tsinghua University.

I work on computer vision and robotics. My current research focus:

Email  /  CV  /  Google Scholar  /  Github  /  LinkedIn

*Equal contribution,    ^†Project leader.

Selected Preprint

We propose iMaC, an image-as-action control paradigm for embodied world models. iMaC converts robot actions into dense motion and contact images through URDF/FK rendering and RGB-D geometry, exposing spatial motion intention and robot-scene contact relations. These image controls enable contact-sensitive future prediction and closed-loop policy evaluation.

We propose F2F-AP, a flow-to-future asynchronous policy for real-time dynamic manipulation. It predicts object flow to synthesize future observations and aligns visual features with future states, allowing policies to compensate for latency and interact with moving objects.

We propose R2RDreamer, a real-to-real demonstration augmentation framework for spatially generalized 2D manipulation policies. R2RDreamer edits incomplete object point clouds and end-effector trajectories in 3D, projects them into occlusion-aware image-space controls, and uses dense-control video completion to synthesize temporally coherent RGB-action demonstrations from limited real data.

We propose a TAsk Planing Agent (TaPA) in embodied tasks for grounded planning with physical scene constraint, where the agent generates executable plans according to the existed objects in the scene by aligning LLMs with the visual perception models.

Selected Publications

We propose a real-to-real 3D data generation framework for robotic manipulation. R2RGen generates spatially diverse manipulation demonstrations for training real-world policies, requiring only one human demonstration without simulator setup.

We propose AwareVLN, a self-aware reasoning framework for vision-language navigation. It triggers structured reasoning at key navigation nodes to understand scene context, task progress, and next-step plans, improving instruction following in simulation and real-world navigation.

We propose a general framework for mobile manipulation, which can be divided into docking point selection and fixed-base manipulation. We model the docking point selection stage as an optimization process, to let the agent move and touch target keypoint under several constraints.

We propose IGL-Nav, an incremental 3D Gaussian localization framework for image-goal navigation. It supports challenging scenarios where the camera for goal capturing and the agent's camera have very different intrinsics and poses, e.g., a cellphone and a RGB-D camera.

We propose TSP3D, an efficient multi-level convolution architecture for 3D visual grounding. TSP3D achieves superior performance compared to previous approaches in both accuracy and inference speed.

We propose UniGoal, a unified graph representation for zero-shot goal-oriented navigation. Based on online 3D scene graph prompting for LLM, our method can be directly applied to different kinds of scenes and goals without training.

We presented ESAM, an efficient framework that leverages vision foundation models for online, real-time, fine-grained, generalized and open-vocabulary 3D instance segmentation.

We propose a training-free object-goal navigation framework by leveraging LLM and VFMs. We construct an online hierarchical 3D scene graph and prompt LLM to exploit structure information contained in subgraphs for zero-shot decision making.

We propose an effective and efficient 3D detector named DSPDet3D for detecting small objects. By scaling up the spatial resolution of feature maps and pruning uninformative scene representaions, DSPDet3D is able to capture detailed local geometric information while keeping low memory footprint and latency.

We propose a model and task-agnostic plug-and-play module, which converts offline 3D scene perception models (receive reconstructed point clouds) to online perception models (receive streaming RGB-D videos).

We propose a weakly-supervised approach for 3D object detection, which makes it possible to train a strong 3D detector with only annotations of object centers. We convert the weak annotations into virtual scenes with synthetic 3D shapes and apply domain adaptation to train a size-aware detector for real scenes.

Selected Projects

Data-efficient Mobile Manipulation

In this project, we study mobile manipulation, where a robot must combine long-horizon navigation with precise local manipulation. Whole-body demonstrations are expensive to collect, and standard navigation usually stops at a coarse region that is still far from the accuracy required by manipulation policies. We therefore focus on two complementary problems: accurate docking that turns navigation into a suitable fixed-base manipulation setup, and scalable data generation that improves policy generalization across viewpoints, object locations, and object geometry for both 2D and 3D policies. Our works are summarized as:

3D Representation for Visual Navigation

In this project, we study how to design a proper representation and how to exploit the representation for general visual navigation. Previous methods mainly focus on BEV map or topological graph, which lacks 3D information to reason fine-grained spatial relationship and detailed color / texture. Therefore, we leverage 3D representation for better modeling of the observed 3D environment. We propose: (1) 3D scene graph as a structural representation for explicit LLM reasoning and unification of different kinds of tasks and (2) 3D gaussians as a renderable representation for accurate image-goal navigation. Our works are summarized as:

Efficient and Online 3D Scene Perception

In this project, we study how to make 3D scene perception methods applicable for embodied scenarios such as robotic planning and interaction. Although various research have been conducted on 3D scene perception, it is still very challenging to (1) process large-scale 3D scenes with both high fine granularity and fast speed and (2) perceive the 3D scenes in an online and real-time manner that directly consumes streaming RGB-D video as input. We solve these problems in below works:

Website Template