Abstract

Transparent objects such as glass containers, test tubes, and plastic bottles are common in robotic manipulation scenarios, but their refractive and reflective surfaces produce incomplete RGB-D geometry and make task-specific grasp selection unreliable. This paper presents an integrated vision-language system for transparent object perception and task-oriented grasping. First, we construct VLM-DRE, a transparent object image instruction dataset with 12,700 images and 38,100 image-instruction-bounding-box triplets. LoRA fine-tuning of Molmo-7B improves target click accuracy from 86.4% to 91.5% and IoU@0.75 from 57.5% to 69.1%. Second, MSR-Net performs monocular depth completion and mask prediction using multi-scale adaptive feature fusion and progressive feature refinement, achieving RMSE 0.066, mAP 98.61%, and IoU 94.12% on Syn-TODD, and RMSE 0.118, mAP 99.02%, and IoU 87.95% on ClearPose. Third, LMF-Net combines RGB-D cross-modal fusion with learnable multi-factor matching to rank AnyGrasp 6-DoF candidates, reaching 77.8% Top-1 and 90.5% Top-3 accuracy on TaskGrasp-Image and improving PRISM-Real success from 61.1% to 68.5%. On a RealSense D435i–Unitree Z1 Pro platform, the complete system obtains 85.4% success with manual clicks and 71.3% with VLM-predicted clicks, supporting perception-to-grasping integration while highlighting target localisation and runtime as deployment bottlenecks.

IPC Classification

Keywords