Abstract

Current digital twin frameworks focused on human–robot collaboration rarely take into account the sensory degradation of real industrial environments, nor do they integrate the operator as an active agent within the system. This research presents a multimodal digital twin framework for a dual-arm collaborative robot at an assembly station; the system was developed using ROS 2 Jazzy and CoppeliaSim as the simulator. The architecture integrates three main components: the first is a perception layer that captures voice commands using Whisper ASR and the state of the workspace using a hybrid YOLO + ViT visual pipeline, both with per-channel metadata; the second consists of a Confidence-Weighted Late Fusion engine that dynamically adjusts the weight of each modality based on real-time signal quality, so that each fusion decision can be reconstructed from the signals that generated it; and the third component is a Reference Resolver that grounds linguistic intent within the visual context of the scene and in the fusion weights, using a local instance of Llama 3.1 8B that does not transmit audio, transcripts, or images outside the system. The framework was evaluated using 210 iterations distributed across seven degradation conditions of increasing severity, comparing adaptive fusion against a baseline of fixed weights (0.5/0.5). Under clean conditions and under visual degradation of any severity, both configurations achieved 100% accuracy. Under severe auditory degradation (SNR 0 dB), adaptive fusion activated the safety gate and refrained from executing most commands (13.3% accuracy), while the fixed-weight baseline executed more commands (60% accuracy) but made three incorrect object selections; under severe dual degradation, the pattern repeated (13.3% vs. 40%, with five incorrect selections in the baseline). The adaptive system made no grounding errors in the 210 executions, compared to eight in the baseline, substituting incorrect execution with conservative abstention when no modality provided a reliable signal. The implementation, featuring a versioned degradation protocol and a fixed seed, provides a reproducible benchmark for evaluating multimodal fusion strategies in human–cobot interaction.

IPC Classification

Keywords