Abstract:
The human visual system did not evolve to optimize tasks such as driving in a blizzard, navigating subterranean sites or rescuing trapped victims from a burning building. For such crucial avenues, we hope to build robots that operate better than humans. But, studies on passive robot vision have largely focused on visible cameras which, much like the human eye, were not designed for adverse conditions. By relying on visible light, we are inadvertently constraining the robots’ abilities. In this thesis, we explore the use of thermal infrared cameras for robot vision in the field under arbitrary lighting and weather conditions. In particular, uncooled microbolometers, which are the affordable type of thermal cameras, suffer from significant image degradation, including motion blur and rolling shutter distortions, in the presence of relative motion between the camera and the scene. The lack of control over exposure time and the lack of global shutter technology limit the use of thermal cameras in robotics. Our work studies the origins of motion blur in microbolometers starting from their physics of image formation and proposes model based algorithms using high end microbolometers. Combining this knowledge of microbolometer physics with the generalizability of learning methods could enable wider adoption of thermal cameras in robotics. Towards this end, we collected the first-of-its-kind outdoor driving dataset in adverse lighting and weather conditions with a state-of-the-art cooled photon detector that captures sharp images without motion blur and without distortions. We then used the regularization capabilities of implicit neural representations to fuse noisy measurements from a cooled thermal camera and a high-resolution lidar to generate dense pseudo-ground truth images, depth maps and optical flow maps that are pixel-wise aligned with the microbolometer. This can enable developing novel deep network algorithms to perform vision tasks using only microbolometer data. We believe these developments could propel thermal cameras to undertake a primary role for safe robot operation in all lighting and weather conditions.