Natural language and Vision

 

Introduction

We study methods to understand the visual information on an image using natural language. For example, we proposed a method that takes an image and a natural language question about the image and provide an accurate natural language answer as the output. In this task, capturing the relationship between a question and visual information is important to achieve good performance. We propose a novel method using an attention mechanism to capture the relationship between natural language and vision.

VQA-example.png

 

References

Duy-Kien Nguyen and Takayuki Okatani, Multi-task Learning of Hierarchical Vision-Language Representation, IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019 [PDF]
 
Duy-Kien Nguyen and Takayuki Okatani, Improved Fusion of Visual and Language Representations by Dense Symmetric Coattention for Visual Question Answering, IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018 [PDF]