TY  - JOUR
AU  - Ruben Tito
AU  - Dimosthenis Karatzas
AU  - Ernest Valveny
PY  - 2023//
TI  - Hierarchical multimodal transformers for Multi-Page DocVQA
T2  - PR
JO  - Pattern Recognition
SP  - 109834
VL  - 144
N2  - Document Visual Question Answering (DocVQA) refers to the task of answering questions from document images. Existing work on DocVQA only considers single-page documents. However, in real scenarios documents are mostly composed of multiple pages that should be processed altogether. In this work we extend DocVQA to the multi-page scenario. For that, we first create a new dataset, MP-DocVQA, where questions are posed over multi-page documents instead of single pages. Second, we propose a new hierarchical method, Hi-VT5, based on the T5 architecture, that overcomes the limitations of current methods to process long multi-page documents. The proposed method is based on a hierarchical transformer architecture where the encoder summarizes the most relevant information of every page and then, the decoder takes this summarized information to generate the final answer. Through extensive experimentation, we demonstrate that our method is able, in a single stage, to answer the questions and provide the page that contains the relevant information to find the answer, which can be used as a kind of explainability measure.
SN  - ISSN 0031-3203
L1  - http://158.109.8.37/files/TKV2022.pdf
UR  - http://dx.doi.org/10.1016/j.patcog.2023.109834
N1  - DAG; 600.155; 600.121
ID  - Ruben Tito2023
ER  -