Amharic visual question answering on Ethiopian tourism

Visual Question Answering (VQA) is a Vision-to-Text (V2T) task that integrates visual 
features of images with natural language questions to generate meaningful responses. 
Most existing research has focused on English, leaving a significant gap for other 
languages, including Amharic. Tourism, a major global industry, relies heavily on 
interactions where visitors seek information about natural, historical, cultural, and 
religious sites. Ethiopia is a remarkable tourist destination, home to unique sites such as 
the Rock-hewn churches of Lalibela and the Castles of Gondar, as well as natural 
phenomena like Simien National Park and Lake Tana. Most visitors are local, creating an 
urgent need for a VQA model that can deliver accurate, culturally relevant information in 
Amharic. Unfortunately, no such model currently exists to assist tourists at these heritage 
sites. This research addresses this gap by developing an Amharic Visual Question 
Answering model specifically tailored for Ethiopian tourism. A new Amharic VQA 
dataset was created using 2,200 diverse images from Ethiopian tourist sites paired with 
6,600 questions in Amharic, covering natural landmarks, historical sites, and religious 
celebrations. Our dataset is collected from various sources, including the UNICCO 
website, the Amhara Tourism office, and online platforms such as Facebook, Free pixel, 
and Instagram. Each image is complemented by three corresponding questions 
formulated by three individual experts and answered by ten candidates. The questions, 
answers, and images are linked through annotations and fed into the model. We used 
ResNet-50 for feature extraction and Bidirectional Gated Recurrent Unit (BiGRU) with 
attention mechanisms, achieving a testing accuracy of 54.98%, demonstrating the model's 
effectiveness in answering questions about Ethiopian heritage. We will expand this 
research using external knowledge to gat answer and description beyond image and 
custom object detection