Federated learning-driven collaborative recommendation system for multi-modal art analysis and enhanced recommendations

• Published in: PEERJ COMPUTER SCIENCE
• Main Authors: Gong, Bei; Mahsan, Ida Puteri; Xiao, Junhua
• Format: Article
• Language: English
• Published: PEERJ INC 2024
• Subjects: Computer Science
• Online Access: https://www-webofscience-com.uitm.idm.oclc.org/wos/woscc/full-recordWOS:001374782600003
• ISSN: 2376-5992
• DOI: 10.7717/peerj-cs.2405

With the rapid development of artificial intelligence technology, recommendation systems have been widely applied in various fi elds. However, in the art fi eld, art similarity search and recommendation systems face unique challenges, namely data privacy and copyright protection issues. To address these problems, this article proposes a cross-institutional artwork similarity search and recommendation system (AI-based Collaborative Recommendation System (AICRS) framework) that combines multimodal data fusion and federated learning. This system uses pre- trained convolutional neural networks (CNN) and Bidirectional Encoder Representation from Transformers (BERT) models to extract features from image and text data. It then uses a federated learning framework to train models locally at each participating institution and aggregate parameters to optimize the global model. Experimental results show that the AICRS framework achieves a fi nal accuracy of 92.02% on the SemArt dataset, compared to 81.52% and 83.44% for traditional CNN and Long Short-Term Memory (LSTM) models, respectively. The fi nal loss value of the AICRS framework is 0.1284, which is better than the 0.248 and 0.188 of CNN and LSTM models. The research results of this article not only provide an effective technical solution but also offer strong support for the recommendation and protection of artworks in practice.