Federated learning-driven collaborative recommendation system for multi-modal art analysis and enhanced recommendations

• Published in: PeerJ Computer Science
• Main Author: Gong B.; Mahsan I.P.; Xiao J.
• Format: Article
• Language: English
• Published: PeerJ Inc. 2024
• Online Access: https://www.scopus.com/inward/record.uri?eid=2-s2.0-85210759435&doi=10.7717%2fpeerj-cs.2405&partnerID=40&md5=7418e8fcf0f994d59a2204bd6441ef4e

With the rapid development of artificial intelligence technology, recommendation systems have been widely applied in various fields. However, in the art field, art similarity search and recommendation systems face unique challenges, namely data privacy and copyright protection issues. To address these problems, this article proposes a cross-institutional artwork similarity search and recommendation system (AI-based Collaborative Recommendation System (AICRS) framework) that combines multimodal data fusion and federated learning. This system uses pre-trained convolutional neural networks (CNN) and Bidirectional Encoder Representation from Transformers (BERT) models to extract features from image and text data. It then uses a federated learning framework to train models locally at each participating institution and aggregate parameters to optimize the global model. Experimental results show that the AICRS framework achieves a final accuracy of 92.02% on the SemArt dataset, compared to 81.52% and 83.44% for traditional CNN and Long Short-Term Memory (LSTM) models, respectively. The final loss value of the AICRS framework is 0.1284, which is better than the 0.248 and 0.188 of CNN and LSTM models. The research results of this article not only provide an effective technical solution but also offer strong support for the recommendation and protection of artworks in practice. © 2024 Gong et al.