Northeastern Toronto Professor Fuses AI and Art to Preserve Historical Murals
Dr. Zheng Zheng was only six years old when painting became one of his favourite hobbies. Despite his artistic talent, Zheng’s career as an engineer took him in a much more technical direction.
Now an Assistant Teaching Professor in the College of Engineering at Northeastern University in Toronto, Zheng is combining his cherished pastime with innovative artificial intelligence (AI) research. His latest project aims to create a platform that utilizes advanced AI techniques to restore damaged murals digitally. It will take murals with issues like cracks, peeling paint, mould, or missing sections and transform them into images that reveal their original appearance.
“Murals are more than just art,” says Zheng, who has extensive data management experience in both academic and industry roles. “They help us understand the history and culture of past civilizations, but restoring them by hand takes a lot of time and effort.
“While AI tools can help, methods that work well on standard images often make murals look unnatural. That’s because murals are often damaged in complex ways, suffering severe deterioration due to environmental exposure, biological decay, and human activities. These types of deterioration are hard for AI to handle properly. We’re hoping to address those shortcomings.”
The Challenges of Mural Restoration Using AI
Current automated restoration methods either blend nearby pixels or copy parts of the image from different parts. To improve these techniques, Zheng is using advanced AI. His system utilizes deep learning algorithms — AI methods that learn from large amounts of data, much like the human brain — to automatically identify damaged areas and restore them with greater accuracy.
The research is addressing five key challenges.
1. Large missing regions: Many AI tools can repair minor or moderate damage, but struggle with large, severely damaged areas.
2. Style and texture consistency: Maintaining the original artistic style, especially in parts with fine details or textures.
3. Computational efficiency: Some of the most powerful AI models require substantial computing power, making them challenging to deploy in real-time or for large-scale tasks.
4. Limited and heterogeneous data: There aren’t many mural images that come with helpful labels from humans — like notes about what’s in the scene, where damage is, or what time period the work originates from — which makes it hard to train AI. Differences in style and meaning across cultures and historical periods make it even more difficult to build reliable models.
5. Integration of Domain Knowledge: Many current methods overlook essential historical and cultural details crucial for creating authentic and accurate restorations.
“What’s exciting about this project is that because mural restorations are the most complex, our algorithm can be used to restore everything from paintings to photos,” says Zheng. “The idea is to have a tool where you can simply upload your mural or photo and get a beautiful, intact result.”
The First Stage of Advancing Deep Learning for Art Restoration
Still in the early stages of his research, Zheng is developing the tool in two phases.
In the project’s first stage, he will utilize a type of AI model known as a transformer-based encoder to analyze the overall layout of damaged mural images.
The tool won’t just examine small details; it will also understand how different parts of the image fit together, even if they’re far apart. This will enable the AI to make more informed decisions when determining how to fill in missing or damaged areas. It will also recognize essential elements of the image, such as shapes, objects, and scenes.
Zheng will also incorporate a system that includes historical and cultural information, ensuring the restoration is guided by relevant context and written texts, not just visual data.
This multimodal integration is expected to enhance the model’s ability to capture and preserve the stylistic and cultural nuances of the artwork.
“Consider one of the most famous AI tools we have, ChatGPT,” says Zheng, who previously served as the technical vice president and the head of AI at a NASDAQ-listed company. “It is based on a large language model (LLM), which means it is designed to understand and generate human-like text based on the vast amounts of language data it was trained on.
We are combining LLM with computer vision (CV), the AI field focused on interpreting images and videos. This ensures that the restoration isn’t just based on the image itself but also on significant cultural and historical context, making the repair more precise and meaningful.”
The Second Stage of Zheng’s Mural Restoration Technology
In Stage 2, Zheng’s system will be trained to enhance the coarse restoration by incorporating fine details and realistic textures. Through a step-by-step process, the tool will progressively make colours and patterns more natural and accurate.
To further enhance the AI’s restoration capabilities, Zheng will develop a method to evaluate its performance. This function will measure how well the restored mural matches the original’s visual appearance, artistic style, and cultural significance. If the restoration strays too far from the original, the AI will be “penalized” and learn to improve next time.
Zheng will also use a special training method that helps the AI make the repaired parts blend in perfectly with the rest of the mural, so well that the restored sections are almost impossible to tell apart from the original artwork.
The Human Touch in Preserving Murals with AI
“Many experts in murals are interested in this project, and we are actively working to connect with them,” says Zheng. “This is very important because it is essential to integrate human knowledge from experts into our work.”
The researcher isn’t just enlisting the help of mural experts. He is also working with students from Northeastern University in Toronto. One of Zheng’s students is working on the mural restoration project, contributing to the literature review.
In January, Zheng also launched the Data Science and Artificial Intelligence (DSAI) Research Lab at Northeastern University in Toronto. The lab enables students to engage in research during their studies.
“We have a lot of students interested in research, and DSAI is a way for them to get some experience in this area,” Zheng says. “The student I am working with is very interested in deep learning. We hope that one day, people will visit museums and see this technique in action, revealing and preserving the profound historical and cultural significance of murals.”
By: Izabela Shubair