Mobile App that Categorizes Art and Spots Forgery

Client

An acclaimed artist with a keen interest in technology

Industry

Entertainment

Services

AI, ML

Technology

Pytorch, SWIFT

Challenge

Establishing the authorship and authenticity of artworks poses a challenge. Our client, a renowned artist, sought to address this hurdle for visitors to his exhibitions by developing an application that would aid viewers in confirming these artistic attributes. In response to the challenge, we undertook research into the potential applications of generative AI in the art world. We set out to apply machine learning to large databases of artworks to verify if they can classify artworks as belonging to certain artists and spot forgery as accurately as art forensics experts would do.

Specifically, we took on the following duties:

●

Identify an appropriate approach to art classification and authentication

●

Design the architecture of a generative AI model powering the solution

●

Train the model on training data and tune it for optimal performance

●

Develop an iOS application and integrate it with the trained and tuned GenAI model

Solution

In the course of two months, the ITRex team conceptualized and developed a proof of concept (PoC) of the solution. The PoC was realized as an iOS application coded in Swift, and built with the help of up-to-date frameworks. The application precisely identifies an artwork’s authorship and assists users in determining its authenticity. The solution relies on StyleGAN, a two-component network featuring a generator and a discriminator. The generator crafts simulated art, while the discriminator distinguishes real artworks from counterfeit ones. The training process continues until the generator creates high-quality forgeries that challenge human judgment. To identify forgery, we fine-tuned a pre-trained StyleGAN on a dataset containing real paintings, man-made forgeries, and AI-generated forgeries. We also built a feature extractor from the GAN's generator network, capable of capturing style features like brushstroke shapes, paint thickness, and color palettes. The training process unfolded as follows:

●

We preselected artists for recognition.

●

We compiled a dataset of their works and other art pieces.

●

To identify artists, we fine-tuned a Variational Autoencoder (VAE) on ResNet and used its decoder as a feature extractor, while training a classifier to recognize the artists.

●

To spot forgeries, we added man-made and AI-generated forgeries to the dataset, splitting it into separate sets for each artist.

●

We fine-tuned StyleGAN, with the generator crafting artificial art and the discriminator distinguishing real from forged. Training continued until the generator produced high-quality forgeries.

●

Following training, we got a feature extractor from StyleGAN, capturing style elements like brushstroke styles, color descriptions, and canvas characteristics that define an artist's work.

●

The trained discriminator from StyleGAN then distinguished between real and counterfeit art, reinforcing authenticity.

Impact

●

We proved that modern smartphone cameras set to macro mode coupled with the generative AI algorithms are enough for the AI model to accurately determine the authorship and authenticity of artworks.

●

When fully productized, the technology has the potential to assist antique buyers, pawn shop workers, and enthusiasts who want to learn more about the items in their home collection.