What is anomaly detection, and how can it benefit your company?

Anomaly detection is a type of data mining that analyzes a company’s data to detect data points that deviate from the established baseline (e.g., the standard behavior of the dataset). These outliers typically indicate incidents, such as technical glitches in equipment, changes in customer preference, and other types of abnormalities, enabling companies to act before the damage is done.

An anomaly is an inconsistent data point that deviates from a familiar pattern. Even though it doesn’t always represent a significant concern, it’s worth investigating to prevent possible escalations. For example, a spike in product sales can be a result of a successful marketing campaign, or it can point to a change in trends and customer behavior, which companies will have to adapt to.

Business data anomalies fall into three outlier categories:

Most companies deal with large volumes of structured and unstructured data, with the latter comprising up to 90% of information generated within a company’s walls. It’s impossible to process all this information manually and generate meaningful insights — especially if we talk about unstructured data, which is made up of images, transactions, free-form text, etc.

Research shows that machine learning techniques are the best choice for processing large unstructured datasets. This field has an extensive number of algorithms, and you can select what suits you best. You can also combine several ML techniques together for optimal results.

There are three main types of AI and machine learning-based anomaly detection techniques.

Anomaly detection relies on artificial intelligence and its subtypes, including machine learning. Here are five ML techniques that are frequently deployed within this context.

Autoencoders are unsupervised artificial neural networks that compress data and then reconstruct it to resemble the original form as closely as possible. These algorithms can effectively ignore noise and reconstruct text, images, and other types of data. An autoencoder has two parts:

When using an autoencoder, pay attention to the code’s size, as it will determine the compression rate. Another important parameter is the number of layers. With fewer layers, the algorithm will be faster, but it could work on fewer features.

This technique is a type of probabilistic graph-based model that calculates probability based on Bayesian inference. Nodes in a graph correspond to random variables, while edges represent conditional dependencies that allow the model to make inferences.

Bayesian networks are used in diagnostics, causal modeling, reasoning, and more. In anomaly detection, this method is particularly useful for detecting subtle deviations that are hard to spot using other techniques. This method can also tolerate missing data during training and will still have solid performance if trained on small datasets.

This is an unsupervised ML clustering technique that detects patterns relying purely on spatial location and distances between neighbors. It compares the density value of a data point to the density of its neighboring data points. An outlier (an anomaly) will have a lower density value than other data populations.

This is a supervised ML algorithm that is commonly used for classification. However, SVM extensions can also operate in an unsupervised environment. This technique uses hyperplanes to divide data points into classes.

Even though SVM typically works with two or more classes, in anomaly detection, it can analyze single-class problems. It learns “the norm” for this one class and determines whether a data point can belong to this class or whether it is an outlier.

GMM is a probabilistic clustering technique. This technique classifies data into different clusters based on probability distribution. It assumes that data points belong to a mix of Gaussian distributions with unknown parameters, and it detects anomalies by spotting data in low-density regions.

Now that you know how anomaly detection works behind the scenes and the AI techniques it relies on, it’s time to study some anomaly detection examples in different industries.

Anomaly detection can benefit the medical sector by helping physicians identify any problems with patient health, detect escalations in inpatients and notify the medical staff before it is too late, and aid in diagnosis and treatment selection. All this reduces the manual work and the cognitive load that doctors experience.

However, anomaly detection has its unique challenges in healthcare.

One issue is that it can be difficult to establish the baseline (i.e., the normal behavior) when it comes to different medical diagrams. For instance, an electroencephalogram of a healthy person varies based on individual characteristics. Researchers identified considerable variations in children, and there are differences in adults depending on the age group and gender.

Another aspect is that ML models must be highly accurate, as people’s lives will depend on their performance.

Medical anomaly detection algorithms can analyze the following information:

One example of anomaly detection comes from a research team in South Africa. They successfully combined autoencoder and extreme gradient boosting techniques to monitor the physiological variables of COVID-19 patients and detect any anomalies that would indicate health degradation.

Another team focused not only on detecting anomalies but also on explaining why the tool flagged them as such. So, first, they used anomaly detection techniques to spot deviations and then deployed aspect mining algorithms to outline a set of features in which a certain data point is considered an outlier.

Sports and entertainment environments rely on extensive video-based security monitoring with hundreds of cameras. Therefore, it wouldn’t be possible for security teams to spot and timely react to accidents if the footage was reviewed manually. Thanks to machine learning, algorithms can analyze videos streaming from each camera at the facility and detect security violations.

As ML models continue to learn on the job, they could spot threats and violations that your human operators couldn’t have noticed. These algorithms can detect vandalism, unrest among spectators, smoke, suspicious objects, and more, and alert security staff so that they have time to act and prevent liabilities and reputational damage.

One such project comes straight out of our portfolio. A US-based entertainment company with gaming rooms located all around the country turned to ITRex to build an ML-driven anomaly detection solution that would integrate into their cloud-based video surveillance system. This application would catch any dangerous and violent behavior, such as breaking slot machines. It would also streamline the administrative process by spotting forgotten items and machines that are out of order.

Our team built a customized ML model using a variational autoencoder. We aggregated a training dataset of 150 videos depicting physical violence and property damage and preprocessed these videos with the OpenCV framework. Then, we relied on the torchvision library to normalize and augment the data and used it to train the ML algorithm.

The resulting solution relied on cross-validation to spot anomalies. For instance, it could identify malfunctioning slot machines by “reading” the error message on the screen and validating it against the available screen templates. The final solution integrated seamlessly into the client’s cloud-based security system, monitored slot machines 24/7, and notified the security personnel whenever it detected an anomaly.

As manufacturing processes become increasingly automated, machinery becomes more complex and the facilities grow larger. Consequently, the traditional monitoring approaches aren’t sufficient anymore.

Anomaly detection techniques can depict different deviations from the norm at your facility and notify you before they escalate and even learn to distinguish between minor issues and pressing concerns.

There are numerous anomaly detection benefits for manufacturing. These tools can spot the following issues:

An example of a manufacturing anomaly detection solution comes from Hemlock Semiconductor, a US-based producer of hyper-pure polysilicon. The company deployed anomaly detection to get visibility into their processes and record any deviations from optimal production patterns. The company reported saving around $300,000 per month in resource consumption.

Anomaly detection can help retailers identify unusual patterns of behavior and use these insights to improve operations and protect their business and customers. AI algorithms can catch on changing client demands and alert retailers to stop acquiring products that will not sell while restocking on items that are in demand. Also, anomalies can represent business opportunities at early stages, allowing retailers to capitalize on them before the competition. In the case of eCommerce, website owners can deploy anomaly detection models to monitor traffic to spot unusual behavior that might signal fraudulent activity.

Additionally, retailers can use anomaly detection techniques to secure their premises. At ITRex, we conducted a series of PoCs to build a solution that can detect expressions of violence, such as fights, in videos streamed by security cameras placed in shopping malls and other public places. The solution relies on the 3D convolutional neural networks anomaly detection method, which was trained on an extensive fight dataset. This type of ML algorithm is known to perform well on action detection tasks.
If you are interested in such a solution, we can show you the full demo to begin with. Then, our team will fine-tune the algorithm and adjust its settings to match the specifics of your location and business, and we will integrate it seamlessly into your existing security system.

As you can see, training custom AI models for spot-on anomaly detection can be a technical challenge. That’s why our team prepared a five-step guide for companies eyeing the novel technology. Scroll down for some expert tips — and consider downloading our business guide to artificial intelligence if you’re new to AI or seek more information about AI applications and project costs.

There are two options here. You are either looking for specific anomalies in your data, or you want to flag everything that deviates from the standard behavior. What you choose here will impact your training data and restrict the selection of AI techniques.

If you want to catch every event that deviates from the baseline, then you will train the model on a large dataset representing normal behavior. For instance, if you are working on driving and traffic safety, then your dataset will be composed of videos showing safe driving.

Suppose you are looking to detect specific anomalies — for instance, car crashes, but not minor violations, such as driving through a red light. In this case, your training dataset will include videos or images of car crashes.

The outcome of the previous step will help you decide which type of data you need.

Collect the data from your internal company sources or use publicly available datasets. Then, clean this data to eliminate duplicates and any incorrect or unbalanced entries. When the dataset is cleaned, you can use scaling, normalization, and other data transformation techniques to make the set suitable for AI algorithms. Split your dataset into three parts:

For more information, check out our detailed guide on how to prepare data for machine learning.

This step is only relevant if you want to build a custom solution. You or your tech vendor will select the best-suited artificial intelligence technique to tackle the business problem. There are three key factors to consider here:

You can either purchase ready-made anomaly detection software or implement a custom system that will correspond to your unique needs and be tailored to the type of anomalies that you are interested in.

You can opt for an off-the-shelf anomaly detection system when you have limited financial resources, no custom training dataset, or no time for model training, and you can find a vendor who already offers a solution that can detect the type of anomalies you are concerned with. But keep in mind that these solutions have built-in assumptions regarding data characteristics and they will perform well as long as these assumptions hold. However, if your company data deviates from that baseline, the algorithm might not detect anomalies with the same accuracy.

If you have sufficient data to train AI algorithms, you can hire a machine learning development company to build and train a custom anomaly detection solution. This option will be designed to meet your business needs and fit into your processes. Another great benefit is that you can still optimize this solution even after deployment. You can tweak its settings to make it work faster or focus on different parameters, depending on your changing business requirements.

You will deploy the anomaly detection solution either locally or in the cloud. If you work with ITRex, we will have two options for you to choose from:

ML algorithms continue to learn on the job, which allows them to adapt to new types of data. But this also means that they can acquire bias and other undesirable tendencies. To avoid this scenario, you can schedule an audit to re-evaluate the algorithms’ performance and implement the necessary adjustments.

At ITRex Group, we have vast experience with machine learning models, such as Beta-Variational Autoencoders (Beta-VAE) and Gaussian Mixture Models (GMM), IoT, data analytics, and data visualization. We’ve implemented these technologies in different industries, so we are aware of the specifics that heavily-regulated sectors, such as healthcare, bring. We use a combination of open-source and proprietary technology, such as data mining tools and ML frameworks, to develop custom solutions and integrate them into your business processes.

Our custom AI-based solutions for anomaly detection can work with both tasks — detecting predefined anomalies and spotting any deviation from the established standard behavior. You can opt for the cloud to save on infrastructure, or we can make the system run locally to support critical applications that don’t tolerate latency.