Machine Learning

Supervised machine learning is a branch of artificial intelligence. In supervised machine learning the machine is taught to learn from the training data and then use this learning to classify new data or predict the future.

The training data in supervised learning are always labelled i.e. the inputs are defined and their corresponding outputs are already known. It means the answer to each problem is already available in the training data. When this training data is fed into the system, machine learning algorithm understands the features, patterns & relationship in the data and uses this learning to correctly answer when it is required to classify a new item or to predict the future.

Supervised machine learning algorithms are classified into classification model and regression models.

Classification Model- In this model, the learning algorithm recognises all the new items and correctly classify them into their correct groups. For example, during the training process, the system will be fed with a bunch of spam emails marking them as “spam” and the learning algorithm will automatically recognize the key features of a spam email from these training data. Next time when a new e mail comes up, the system can quickly identify whether it is a spam email or not. 

Regression Model – In this model the machine is trained to learn the trend, pattern & relationship in the training data and uses this learning to predict a numerical number like price, interest rate, rain fall or temperature. 

Regression model is used to predict a target value i.e. dependent value by using one or more input values i.e. independent variables.

This model can be used to do the following type of predictions:

Time series-based prediction - In this method, when training data for a long period of time is fed into the system, the machine learning algorithm recognizes the sequence, time interval, seasonality and cyclical behavior in the occurrence of events or their increasing or decreasing trends. These insights are used by the system to predict weather condition, website traffic, consumer demand or stock price for the next hour, day, week, month or year.

Causation based prediction – In this method when the training data is fed to the system, the machine learning algorithm recognises the cause and effect relationship in the data i.e. which cause leads to what effect and how much  One effect can have one cause or multiple causes. These insights are then used by the system to predict the output (i.e. effect) given set of inputs (i.e. the causes). For example, predicting the sales volume when price is increased or decreased, or predicting time to reach home when the time, route and whether condition is selected.

In many cases both time series-based & causation-based predictions are used simultaneously. For example, machine learning algorithm would predict the demand for housing based on the increasing trend in the time series data and then based on causation relationship, it would predict the house prices for that volume of demand.

Following are the commonly used regression techniques, though linear regression and logistic regression techniques are the most popular.

· Linear regression - It establishes relationship between the dependent variable and the independent variables using a best fit straight line

· Polynomial regression - In this method relationship is between dependent & independent variables are established using a curve that fits all data points.

· Logistic regression - This method is used when the dependent variable is binary in nature i.e. pass or fail, win or lose, true or false.

· Stepwise regression - This technique is used when segregation of significant variables is required from a list of multiple independent variables.

· Ridge regression - This technique is used when there is multicollinearity issue in the data (i.e. independent variables are highly co-related with each other).

· Lasso regression - It is similar to ridge regression. Additionally, it is also capable of reducing variability and improving accuracy of linear models.

· Elastic net regression – This method is a mix of ridge regression and lasso regression, to reduce over fitting issues in linear regression.

Some of the common application of supervised learning is given below.

Safety & security – recognizing voice, face, handwriting & fingerprints, identifying suspicious people & movements in CCTV cameras, recognizing hazardous conditions, improving lie detection testing.

Health care – identifying and predicting diseases from medical scans & pathological samples of patients, sample testing & approval of new drugs, suggesting drugs and surgical procedures from patient medical conditions.

Environment – Forecasting weather condition and natural calamities like cyclone & earthquakes, recognizing crop suitability from soil samples, identifying mineral & oil deposits from soil condition, classifying intensity of natural calamities and threat levels, identifying water & air pollution levels.

Business application – Forecasting, sales volume, inventory requirement, manpower requirement & fund requirement, classifying sales leads based on their potential, suggesting suitable products based on customer profile, identifying defective items while receiving from suppliers or delivering to customers, recognising customer & employee sentiments from their facial expression, body language, tweets, comments & emails, recognizing employee personality & motives during recruitment, identifying suspicious banking transactions, identifying equipment condition from pressure, vibration & temperature monitoring.

Unsupervised Machine Learning is another part of Artificial intelligence. In this case the machine is not trained on any labelled training data. It is not taught to get the dependent variable (i.e. answer to a question) based on given independent variables (i.e. the inputs). On the contrary the machine analyses live data and learn on a real time basis to provide hidden pattern in data.

It is mostly used in descriptive analysis to work on its own to discover features, patterns, and relationship in data. This machine learning algorithm is not designed to give answer to a question or predict the future events, rather it is designed to bring valuable insights from live data to support decision making, without any prior training.

Following are the most common application of unsupervised learning:

· Clustering

· Association rule

· Anomaly detection

· Latent variable models

Clustering - It automatically splits data into groups based on similarity in their features. For example, when given a new set of sales data, the machine learning algorithm can create group wise sales like customer segment wise sales or product family wise sales. Machine does this analysis on its own without having any prior knowledge on the type of customer segments and product families and the rules to allocate sales transaction to these groups.

Hierarchical clustering is used to find groups and subgroups in data allowing the user to drill down from the top to the groups and then subgroups and to individual items. At the first layer each group will be dissimilar to each other, then with in each group the sub-groups will be dissimilar and then with in the subgroup each item will be dissimilar. For example, the learning algorithm will divide the total sales into geography wise sales and within each geography it will divide the sale into customer segment wise sales and then with in each customer segment it will divide the sales into product family wise and then product wise.

There are two type of hierarchical clustering, Divisive and Agglomerative.

Divisive clustering – This is a top down clustering method in which all the data are assigned to a single cluster and then this single cluster is partitioned to least similar clusters. Then each of these cluster is further partitioned to least similar clusters. This process is continued till there is one cluster which cannot be partitioned further.

Agglomerative clustering – This is a bottom up method in which each data is assigned to its own cluster and then based on similarity between the clusters, similar clusters are joined. This process is repeated till there is only a single cluster left.

Association Rule –  In association rule the machine learning algorithm discovers association relationship in the data. It identifies those patterns in data which reflect when one event takes place another event also without any visible cause and effect relationship between them.

For example, association rule algorithm can be used to analyse sales data like which products customer buys together or what % of customer who buys “X” also buys “Y” or what % of customers who buys “X & Y” also buys “Z”. This is based on the logic that customers many times buy different products together like in when they buy bread they also by butter and when they buy bread & butter they also buy eggs or when the buy eggs they also buy milk. Association rule discovers all these relationship in data which can used for marketing campaign, pricing and retail outlet display decisions.

Association rule algorithm can also be used to identify what are the news, stories and topics are usually followed or searched by different people. These insights can be used by social media, search engines and e commerce platforms to provide customised news, stories and product suggestions to individual customers.

In medical field also association rules can be used to identify which medical condition what pathological test results and which patient profiles are associated. These insights can be used to predict and cure diseases.

For predicting and solving equipment failures association rule algorithm can also be used for identifying association between equipment failures and equipment temperature, vibration, pressure, running hour, weather condition and other operating parameters.

Anomaly detection – This algorithm is used to identify abnormal or rare events which are different than rest of the data points. For example, anomaly detection techniques are used to identify abnormal or suspicious transactions to detect debit card or credit card frauds.

Without being trained on a set of training data, this unsupervised machine leaning algorithm can continuously scan the incoming live data and identify any abnormal transaction, or event or activity with is different than rest of the data.

Anomaly detection can also be used as a part of supervised machine learning if the machine is trained to detect unusual transactions by prior training using labelled training data.

Some application of anomaly detection technique is given below:

Banking & Finance - Identifying suspicious transactions in debt card, credit card, net-banking and other online payment gateways. 

Computer & IT System -  Tracking suspicious user access to servers & computer networks and identifying abnormal hardware and software performance resulting from virus attack. 

Medical & Healthcare - Identifying abnormal medical condition and test results of patients signaling a possible disease or organ failure.

Equipment Maintenance - Identifying unusual temperature, pressure, vibration and noise in industrial, transport, medical & other equipment indicating possible equipment failures. 

Environment - Tracking abnormal change in air pressure, temperature & moisture indicating possible cyclone and natural disaster.

Safety & Security - Identifying abnormal people, objects, activity or crowd formation in CC TV cameras, drone & sate light images indicating possible security threats.

Latent variable models - Latent variables are not directly observed in data but can be inferred from other observable variables. Machine learning algorithm can be used to reduce the dimensionality of data. Many observable variables in data can be aggregated in a model to represent an under laying concept in data. 

Many times latent variables cannot be directly measured. These are hidden variables. Linking these variables to other observable variables, the values of the latent variables can be inferred from the measurement of the observable variable. 

For example, ease of doing business in a country can be a latent variable the value of which can be inferred by the machine learning algorithm from the measurement of other observed variables like number of license required to start a business, private sector share in GDP, private sector employment % to total employment and foreign direct Investment growth.