Business Intelligence System
Business Intelligence Systems are rapidly becoming the key to gain competitive advantage and achieve superior business performance.
This paper explains the key components of a business intelligence system and how it can support management in increasing growth and profitability.
Descriptive analysis presents past data to the user in a more easy to understand way. It shows data in the form of reports & graphs explaining what has happened without explaining why it has happened.
Different types of descriptive analysis are:
Key Performance Indicator analysis - It shows the sum, average or % change in the number of events, actions or results over a time compared to the previous period or target. Key performance indicators like total revenue, cost, profit or ROE for a month, quarter or year and the % change compared to the same period in previous period or to the budget.
Cluster Analysis - It groups the events, actions & results into groups and sub group. For example grouping product wise sales into product groups and sub groups so that total sales is presented group wise with drill down analysis to sub groups and to individual products.
Time series analysis - It shows the trend, seasonality and cyclical behavior of events, actions & results over a time period. For example month wise, quarter wise or year wise sales figure highlighting overall increasing or decreasing trend and the impact of seasons or business cycles.
Most of the business dashboards use descriptive analysis to show business results in bar charts, line charts, pie charts or in tabular forms. These dashboards collect data from the organisation's ERP system or from the web and present the analysis to the users after manipulating through predefined logic.
Diagnostic analysis finds the hidden relationship in data. It tries to establish relationship between different variables to solve a business problem or to identify a business opportunity.
Different types of diagnostic analysis are:
Causation analysis - It shows the cause and effect relationship between events, actions & results. Through drill down facility, it goes to the root cause and answer why it has happened if something good or bad has happened. For example if sales is down compared to last year, causation analysis answers whether due due to product mix change or volume change or due to price change. If it is due to drop in volume, it further drills down to the product, customer and the sales manager where this drop has happened. It can further dill down to answer if the volume for that product is down due to lack of leads or it is down due to lost sales.
Association analysis - It shows the association relationship between two variables in the data. Association relationship is said to be present when one event takes place simultaneous when other event takes place without any established cause and effect relationship between them. For example when customer buy one product they also buy another product or when sales to a customer drops and the overdue receivable from that customer increases. Two variables are associated when both variables have same cause or both have the same effect.
Advanced dashboards provide causation and association relationship in data through drill down facility from the effect to the cause or showing the change in both the associated variables in one chart.
Predictive analysis predicts the future events and results based on the pattern in the past data.
Predictive analysis are classified into:
Time series based prediction - These predictions assume that history repeats itself. Using this method, BI systems predict the future through:
- understanding from the past data which event is repeating itself, after what duration and at what rate of increase or decrease and then
- applying these insights to predict the future events. Seasonality, cyclical behaviour and increasing or decreasing trend in past data are the basis of such predictions.
For example it predicts the sales of a future month based on increasing or decreasing trend of sales of that month in the past period by identifying the seasonal fluctuation in sales in the past.
Causation based prediction - These predictions assume that there is always a reason for something to happen. Using this method, BI systems predict the future through:
- understanding from the past data or from common belief which cause creates what effect, how much and after what duration and then
- applying these insights to the current cause (i.e. current actions & events) to predict the future events.
For example it predicts the sales of coming month by applying the past average closing lead time and conversion % on the current enquiry in hand.
Association based prediction - These predictions assume that some events are associated and they arise together. They may not have any cause & effect relationship between them, but they may have a common cause or they may have a common effect. Using this method, BI systems predict the future through:
- understanding from the past data or from common belief which events are associated and to what extent and then
- applying this insight on the happening of one event to predict the happening of the associated event .
For example when a customer buys one product on the web, it automatically predicts what are the other products that customer might buy based on association relationship and displays those products to the customer.
Highly advanced dashboards provide prediction of upcoming events in the coming week, month or year to enable proactive decisions for capitalising on opportunities and mitigating risks.
Prescriptive analysis recommends the most appropriate action to address an opportunity or risk after analysing all possible options.
Prescriptive analysis can be reactive i.e. after conducting descriptive analysis or diagnostic analysis or can be proactive i.e. after the predictive analysis.
Reactive prescriptive analysis- Reactive prescriptive analysis is always based on the insights from descriptive or diagnostic analysis. i.e. based on analysis on what has happened and why it has happened.
Descriptive analysis identifies opportunities & risks by summarizing what has happened and prescriptive analysis evaluate possible actions to address them and recommend the most appropriate action.
For example, descriptive analysis will monitor and alert when an order is received from customer and then prescriptive analysis will recommend from which warehouse the product should be delivered based on ware house distance from the customer, self-life of the product and the availability of the delivery resources. It can also go one step ahead and automatically allocate the order to the concerned warehouse and delivery resources.
Diagnostic analysis on the other hand identifies opportunities & risks by understanding why some thing has happened by going to the root cause and then prescriptive analysis recommend the optimal solution to address them.
For example, descriptive analysis would identify and create an alert when delivery timeline for any customer order is missed. Diagnostic analysis would then track the reason for this delay by finding out if the order is produced and waiting for delivery or is laying at any stage of production . Prescriptive analysis would then evaluate possibility of reallocating production & delivery resources to speed up the delivery and recommend the optimal solution. It can also automatically adjust the production & delivery planning in addition to communicating the customer with an apology note and the expected delivery date.
Proactive prescriptive analysis - Proactive prescriptive analysis follows predictive analysis i.e. it recommends the most optimal action plan when an opportunity or a risk is predicted.
For example, descriptive analysis would use sensors to regularly track and alert any abnormal increase in the temperature, pressure, vibration or noise in the equipments. Predictive analysis will use these alerts and predict the risk of equipment failures and subsequent loss of production. Prescriptive analysis would recommend the most optimal action by analysing options like changing the preventive maintenance plan or changing a spare part or calling the maintenance agency for repair or replacing the equipment. It can also go ahead and automatically communicate with the maintenance agency of a repair is needed.
Supervised machine learning is a branch of artificial intelligence. In supervised machine learning the machine is taught to learn from the training data and then use this learning to classify new data or predict the future.
The training data in supervised learning are always labelled i.e. the inputs are defined and their corresponding outputs are already known. It means the answer to each problem is already available in the training data. When this training data is fed into the system, machine learning algorithm understands the features, patterns & relationship in the data and uses this learning to correctly answer when it is required to classify a new item or to predict the future.
Supervised machine learning algorithms are classified into classification model and regression models.
Classification Model- In this model, the learning algorithm recognises all the new items and correctly classify them into their correct groups. For example, during the training process, the system will be fed with a bunch of spam emails marking them as “spam” and the learning algorithm will automatically recognise the key features of a spam email from these training data. Next time when a new e mail comes up, the system can quickly identify whether it is a spam email or not.
Regression Model – In this model the machine is trained to learn the trend, pattern & relationship in the training data and uses this learning to predict a numerical number like price, interest rate, rain fall or temperature.
Regression model is used to predict a target value i.e. dependent value by using one or more input values i.e. independent variables.
This model can be used to do the following type of predictions:
Time series-based prediction - In this method, when training data for a long period of time is fed into the system, the machine learning algorithm recognises the sequence, time interval, seasonality and cyclical behaviour in the occurrence of events or their increasing or decreasing trends. These insights are used by the system to predict weather condition, website traffic, consumer demand or stock price for the next hour, day, week, month or year.
Causation based prediction – In this method when the training data is fed to the system, the machine learning algorithm recognises the cause and effect relationship in the data i.e. which cause leads to what effect and how much One effect can have one cause or multiple causes. These insights are then used by the system to predict the output (i.e. effect) given set of inputs (i.e. the causes). For example, predicting the sales volume when price is increased or decreased, or predicting time to reach home when the time, route and whether condition is selected.
In many cases both time series-based & causation-based predictions are used simultaneously. For example, machine learning algorithm would predict the demand for housing based on the increasing trend in the time series data and then based on causation relationship, it would predict the house prices for that volume of demand.
Following are the commonly used regression techniques, though linear regression and logistic regression techniques are the most popular.
· Linear regression - It establishes relationship between the dependent variable and the independent variables using a best fit straight line
· Polynomial regression - In this method relationship is between dependent & independent variables are established using a curve that fits all data points.
· Logistic regression - This method is used when the dependent variable is binary in nature i.e. pass or fail, win or lose, true or false.
· Stepwise regression - This technique is used when segregation of significant variables is required from a list of multiple independent variables.
· Ridge regression - This technique is used when there is multicollinearity issue in the data (i.e. independent variables are highly co-related with each other).
· Lasso regression - It is similar to ridge regression. Additionally, it is also capable of reducing variability and improving accuracy of linear models.
· Elastic net regression – This method is a mix of ridge regression and lasso regression, to reduce overfitting issues in linear regression.
Some of the common application of supervised learning is given below.
Safety & security – recognising voice, face, handwriting & fingerprints, identifying suspicious people & movements in CCTV cameras, recognising hazardous conditions, improving lie detection testing.
Health care – identifying and predicting diseases from medical scans & pathological samples of patients, sample testing & approval of new drugs, suggesting drugs and surgical procedures from patient medical conditions.
Environment – Forecasting weather condition and natural calamities like cyclone & earthquakes, recognising crop suitability from soil samples, identifying mineral & oil deposits from soil condition, classifying intensity of natural calamities and threat levels, identifying water & air pollution levels.
Business application – Forecasting, sales volume, inventory requirement, manpower requirement & fund requirement, classifying sales leads based on their potential, suggesting suitable products based on customer profile, identifying defective items while receiving from suppliers or delivering to customers, recognising customer & employee sentiments from their facial expression, body language, tweets, comments & emails, recognising employee personality & motives during recruitment, identifying suspicious banking transactions, identifying equipment condition from pressure, vibration & temperature monitoring.
Unsupervised Machine Learning is another part of Artificial intelligence. In this case the machine is not trained on any labelled training data. It is not taught to get the dependent variable (i.e. answer to a question) based on given independent variables (i.e. the inputs). On the contrary the machine analyses live data and learn on a real time basis to provide hidden pattern in data.
It is mostly used in descriptive analysis to work on its own to discover features, patterns, and relationship in data. This machine learning algorithm is not designed to give answer to a question or predict the future events, rather it is designed to bring valuable insights from live data to support decision making, without any prior training.
Following are the most common application of unsupervised learning:
· Clustering
· Association rule
· Anomaly detection
· Latent variable models
Clustering - It automatically splits data into groups based on similarity in their features. For example, when given a new set of sales data, the machine learning algorithm can create group wise sales like customer segment wise sales or product family wise sales. Machine does this analysis on its own without having any prior knowledge on the type of customer segments and product families and the rules to allocate sales transaction to these groups.
Hierarchical clustering is used to find groups and subgroups in data allowing the user to drill down from the top to the groups and then subgroups and to individual items. At the first layer each group will be dissimilar to each other, then with in each group the sub-groups will be dissimilar and then with in the subgroup each item will be dissimilar. For example, the learning algorithm will divide the total sales into geography wise sales and within each geography it will divide the sale into customer segment wise sales and then with in each customer segment it will divide the sales into product family wise and then product wise.
There are two type of hierarchical clustering, Divisive and Agglomerative.
Divisive clustering – This is a top down clustering method in which all the data are assigned to a single cluster and then this single cluster is partitioned to least similar clusters. Then each of these cluster is further partitioned to least similar clusters. This process is continued till there is one cluster which cannot be partitioned further.
Agglomerative clustering – This is a bottom up method in which each data is assigned to its own cluster and then based on similarity between the clusters, similar clusters are joined. This process is repeated till there is only a single cluster left.
Association Rule – In association rule the machine learning algorithm discovers association relationship in the data. It identifies those patterns in data which reflect when one event takes place another event also without any visible cause and effect relationship between them.
For example, association rule algorithm can be used to analyse sales data like which products customer buys together or what % of customer who buys “X” also buys “Y” or what % of customers who buys “X & Y” also buys “Z”. This is based on the logic that customers many times buy different products together like in when they buy bread they also by butter and when they buy bread & butter they also buy eggs or when the buy eggs they also buy milk. Association rule discovers all these relationship in data which can used for marketing campaign, pricing and retail outlet display decisions.
Association rule algorithm can also be used to identify what are the news, stories and topics are usually followed or searched by different people. These insights can be used by social media, search engines and e commerce platforms to provide customised news, stories and product suggestions to individual customers.
In medical field also association rules can be used to identify which medical condition what pathological test results and which patient profiles are associated. These insights can be used to predict and cure diseases.
For predicting and solving equipment failures association rule algorithm can also be used for identifying association between equipment failures and equipment temperature, vibration, pressure, running hour, weather condition and other operating parameters.
Anomaly detection – This algorithm is used to identify abnormal or rare events which are different than rest of the data points. For example, anomaly detection techniques are used to identify abnormal or suspicious transactions to detect debit card or credit card frauds.
Without being trained on a set of training data, this unsupervised machine leaning algorithm can continuously scan the incoming live data and identify any abnormal transaction, or event or activity with is different than rest of the data.
Anomaly detection can also be used as a part of supervised machine learning if the machine is trained to detect unusual transactions by prior training using labelled training data.
Some application of anomaly detection technique is given below:
Banking & Finance - Identifying suspicious transactions in debt card, credit card, net-banking and other online payment gateways.
Computer & IT System - Tracking suspicious user access to servers & computer networks and identifying abnormal hardware and software performance resulting from virus attack.
Medical & Healthcare - Identifying abnormal medical condition and test results of patients signaling a possible disease or organ failure.
Equipment Maintenance - Identifying unusual temperature, pressure, vibration and noise in industrial, transport, medical & other equipment indicating possible equipment failures.
Environment - Tracking abnormal change in air pressure, temperature & moisture indicating possible cyclone and natural disaster.
Safety & Security - Identifying abnormal people, objects, activity or crowd formation in CC TV cameras, drone & sate light images indicating possible security threats.
Latent variable models - Latent variables are not directly observed in data but can be inferred from other observable variables. Machine learning algorithm can be used to reduce the dimensionality of data. Many observable variables in data can be aggregated in a model to represent an under laying concept in data.
Many times latent variables cannot be directly measured. These are hidden variables. Linking these variables to other observable variables, the values of the latent variables can be inferred from the measurement of the observable variable.
For example, ease of doing business in a country can be a latent variable the value of which can be inferred by the machine learning algorithm from the measurement of other observed variables like number of license required to start a business, private sector share in GDP, private sector employment % to total employment and foreign direct Investment growth.