There is a danger here, the easy use of GUI tools on large amount of data available is tempting and which makes users tempt to use black box methodologies available in the tools to solve their business problems. They mistakingly assume data mining is all about using a tool and running the data on the tool, this can be very hazardous and dangerous as actions are taken on business decisions.

Little knowledge is dangerous while applying powerful models.

Since the underlying logic used for model building in automated tools are unknown (their internal assumptions are unclear). Therefore it is very easy to do it in the wrong way assuming it is the right solution. Such mistakes are still made in companies.

People sometimes might argue that there is always a simpler solution to problems. I am the person who belives in Occam’s Razor, which means simplest explanation is always the best way. This means the interpretation of the model and the final solution should be as simple as possible. However,  method of arriving at solution must be as detail as possible taking every factor, element into consideration. Analysts must not make the mistake of over simplyfying the method and make too many assumptions.

A lot of knowledge workers believe that predictive modeling process can be automated by using tools and statistical software. These software are certainly useful but cannot replace the intellectual  of the analyst.

The processof building the model must be as detailed as possible, it has to tried & tested on various data and measured by various parameters. All possible factors must be considered while building the model.

Therefore Analyst must understand the underlying algorithm , process of arriving score, model design etc.

More on such articles regarding online marketing, data mining, data modeling can be found at http://www.praveenkodur.com/blog

Even a using a tool as simple as a spreadsheet will be extremely helpful in providing a mental representation of business situation. Most commonly used statistical techniques can be implemented in a spreadsheet as simple as MS EXCEL.

There are some very important techniques to business intelligence analysis. Most importantly defining the objective and performance indicators, these are metrics that are used to estimate performance of an object (entity). The next is developing mathematical relationships between variables and metrics through finding patterns. The last is What-If analysis is by determining variations in the output metric by changing the input variables.

The advantage of using mathematical models is beyond increasing performance and ROI. It helps knowledge workers in deeper analysis of the business and underlying product/domain. This will increase awareness in the company, knowledge transfer within the company,  and higher desire to learn better things. It encourages intellectual thinking within the company and promote people with good analytical skills who can offer great value to the company.\

There are many techniques like regression and classification, which are some of the popular mathematical models,however predictive analytics are not limited to these methods.

Regression:      Linear Regression, kNN, CART, Neural Net

Classification: Logistic Regression, Bayesian Methods, Discriminant Analysis, Neural Net, kNN, CART.

There are some limitations and advantages of each of the methods. The right model and right mathematical technique to be choosen for each problem. The underlining business value that needs to be increased with each of the techniques.

Best techniques are formulates after and testing and evaluating each approach and measuring the impact of success or performance.  All mathematical models use simple statistical techniques, however the value is in mapping the business problem into a mathematical problem, this requires some intellectual talent.

Models developed can be extremely useful in business critical process like sales, marketing and product.

More on such topics will be published on http://www.praveenkodur.com/blog/

Focus for many companies such as Banks, Insurance companies, Telecommunication companies is to track customer end to end, from the time money spent on the customer acquisition to the time revenue is generated from the consumer for product/service until the customer attrites from the firm. Every activity in the chain must be closely observed to evaluate the true value of customer.  Companies worldwide are creating process to deal with customers  individually.  This can help them devout more attention to customers who are more valuable to the business and let go customers not valuable to business.

Data Mining requires a lot of effort/techniques and focus to centre their business around the customer than a product. Companies have to constantly keep a watch on what their customers are doing, keep in mind their past actions, discover knowledge from their actions (gain knowledge)  and finally use the knowledge cleverly to make decisions to make profit.

However data mining is not always beneficial for the user, consider the fact a model recommends a service to a user instead of the product A. however if the business makes more profit on product and very negligible amount on service. Model recommendations may not be implemented. However it helps in understanding such customers.

Good question to ask is how can a consumer company with large base of customers can individually deal with each consumer. This can be accomplished intelligently by deploying effective technology solutions which is customisable based on data mining models and techniques. Nowadays the customer is the data entry operator who enters data into the system at various points and these are captured by the system.  Consider your bank, your touch points are ATM, Bank Branch, customer care who responds on phone, mail, hard mail & lastly your account /loan/credit card that you hold of the bank. Each transaction records your behaviour which can be an additional knowledge that bank is made aware of, this knowledge can be used by the bank to learn more about you and customise the next interaction or next touch point instance with you.

The transaction system records every instance of transaction of the user enabling the bank to analyse the nature of every transaction and update your profile with rich insights. The knowledge discovery doesnt end here. It needs support of data warehousing system along with extremely good data mining models to be able to take action / make decisions and deal with each customer. More on this will be written in more detail in coming articles.

The blog  http://www.praveenkodur.com/blog/ will be updated with more such analytics, online marketing articles.

Some of the common uses of analytical models are:

Acquiring new customers: Acquiring new customers is generally a big cost for the company, with increase in prices on various acquisition channels, companies find it hard to reduce cost in acquiring good profitable customers. Predictive modeling to certain extent becomes very useful in reducing costs in acquiring right customers and increase profitability. It also helps in designing marketing offers, special campaigns for customers to reach out to them.

Predictive models use historical data of customer attributes to understand relationships between attributes and their specific response or behavior. The output of predictive model is generally to predict a future response of the customer with their present data. These models are generally used to rank a list of prospective customers on the likelihood of their predicted response. This is very useful as it helps take present decisions. Other factors can also be fit into the response such as risk of acquiring customers like credit risk, or the cost of retaining customer. We can also predict response to each specific product, which helps in better targeting the customer and increases chances of acquiring them or even for cross selling products

(http://www.stochasticsolutions.com/pdf/CrossSell.pdf).

Additionally, there might be situations where it might not be easy to connect both customer and the response desired. Additionally there may be situations where various responses of customers are also valuable. For example, a customer might purchase a product after visiting the site 4 times, each time the customer performs a action like search, sending an enquiry, contacting user, reading knowledge material etc. Each action may be valuable to business therefore need to included in the model. In all such cases we need to use proxies to understand purchase behavior. Model building is a tedious task, but very worthy effort in increasing profitability.

In process there are additional outputs generated like customer profiling, customer segmentation, clustering and affinity pairs which is very valuable in developing products for customers.

http://en.wikipedia.org/wiki/Predictive_analytics

Customers Retention: The second problem is of retaining existing customers, companies are willing to provide offers to retain customers. The data mining problem will translate into finding the customers at risk (or customers who are looking to switch) and additionally identify those customers who are more likely to change behaviour due the marketing offer given by the company. (http://www.stochasticsolutions.com/pdf/FinanceRetention.pdf).

The data useful will be behavior of customers before attrition for certain time duration along with their demographics, attitudinal & behavioral patterns.

Predicting customer attrition rate is separate from predicting their behavioral change because of promotional offer. These models are extremely useful where markets are saturated and acquiring new customers becomes increasingly difficult.

Marketing techniques for retaining customer could backfire sometimes, resulting in loss of customer due to persuation from the company. There are also certain customers who would attrite irrespective of any marketing offered to them. Capturing this behavioral difference can be done through Control Groups, Test Groups & Hold Groups. This method is called as Differential Response modeling or Incremental impact model or Uplift model or Net model. Here is more info on the same: http://en.wikipedia.org/wiki/Uplift_modelling, and few white papers on the same http://www.stochasticsolutions.com/pdf/SavedAndDrivenAway.pdf. Customers in Control Group are randomly targeted, Customers in Test Groups are targeted based on model, Hold Group Customers are not considered for targeting of offer. Results of test and experiment is used in building the Net Model.

The industries where these techniques will be highly useful are Financial Services, Retail, Telecom, Internet Companies and Software Houses.

Do check back for more articles on this topic.

List of resources to find more information about Analytics

1) http://www.destinationcrm.com/Articles/CRM-News/Daily-News/Predictive-Analytics-Can-Pinpoint-Profitable-Customers-52164.aspx

2) http://scientificmarketer.com/search/label/response

3) http://www.redclaymedia.com/response_modeling.php

4) http://stochasticsolutions.com/retention.html

5) http://www.information-management.com/specialreports/2008_62/10000747-1.html?ET=dmreview:e323:1015879a:&st=email

6) http://www.information-management.com/issues/2007_52/10001990-1.html

7) http://www.predictiveanalyticsinsight.com/articles/callcenter.htm

http://www.predictiveanalyticsworld.com/predictive_analytics.php

9) http://www.marketingprofs.com/4/shearer1.asp

10) http://semphonic.blogs.com/semangel/2009/01/predictive-analytics-getting-a-legup-on-where-analytics-is-headed-.html

The Original article is present on http://www.praveenkodur.com/blog

1) When the dependent variable contains missing values, simply eliminate the records.

2) Correctly Identify slices of data and Substitute with measure of central tendency like Median, Mean & Mode. Identifying the right slice is also important. You can group by various parameters and take a central tendency. Choose the one with highest bias (chi-square)

3) If the missing value forms a Normal distribution pattern, find the missing value by normal inverse function.

4) Treating the missing values as a dependent variable in a regression equation. Use the multiple linear regression function to impute the missing variable. You can try other methods instead of regression like classification, decision tree etc.

5) Use business logic to understand the missing values.

6) Check the data capturing process, there could a error present at source of data entry. Also it helps identify if the missing data points are at random or non-random. If it is random missing error then you can use simple imputations, however if it is non-random then you need advanced techniques to impute values. Also look at bias in the particular column, if the bias is significant then you need advanced techniques. If bias is minimal then you can proceed with simple imputation.

7) Identify the list of possible values for the missing data set. Try and replace each possible value and create different data sets and build the model. Calculate differences in accuracies and consistency based on different substitutes. This way you can even add variation of the values into the missing element and remove bias.

Use regression to determine the distribution of the values in place of missing values. Create a What-If scenario by imputing every range of value.

9) Do nothing remove missing values and duplicate records of sample data set to increase the size of the data set.

10) Measure similarlity of records like vectors. The similarity is the cosine function between records, and find similar records to the missing data values.

11) Use logistic regression to measure likelihood of observed or likelihood of missing. If value missing the output is 0, else 1. The rest of the variables (non-missing) act as independent variables. This does not predict anything but only a likelihood of finding the variable missing. Records with same probability or closest probability is considered similar and missing data is donated.

Multiple imputation generally yields better results but it requires high-end statistical software for computation. It becomes necessary to use the help of statistical software.

This article is originally found on Praveen Kodur http://www.praveenkodur.com/blog/.