![\"Mahout-logo\"](\"https:\/\/trifork.nl\/articles\/wp-content\/uploads\/sites\/3\/2014\/01\/mahout-logo-400.png\"){kind=logo}
This blog features classification in Mahout and the underlying concepts. I will explain the basic classification process, training a Logistic Regression model with Stochastic Gradient Descent and a give walkthrough of classifying the Iris flower dataset with Mahout.<\/p>\n
<\/p>\n
One of my previous blogs focused on text clustering in Mahout<\/a>. Clustering is an example of unsupervised learning. The clustering algorithm finds groups within the data without being told what to look for upfront. This contrasts with classification, an example of supervised machine learning, which is the process of determining to which class an observation belongs. A common application of classification is spam filtering. With spam filtering we use labeled data to train the classifier: e-mails marked as spam or ham. We then can test the classifier to see whether it has done a good job of detecting spam from e-mail messages it hasn’t seen during the training phase.<\/p>\n Classification is a deep and broad subject with many different algorithms and optimizations. The basic process however remains the same:<\/p>\n Before we dive into Mahout let’s look at how Logistic Regression and Stochastic Gradient Descent work. This is very short and superficial introduction to this topic but I hope it gives enough of an idea how the algorithms work in order to follow the example later on. I have included links here and there to Wikipedia and videos of the Coursera Machine Learning course for more information.<\/p>\n Before I discuss Logistic Regression and SGD let’s look it’s foundation, the logistic function. The logistic function<\/a> is an S-shaped function whose range lies between 0 and 1, which makes it useful to model probabilities. When used in classification, an output close to 1 can indicate that an item belongs to a certain class. See the formula and graph below.<\/p>\n $latex \\frac{1}{1+e^{-x}}&s=3$ <\/p>\n Logistic function<\/i><\/p>\n Logistic Regression builds upon the logistic function. In contrast to the logistic function above which has a single x value as input, a Logistic Regression model allows many input variables: a vector of variables. Additionally, it consists of weights or coefficients for each input variable. The resulting Logistic Regression model looks like this:<\/p>\n $latex \\frac{1}{1+e^{-(\\beta_0 + \\beta_{1} x_{1} + \\beta_{2} x_{2} + … + \\beta_{n} x_{n})}}&s=3$ <\/p>\n Logistic regression model<\/i><\/p>\n The goal now is to find the values for $$\\beta$$s, the regression coefficients, in such a way that the model can classify data with high accuracy. The classifier is accurate if the difference between observed and actual probabilities is low. This difference is also called the cost. By minimizing the cost function of the Logistic Regression model we can learn the values of the $$\\beta$$ coefficients. See the following Coursera video on minimizing the cost function<\/a>.<\/p>\n The minimum of the cost function of Logistic Regression cannot be calculated directly, so we try to minimize it via Stochastic Gradient Descent<\/a>, also known as Online Gradient Descent. In this process we descend along the cost function towards its minimum for each training observation we encounter. As a result, the $$\\beta$$ coefficients are updated at every step and eventually as we keep taking steps closer to the minimum the cost is reduced and our model improves.<\/p>\n Now that you have a general idea about Logistic Regression and Stochastic Gradient Descent let’s look at an example. The Mahout source comes with a great example to demonstrate the classification process described above. The unit test The code follows most of the steps of the classification process described above. To follow along make sure you have checked out the Mahout source code<\/a>. Open See the code snippet below. The first part of the code concerns with setting up a few data structures and load the test and training sets. We also create two separate In the Now that all the proper data structures are in place let’s train the Logistic Regression model. The The basic classification process<\/h2>\n
\n
Logistic Regression & Stochastic Gradient Descent<\/h2>\n
The Logistic function<\/h3>\n
![\"Logistic](\"https:\/\/trifork.nl\/articles\/wp-content\/uploads\/sites\/3\/2014\/01\/logistic-curve.png\")
<\/a>
\nGraph of the logistic function<\/i><\/p>\nLogistic Regression model<\/h3>\n
Stochastic Gradient Descent<\/h3>\n
Classifying the Iris flower dataset<\/h2>\n
\n![\"Iris](\"https:\/\/trifork.nl\/articles\/wp-content\/uploads\/sites\/3\/2014\/01\/iris_setosa.jpg\")
<\/a>![\"Iris](\"https:\/\/trifork.nl\/articles\/wp-content\/uploads\/sites\/3\/2014\/01\/Iris_versicolor.jpg\")
<\/a>![\"Iris](\"https:\/\/trifork.nl\/articles\/wp-content\/uploads\/sites\/3\/2014\/01\/iris_virginica.jpg\")
<\/a>OnlineLogisticRegressionTest<\/code> contains a test case for classifying the well-known Iris flower dataset<\/a>. This is a small dataset from 1936 of 150 flowers with 3 different species: Setosa<\/i>, Versicolor<\/i> and Virginica<\/i> and the width and length of their sepals and petals. The dataset is used as a benchmark for testing classification and clustering algorithms.<\/p>\nOnlineLogisticRegressionTest<\/code> and look at the iris()<\/code> test case. In the unit test a classifier is trained which can classify the flowers’ species based on dimensions of the sepals and petals. In the next sections I show how Mahout can be used to train and test the classifier and finally assert its accuracy.<\/p>\n<\/div>\nSetup and parsing the dataset<\/h3>\n
List<\/code>s, data<\/code> and target<\/code>, for the features of the dataset and the classes we want to predict. Note that the type of the target<\/code> List<\/code> is Integer<\/code> because the classes of species will be encoded to Integer<\/code>s via the dictionary<\/code> based on the order they are processed in the dataset. More on that later. Another List<\/code> called order<\/code> is created to shuffle the contents of the dataset.<\/p>\nfor<\/code> loop on line 24 we iterate over the lines from the CSV file. In lines 29 to 36 each line is splitted on comma’s into separate fields and all fields put into a vector, which is added to data<\/code>, the list of vectors. On line 30, the first position of the vector, which corresponds to $$\\beta_{0}$$ is set to 1. $$\\beta_{0}$$ is also known as the intercept term. Look at the regression graph on the following link to see why we need the intercept<\/a>. The target variable is the 5th position in the CSV file, hence we use Iterables.get(values, 4)<\/code> on line 39 to obtain it and add it to target<\/code>. The other values of the data<\/code> vector are the remaining variables.<\/p>\n\n @Test\n public void iris() throws IOException {\n\n \/\/ Snip ...\n\n RandomUtils.useTestSeed();\n Splitter onComma = Splitter.on(",");\n\n \/\/ read the data\n List raw = Resources.readLines(Resources.getResource("iris.csv"), Charsets.UTF_8);\n\n \/\/ holds features\n List data = Lists.newArrayList();\n\n \/\/ holds target variable\n List target = Lists.newArrayList();\n\n \/\/ for decoding target values\n Dictionary dict = new Dictionary();\n\n \/\/ for permuting data later\n List order = Lists.newArrayList();\n\n for (String line : raw.subList(1, raw.size())) {\n \/\/ order gets a list of indexes\n order.add(order.size());\n\n \/\/ parse the predictor variables\n Vector v = new DenseVector(5);\n v.set(0, 1);\n int i = 1;\n Iterable values = onComma.split(line);\n for (String value : Iterables.limit(values, 4)) {\n v.set(i++, Double.parseDouble(value));\n }\n data.add(v);\n\n \/\/ and the target\n target.add(dict.intern(Iterables.get(values, 4)));\n }\n\n \/\/ randomize the order ... original data has each species all together\n \/\/ note that this randomization is deterministic\n Random random = RandomUtils.getRandom();\n Collections.shuffle(order, random);\n\n \/\/ select training and test data\n List train = order.subList(0, 100);\n List test = order.subList(100, 150);\n logger.warn("Training set = {}", train);\n logger.warn("Test set = {}", test);\n<\/pre>\nTraining the Logistic Regression model<\/h3>\n
iris<\/code> test will perform 200 runs. This means that it creates 200 instances of the LR algorithm. Also it will do 30 passes through the training set for each run to improve accuracy of the classifier. Mahout’s Logistic Regression code is based on the pseudocode in the appendix of Bob Carpenter’s paper on Stochastic Gradient Descent<\/a>.<\/p>\n