{"id":3225,"date":"2011-06-21T12:27:40","date_gmt":"2011-06-21T10:27:40","guid":{"rendered":"http:\/\/blog.jteam.nl\/?p=3225"},"modified":"2011-06-21T12:27:40","modified_gmt":"2011-06-21T10:27:40","slug":"running-mahout-in-the-cloud-using-apache-whirr","status":"publish","type":"post","link":"https:\/\/trifork.nl\/blog\/running-mahout-in-the-cloud-using-apache-whirr\/","title":{"rendered":"Running Mahout in the Cloud using Apache Whirr"},"content":{"rendered":"

This blog shows you how to run Mahout in the cloud, using Apache Whirr<\/a>. Apache Whirr is a promosing Apache incubator project for quickly launching cloud instances, from Hadoop to Cassandra, Hbase, Zookeeper and so on. I will show you how to setup a Hadoop cluster and run Mahout jobs both via the command line and Whirr’s Java API (version 0.4).<\/p>\n

<\/p>\n

What is Whirr?<\/h2>\n

Apache Whirr is a set of libraries and tools that allow you to run different services in the cloud. It currently has support for Amazon EC2 and Rackspace Cloud Servers. Services it currently supports are: Cassandra, Hadoop, Zookeeper, HBase, ElasticSearch and Voldemort.<\/p>\n

With whirr you can easily start these cloud services via the command line or a Java API. For example, you specify the following property file:<\/p>\n

&lt;br&gt;\nwhirr.service-name=hadoop&lt;br&gt;\nwhirr.cluster-name=test-cluster&lt;br&gt;\nwhirr.instance-templates=1 hadoop-jobtracker+hadoop-namenode, 2 hadoop-datanode+hadoop-tasktracker&lt;br&gt;\nwhirr.provider=aws-ec2&lt;br&gt;\nwhirr.location-id=eu-west-1&lt;br&gt;\nwhirr.hardware-id=m1.small&lt;br&gt;\nwhirr.image-id=eu-west-1\/ami-1b9fa86f&lt;br&gt;\nwhirr.identity=${env.AWS_ACCESS_KEY}&lt;br&gt;\nwhirr.credential=${env.AWS_SECRET_KEY}&lt;br&gt;\nwhirr.private-key-file=\/home\/frank\/.ssh\/id_rsa_whirr&lt;br&gt;\nwhirr.public-key-file=\/home\/frank\/.ssh\/id_rsa_whirr.pub&lt;br&gt;\nwhirr.cluster-user=frank&lt;br&gt;\n<\/pre>\n
and then run the following to start a cluster.<\/p>\n
&lt;br&gt;\n$ whirr launch-cluster --config [path-to-property-file]&lt;br&gt;\n<\/pre>\n
When the cluster is started you have to run the proxy script in ~\/.whirr\/[clustername]\/hadoop-proxy.sh<\/code> and then you can SSH into the cluster.<\/p>\n
Why use Whirr with Mahout<\/h2>\n
Below are some reasons for using Whirr with Mahout specifically. The first is the principle of ‘convention over configuration’. This means that you specify what kind of cluster you want, not the specifics of how to create it. When using Mahout you mostly want to run Hadoop jobs, so you want to start a Hadoop cluster, which is supported out-of-the-box by Whirr.<\/p>\n
The second reason is that Whirr enables transparant job submission. It generates a hadoop-site.xml<\/code> in ~\/.whirr\/<\/code> on your local machine at startup. By pointing HADOOP_CONF_DIR<\/code> to this directory you can transparently launch jobs from your local machine to the cluster. It’s almost as if your local machine is the cluster. This transparency is of course a Hadoop feature, but Whirr enables this automatically. Compare this with running Mahout on Amazon’s Elastic Map Reduce where you have manual steps like uploading the Mahout jar and referencing the jar’s location in any subsequent command line parameters. With Whirr you can run Mahout jobs from Java without a separate API for launching jobs on Amazon. More on running Mahout jobs from Java later on.<\/p>\n
Getting started with Whirr<\/h2>\n
Step 1 – Prequisites<\/h3>\n
To get started you first need a Amazon Web Services credentials and install Whirr. Check out Whirr founder Tom White’s Whirr in 5 minutes<\/a> to see how to get everything up and running.<\/p>\n
Step 2 – Overriding Hadoop properties<\/h3>\n
Whirr allows you to override Hadoop properties<\/a>. For instance, common changes are increasing heap space and ulimit for tasks. To specify this add the following to Whirr’s property file:<\/p>\n
&lt;br&gt;\nhadoop-mapreduce.mapred.child.java.opts=-Xmx1000m&lt;br&gt;\nhadoop-mapreduce.mapred.child.ulimit=1500000&lt;br&gt;\n<\/pre>\n
The prefix hadoop-mapreduce<\/code> specifies that these properties will be written to the mapred-site.xml<\/code> file on all machines in the cluster. You can also use the hadoop-common<\/code> and hadoop-hdfs<\/code> prefix to specify properties belonging to core-site.xml<\/code> and hdfs-site.xml<\/code>, respectively. This additional configuration will not be added to the hadoop-site.xml<\/code> on your local machine, but that’s not a problem.<\/p>\n
Step 3 – Whirr’s Java API<\/h3>\n
You can also use Whirr’s Java API to start a cluster. The snippet below shows how to launch and destroy a cluster based on a given Whirr property file.<\/p>\n
&lt;br&gt;\nClusterSpec clusterSpec = new ClusterSpec(new PropertiesConfiguration(whirrConfigFile), false);&lt;br&gt;\nService service = new Service();&lt;\/p&gt;\n&lt;p&gt;Cluster cluster = service.launchCluster(clusterSpec);&lt;\/p&gt;\n&lt;p&gt;HadoopProxy proxy = new HadoopProxy(clusterSpec, cluster);&lt;br&gt;\nproxy.start();&lt;\/p&gt;\n&lt;p&gt;\/\/ Launch jobs&lt;\/p&gt;\n&lt;p&gt;proxy.stop();&lt;br&gt;\nservice.destroyCluster(clusterSpec);&lt;br&gt;\n<\/pre>\n
The constructor loads the specified Whirr property file, launches the cluster and the Hadoop proxy. The hadoop proxy is needed to be able to access the cluster and the JobTracker or NameNode UI at http:\/\/[jobtracker]:50030 and http:\/\/[namenode]:50070 respectively.<\/p>\n
Step 4 – Building the Mahout job jar<\/h3>\n
Before we can run jobs we need to build the Mahout job jar. A job jar is a Hadoop convention and is a jar that has a lib folder with job-specific dependencies. The Mahout job jar has Lucene as one of its dependencies for example. When you submit a job with Hadoop it will look for a jar on the classpath and submit it to the cluster. If your job jar is not found it might select a normal jar without<\/strong> the dependencies and you will get ClassNotFoundExceptions<\/code>. To build the Mahout example job jar, run the following commands:<\/p>\n
&lt;br&gt;\n$ svn co http:\/\/svn.apache.org\/repos\/asf\/mahout\/trunk mahout&lt;br&gt;\n$ cd mahout&lt;br&gt;\n$ mvn clean install -DskipTests=true&lt;br&gt;\n<\/pre>\n
If you want to run a mahout job from your IDE you need to put the mahout job jar<\/strong> on the classpath. In IntelliJ you can add the job jar via ‘Project Structure’ > ‘Dependencies’ > ‘Add Single-Entry Module Library’. If you don’t add the correct job jar to the classpath you get the following warning<\/p>\n
\n“No job jar file set.  User classes may not be found. See JobConf(Class) or JobConf#setJar(String)”<\/p><\/blockquote>\n
Step 6 – Uploading data<\/h3>\n
Use the snippet below to create a directory on HDFS and upload data to the cluster.<\/p>\n
&lt;br&gt;\n$ export HADOOP_CONF_DIR=~\/.whirr\/cluster-name&lt;br&gt;\n$ hadoop fs -mkdir input&lt;br&gt;\n$ hadoop fs -put seinfeld-scripts input&lt;br&gt;\n<\/pre>\n
Now you can upload some data to the cluster via the command line, for instance the Seinfeld dataset from one of my earlier blogs<\/a><\/p>\n
Step 7 – Loading Hadoop Configuration in Java<\/h3>\n
The next step is to load a Hadoop Configuration<\/code> object that points to your cluster. Use the following:<\/p>\n
&lt;br&gt;\nPropertiesConfiguration props = new PropertiesConfiguration(whirrConfigFile);&lt;br&gt;\nString clusterName = props.getString("whirr.cluster-name");&lt;\/p&gt;\n&lt;p&gt;Configuration configuration = new Configuration();&lt;br&gt;\nconfiguration.addResource(new Path(System.getProperty("user.home"), ".whirr\/" + clusterName + "\/" + "hadoop-site.xml"));&lt;br&gt;\n<\/pre>\n
This configuration object can now be passed into your Mahout jobs<\/p>\n
Step 5 – Run!<\/h3>\n
The cluster is running, the job jar is built, you can now run a job via your IDE or the command line. To run from Java you can use the ToolRunner to run Mahout’s Driver<\/code> classes. The arguments to ToolRunner<\/code> are the Configuration<\/code> object loaded with values from Whirr and the parameters of the job in a String[]<\/code><\/p>\n
&lt;br&gt;\nString[] seq2SparseParams = new String[] {&lt;br&gt;\n"--input", textOutputPath.toString(),&lt;br&gt;\n"--output", sparseOutputPath,&lt;br&gt;\n"--weight", "TFIDF",&lt;br&gt;\n"--norm", "2",&lt;br&gt;\n"--maxNGramSize", "2",&lt;br&gt;\n"--namedVector",&lt;br&gt;\n"--maxDFPercent", "50",&lt;br&gt;\n};&lt;\/p&gt;\n&lt;p&gt;ToolRunner.run(configuration, new SparseVectorsFromSequenceFiles(), seq2SparseParams);&lt;br&gt;\n<\/pre>\n
or you can run mahout from the command line. Enjoy!<\/p>\n
P.S. Don’t forget to shutdown your cluster \ud83d\ude09<\/p>\n
&lt;br&gt;\n$ whirr destroy-cluster --config [path-to-property-file]&lt;br&gt;\n<\/pre>\n","protected":false},"excerpt":{"rendered":"
This blog shows you how to run Mahout in the cloud, using Apache Whirr. Apache Whirr is a promosing Apache incubator project for quickly launching cloud instances, from Hadoop to Cassandra, Hbase, Zookeeper and so on. I will show you how to setup a Hadoop cluster and run Mahout jobs both via the command line […]<\/p>\n","protected":false},"author":6,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"content-type":"","footnotes":""},"categories":[40,15],"tags":[42,251,236,115,252,74,158,253],"class_list":["post-3225","post","type-post","status-publish","format-standard","hentry","category-mahout","category-enterprise-search","tag-apache","tag-apache-hadoop","tag-apache-mahout","tag-cloud","tag-cluster","tag-hadoop","tag-recommendations","tag-whirr"],"acf":[],"yoast_head":"\nRunning Mahout in the Cloud using Apache Whirr - Trifork Blog<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/trifork.nl\/blog\/running-mahout-in-the-cloud-using-apache-whirr\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Running Mahout in the Cloud using Apache Whirr - Trifork Blog\" \/>\n<meta property=\"og:description\" content=\"This blog shows you how to run Mahout in the cloud, using Apache Whirr. Apache Whirr is a promosing Apache incubator project for quickly launching cloud instances, from Hadoop to Cassandra, Hbase, Zookeeper and so on. I will show you how to setup a Hadoop cluster and run Mahout jobs both via the command line […]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/trifork.nl\/blog\/running-mahout-in-the-cloud-using-apache-whirr\/\" \/>\n<meta property=\"og:site_name\" content=\"Trifork Blog\" \/>\n<meta property=\"article:published_time\" content=\"2011-06-21T10:27:40+00:00\" \/>\n<meta name=\"author\" content=\"frank\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"frank\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"7 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\/\/trifork.nl\/blog\/running-mahout-in-the-cloud-using-apache-whirr\/\",\"url\":\"https:\/\/trifork.nl\/blog\/running-mahout-in-the-cloud-using-apache-whirr\/\",\"name\":\"Running Mahout in the Cloud using Apache Whirr - Trifork Blog\",\"isPartOf\":{\"@id\":\"https:\/\/trifork.nl\/blog\/#website\"},\"datePublished\":\"2011-06-21T10:27:40+00:00\",\"author\":{\"@id\":\"https:\/\/trifork.nl\/blog\/#\/schema\/person\/00fad6c5829f6770345f23ccace2e54f\"},\"breadcrumb\":{\"@id\":\"https:\/\/trifork.nl\/blog\/running-mahout-in-the-cloud-using-apache-whirr\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/trifork.nl\/blog\/running-mahout-in-the-cloud-using-apache-whirr\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/trifork.nl\/blog\/running-mahout-in-the-cloud-using-apache-whirr\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/trifork.nl\/blog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Running Mahout in the Cloud using Apache Whirr\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/trifork.nl\/blog\/#website\",\"url\":\"https:\/\/trifork.nl\/blog\/\",\"name\":\"Trifork Blog\",\"description\":\"Keep updated on the technical solutions Trifork is working on!\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/trifork.nl\/blog\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/trifork.nl\/blog\/#\/schema\/person\/00fad6c5829f6770345f23ccace2e54f\",\"name\":\"frank\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/trifork.nl\/blog\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/5c39a948f2b70fa900b25dc79cde8643?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/5c39a948f2b70fa900b25dc79cde8643?s=96&d=mm&r=g\",\"caption\":\"frank\"},\"url\":\"https:\/\/trifork.nl\/blog\/author\/frank\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Running Mahout in the Cloud using Apache Whirr - Trifork Blog","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/trifork.nl\/blog\/running-mahout-in-the-cloud-using-apache-whirr\/","og_locale":"en_US","og_type":"article","og_title":"Running Mahout in the Cloud using Apache Whirr - Trifork Blog","og_description":"This blog shows you how to run Mahout in the cloud, using Apache Whirr. Apache Whirr is a promosing Apache incubator project for quickly launching cloud instances, from Hadoop to Cassandra, Hbase, Zookeeper and so on. I will show you how to setup a Hadoop cluster and run Mahout jobs both via the command line […]","og_url":"https:\/\/trifork.nl\/blog\/running-mahout-in-the-cloud-using-apache-whirr\/","og_site_name":"Trifork Blog","article_published_time":"2011-06-21T10:27:40+00:00","author":"frank","twitter_card":"summary_large_image","twitter_misc":{"Written by":"frank","Est. reading time":"7 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/trifork.nl\/blog\/running-mahout-in-the-cloud-using-apache-whirr\/","url":"https:\/\/trifork.nl\/blog\/running-mahout-in-the-cloud-using-apache-whirr\/","name":"Running Mahout in the Cloud using Apache Whirr - Trifork Blog","isPartOf":{"@id":"https:\/\/trifork.nl\/blog\/#website"},"datePublished":"2011-06-21T10:27:40+00:00","author":{"@id":"https:\/\/trifork.nl\/blog\/#\/schema\/person\/00fad6c5829f6770345f23ccace2e54f"},"breadcrumb":{"@id":"https:\/\/trifork.nl\/blog\/running-mahout-in-the-cloud-using-apache-whirr\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/trifork.nl\/blog\/running-mahout-in-the-cloud-using-apache-whirr\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/trifork.nl\/blog\/running-mahout-in-the-cloud-using-apache-whirr\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/trifork.nl\/blog\/"},{"@type":"ListItem","position":2,"name":"Running Mahout in the Cloud using Apache Whirr"}]},{"@type":"WebSite","@id":"https:\/\/trifork.nl\/blog\/#website","url":"https:\/\/trifork.nl\/blog\/","name":"Trifork Blog","description":"Keep updated on the technical solutions Trifork is working on!","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/trifork.nl\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/trifork.nl\/blog\/#\/schema\/person\/00fad6c5829f6770345f23ccace2e54f","name":"frank","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/trifork.nl\/blog\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/5c39a948f2b70fa900b25dc79cde8643?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/5c39a948f2b70fa900b25dc79cde8643?s=96&d=mm&r=g","caption":"frank"},"url":"https:\/\/trifork.nl\/blog\/author\/frank\/"}]}},"_links":{"self":[{"href":"https:\/\/trifork.nl\/blog\/wp-json\/wp\/v2\/posts\/3225","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/trifork.nl\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/trifork.nl\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/trifork.nl\/blog\/wp-json\/wp\/v2\/users\/6"}],"replies":[{"embeddable":true,"href":"https:\/\/trifork.nl\/blog\/wp-json\/wp\/v2\/comments?post=3225"}],"version-history":[{"count":0,"href":"https:\/\/trifork.nl\/blog\/wp-json\/wp\/v2\/posts\/3225\/revisions"}],"wp:attachment":[{"href":"https:\/\/trifork.nl\/blog\/wp-json\/wp\/v2\/media?parent=3225"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/trifork.nl\/blog\/wp-json\/wp\/v2\/categories?post=3225"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/trifork.nl\/blog\/wp-json\/wp\/v2\/tags?post=3225"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}