{"id":476,"date":"2009-07-22T20:47:39","date_gmt":"2009-07-22T19:47:39","guid":{"rendered":"http:\/\/blog.jteam.nl\/?p=476"},"modified":"2009-07-22T20:47:39","modified_gmt":"2009-07-22T19:47:39","slug":"enterprise-search-introduction-to-solr","status":"publish","type":"post","link":"https:\/\/trifork.nl\/blog\/enterprise-search-introduction-to-solr\/","title":{"rendered":"Enterprise Search: Introduction to Solr"},"content":{"rendered":"

From day one, we at JTeam were very much occupied with pushing new revolutionary open source technologies that can bring real value to us and to our customers. We were there when Spring just started and we helped making it what it is today. We were one of the first companies to use Hibernate in real world projects (I reckon the first version we used was 0.4), and contributed to (back then) innovative new front end technologies like Ajax and DWR. With time, these technologies became mainstream and for a while it seemed that they just fulfilled every bit of our needs where JEE development is\u00a0 concerned. Yet something was still missing. About 3 years ago, we started noticing a new and growing trend in the market \u2013 a new demand \u2013 demand for search. Customers started paying more attention to the \u201cfindability\u201d aspect in their offerings, be it an e-commerce website offering faceted navigation to its users, or proprietary search solutions on top large service management systems. The trend was obvious, the demand was there, and we had to deliver. We started by implementing our own custom solutions based on the brilliant Lucene library, but then came Solr and once again revolutionized our JEE development.<\/p>\n

My goal in this post is to introduce you to Solr. Not too fancy, but to give you just a taste and enough information to at least get started with it. In future posts, I hope to expand on this and show you how you can leverage some of Solr’s features to implement some really cool stuff.
\n<\/p>\n

So what is Solr?<\/h2>\n

Solr is an apache project which can be described in many ways. Some like to see it as an open source enterprise search platform rivaling well established commercial offerings like Autonomy, Fast, and Dieselpoint. But it’s very hard to get a really good grasp of what Solr is and what it does with this definition, especially as the phrase \u201cEnterprise Search Platform\u201d has different meanings for different people. I like to view Solr as an attempt to gather all the experience people have had over the years developing search solutions with Lucene and build a search platform based on these experiences and all the known (and unknown) best practices. <\/em><\/p>\n

NOTE: For those who are not familiar with Lucene, it is a low level IR (Information Retrieval) library implemented in Java. With Lucene you can index textual data and execute free text search on it in a highly performant manner.<\/em><\/em><\/p>\n

So what’s the purpose of it all? Well, I guess it depends on who you’re asking and on the context in which search is applied. For example, if you’re a web developer looking to integrate search in a web site, Solr can be a perfect fit for you. It can be installed as a standalone server and exposes its search functionality via a REST-like API (which also makes it language independent so it doesn’t matter whether you develop in Java, PHP, .NET or any other preferred language\/platform). Also note that the core search functionality that is supported out of the box by Solr is probably more than enough to what most websites require.<\/p>\n

That’s all well, but Solr is by far not limited to website search. It is not for nothing that it is often compared with Fast, Dieselpoint or other enterprise search platforms. You can build quite large scale and complex search solutions with it and the list of high profile companies (CNet,\u00a0 AOL, Digg, and more) already using it to power their search requirements is a testimony for that.<\/p>\n

Getting Solr up and running<\/h2>\n

You can get Solr up and running in practically no time. You’ll first need to download it from the following site: http:\/\/www.apache.org\/dyn\/closer.cgi\/lucene\/solr.<\/p>\n

Once downloaded, you can extract the compressed archive (zip or gzip \u2013 depending one your platform) anywhere you want on your file system. I would now like you to pay attention to two folders in the extracted directory \u2013 dist<\/em> and example<\/em>.<\/p>\n

As I already mentioned, Solr is essentially a search server implemented in Java. It is actually, a standard web application that can be deployed in a normal servlet container such as Tomcat. In the dist<\/em> folder, you can find the war file for this server (along with several other jar files which serve for different purposes… well… I’ll cover them in a later post). But as you probably know, it’s always a bit of a hustle to work with war files \u2013 you first need to set up a servlet container, then you need to set up a few environment variables, then deploy the war, etc, etc, etc… Luckily, the developers of Solr acknowledged that and decided to make it even easier for you to get started with it, hence the example<\/em> folder. This folder contains a Solr distribution bundled with a Jetty server. Here is the layout of the example folder:<\/p>\n

\"Example
Example Folder Layout<\/figcaption><\/figure><\/p>\n

Now, all you need to do to start Solr server, is run the following command from this folder:<\/p>\n

\njava -jar start.jar\n<\/pre>\n
Congrats! You are now running a Solr server.<\/p>\n
Interacting with Solr<\/h2>\n
Now that you have Solr up and running, it’s time to do something with it. As a search service, the two main operations that Solr supports are indexing and searching. First, you need to send Solr data to index after which you’ll be able to perform free text search on this data. But what is this data and how does it look like?<\/p>\n
A world of Documents<\/h3>\n
In Java, we’re used to model the world in terms of objects and properties. In the IR world, the world is modeled as Documents and fields. A Document represents a unit of data and is made up of one or more fields. A field is a simple text based name-value pair which holds the actual data. For example, a web page can be represented as a document with 3 fields – URL, title, and body:<\/p>\n
\"webpage\"<\/p>\n
And here’s how you can model a person as a Document:<\/p>\n
\"person\"<\/p>\n
Indexing Documents<\/h3>\n
Now that you understand how data is represented in Solr, it is time to send Solr a few documents to be indexed. For that, we’ll used yet another useful tool that Solr ships with. In the example\/exampledocs<\/em> directory you will find several XML files and a post.jar<\/em> file. The latter, is a tool which can post document files to Solr to be indexed. If you open one of the XML files, you’ll see that each file actually holds an XML structure that represents an \u201cadd\u201d command. The post.jar<\/em> accepts a list of files as an argument and send these files to Solr using HTTP POST request to a dedicated \u201cupdate\u201d URL. Make sure that Solr is running, and execute the following command:<\/p>\n
\njava -jar post.jar *.xml\n<\/pre>\n
When executed, this command will send all documents in all XML files to Solr. After they’ve all been sent, a \u201ccommit\u201d command is sent which makes these documents available for search.<\/p>\n
NOTE: without committing the documents will still be indexed, but they will not be available for search until either a \u201ccommit\u201d is executed, or the Solr server is restarted. Luckily, the post.jar tool sends a \u201ccommit\u201d request automatically after sending the documents.<\/em><\/p>\n
Searching for documents<\/h3>\n
We managed to index some documents in Solr, all that is left now is to search for them. Just like Solr has a dedicated URL for indexing documents, it also has a dedicated URL for searching for documents (actually, there can be more than one such URL but I won’t get into that right now). By default, the search URL is:<\/p>\n
http:\/\/localhost:8983\/solr\/select<\/a><\/pre>\n
But (as you may already have tried and realized) opening this URL in the browser results in an exception. The reason for this is quite simply due to the fact that no search query was specified. To do that, just add a \u201cq\u201d request parameter which holds the free text you want to search, for example:<\/p>\n
http:\/\/localhost:8983\/solr\/select?q=*:*<\/a><\/pre>\n
NOTE: the \u201c*.*\u201d represents a \u201cmatch all\u201d query. When executing this query, it’s as if you’re asking Solr to return all the documents it has indexed.<\/em><\/p>\n
The result returned from Solr for this query is an XML document containing some meta data over the request and query execution (e.g. how long did the search take), and also a list of the matched documents (also referred to as \u201csearch hits\u201d). Notice, that it’s only a partial list of the documents \u2013 while the \u201cnumFound\u201d<\/em> attribute of the <result><\/em> element shows that 26 documents were found, only 10 are actually returned. There’s a very good reason for that \u2013 as Solr is designed to index millions of documents, it make very little sense to return such large search results at once. Therefore, Solr returns the results one \u201cpage\u201d at a time. By default the page size is 10 and if not specified otherwise the first page is returned (that is, the first 10 documents). You can control this behavior by providing 2 extra parameters \u2013 rows<\/em> (determine the page size) and start<\/em> (determines the zero-based index of the first document in the page):<\/p>\n
http:\/\/localhost:8983\/solr\/select?q=*:*&rows=5&start=1<\/a><\/pre>\n
Understanding and controlling the returned result<\/h3>\n
As you can see in the returned XML, each document in the result is returned with its fields. When configuring Solr, one can specify the schema of the index. I will not go into it right now, but in general, the schema determines what fields (name and type) a document is expected to have and also how should Solr handle them. Fields can be handled in 3 ways:<\/p>\n