[You are receiving this because you have subscribed to this free newsletter as
lightheartedlibrarians@.... If you would like to unsubscribe,
just go to BatesInfo.com/subscribe.html or email me.]


* * * Clustering On Demand * * *

I was recently doing some research for a client on the topic of social capital
(see, for example, Robert Putnam's book, Bowling Alone). It's a difficult topic
to search and, of course, I retrieved kajillions of results from several search
engines. I went through as many of them as I had the patience for, and I tried a
number of refinements to further focus my search. But I found it difficult to
find what I wanted in the major search engines.

Then I remembered hearing about Carrot2 (http://demo.carrot2.org), an open
source search-results-clustering engine, just recently out in beta. In a
nutshell, it takes search results, analyzes them and, on the fly, creates groups
of the most common concepts or terms from those results. Since this is all done
by algorithms rather than by humans, expect the odd result every once in a
while, but I found the clusters to be consistently useful.

Carrot2's default is to search the web using eTools.ch, a Swiss meta-search
engine that queries 10 search engines, including Google, Yahoo, Ask and MSN.
However, since eTools only returns the top 20 results from each search engine, I
prefer not to use eTool search results. Instead, you can click a tab to limit
your search to Google, Yahoo, MSN, Wikipedia, PubMed and a few other finding
tools. Because clustering is a computationally intensive process, Carrot2 limits
the search results by default to the top 100 results from any of the search
engines. However, you can click the Show Options link and set Carrot2 to search
and sort up to 400 results. (Note that increasing the number of search results
also increases the number of results from each search engine when using the
eTools meta-search engine from 20 to 40.)

Geek that I am, I find it even more intriguing that, under that "Show Options"
link is a pull-down menu that lets you select which of six different sorting
algorithms you want to use. The clustering results are dramatically different
(although keep in mind that the search results themselves stay the same -- only
the clusters change). With my "social capital" search, I was able to see a
variety of groupings of my search results, and identify some of the key writers
and terms.

Carrot2 may not be your day-to-day search tool, but it is tremendously useful
for those searches in which it is difficult to sift the wheat from the chaff.

****************************

"Can I publish or reproduce this InfoTip?" Be my guest. Just make sure you
credit the source, Bates Information Services, and include the URL,
www.BatesInfo.com/tip.html.

In addition to InfoTips, I've got a personal blog, Librarian of Fortune
(www.LibrarianOfFortune.com)

A version of this InfoTip with live links is available at
www.batesinfo.com/tip.html An RSS feed for my InfoTip is at
www.batesinfo.com/tip.rss

If you want to see where I will be speaking next, check out
www.BatesInfo.com/new.html

Do you need value-added research or training services?

Contact me at:
Mary Ellen Bates
Bates Information Services Inc.
+1 303.772.7095
mailto:mbates@...
www.BatesInfo.com