Broken Wing

Testing a search engine

Since I am doing research in natural language processing, I am always interested (and skeptical) when a new product makes claims about its language understanding abilities. I became aware of the Brainboost search engine from Greet Machine, so I decided to test it out a little bit. The idea is that you enter a question and instead of getting links you get answers. Here are some results from some preliminary testing I did.

Query:What is the surface of Venus made of?
Result:
(1) Continents and lowlands - Only 8% of the surface of Venus is made of continental highlands
(2) Venus - - The surface of Venus is made of a thick crust of dried up volcano lava.
Analysis: (1) is tough to make sense out of, and is only a partial answer. (2), on the other hand is muy impressiv.

Query:Who is Scrooge McDuck's nemesis?
Result: Limited supplies ) - $6.00 Disney’s DuckTales #2 (July 1990) - "Scrooges Quest" Part 2 - "The Nemesis of Napalabama" Scrooges search for Webby takes our heroes to the Howling Hills of Pandamonia.
Analysis: Not very impressive. The correct answer, of course, is Flintheart Glumgold.

Query:Where is AAMAS being held this year?
Result:
(1)Sounds very interesting and Ill certainly be trying to submit a paper for this! (the fact AAMAS is held in Melbourne this year has nothing to do with it!) posted by Jamie at 5:56 PM << Back to blog.
(2) The award carries an honorarium, along with an invitation to give a talk at the annual Autonomous Agents and Multi - Agent System (AAMAS) Conference, which this year will be held at Utrecht University, the Netherlands July 25 - 29.
Analysis: This search makes it clear that the search technique is still using a "bag of words" approach instead of some kind of semantic analysis. Notice that the string "this year" appears in both results, and also that the two results give two different answers. "This year" could be resolved into the current year in a fully understanding system. Incidentally, if I changed the search to "Where is AAMAS 2005 being held," the search works perfectly.

Query:How do you make an old fashioned?
The result of this query was a polite request to limit queries to factual questions. I think the search engine got thrown off by the word "you," thinking I was trying to make smalltalk, so I gave it the benefit of the doubt and tried again.
Query (try 2):How is an old fashioned made?
Results
(1) The Old Fashioned is made by taking an orange wheel and a couple of maraschino cherries with a dash of sugar syrup and Angostura bitters and "muddling" them - basically squishing them all together in the bottle of the glass.
(2) For instance, an Old Fashioned is made by muddling an orange and a cherry with a dash of bitters then filling the glass with bourbon or whiskey and a splash of soda water. So that makes 4 non - alcoholic ingredients to 1 alcoholic ingredient, but I have yet to meed a chick who likes it.
Analysis: Result (1) seems to be incomplete - there is no mention of adding liquor or soda. Result (2) is a little better, plus it has that hilarious last line. I could probably figure out how to make it from answer (2), but I'll let #1 Dad be the judge of whether it's a good recipe or not.

So, with the factual questions out of the way, I moved on to more opinion and values-based questions:

Query:Why don't girls like me?
Answer: Nice Girls Don`t Get Drunk. - - Like a good girl I then lifted my dress over my backside and leant forward, ready to be screwed by my new black lover, who began to pull my panties down.
Analysis:I'm not quite sure what to make of this. I'm unsatisfied with the response yet strangely intrigued.

Query:Who is the best wide receiver in the NFL draft?
Answer:# Charles Rogers, WR Michigan State - Not only is Charles Rogers rated as the top NFL Draft prospect at wide receiver, many scouts feel he is the best player in the draft.
Analysis:Not only is this two year old information, but it completely neglects Chris Samp (WR - Winona State), who is clearly the cream of this year's class.

Query:If God is infinitely good, why do bad things happen?
Result: Just as human parents operate according to a higher wisdom than children do, so God operates according to an infinitely higher wisdom than we do.
Analysis: That really makes me feel a lot better. Posted by mill1991 at March 16, 2005 09:45 AM | TrackBack