Search Wikia gets slammed, here’s our review

Search Wikia search engine has received mixed reviews, to say the least, since launching this morning.

We’ve covered some of its business and social aspects and an initial look when it first launched, but have since had more time to tinker with it.

The harsh reviews resulted in Search Wikia putting up a temporary disclaimer at its home page, http://search.wikia.com, conceding that the quality of its search results is poor (screenshot below). This was not the case last night when the landing page took users straight to a search box.

To find the search box, you now go to http://alpha.search.wikia.com/.

We decided to try a very popular term and a very obscure term. The goal of the popular term test is to figure out whether the engine finds the most “obvious” sources and returns them in a reasonable order. Conversely, the obscure term is to figure out whether the search engine can actually scale to the vast universe of queries individual users are likely to try — like they would at major search engines like Google. Finally, thirdly, we also tried a newsworthy query to gauge the freshness of the content returned.

Query: Las Vegas

We’ll assume Las Vegas is among the more popular travel-based queries in the United States. As an end user, I expected (personal expectations are key) to see a diversity of sources that would allow me to read about the city’s relevant attractions, show me travel and lodging options, recent and/or relevant reviews of user trips (blog posts) and news items related to Las Vegas. Here are Wikia’s results on the query (screenshot below). This results page is an example of what most new search engines must battle; it is not enough to have the best algorithm, best user interface or even the most comprehensive index. A search engine must deal with spam results, porn results and foreign language filtering. Not having these features available can completely kill relevancy of results.

Three out of the ten results below are foreign language results and completely useless to most U.S users. The second result redirects to relocation information about Las Vegas and the site itself is not a major travel site like Y! Travel or TripAdvisor. Surprisingly enough, the Wikipedia page about Las Vegas is not among the top ten even though Wales confirmed to us that Search Wikia is “deep-crawling” Wikipedia and therefore likely to rank Wikipedia very high among search results.

A straight comparison to Google is unfair, so we won’t include screenshots, but click here to find Google’s take on “Las Vegas”.

Query: Apocrine Hidrocystoma of the eyelid

This is an obscure query but one that users are actually enter into search engines. It is queries like these that expose a search engine’s comprehensiveness or lack thereof and test its ability to serve an essentially infinite universe of queries.

Here are Wikia’s results on the query (screenshot below).

This shows that Search Wikia’s index needs to grow substantially – and fast – if they want to achieve web scale. By comparison, we also checked out results from GoogleYahoo! Search. Note that both search engines find the same first result from eMedicine. Google finds 640 results while Yahoo! Search tops out at 152 results. By comparison, a popular term like Las Vegas generates hundreds of thousands of results.

Query: Barack Obama

This is an interesting query because not only is it extremely popular but also very newsworthy. As an end user, I expect to see Obama’s “official” website, some popular news stories about him, references to the books he has written or even his MySpace page. Most importantly, I would want to see some mention of his recent surprise victory in Iowa (again, personal expections are absolutely critical).

Here are Wikia’s results on the query (screenshot below).

The results are better than the first two queries we tried. The first result is a blog site. This site is clearly not the most “obvious” site about Barack Obama as an end user. Web Analytics firm Compete.com reports that the site receives less than 1000 visits a month, making it insignificant from a traffic point of view. Google also reports that the number of sites linking to the first result is only 60 (links to a site may also be roughly thought of as a measure of popularity of the site).

By contrast, click here to find Google’s take on “Barack Obama”. The first result (http://www.barackobama.com, the official site) has 6230 other pages linking to it. Also check out Yahoo!’s take on “Barack Obama”.

Conclusion

We applaud Search Wikia’s attempt to revolutionize the search space by creating an open source, freely licensable search engine with room for humans to contribute in the search results. In our review, we found all the problems we expected to find at such a nascent stage – spam, pornography, foreign language results and lack of coverage are all problems that Search Wikia needs to take on aggressively to build a killer product.

Saumil Mehta is a contributing writer for VentureBeat. Disclosure: He is product manager at Kosmix RightHealth (http://www.righthealth.com), which is also a search engine company. The opinions expressed here are his own.