Options for making MIT's web pages searchable

What's needed is a facility for searching MIT web pages. Being able to type "kerberos" and find pages that mention it is much easier than wandering around the web.

Searching is a much-needed service for users of MIT's CWIS. It's important to look at options for providing this service and choose one that will best meet the need.

There are hundreds of possible solutions, but they fall into four basic categories.

Point users at search services outside MIT

The simplest solution from a development standpoint is to do nothing. Tell people to use InfoSeek or any of the multitude of searching services out there. If they want to find MIT's reengineering pages, they can search for "MIT reengineering". If they want to find the CP's pages, they can search for "MIT police".

Advantage: requires no development effort.

Disadvantage: adds dependence outside MIT.

Use a robot like webcrawler

There's a World Wide Web Robots, Wanderers, and Spiders site in England with a long list of robots.

Advantage: there are plenty to choose from.

Disadvantage: possible detriment to server performance.

The detriment to server performance results from the fact that normal operation of the web server depends on use of the AFS cache. Many clients tend to request the same pages, which web.mit.edu retrieves from local disk rather than from an AFS server. Robots tend to flush the cache by asking for all pages in the tree, one after another.

Use a local-disk indexer like WWWWAIS

There's a page on yahoo that lists many of these. They too have their advantages and disadvantages.

Advantage:

• faster, more efficient than web robots
Disadvantages:
• require that you
  • index all files under one directory (e.g. /afs/athena/)
  • OR enumerate all filenames to index
  • then map filenames back to URL's
• no mechanism for combining indexes from multiple servers, which is important if we want to cooperate with other MIT servers, e.g. Mech E.

Use harvest

In looking at the previous two options, it's clear that we want a product that will walk pages from the MIT home page, not traverse all files and directories in MIT's AFS cells. Yet we want it to get pages out of AFS directly, not go through the server. The only product that can do this is Harvest.

Who wrote Harvest? See the press release.

The central parts of the Harvest system are the gatherer, which compiles a summary of information in web pages, and the broker, which collects that information from the gatherer and builds a searchable index. One uses the broker when doing a search. Read Distributing the Gathering and Brokering Processes to learn how Harvest can be configured to gather either across the network or from local disk.

There's a Technical Discussion of the Harvest System that explains other design decisions in Harvest that make it scalable. For example, there's a system for efficiently replicating indexing information, so that searches aren't bottlenecked by one server.

You can try my experimental broker through this query page.