The WebAnts(tm) Approach

Given the issues that complicate information discovery on the internet, the WebAnts project is working to develop a cooperative explorer; that is, an explorer (or ant) which will share results with other explorers and not duplicate their efforts. This model could be used for either searching or indexing; at the moment, we are concentrating on searching, until we have settled on a design for the index.

For searching, we expect this cooperation to be very effective, in that the different ants may be directed based on others' results. If one ant finds a document that satisfies the search criteria, it can share the references from that document with any other ants that are not currently exploring hits of their own. And as each ant explores a document, it lets other ants know, so that they do not have to examine the same document. In this way, the information can be gathered more effectively, minimizing wasted time on the parts of the various ants. When a particular corner of the web stops satisfying the criteria, an ant returns to a more general search to find other sources of information. (Note the parallels between this and the manner in which biological ants leave chemical trails to sources of food and cooperate in the harvesting; hence the metaphor).

For indexing, this cooperation allows each indexer to conserve resources, and also to distribute the indexing load between different explorers. Each index server would provide all the information gathered by one of the ants during exploration. When querying, a user could restrict the query to the local ant or allow it to propagate to the entire colony. Admittedly, this does not solve the fan-in, fan-out problem at the macro level, but it does serve to reduce the bottleneck effect and to better distribute the net load. This is not unlike the idea underlying the Übernet/Alibi project.

One issue that the WebAnts project does not yet entirely address, but which any indexer must eventually deal with, is that the size of an index will grow proportionally to the size of the web. The storage, retrieval, and distribution of information on the scale will no doubt prove a compelling challenge in coming years.

Back up to the WebAnts home page

Last updated 06-Feb-95 by John Leavitt (