WebAnts(tm) Motivating Issues

There are several motivating issues behind the WebAnts project.

The first is that information discovery on the web (including gopher-space and ftp-space) is currently too large a task. There are somewhere between 5000 and 10,000 web servers at this point, a similar number (or more) of gopher servers, and at least an order of magnitude more ftp servers. And all these numbers are increasing. Add to this the growing number of users on the web, and the fan-in and fan-out quickly become extraordinary. However, even with only a single explorer this scale is a problem, as the local portion of the network (as well as any topological bottlenecks) will sustain considerable traffic during exploration. Furthermore, such an exploration (if intended to be comprehensive) will necessarily take a long time (progressively longer as the net grows).

The second motivating issue is that it is undesirable to rely on a single site for such services, as this would create a bottleneck. Creating a single explorer site does not solve the fan-in, fan-out nature of the information discovery problem, but rather exacerbates it, making that one site a bottleneck for all users.

The final issue is that it is undesirable for multiple explorers to examine the same sites. If exploring the web alone is a problem, having a number of non-cooperating, and therefore redundant, explorers is worse. Not only does it cause unnecessary load on the servers, but it fails to provide a reasonable service to the user. This is, in fact, almost what we have now. To search for information on a general topic, a user cannot rely on a single search engine, since it does not explore everything. Neither can a user merely combine the results of several search engines together, since this inevitably yields repeated hits.

Back up to the WebAnts home page

Last updated 06-Feb-95 by John Leavitt (jrrl@cmu.edu)