Reference

Reference


http://info.webcrawler.com/mailing-lists/robots/info.html
http://info.webcrawler.com/mak/mak.html


Gray, M. "Growth of the World-Wide Web," Dec. 1993.

<URL: http://www.mit.edu:8001/aft/sipb/user/mkgray/ht/web-growth.html


 Berners-Lee, T., D. Conolly at al., "HyperText Markup Language
Spacification 2.0". Work in progress of the HTML working group of the
IETF. URL:
ftp://nic.merit.edu/documents/internet-drafts/draft-ietf-html-spec-00.txt

 Berners-Lee, T., R. Fielding, F. Nielsen. "HyperText Transfer
Protocol". Work in progress of the HTTP working group of the
IETF. URL:
ftp://nic.merit.edu/documents/internet-drafts/draft-fielding-http-spec-00.txt

 Spero, S. "Analysis of HTTP Performance problems" July 1994 
URL: http://sunsite.unc.edu/mdma-release/http-prob.html

 Spero, S. "Progress on HTTP-NG".
URL: http://info.cern.ch/hypertext/www/Protocols/HTTP-NG/http-ng-status.html

 Koster, M., "A Standard for Robot Exclusion,"
URL: http://info.webcrawler.com/mak/projects/robots/exclusion.html

 Koster, Martijn. "Guidelines for Robot Writers". 1993. 
URL: http://info.webcrawler.com/mak/projects/robots/guidelines.html

 Koster, Martijn. "WWW Robots, Wanderers and Spiders".
URL: http://info.webcrawler.com/mak/projects/robots/robots.html

 Bowman, Mic, Peter B. Danzig, Darren R. Hardy, Udi Manber and
Michael F. Schwartz. "Harvest: Scalable, Customizable Discovery and
Access System". Technical Report CU-CS-732-94, Department of Computer
Science, University of Colorado, Boulder, July 1994. 
URL: http://harvest.cs.colorado.edu/