WebMon is an HTTP tracing and capture tool. It can be used to examine the HTTP dialogue between a web browser and an web server. It provides a variety of features & functions:
WebMon's most popular use is to provide the input files for the webclient tool.
webmon
does not try to match the connection behavior of the browser
in how it fetches pages. It will open distinct and unique sockets
for each web-page GIF fetch, instead of reusing an existing socket
like many web browsers. In particular, it does not emulate or
the HTTP "KeepAlive" protocol, nor does it use multiple parallel
connections to the web server. Thus, it cannot be used to diagnose
conditions. In most respects, webmon
appears to be a single-threaded
process to the web server.
In the default mode of operation, webmon
acts as a proxy server.
Therefore, you must configure your browser to use webmon
as a proxy,
as described in the previous chapter. Specific example usages are given
in the next section, below.
XXX this section needs more work ...
The following command-line options are defined:
Usage: webmon [options]
-a, --act-as-proxy behave as a proxy server
-A, --alarm=<time> turns on timeout alarms (delay 'time')
-d, --debug print basic debugging messages
-D, --Debug turns on verbose debugging
-E, --Debug-debug turns on extra verbose debugging
-e, --print-each print individual response time observations
-h, --help print this message
-n, --no-xlation disable host/url translation/rewriting
-p, --listen-port=<port> specifies the port to listen to
-P, --proxy=<proxy:port> specifies the proxyserver and port
-q, --debug-time turns on debugging output for timing statistics
-Q, --Debug-time turns on verbose timing debug output
-r, --report-file=<file> specifies the report file name
-t, --trace-file=<file> write HTTP traces to file
-U, --user-agent=<string> specify value of 'User-Agent:' in HTTP header
string must be enclosed in single quotes
-v, --request-file=<file> record browser requests in this file
-w, --webserver=<server:port> specifies the webserver and port
Flags without short-form equivalents:
--access-log=<file> write webserver-style access log
--no-bug-compat enable strict conformance to HTTP standards
--quiet minimize messages written to stdout
--show-progress write out each URL as its fetched
--trace-client write exchanges with client to trace file
--trace-server write exchanges with server to trace file
--version print version info and exit
SSL options:
--ssl-server use SSL to connect to server
-S, --server-cipher=NUM force use of a specific cipher
--server-timeout=NUM set SSL session cache timeout (seconds)
--ssl-browser use SSL to connect to browser
--browser-cipher=NUM force use of a specific cipher
--browser-timeout=NUM set SSL session cache timeout (seconds)
--certif=<filename> specify file containing certificates
-k, --keyring=<filename> specify file containing keys
-K, --password=<passwd> specify keyring password
-N, --distinguish=<name> specify a distinguished name
Cipher suites may be specified in the form SSLV2:N with N 1,2,3,..7
or SSLV3:NN for NN=00, 01, 02 ... 06 or 0A.If a cipher suite has been
specified, then no cipher suite negotiation is performed.
The Cipher-suite numbering convention follows that of the SSLv3 spec. Below is a partial listing; see the spec for a full listing.
SSLV2 Cipher suite names and N values:
SSL2_RC4_128_WITH_MD5 = 1
SSL2_RC4_128_EXPORT_40_WITH_MD5 = 2
SSL2_RC2_128_CBC_WITH_MD5 = 3
SSL2_RC2_128_CBC_EXPORT40_WITH_MD5 = 4
SSL2_IDEA_128_CBC_WITH_MD5 = 5
SSL2_DES_64_CBC_WITH_MD5 = 6
SSL2_DES_192_EDE3_CBC_WITH_MD5 = 7
The Cipher values +rc4export and +rc2export in the magnus.conf
file
for the webserver appear to correspond to cipher suite specifications
of 2 and 4, respectively. (Yes, they are backwards....)
SSLV3 Cipher suite names and NN values:
SSL_NULL_WITH_NULL_NULL = 0x0000
SSL_RSA_WITH_NULL_MD5 = 0x0001
SSL_RSA_WITH_NULL_SHA = 0x0002
SSL_RSA_EXPORT_WITH_RC4_40_MD5 = 0x0003
SSL_RSA_WITH_RC4_128_MD5 = 0x0004
SSL_RSA_WITH_RC4_128_SHA = 0x0005
SSL_RSA_EXPORT_WITH_RC2_CBC_40_MD5 = 0x0006
SSL_RSA_WITH_DES_CBC_SHA = 0x0009
SSL_RSA_WITH_3DES_EDE_CBC_SHA = 0x000A
OpenSSL supports a large number of additional ciphers not listed here.
Webmon can be used in one of three different ways (which may seem similar, but have important, fundamental differences). Lets term these the "proxy" mode, the "rewrite" mode, and the "passthru" mode.
For most users, it is suggested that the "proxy" mode be used
in normal operation, as it is the simplest to configure, understand
and use. In this mode, webmon
acts as an ordinary web proxy.
That is, it will listen for URL requests that have a server name
embedded in them, and will contact these servers on behalf of the
browser. Thus, a typical HTTP request line sent by the browser
to webmon
will look like
GET http://some.where.com/some/file.html HTTP/1.0
Upon receipt, webmon
will contact the host
some.where.com
on port 80 and will issue it the usual
HTTP request
GET /some/file.html HTTP/1.0
. It will then wait for
a response, and pass this response back to the browser, unmodified.
Essentially all proxies work in this way.
When webmon
is used in proxy mode to access secure (encrypted)
web sites, it does do something unusual. Instead of passing the
secure connection request to the secure server, it will instead present
it's own credentials, initiating a "man-in-the-middle" attack.
This allows webmon
to decrypt the traffic from the browser
and record it, i.e. to snoop on it. It does this by listening
for the CONNECT
HTTP directive, and instead of connecting,
it responds with its own certificate, as if it were the end host.
As long as the browser accepts the certificate, webmon
will
be able to snoop on the browsers traffic.
Note that webmon
is distributed with a working example
certificate in the examples
directory. This certificate
contains an export-grade (512-bit) private key and a signature
from a fictitious certification authority. This signing
authority will not be recognized as official by most browsers,
and thus, most browsers will put up warning dialogs when presented
with this certificate. If a signed certificate were obtained
from the usual recognized authorities, webmon
could be used
to snoop on encrypted traffic without the users knowledge
(provided the user had configured to use webmon
as a proxy).
To use webmon
in the "proxy" mode, specify the -a
and
-n
command line flags.
Another mode of operation for webmon
is the "rewrite" mode.
In this mode, the browser accesses webmon
as if it were the
true server, i.e. it uses webmon
's URL as the server URL.
As in all of the other modes this mode, webmon
will pass
the request on to the true server, and wait for a reply.
When it receives the reply, it will "rewrite" it to blank out
all embedded URL's, and convert them into local URL's. In this
way, it can keep the browser from ever "escaping" to a different
server: any link that the user clicks on will always be a
link that is "local" to the webmon
server, and thus the
browser will always direct this request back to webmon
.
In "rewrite" mode, the browser *must* be specifically aimed at
webmon, and the port that webmon listens to, and webmon must in
turn be aimed at a specific website. One must tell webmon
about an initial web server with the -w
flag, thus for
example: webmon -w www.fictitious.com
. Do not
use the -a
or -n
flags in this mode; they are
mutually incompatible. Then, in order for the browser to
access the site www.fictitious.com, the user must
request the URL http://your.webmonhost.yourdomain.com:5080/
.
Similarly, if the user wants to view
http://www.fictious.com/some/page.html
, they should
request
http://your.webmonhost.yourdomain.com:5080/some/page.html
.
As long as the user just clicks on links, and does not type
any in, webmon
will capture the surfing session and record it.
Note that in "rewrite" mode, webmon
needs to perform
some amount of parsing of HTML pages in order to find links
and re-write them. It is a very simple parser, and it can
get confused by more complex sites. It does understand some
basic JavaScript, and forms, but it can be confused by layers
and ilayers, and will occasionally trip over a
<base href=
tag. Because of this, the rewrite mode is
not the recommended mode of operation.
Note that a certain amount of SSL testing can be done in rewrite
mode that is difficult to do in other modes. In this mode,
webmon
can accept unencrypted connections, and convert
them to encrypted connections; it can also do the vice-versa.
The --ssl-server
flag is used to indicate that an SSL session
should be negotiated with the server; the --ssl-browser
flag
is used to negotiate an SSL session with the client. Note that
a certificate is needed when talking SSL to the client. Instead of
allowing SSL to negotiate a cipher suite, the --server-cipher
and --browser-cipher
flags can be used to explicitly specify
an SSLv2 or SSLv3/TSLv1 cipher suite.
The "passthru" mode is the most transparent mode of operation
that webmon
has. In this mode, all re-writing is disabled.
It neither re-writes the requests (as is done in proxy mode) nor
does it rewrite the responses (as it does in rewrite mode).
It simply passes any received request to an indicated server.
This is done by using the -w
and the -n
command-line
flags together.
The most useful form of this mode is when the -w
flag is
used to specify a proxy (and not the -P
flag),
and the browser is told that webmon
is a proxy. In this
configuration, webmon
will simply record all the traffic
between the browser and the proxy, without changing any of it.
Note that in pass-through mode, webmon
will still listen
for the CONNECT
protocol directive used for establishing
encrypted sessions, and will initiate a man-in-the-middle attack
on the traffic. In this way, it can still snoop on the encrypted
traffic.
All three of the modes described above can be used with either
socks or proxy style firewalls. Firewall operation proceeds
as described elsewhere in this document, and webmon
will
always "do the right thing" in such situations. These modes
are not further described here, as they serve only to confuse
conversation, whereas their operation is transparent.
One typically wants to use a number of command-line flags
with webmon
, and it is easiest to put this into a script
batch file, and treat it as a config file.
The example file examples/run.webmon
illustrates such a file.
Various important switch settings are reviewed below.
To configure webmon
to act as an HTTP proxy, listening to
port 5080, and recording the session to a file my.session
,
start webmon
as follows:
webmon -a -n -p 5080 -v my.session
Note that long-form versions of these flags can be used: these require more typing, but can serve as memory aids to make the file more readable:
webmon --act-as-proxy \
--no-xlation \
--listen-port=5080 \
--request-file=my.session
When webmon starts it will print a summary of the command-line options, and then indicate that it is listening to a particular port. The tail end of this message will look like so:
Info: webmon: listening for connections at yourmachinename:5080
Info: webmon: waiting for a connection.......
XXX UNDER CONSTRUCTION Finish writing this section...
To collect timing statistics and put them in the file webmon.report
,
the command would look like the following:
webmon -a -n -p 5080 -r webmon.report
To trace traffic ...
webmon -a -n -p 5080 -t webmon.session
or use the flags ...
webmon -a -n -p 5080 --trace-server -t webmon.session
webmon -a -n -p 5080 --trace-client -t webmon.session
To do everything as above, but connect to the server using SSLV3 and cipher suite 4 (RC4_128_MD5):
webmon -a -n -p 5080 -S SSLV3:04 -v webmon.session
or use
start webmon with the --ssl-server
option (use SSL), and ...
The script file examples/run.webmon
has been included here to
give you examples of how to use webmon. You can use these scripts
directly, if you like. Take a look at them to see how they are used.
webmon
prints the following statistics to its console and to the
report file. The existence of a report file (-r
option) tells
webmon to calculate and print these timing statistics:
elapsed time from connect until all data for the indicated request to arrive. This time does *not* include time spent to resolve host names into IP addresses.
time from connect until the first message containing body data is received.
time required to establish a connection to the host for this request. If you are using SSL, it includes the SSL connect time.
time from when the request was sent to the host until the HTTP header was received.
number of header and body bytes received.
time from when the header was received until the last byte of body was received.
In addition, the following statistics are placed in the report file, for each request, if SSL is enabled:
The time spent in the SSL code running on the local host. This time is indicative of CPU cycles spent on the client that are independent of any processing occurring on the server.
The time spent in TCP/IP and waiting for the response from the host during an SSL connection handshaking & negotiation.
The time spent in local SSL code during the processing for the HTTP header.
The time spent in TCP/IP and waiting for the HTTP header to be returned.
The time spent in SSL during transfer of the body of the message.
The time spent in TCP/IP and waiting for the body to be returned.
The following invariants are true about the various statistics:
response time = connect time + header delay + transfer time
connect time = SSLConnectOvhd + Network Delay Connect
header delay = SSLHeaderOvhd + Network Delay Header
transfer time = SSLTransfrOvhd + Network Delay Transfer