Next Previous Contents

4. Recording Sessions with WebMon

WebMon is an HTTP tracing and capture tool. It can be used to examine the HTTP dialogue between a web browser and an web server. It provides a variety of features & functions:

WebMon's most popular use is to provide the input files for the webclient tool.

Limitations

webmon does not try to match the connection behavior of the browser in how it fetches pages. It will open distinct and unique sockets for each web-page GIF fetch, instead of reusing an existing socket like many web browsers. In particular, it does not emulate or the HTTP "KeepAlive" protocol, nor does it use multiple parallel connections to the web server. Thus, it cannot be used to diagnose conditions. In most respects, webmon appears to be a single-threaded process to the web server.

4.1 Using Webmon

In the default mode of operation, webmon acts as a proxy server. Therefore, you must configure your browser to use webmon as a proxy, as described in the previous chapter. Specific example usages are given in the next section, below.

XXX this section needs more work ...

The following command-line options are defined:


Usage: webmon [options]

-a, --act-as-proxy        behave as a proxy server
-A, --alarm=<time>        turns on timeout alarms (delay 'time')
-d, --debug               print basic debugging messages
-D, --Debug               turns on verbose debugging
-E, --Debug-debug         turns on extra verbose debugging
-e, --print-each          print individual response time observations
-h, --help                print this message
-n, --no-xlation          disable host/url translation/rewriting
-p, --listen-port=<port>  specifies the port to listen to
-P, --proxy=<proxy:port>  specifies the proxyserver and port
-q, --debug-time          turns on debugging output for timing statistics
-Q, --Debug-time          turns on verbose timing debug output
-r, --report-file=<file>  specifies the report file name
-t, --trace-file=<file>   write HTTP traces to file
-U, --user-agent=<string> specify value of 'User-Agent:' in HTTP header
                          string must be enclosed in single quotes
-v, --request-file=<file> record browser requests in this file
-w, --webserver=<server:port> specifies the webserver and port

Flags without short-form equivalents:
--access-log=<file>       write webserver-style access log              
--no-bug-compat           enable strict conformance to HTTP standards
--quiet                   minimize messages written to stdout
--show-progress           write out each URL as its fetched
--trace-client            write exchanges with client to trace file
--trace-server            write exchanges with server to trace file
--version                 print version info and exit

SSL options:

      --ssl-server           use SSL to connect to server
  -S, --server-cipher=NUM    force use of a specific cipher
      --server-timeout=NUM   set SSL session cache timeout (seconds)
      --ssl-browser          use SSL to connect to browser
      --browser-cipher=NUM   force use of a specific cipher
      --browser-timeout=NUM  set SSL session cache timeout (seconds)
      --certif=<filename>    specify file containing certificates
  -k, --keyring=<filename>   specify file containing keys
  -K, --password=<passwd>    specify keyring password
  -N, --distinguish=<name>   specify a distinguished name

Cipher suites may be specified in the form SSLV2:N with N 1,2,3,..7
or SSLV3:NN for NN=00, 01, 02 ... 06 or 0A.If a cipher suite has been
specified, then no cipher suite negotiation is performed.

4.2 SSL Cipher-Suite Codes

The Cipher-suite numbering convention follows that of the SSLv3 spec. Below is a partial listing; see the spec for a full listing.

SSLV2 Cipher suite names and N values:


SSL2_RC4_128_WITH_MD5 =                 1
SSL2_RC4_128_EXPORT_40_WITH_MD5 =       2
SSL2_RC2_128_CBC_WITH_MD5 =             3
SSL2_RC2_128_CBC_EXPORT40_WITH_MD5 =    4
SSL2_IDEA_128_CBC_WITH_MD5 =            5
SSL2_DES_64_CBC_WITH_MD5 =              6
SSL2_DES_192_EDE3_CBC_WITH_MD5 =        7

The Cipher values +rc4export and +rc2export in the magnus.conf file for the webserver appear to correspond to cipher suite specifications of 2 and 4, respectively. (Yes, they are backwards....)

SSLV3 Cipher suite names and NN values:


SSL_NULL_WITH_NULL_NULL =                   0x0000
SSL_RSA_WITH_NULL_MD5 =                     0x0001
SSL_RSA_WITH_NULL_SHA =                     0x0002
SSL_RSA_EXPORT_WITH_RC4_40_MD5 =            0x0003
SSL_RSA_WITH_RC4_128_MD5 =                  0x0004
SSL_RSA_WITH_RC4_128_SHA =                  0x0005
SSL_RSA_EXPORT_WITH_RC2_CBC_40_MD5 =        0x0006
SSL_RSA_WITH_DES_CBC_SHA =                  0x0009
SSL_RSA_WITH_3DES_EDE_CBC_SHA =             0x000A

OpenSSL supports a large number of additional ciphers not listed here.

4.3 Theory of Operation

Webmon can be used in one of three different ways (which may seem similar, but have important, fundamental differences). Lets term these the "proxy" mode, the "rewrite" mode, and the "passthru" mode.

Proxy Mode

For most users, it is suggested that the "proxy" mode be used in normal operation, as it is the simplest to configure, understand and use. In this mode, webmon acts as an ordinary web proxy. That is, it will listen for URL requests that have a server name embedded in them, and will contact these servers on behalf of the browser. Thus, a typical HTTP request line sent by the browser to webmon will look like GET http://some.where.com/some/file.html HTTP/1.0 Upon receipt, webmon will contact the host some.where.com on port 80 and will issue it the usual HTTP request GET /some/file.html HTTP/1.0. It will then wait for a response, and pass this response back to the browser, unmodified. Essentially all proxies work in this way.

When webmon is used in proxy mode to access secure (encrypted) web sites, it does do something unusual. Instead of passing the secure connection request to the secure server, it will instead present it's own credentials, initiating a "man-in-the-middle" attack. This allows webmon to decrypt the traffic from the browser and record it, i.e. to snoop on it. It does this by listening for the CONNECT HTTP directive, and instead of connecting, it responds with its own certificate, as if it were the end host. As long as the browser accepts the certificate, webmon will be able to snoop on the browsers traffic.

Note that webmon is distributed with a working example certificate in the examples directory. This certificate contains an export-grade (512-bit) private key and a signature from a fictitious certification authority. This signing authority will not be recognized as official by most browsers, and thus, most browsers will put up warning dialogs when presented with this certificate. If a signed certificate were obtained from the usual recognized authorities, webmon could be used to snoop on encrypted traffic without the users knowledge (provided the user had configured to use webmon as a proxy).

To use webmon in the "proxy" mode, specify the -a and -n command line flags.

Rewrite Mode

Another mode of operation for webmon is the "rewrite" mode. In this mode, the browser accesses webmon as if it were the true server, i.e. it uses webmon's URL as the server URL. As in all of the other modes this mode, webmon will pass the request on to the true server, and wait for a reply. When it receives the reply, it will "rewrite" it to blank out all embedded URL's, and convert them into local URL's. In this way, it can keep the browser from ever "escaping" to a different server: any link that the user clicks on will always be a link that is "local" to the webmon server, and thus the browser will always direct this request back to webmon.

In "rewrite" mode, the browser *must* be specifically aimed at webmon, and the port that webmon listens to, and webmon must in turn be aimed at a specific website. One must tell webmon about an initial web server with the -w flag, thus for example: webmon -w www.fictitious.com. Do not use the -a or -n flags in this mode; they are mutually incompatible. Then, in order for the browser to access the site www.fictitious.com, the user must request the URL http://your.webmonhost.yourdomain.com:5080/. Similarly, if the user wants to view http://www.fictious.com/some/page.html, they should request http://your.webmonhost.yourdomain.com:5080/some/page.html. As long as the user just clicks on links, and does not type any in, webmon will capture the surfing session and record it.

Note that in "rewrite" mode, webmon needs to perform some amount of parsing of HTML pages in order to find links and re-write them. It is a very simple parser, and it can get confused by more complex sites. It does understand some basic JavaScript, and forms, but it can be confused by layers and ilayers, and will occasionally trip over a <base href= tag. Because of this, the rewrite mode is not the recommended mode of operation.

Note that a certain amount of SSL testing can be done in rewrite mode that is difficult to do in other modes. In this mode, webmon can accept unencrypted connections, and convert them to encrypted connections; it can also do the vice-versa. The --ssl-server flag is used to indicate that an SSL session should be negotiated with the server; the --ssl-browser flag is used to negotiate an SSL session with the client. Note that a certificate is needed when talking SSL to the client. Instead of allowing SSL to negotiate a cipher suite, the --server-cipher and --browser-cipher flags can be used to explicitly specify an SSLv2 or SSLv3/TSLv1 cipher suite.

Passthru Mode

The "passthru" mode is the most transparent mode of operation that webmon has. In this mode, all re-writing is disabled. It neither re-writes the requests (as is done in proxy mode) nor does it rewrite the responses (as it does in rewrite mode). It simply passes any received request to an indicated server. This is done by using the -w and the -n command-line flags together.

The most useful form of this mode is when the -w flag is used to specify a proxy (and not the -P flag), and the browser is told that webmon is a proxy. In this configuration, webmon will simply record all the traffic between the browser and the proxy, without changing any of it.

Note that in pass-through mode, webmon will still listen for the CONNECT protocol directive used for establishing encrypted sessions, and will initiate a man-in-the-middle attack on the traffic. In this way, it can still snoop on the encrypted traffic.

Firewall Operation

All three of the modes described above can be used with either socks or proxy style firewalls. Firewall operation proceeds as described elsewhere in this document, and webmon will always "do the right thing" in such situations. These modes are not further described here, as they serve only to confuse conversation, whereas their operation is transparent.

4.4 Example Usage

One typically wants to use a number of command-line flags with webmon, and it is easiest to put this into a script batch file, and treat it as a config file. The example file examples/run.webmon illustrates such a file. Various important switch settings are reviewed below.

To configure webmon to act as an HTTP proxy, listening to port 5080, and recording the session to a file my.session, start webmon as follows:


         webmon -a -n -p 5080 -v my.session

Note that long-form versions of these flags can be used: these require more typing, but can serve as memory aids to make the file more readable:


         webmon --act-as-proxy          \
                --no-xlation            \
                --listen-port=5080      \
                --request-file=my.session

When webmon starts it will print a summary of the command-line options, and then indicate that it is listening to a particular port. The tail end of this message will look like so:


         Info: webmon: listening for connections at yourmachinename:5080
         Info: webmon: waiting for a connection.......

XXX UNDER CONSTRUCTION Finish writing this section...

To collect timing statistics and put them in the file webmon.report, the command would look like the following:


          webmon -a -n -p 5080 -r webmon.report

To trace traffic ...


           webmon -a -n -p 5080 -t webmon.session

or use the flags ...


           webmon -a -n -p 5080 --trace-server -t webmon.session
           webmon -a -n -p 5080 --trace-client -t webmon.session

To do everything as above, but connect to the server using SSLV3 and cipher suite 4 (RC4_128_MD5):


           webmon -a -n -p 5080 -S SSLV3:04 -v webmon.session

or use start webmon with the --ssl-server option (use SSL), and ...

The script file examples/run.webmon has been included here to give you examples of how to use webmon. You can use these scripts directly, if you like. Take a look at them to see how they are used.

4.5 Statistics Reported by Webmon

webmon prints the following statistics to its console and to the report file. The existence of a report file (-r option) tells webmon to calculate and print these timing statistics:

response time

elapsed time from connect until all data for the indicated request to arrive. This time does *not* include time spent to resolve host names into IP addresses.

first data response time

time from connect until the first message containing body data is received.

connect time

time required to establish a connection to the host for this request. If you are using SSL, it includes the SSL connect time.

header delay

time from when the request was sent to the host until the HTTP header was received.

bytes transferred

number of header and body bytes received.

transfer time

time from when the header was received until the last byte of body was received.

In addition, the following statistics are placed in the report file, for each request, if SSL is enabled:

SSLConnectOvhd

The time spent in the SSL code running on the local host. This time is indicative of CPU cycles spent on the client that are independent of any processing occurring on the server.

Network Delay Connect

The time spent in TCP/IP and waiting for the response from the host during an SSL connection handshaking & negotiation.

SSLHeaderOvhd

The time spent in local SSL code during the processing for the HTTP header.

Network Delay Header

The time spent in TCP/IP and waiting for the HTTP header to be returned.

SSLTransfrOvhd

The time spent in SSL during transfer of the body of the message.

Network Delay Transfer

The time spent in TCP/IP and waiting for the body to be returned.

The following invariants are true about the various statistics:



response time = connect time + header delay + transfer time

connect time  = SSLConnectOvhd + Network Delay Connect

header delay  = SSLHeaderOvhd + Network Delay Header

transfer time = SSLTransfrOvhd + Network Delay Transfer


Next Previous Contents