Chapter 2

HTTP server

The SUnet HTTP Server is a complete industrial-strength implementation of the HTTP 1.0 protocol. It is highly configurable and allows the writing of dynamic web pages that run inside the server without going through complicated and slow protocols like CGI or Fast/CGI.

2.1  Starting and configuring the server

All procedures described in this section are exported by the httpd structure.

The Web server is started by calling the httpd procedure, which takes one argument, an options value:

This procedure starts the server. The options argument specifies various configuration parameters, explained below.

The server's basic loop is to wait on the port for a connection from an HTTP client. When it receives a connection, it reads in and parses the request into a special request data structure. Then the server forks a thread which binds the current I/O ports to the connection socket, and then hands off to the top-level request handler (which must be specified in the options). The request handler is responsible for actually serving the request -- it can be any arbitrary computation. Its output goes directly back to the HTTP client that sent the request.

Before calling the request handler to service the request, the HTTP server installs an error handler that fields any uncaught error, sends an error reply to the client, and aborts the request transaction. Hence any error caused by a request handler will be handled in a reasonable and robust fashion.

This specifies the port on which the server listens. Defaults to 80.

This specifies the current directory of the server. Note that this is not the document root directory. Defaults to /.

This specifies the fully-qualified domain name the server uses in automatically generated replies, or #f if the server should query DNS for the fully-qualified domain name.. Defaults to #f.

This specifies the port number the server uses in automatically generated replies or #f if the reported port is the same as the port the server is listening on. (This is useful if you're running the server through an accelerating proxy.) Defaults to #f.

This specifies the email address of the server administrator the server uses in automatically generated replies. Defaults to #f.

This specifies the request handler of the server to which the server delegates the actual work. More on that subject below in Section 2.5. This parameter must be specified.

This specifies a limit on the number of simultaneous requests the server servers. If that limit is exceeded during operation, the server will hold off on new requests until the number of simultaneous requests has sunk below the limit again. If this parameter is #f, no limit is imposed. Defaults to #f.

This specifies the name of a log file for the server where it writes Common Log Format logging information. It can also be a port in which case the information is logged to that port, or #f for no logging. Defaults to #f.

To allow rotation of log files, the server re-opens the log file whenever it receives a USR1 signal.

This specifies whether the server will log information about incoming to the Unix syslog facility. Defaults to #t.

This specifies whether the server writes the domain names rather than numerical IPs to the output log it produces. Defaults to #t.

To avoid paranthitis, the make-httpd-options procedure eases the construction of the options argument:

This constructs an options value from an argument list of parameter transformers and parameter values. The arguments come in pairs, each an option transformer from the list above, and a value for that parameter. Make-httpd-options returns the resulting options value.

For example,

(httpd (make-httpd-options
         with-request-handler (rooted-file-handler "/usr/local/etc/httpd")
         with-root-directory "/usr/local/etc/httpd"))

starts the server on port 80 with /usr/local/etc/httpd as its root directory and lets it serve any file out from this directory.

2.2  Requests

Request handlers operate on requests which contain the information needed to generate a page. The relevant procedures to dissect requests are defined in the httpd-requests structure:

The procedure inspect request values. Request? is a predicate for requests. Request-method extracts the method of the HTTP request; it's a string such as "GET", "PUT". Request-uri returns the escaped URI string as read from request line. Request-url returns an HTTP URL value (see the description of the url structure in 4). Request-version returns (major . minor) integer pair representing the version specified in the HTTP request. Request-headers returns an association lists of header field names and their values, each represented by a list of strings, one for each line. Request-socket returns the socket connected to the client.¹

2.3  Responses

A path handler must return a response value representing the content to be sent to the client. The machinery presented here for constructing responses lives in the httpd-responses structure.

This procedure constructs a response value. Status-code is an HTTP status code (more on that below). Maybe-message is a a message elaborating on the circumstances of the status code; it can also be #f meaning that the server should send a default message associated with the status code. Seconds natural number indicating the time the content was created, typically the value of (time). Mime is a string indicating the MIME type of the response (such as "text/html" or "application/octet-stream"). Extras is an association list with extra headers to be added to the response; its elements are pairs, each of which consists of a symbol representing the field name and a string representing the field value. Body represents the body of the response; more on that below.

This is a helper procedure for constructing HTTP redirections. The server will serve the new file indicated by location. Location must be URI-encoded and begin with a slash.

This is a helper procedure for constructing error responses. code is status code of the response (see below). Request is the request that led to the error. Message is an optional string containing an error message written in HTML, and extras are further optional arguments containing further message lines to be added to the web page that's generated.

Make-error-response constructs a response value which generates a web page containg a short explanatory message for the error at hand.

The status-code syntax returns a status code where <.name.> is the name from Table 1. Name->status-code also returns a status code for a name represented as a symbol. For a given status code, status-code-number extracts its number, and status-code-message extracts its associated default message.

2.4  Response Bodies

A response body represents the body of an HTTP response. There are several types of response bodies, depending on the requirements on content generation.

This constructs a response body from a writer -- a procedure that prints the page contents to a port. The proc argument must be a procedure accepting an output port (to which proc prints the body) and the options value passed to the httpd invocation.

This constructs a response body from a reader/writer -- a procedure that prints the page contents to a port, possibly after reading input from the socket of the HTTP connection. The proc argument must be a procedure accepting three arguments: an input port (associated with the HTTP connection socket), an output port (to which proc prints the body), and the options value passed to the httpd invocation.

2.5  Request Handlers

A request handler generates the actual content for a request; request handlers form a simple algebra and may be combined and composed in various ways.

A request handler is a procedure of two arguments like this:

Req is a request. The path argument is the URL's path, parsed and split at slashes into a string list. For example, if the Web client dereferences URL

http://clark.lcs.mit.edu:8001/h/shivers/code/web.tar.gz

then the server would pass the following path to the top-level handler:

("h" "shivers" "code" "web.tar.gz")

The path argument's pre-parsed representation as a string list makes it easy for the request handler to implement recursive operations dispatch on URL paths.

The request handler must return an HTTP response.

2.5.1  Basic Request Handlers

The web server comes with a useful toolbox of basic request handlers that can be used and built upon. The following procedures are exported by the httpd-basic-handlers structure:

This request handler always generated a not-found error response, no patter what the request is.

The request handler returned by this procedure first calls predicate on its path and request; it then acts like handler if the predicate returned a true vale, and like default-handler if the predicate returned #f.

The request handler returned by this procedure compares the host name specified in the request with hostname: if they match, it acts like handler, otherwise, it acts like default-handler.

The request handler returned by this procedure first calls predicate on its path; it then acts like handler if the predicate returned a true vale, and like default-handler if the predicate returned #f.

This constructs a request handler that calls handler on its argument if path-prefix (a string) is the first element of the requested path; it calls handler on the rest of the path and the original request. Otherwise, the handler acts like default-handler.

This procedure takes as arguments an alist mapping strings to path handlers, and a default request handler, and returns a handler that dispatches on its path argument. When the new request handler is applied to a path

("foo" "bar" "baz")

it uses the first element of the path -- foo -- to index into the alist. If it finds an associated request handler in the alist, it hands the request off to that handler, passing it the tail of the path, in this case

("bar" "baz")

On the other hand, if the path is empty, or the alist search does not yield a hit, we hand off to the default path handler, passing it the entire original path,

("foo" "bar" "baz")

This procedure is how you say: ``If the first element of the URL's path is `foo', do X; if it's `bar', do Y; otherwise, do Z.'' The slash-delimited URI path structure implies an associated tree of names. The request-handler system and the alist dispatcher allow you to procedurally define the server's response to any arbitrary subtree of the path space.

Example: A typical top-level request handler is

(define ph
  (alist-path-dispatcher
      `(("h"       . ,(home-dir-handler "public_html"))
        ("cgi-bin" . ,(cgi-handler "/usr/local/etc/httpd/cgi-bin"))
        ("seval"   . ,seval-handler))
      (rooted-file-handler "/usr/local/etc/httpd/htdocs")))

This means:

• If the path looks like ("h" "shivers" "code" "web.tar.gz"), pass the path ("shivers" "code" "web.tar.gz") to a home-directory request handler.

• If the path looks like ("cgi-bin" "calendar"), pass ("calendar") off to the CGI request handler.

• If the path looks like ("seval" ...), the tail of the path is passed off to the code-uploading seval path handler.

• Otherwise, the whole path is passed to a rooted file handler, who will convert it into a filename, rooted at /usr/local/etc/httpd/htdocs, and serve that file.

2.5.2  Static Content Request Handlers

The request handlers described in this section are for serving static content off directory trees in the file system. They live in the httpd-file-directory-handlers structure.

The request handlers in this section eventually call an internal procedure named file-serve for serving files which implements a simple directory-generation service using the following rules:

• If the filename has the form of a directory (i.e., it ends with a slash), then file-serve actually looks for a file named index.html in that directory.

• If the filename names a directory, but is not in directory form (i.e., it doesn't end in a slash, as in ``/usrinclude'' or ``/usrraj''), then file-serve sends back a ``301 moved permanently'' message, redirecting the client to a slash-terminated version of the original URL. For example, the URL http://clark.lcs.mit.edu/ shivers would be redirected to http://clark.lcs.mit.edu/ shivers/

• If the filename names a regular file, it is served to the client.

The httpd-file-directory-handlers all take an options value as an argument, similar to the options for httpd itself.

The options argument can be constructed through a number of procedures with names of the form with-.... Each of these procedures either creates a fresh options value or adds a configuration parameter to an old options argument. The configuration parameter value is always the first argument, the (old) options value the optional second one. Here they are:

This specifies a procedure for determining the MIME content type (``text/html,'' ``application/octet-stream'' etc.) from a file name. Proc takes a file name as an argument and must return a string. (This is relevant in directory listings.) The default is a procedure able to handle the more common file extensions.

This specifies a procedure for determining the MIME content encoding (if the file is compressed, gzipped, etc.) from a file name. (This is relevant in directory listings.) Proc takes a file name as an argument and must return two values: the equivalent, unencoded file name (i.e., without the trailing .Z or .gz) and a string representing the content encoding.

This specifies a procedure for determining the icon to be displayed next to a file name in a directory listing. Proc takes a file name as an argument and must return a URL for the corresponding icon or #f.

This specifies a file name (or its absence) for the special icon that must be as wide as the icons returned by the previous procedure but that is blank.

This specifies a file name (or its absence) for the special icon that is displayed next to the ``parent directory'' link in directory listings.

This specifies a file name (or its absence) for the special icon that is displayed next to the unknown entries in directory listings.

The make-file-directory-options procedure eases the construction of the options argument:

This constructs an options value from an argument list of parameter transformers and parameter values. The arguments come in pairs, each an option transformer from the list above, and a value for that parameter. Make-file-directory-options returns the resulting options value.

This returns a request handler that serves files from a particular root in the file system. Only the GET operation is provided. The path argument passed to the handler is converted into a filename, and appended to root. The file name is checked for .. components, and the transaction is aborted if it does. Otherwise, the file is served to the client.

Dito, but also serve directory indices for directories without index.html.

This procedure builds a request handler that does basic file serving out of home directories. If the resulting request-handler is passed a path of the form (user . file-path), then it serves the file subdir/file-path inside the user's home directory.

The request handler only handles GET requests; the filename is not allowed to contain .. elements.

This returns request handler that examines the car of the path. If it is a string beginning with a tilde, e.g., " ziggy", then the string is taken to mean a home directory, and the request is served similarly to a home-dir-handler request handler. Otherwise, the request is passed off in its entirety to the default-request-handler.

2.6  CGI Server

The procedure(s) described here live in the httpd-cgi-handlers structure.

Returns a request handler for CGI scripts located in bin-dir. Cgi-bin-dir specifies the value of the PATH variable of the environment the CGI scripts run in. It defaults to

/bin:/usr/bin:/usr/ucb:/usr/bsd:/usr/local/bin

The CGI scripts are called as specified by CGI/1.1².

Note that the CGI handler looks at the name of the CGI script to determine how it should be handled:

• If the name of the script starts with `nph-', its reply is read, the RFC 822-fields like Content-Type and Status are parsed and the client is sent back a real HTTP reply, containing the rest of the script's output.

• If the name of the script doesn't start with `nph-', its output is sent back to the client directly. If its return code is not zero, an error message is generated.

2.7  Scheme-Evaluating Request Handlers

The httpd-seval-handlers structure contains a handler which demonstrates how to safely evaluate Scheme code uploaded from the client to the server.

This request handler is suitable for receiving code entered into an HTML text form. The Scheme code being uploaded is being POSTed to the server (from a form). The code should be URI-encoded in the URL as program=<stuff>. stuff must be an (URI-encoded) Scheme expression which the handler evaluates in a separate subprocess. (It waits for 10 seconds for a result, then kills the subprocess.) The handler then prints the return values of the Scheme code.

The following structures define environments that are R5RSwithout features that could examine or effect the file system. You can also use them as models of how to execute code in other protected environments in Scheme 48.

2.7.1  The loser structure

The loser package exports only one procedure:

Raises an error like ``Illegal call name''.

2.7.2  The toothless structure

The toothless structure contains everything of R5RSexcept that following procedure cause an error if called:

• call-with-input-file

• call-with-output-file

• load

• open-input-file

• open-output-file

• transcript-on

• with-input-from-file

• with-input-to-file

• eval

• interaction-environment

• scheme-report-environment

2.7.3  The toothless-eval structure

Creates a brand-new structure, imports the toothless structure, and evaluates expression in it. When the evaluation is done, the environment is thrown away, so expression's side-effects don't persist from one eval-safely call to the next. If expression raises an error exception, eval-safely returns #f.

2.8  Writing Request Handlers

2.8.1  Parsing HTML Forms

In HTML forms, field data are turned into a single string, of the form <.name.>=<.val.>&<.name.>=<.val.>.... The parse-html-forms structure provides simple functionality to parse these strings.

This parses "foo=x&bar=y" into (("foo" . "x") ("bar" . "y")). Substrings are plus-decoded (i.-e. plus characters are turned into spaces) and then URI-decoded.

This implementation is slightly sleazy as it will successfully parse a string like "a&b=c&d=f" into (("a&b" . "c") ("d" . "f")) without a complaint.

2.9  SSL encryption with Apache

Network traffic with a HTTP server is usually encrypted and protected from manipulation using the cryptographic algorithm provided by an implementation of the secure socket layer, SSL for short. SUnet does not have support for SSL yet. However, an Apache web-server with SSL support can be configured as a proxy. In this setup the Apache web-server accepts encrypted requests and forwards them to a SUnet web-server running locally. This section describes how to set up Apache as an encrypting proxy, assuming the reader has basic knowledge about Apache and its configuration directives.

The following excerpt shows a minimalist SSL virtual host that forwards requests to a SUnet server.

<VirtualHost 134.2.12.82:443>
  DocumentRoot "/www/some-domain/htdocs"
  ServerName www.some-domain.de
  ServerAdmin admin@some-domain.de
  ErrorLog /www/some-domain/logs/error_log

  ProxyRequests off
  ProxyPass / http://localhost:8080/
  ProxyPassReverse / http://localhost:8080/

  SSLEngine on
  SSLRequireSSL

  SSLCertificateFile /www/some-domain/cert/some-domain.cert
  SSLCertificateKeyFile /www/some-domain/cert/some-domain.key
</VirtualHost>

First, a virtual host is added to Apache's configuration file. This virtual host listens for incoming connections on port 443, which is the standard port for encrypted HTTP traffic. SSLRequireSSL ensures that server accepts encrypted connections only.

In terms of the Apache documentation, the web-server acts as a so called reverse proxy. The option ProxyRequests has a misleading name. Setting this option to off does only turns off Apache's facility to act as a forward proxy and has no effect on the configuration directives for reverse proxies. Actually, turning on ProxyRequests is dangerous, because this turns Apache into a proxy server that can be used from anywhere to access any site that is accessible to the Apache server.

In this setting, all requests get forwarded to a SUnet web-server which listens for incoming connections on localhost port 8080 only, thus, it is not reachable from a remote machine. Apache forwards all requests to the host and port specified by the ProxyPass directive. ProxyPassReverse specifies how Location-Header fields of HTTP redirect messages send by the SUNet server are translated.