copyright(c) 2000-2001. Franz Inc
Introduction
Running AllegroServe
Starting the Server
start
Shutting Down the Server
shutdown
Publishing Information
publish-file
publish-directory
publish
publish-prefix
publish-multi
Generating a Computed Response
with-http-response
with-http-body
get-request-body
header-slot-value
reply-header-slot-value
request-query
request-query-value
Request Object Readers and Accessors
request-method
request-uri
request-protocol
request-socket
request-wserver
request-raw-request
request-reply-code
request-reply-date
request-reply-headers
request-reply-content-length
request-reply-plist
request-reply-strategy
request-reply-stream
CGI Program Execution
run-cgi-program
Form Processing
get-multipart-header
parse-multipart-header
get-multipart-sequence
get-all-multipart-data
form-urlencoded-to-query
query-to-form-urlencoded
Authorization
get-basic-authorization
set-basic-authorization
password-authorizer
location-authorizer
function-authorizer
Cookies
set-cookie-header
get-cookie-values
Variables
*aserve-version*
*default-aserve-external-format*
*http-response-timeout*
*mime-types*
AllegroServe Request Processing Protocol
handle-request
standard-locator
unpublish-locator
authorize
failed-request
denied-request
process-entity
Client Functions
do-http-request
client-request
cookie-jar
make-http-client-request
read-client-response-headers
client-request-read-sequence
client-request-close
uriencode-string
Proxy
Cache
Request Filters
Virtual Hosts
Timeouts
wserver-io-timeout
wserver-response-timeout
Miscellaneous
ensure-stream-lock
map-entities
Running AllegroServe as a Service on Windows NT
Using International Characters in AllegroServe
Debugging
net.aserve::debug-on
net.aserve::debug-off
AllegroServe is a webserver written at Franz Inc. AllegroServe is designed to work with the htmlgen system for generating dynamic html, as one of the big advantages of a web server written in Common Lisp is the ability to generate html dynamically. In this document we'll consider the web server and dynamic html generation to be parts of the same product.
The design goals of AllegroServe are:
Running AllegroServe requires that you
We mention publish twice to emphasize that you can publish urls before and after you start the server.
The function net.aserve:start is used to start the server running.
(start &key port host
listeners chunking keep-alive server setuid setgid
debug proxy proxy-proxy
cache restore-cache accept-hook ssl ssl-password
os-processes
external-format)
If no arguments are given then start starts a multi-threaded web server on port 80, which is the standard web server port. If you are running this on Unix then you can only allocate port 80 if you are logged in as root or have made Lisp a set-user-id root program.
There are quite a few keyword arguments to start, but in practice you only need be concerned with :port and :listeners. The arguments have the following meanings:
(shutdown &key server save-cache)
This shuts down the web server given (or the most recently started web server if no argument is passed for server). If save-cache is given then it should be the name of a file to which the current state of the proxy cache will be written. The save-cache file will only contain in-memory information about the cache. The cache usually consists of disk files as well and in order to maintain the complete state of the cache these files must be saved by the user as well. The information in the save-cache file refers to the disk cache files so those disk cache files must exist and be in the same state and location should the user choose to restore the state of the cache.
Once the server is started it will accept requests from http clients, typically web browsers. Each request is parsed and then AllegroServe searches for an object to handle that request. That object is called an entity. If an entity is found, it is passed the request and is responsible for generating and sending a response to the client. If an entity can't be found then AllegroServe sends back a response saying that that request was invalid.
Publishing is the process of creating entities and registering them in the tables scanned by AllegroServe after a request is read.
A request from an http client contains a lot of information. The two items that determine which entity will handle the request are
A request contains other information and while that information isn't used to determine which entity will handle the request it can be used by the entity handling the request in any way it sees fit.
The following functions create entities and specify which requests they will handle. An entity is distinguished by the path and host values passed to the particular publish function. When a publish is done for a path and host for which there is already an entity assigned, the old entity is replaced by the new entity.
(publish-file &key
path host port file content-type class preload cache-p remove
authorizer server timeout plist)
This creates an entity that will return the contents of a file on the disk in response to a request. The url and file must be given, the rest of the arguments are optional.. The arguments have these meanings:
The function that handles requests for files will respond correctly to If-Modified-Since header lines and thus minimizes network traffic.
Example:
This will work on Unix where the password file is stored in /etc.
(publish-file :path "/password" :file "/etc/passwd" :content-type "text/plain")
(publish-directory
&key prefix host port destination remove authorizer server
indexes filter timeout plist publisher access-file)
publish-directory is used to publish a complete directory tree of files. This is similar to how web servers such as Apache publish files. AllegroServe publishes the files in the directory tree in a lazy manner. As files in the tree are referenced by client requests entities are created and published.
publish-directory creates a mapping from all urls whose name begins with prefix to files stored in the directory specified by the destination. The host, port, remove, authorizer, plist and server arguments are as described above for publish-file. The timeout argument defaults as described in publish-file. The access-file argument names the access file name which will be used in this directory tree. When a request comes in for which there isn't an entity that matches it exactly, AllegroServe checks to see if a prefix of the request has been registered. If so, and if the resulting entity is a directory-entity as created by this function, then it strips the prefix off the given request and appends the remaining part of the request to the destination string. It then publishes that (normally using publish-file and computing the content-type from the file type). Next that file-entity is made to handle the request in the normal manner.
If a request comes that maps to a directory rather than a file then AllegroServe tries to locate an index file for that directory. The indexes argument specifies a list of index files to search for. By default the list consists of two filenames "index.html" and "index.htm".
The valueof the filter argument is a function of four values: req ent filename and info. req and ent are the request and entity objects that describe the current client request. filename is the name of a known file on the current machine which is being requested by the current request. info is the list of access information for this file.
If the filter returns nil then the normal operation is done by the directory-entity handler: the selected file is published and then the request to access it processed (and subsequent access using that url will just return the file and never go through the filter again).
If the filter chooses to handle the request for the file itself it must generate a response to the request and then return a non-nil value. To avoid subsequent calls to the filter for this file the filter may choose to publish a handler for this url. If the filter wants to forbid access to this file a handy way to to call (failed-request req) and the standard "404 Not found" will be sent back to the client.
The publisher argument can be used to specify exactly what happens when a request comes that's handled by the directory-entity and a file is located on the disk that matches the incoming url. Nomally a publish-file is done to add that file. You may want to publish some other kind of entity to represent that file. The publisher argument, if non-nil, must be a function of four arguments: req ent filename info. The filename is a string naming the file that's been matched with the request. info is the list of access information for this file. The publisher function must return an entity to be processed to send back a response. The publisher function may wish to publish that entity but it need not do so.
When files are accessed and automatically published you may wish to set some of the parameters of the entity that is published. As mentioned above you can define a publisher function that has complete control in publishing the entity. A less powerful but easier to use alternative is to place access files in the directory tree being published. An access file specifies information that you want passed to the publisher function. You can modify these access files while the directory tree is published and their latest values will be used for publishing subsequent files. This is similar to they way Apache controls its publishing with .htaccess files (except that in AllegroServe once a file is published the access files have no effect on it).
The name of an access file in AllegroServe is controlled by the :access-file argument to publish-directory. We'll assume the name chosen is access.cl in this document. If no :access-file argument is given to publish-directory then no access file checking is done. When a file is about to be published all access files from the destination directory all the way down to the directory containing the file to be published are read and used. For example if the destination in a publish-directory was given as "/home/joe/html/" and an http request comes in which references the file "/home/joe/html/pics/archive/foo.jpg" then AllegroServe will check for access files at all of these locations and in this order
The information is collected as successive access files are read. The new information is placed before the existing information thus causing subdirectory access files to possibly shadow information in access files in directories above it. Also superdirectory access file information is automatically eliminated if it isn't marked as being inherited.
The publisher function receives the collected information and can do with it what it wishes. We'll describe what the built-in publisher function does with the information.
When we speak of information in access files we are purposely being vague. We define what information must look like and what the standard publisher function does with certain information but we allow users to define their own kinds of information and use that in their own publisher function.
Each access file consists of zero or more Lisp forms (and possibly lisp style comments). Each form is a list beginning with a keyword symbol and then followed by a property-list-like sequence of keywords and values. Nothing in the form is evaluated. The form cannot contain #. or #,. macros.
One information form is used by AllegroServe's directory publisher code to decide if it's permitted to descend another directory level:
(:subdirectories :allow allow-list :deny deny-list :inherit inherit-value)
As AllegroServe descends from the destination directory toward the directory containing the file to be accessed it stops at each directory level accumlates the access information and then tests to see if it can descend further based on the :subdirectories information. If it cannot descend into the next subdirectory it gives up immediately and a 404 - Not Found response is returned. See the section Allow Deny processing below for a description of how it uses the :allow and :deny values.
These other information forms are used by the standard publisher function. Each takes an :inherit argument which defaults to false. Information not given with ::inherit t will be eliminated as AllegroServe descends directory levels.
| name | args | meaning |
|---|---|---|
| :ip | :patterns :inherit |
specifies a location-authorizer restriction on which machines can see published files. The value of the :patterns argument has the same form as the :patterns slot of a location-authorizer. |
| :password | :realm :allowed :inherit |
specifies a password-authorizer restriction on access to published files. See the password-authorizer documentation for a description of the :realm and :allowed arguments |
| :files | :allow :deny :inherit |
specifies which files are visible to be published. To be visible a file must be allowed and not denied. What is tested is the filename only (that is the part after the last directory separator in the files's complete name). See below for the rules on how allow and denied is used. |
| :mime | :types :inherit |
specifies which mime types are to be associated with which file types. This list takes precedence over the built-in list inside AllegroServe. :types is a list of mime specifiers. A mime specifier is a list beginning with a string giving the mime type followed by the files types that should map to that mime type. A file type in a list (e.g. ("ReadMe")) refers to the whole file name rather than the type component. |
The :files and :subdirectories information are used to determine if a file or subdirectory of a given name is accessible. AllegroServe will collect all the access file information for the directory containing the file or subdirectory and for all directories above it up to the directory given as the destination argument to publish-directory. Information from superdirectories will only be used :inherit t is given for that information.
The rules is it that a given name is accessible if it is allowed and not denied. That is the filename or directory name must match one of the allow clauses and none of the deny clauses. There may be multiple allow and deny clauses since there may be multiple information forms of the type :files or :subdirectories. Each allow or deny argument can be a string or a list of strings or nil (which is the same as that argument not being given). The strings are regular expressions (which are not exactly like unix shell wildcard filename expressions). In particular ".*" is the regular expression that matches anything.
The special cases are the following
.
Here is a sample access file:
; only connections to localhost will be able to access the
files
(:ip :patterns ((:accept "127.1") :deny) :inherit t)
(:password :realm "mysite"
:allowed (("joe" .
"mypassword")
("sam" . "secret"))
:inherit t) ; applies
to subdirectories
; publish html and cgi files, but not those beginning with a period
(:files :allow ("\\.html$" "\\.cgi$") :deny ("^\\."))
; specify mime type for non-standard file extensions. Also
; specify that a file named exactly ChangeLog should be given
; mime type "text/plain"
(:mime :types (("text/jil" "jil" "jlc")
("text/plain" "cl" ("ChangeLog"))))
(publish &key path host port content-type function class format remove server authorizer timeout plist)
This creates a mapping from a url to a computed-entity, that is an entity that computes its response every time a request comes in. The path, host, port, remove, server , authorizer and class arguments are as in the other publish functions. The timeout argument defaults to nil always. The content-type sets a default value for the response to the request but this can be overridden. The format argument is either :text (the default) or :binary and it specifies the kind of value that will be sent back (after the response headers, which are always in text). This value is only important if the response is generated in a particular way (described below).
The function argument is a function of two arguments: an object of class http-request that holds a description of the request, and an object of class entity that holds this entity which is handling the request. This function must generate a response to the http request, even if the response is only that the request wasn't found.
(publish-prefix
&key prefix host port content-type function class
format remove server authorizer timeout plist)
This is like publish except that it declares function to be the handler for all urls that begin with the string prefix. Note however that prefix handlers have lower priority than exact handlers. Thus if you declare a prefix handler for "/foo" and also a specific handler for "/foo/bar.html" then the specific handler will be chosen if "/foo/bar.html" is found in an http request. Typically a prefix handler is used to make available a whole directory of files since their complete names being with a common prefix (namely the directory in which the files are located). If you want to publish a whole directory then you probably want to use publish-directory since it has a number of features to support file publishing.
(publish-multi &key path host port content-type items class remove server authorizer timeout)
Some web pages are created from information from various sources. publish-multi allows you to specify a sequence of places that supply data for the combined web page. The data for each page is cached by publish-multi so that minimal conputation is required each time the page is requested.
The host, port, content-type, class, remove, server authorizer and timeout arguments are the same as those of the other publish functions. The items argument is unique to publish-multi and is a list of zero or more of the following objects
Here's an example where we create a page from a fixed header and trailer page with a bit of dynamic content in the middle.
(publish-multi :path "/thetime" :items (list "header.html" #'(lambda (req ent old-time old-val) (declare (ignore req ent old-time old-val)) (with-output-to-string (p) (html-stream p :br "The time is " (:princ (get-universal-time)) (:b "Lisp Universal Time") :br))) "footer.html"))
There are a variety of ways that a response can be sent back to the http client depending on whether keep-alive is being done, chunking is possible, whether the response is text or binary, whether the client already has the most recent data, and whether the size of the body of the response is known before the headers are sent. AllegroServe handles the complexity of determining the optimal response strategy and the user need only use a few specific macros in the computation of the response in order to take advantage of AllegroServe's strategy computation
Here's a very simple computed response. It just puts "Hello World!" in the browser window:
(publish :path "/hello"
:content-type "text/html"
:function #'(lambda (req ent)
(with-http-response (req ent)
(with-http-body (req ent)
(html "Hello World!")))))
This example works regardless of whether the request comes in from an old HTTP/0.9 browser or a modern HTTP/1.1 browser. It may or may not send the response back with Chunked transfer encoding and it may or may not keep the connection alive after sending back the response. The user code doesn't have to deal with those possibilities, it just uses with-http-response and with-http-body and the rest is automatic. The html macro is part of the htmlgen package that accompanies AllegroServe. In the case above we are being lazy and not putting out the html directives that should be found on every page of html since most browsers are accommodating. Here's the function that generates the correct html:
(publish :path "/hello2"
:content-type "text/html"
:function #'(lambda (req ent)
(with-http-response (req ent)
(with-http-body (req ent)
(html
(:html
(:body "Hello World!")))))))
The function above generates: <html><body>Hello World!</body></html>.
The macros and functions used in computing responses are these:
(with-http-response
(req ent &key timeout check-modified format response content-type)
&rest body)
This macro begins the process of generating a response to an http request and then runs the code in the body which will actually send out the response. req and ent are the request and entity objects passed into the function designated to compute the response for the request. timeout sets a time limit for the computation of the response. If timeout is nil then the entity ent is checked for a timeout value. If that value is also nil then the timeout value is retreived from the current wserver object using wserver-response-timeout. If check-modified is true (the default) then the last-modified time stored in the entity object will be compared against the if-modified-since time of the request and if that indicates that the client already has the latest copy of this entity then a not-modified response will be automatically returned to the client and the body of this macro will not be run. response is an object containing the code and description of the http response we wish to return. The default value is the value of *response-ok* (which has a code of 200 and a string descriptor "OK"). content-type is a string describing the MIME type of the body (if any) sent after the headers. It has a form like "text/html". If content-type isn't given here then the content-type value in the entity (which is set in the call to publish) will be used.
The format argument specifies whether the code that writes the body of the response will want to write :text (e.g. write-char) or :binary (e.g. write-byte) when it writes the data of the body of the response. Based on the value of the format argument, AllegroServe will create the correct kind of response stream. If format is not specified here it will default to the value specified when publish was called to create the entity. If not :format argument was passed to publish then :binary format is assumed. If :binary is specified then you can write both text and binary to the stream since Allegro's binary streams also support text calls as well. If you specify :text then you may end up with a stream that supports only text operations.
An http response consists of a line describing the response code, followed by headers (unless it's the HTTP/0.9 protocol in which case there are no headers), and then followed by the body (if any) of the response. with-http-response doesn't normally send anything to the client. It only does so when it determines that the if-modified-since predicate doesn't hold and that it must send back a not-modified response. Thus is not enough to just call with-http-response in your response function. You must always call with-http-body inside the call to with-http-response.
(with-http-body (req ent &key format headers external-format) &rest body)
This macro causes the whole http response to be sent out. The macro itself will send out everything except the body of the response. That is the responsibility of the code supplied as the body form of the macro. In cases where there is no body to the response being sent it is still necessary to call with-http-body so that the other parts of the response are sent out, e.g. at a minimum you should put (with-http-body (req ent)) in the body of a with-http-response.
The body forms may not be executed! If the request is an http head request then the browser wants only the headers returned. The with-http-body macro will not evaulate the body forms. You must be aware of this and should never put code in the body form that absolutely must be executed when a request is given.
The headers argument is a list of conses, where the car is the header name (a keyword symbol) and the cdr is the header value. These headers are added to the headers sent as part of this response.
Within the body forms the code calls (request-reply-stream req) to obtain a stream to which it can write to supply the body of the response. The external-format of this stream is set to the value of the external-format argument (which defaults to the value of *default-aserve-external-format*). The variable *html-stream* is bound to the value of (request-reply-stream req) before the body is evaluated. This makes it easy to use the html macro to generate html as part of the response.
Note: there used to be a :format argument to with-http-body. That argument was never used by with-http-body. The :format argument has been moved to with-http-response so that it can now have an effect on the stream created.
Return the body of the request as a string. If there is no body the return value will be an empty string. The result is cached inside the request object, so this function can be called more than once while processing a request. The typical reason for there to be a body to a request is when a web browser sends the result of a form with a POST method.
(header-slot-value request header-name)
Return the value given in the request for the given header-name (a keyword symbol). If the header wasn't present in this request then nil will be returned. header-slot-value is a macro that will expand into a fast accessor if the header-name is a constant naming a known header slot
In older versions of aserve the header-name was a string..
(reply-header-slot-value request header-name)
Return the value associated with the header header-name in the reply sent back to the client. This function is setf'able and this is the preferred way to specify headers and values to be sent with a reply.
(request-query request &key uri post external-format)
Decode and return an alist of the query values in the request. Each item in the alist is a cons where the car is a string giving the name of the argument and the cdr is a string giving the value of the argument.
The query string is in one or both of two places:
request-query will by default look in both locations for the query string and concatenate the results of decoding both query strings. If you would like it to not check one or both of the locations you can use the :uri and :post keyword arguments. If uri is true (and true is the default value) then the query string in the uri is checked. If post is true (and true is the default value) and if the request is a POST then the body of the post form will be decoded for query values.
The external-format is used in the conversion of bytes in the form to characters. It defaults to the value of *default-aserve-external-format*.
A query is normally a set of names and values.
http://foo.com/bar?a=3&b=4 yields a query alist (("a"
. "3") ("b" . "4")).
If a name doesn't have an associated value then the value in the alist is the empty
string.
http://foo.com/bar?a&b=&c=4 yields a query alist (("a"
. "") ("b" . "") (c . "4"))
.
(request-query-value key request &key uri post external-format test)
This combines a call to request-query to retrieve the alist of query keys and values, with a call to assoc to search for the specific key, and finally with a call to cdr to return just the value from the assoc list entry. The test argument is the function to be used to test the given key against the keys in the assoc list. It defaults to #'equal.
If the given key is not present in the query nil is returned. If the given key is present in the query but doesn't have an associated value then the empty string is returned.
The request object contains information about the http request being processed and it contains information about the response that is being computed and returned to the requestor. The following functions access slots of the request object. Those with names beginning with request-reply- are accessing the slots which hold information about the response to the request. When a function is listed as an accessor that means that it can be setf'ed as well as used to read the slot value.
(request-method request) - reader - a keyword symbol naming the kind of request, typically :get, :put or :post.
(request-uri request) - reader - a uri object describing the request. If the request contains a "Host:" header line then the value of this header is placed in the uri-host and uri-port slots of this uri object.
(request-protocol request) - reader - a keyword symbol naming the http protocol requested. It is either :http/0.9, :http/1.0 or :http/1.1.
(request-protocol-string request) - reader - a string naming the http protocol requested. It is either "HTTP/0.9", "HTTP/1.0" or "HTTP/1.1".
(request-socket request) - reader - the socket object through which the request was made and to which the response must be sent. This object can be used to determine the IP address of the requestor.
(request-wserver request) - reader - the wserver object describing the web server taking this request
(request-raw-request request) - reader - a string holding the exact request made by the client
(request-reply-code request) - accessor - the value describes the response code and string we will return for this request. See the value of the argument response in with-http-response for more information.
(request-reply-date request) - accessor - the date the response will be made (in Lisp's universal time format). This defaults to the time when the request arrived.
(request-reply-headers request) - accessor - an alist of some of the headers to send out with the reply (other headers values are stored in specific slots of the request object). Each entry in the alist is a cons where the car is a keyword symbol holding the header name and the cdr is the value (it is printed using the ~a format directive). Typically request-reply-headers isn't used, instead the headers to be sent are passed as the :header argument to with-http-body, or (setf reply-header-slot-value) is called.
(request-reply-content-length request) - accessor - the value to send as the Content-Length of this response. This is computed automatically by AllegroServe and thus a user program shouldn't have to set this slot under normal circumstances.
(request-reply-plist request) - accessor - this slot holds a property list on which AllegroServe uses to store less important information. The user program can use it as well.
(request-reply-strategy request) - accessor - the strategy is a list of symbols which describe how AllegroServe will build a response stream and will send back a response. More details will be given about the possible strategies at a future time.
(request-reply-stream request) - accessor - This is the stream to be used in user code to send back the body of the response. This stream must be used instead of the value of request-socket.
The Common Gateway Interface (CGI) specification allows web servers to run programs in response to http requests and to send the results of the execution of those programs back the web client. The CGI programs finds information about the request in its environment variables and, in the case of a put or post request, the body of the request is sent to standard input of the program.
CGI is a clumsy and slow protocol for extending the behavior or a web server and is falling out of favor. However there are legacy CGI applications you may need to call from AllegroServe. You invoke an external program using the CGI protocol with the run-cgi-program function.
(run-cgi-program req
ent program &key path-info path-translated
script-name query-string
auth-type timeout error-output env)
In response to an http request, this runs program which must be a string naming an exectuable program or script followed optionally by command line arguments to pass to that program. Before the program is run the environment variables are set according the the CGI protocol. The timeout argument is how long AllegroServe should wait for a response from the program before giving up. The default is 200 seconds. The error-output argument specifies what should be done with data the cgi program sends to its standard error. This is described in detail below. The other keyword arguments allow the caller to specify values for the CGI environment variables that can't be computed automatically. path-info specifies the PATH_INFO environment variable, and similarly for path-translated, script-name, query-string and auth-type. If query-string is not given and the uri that invoked this request contains a query part then that query part is passed in the QUERY_STRING environment variable. If script-name is not given then its value defaults to the path of the uri of the request. If you wish to add or modify the environment variables set for the cgi process you can specify a value for env. The value of env should be a list of conses, the car of each cons containing the environment variable name (a string) and the cdr of each cons containing the environment variable value (a string). env is checked after all the standard environment variables are computed and the value given in env will override the value computed automatically.
cgi programs send their result to standard output (file descriptor 1 on Unix). If they encounter problems they often send informative messages to standard error (file descriptor 2 on Unix). The error-output argument to run-cgi-program allows the caller to specify what happens to data sent to standard error. The possibile values for error-output are:
| nil | The cgi program's standard error is made the same as the Lisp process' standard error. This standard error may not be the same as the current binding of *standard-error*. |
| pathname or string | A file with the given name is opened and standard error is directed to that file. |
| :output | Standard error is directed to the same place as standard output thus the error messages will be mixed into the result of running the cgi program. |
| symbol or function | The function is run whenever there is data available to be read from standard error. It must read that data. It must return a true value if it detected an end of file during the read and nil otherwise. The function takes arguments: req ent stream |
A typical way of publishing a CGI page is this:
(publish :path "/cgi/myprog"
:function #'(lambda (req ent)
(run-cgi-program req ent "/server/cgi-bin/myprog")))
If you're concerned about capturing the error output then here's an example where we supply a function to collect all the error output into a string. Once collected we simply print it out here but in a real web server you would want to store it in a log file.
(defun cgierr (req ent)
(let ((error-buffer (make-array 10
:element-type 'character
:adjustable t
:fill-pointer 0)))
(net.aserve:run-cgi-program
req ent
"aserve/examples/cgitest.sh 4"
:error-output
#'(lambda (req ent stream)
(declare (ignore req ent))
(let (eof)
(loop
(let ((ch (read-char-no-hang stream nil :eof)))
(if* (null ch) then (return))
(if* (eq :eof ch)
then (setq eof t)
(return))
(vector-push-extend ch error-buffer)))
eof
)))
(format t "error buffer is ~s~%" error-buffer)
))
Note: The ability to run CGI programs from AllegroServe was due to features added in Allegro Common Lisp version 6.1. This will not work in earlier versions of Allegro CL.
Forms are used on web pages in order to allow the user to send information to the web server. A form consists of a number of objects, such as text fields, file fields, check boxes and radio buttons. Each field has a name. When the user takes a certain action, the form data is encoded and sent to the web server. There are three ways that data can be sent to the web server. The method used is determined by the attributes of the <form> tag that defines the form
If you create a form with <form method="post" enctype="multipart/form-data"> then your url handler must do the following to retrieve the value of each field in the form:
It's important to retrieve all of the data sent with the form, even if that data is just ignored. This is because there may be another http request following this one and it's important to advance to the beginning of that request so that it is properly recognized.
Details on the functions are given next.
(get-multipart-header request)
This returns nil or the MIME headers for the next form field in alist form. If nil is returned then there is no more form data. See parse-multipart-header for a simple way to extratacting information from the header.
For an input field such as <input type="text" name="textthing"> the value returned by get-multipart-header would be
((:content-disposition
(:param "form-data" ("name" . "textthing"))))
For an input field such as <input type="file" name="thefile"> the value returned by get-multipart-header would be something like
((:content-disposition
(:param "form-data" ("name" . "thefile")
("filename" . "C://down//550mhz.gif")))
(:content-type "image/gif"))
Note that the filename is expressed in the syntax of the operating system on which the web browser is running. This syntax may or may not make sense to the Lisp pathname functions of the AllegroServe web server as it may be running on a totally different operating system.
(parse-multipart-header header)
This take the value of get-multipart-header and returns values that describe the important information in the header.
The first value returned is
(get-multipart-sequence request buffer &key start end external-format)
This retrieves the next chunk of data for the current form field and stores it in buffer. If start is given then it specifies the index in the buffer at which to begin storing the data. If end is given then it specifies the index just after the last index in which to store data.
The return value is nil if there is no more data to return, otherwise it is the index one after the last index filled with data in buffer.
The buffer can be a one dimensional array of character or of (unsigned-byte 8). For the most efficient transfer of data from the browser to AllegroServe, the program should use a 4096 byte (unsigned-byte 8) array.
If the buffer is a character array then the data is converted from get-multipart-sequence's (unsigned-byte 8) array to characters using the given external-format (which defaults to the value of *default-aserve-external-format*).
get-multipart-sequence may return before filling up the whole buffer, so the program should be sure to make use of the index returned by get-multipart-sequence.
(get-all-multipart-data request &key type size external-format limit)
This retrieves the complete data object following the last multipart header. It returns it as a lisp object. If type is :text (the default) then the result is returned as a lisp string. If type is :binary then the result is returned as an array of element-type (unsigned-byte 8). size (which defaults to 4096) is the size of the internal buffers used by this function to retrieve the data. You usually won't need to specify a value for this but but if you know the values retrieved are either very small or very large it may may the operation run faster to specify an appropriate size. external-format is used when type is :text to convert the octet stream into characters. It defaults to the value of *default-aserve-external-format*. limit can be given an integer value that specifies the maximum size of data you're willing to retrieve. By default there is no limit. This can be dangerous as a user may try to upload a huge data file which will take up so much Lisp heap space that it takes down the server. If a limit is given and that limit is reached, get-all-multipart-data will continue to read the data from the client until it reaches the end of the data, however it will not save it and will return the symbol :limit to indicate that the data being send to the sever exceeded the limit. It will return a second value which is the size of the data the client tried to upload to the server. If your application intends to handle very large amounts of data being uploaded to the server you would be better off using get-multipart-sequence since with that you can write the data buffer by buffer to the disk instead of storing it in the Lisp heap.
In AllegroServe the information sent to the web server as a result of filling out a form is called a query. We store a query as a list of conses, where the car of the cons is the name (a string) and the cdr of the cons is the value (another string). When a query is transmitted by the web browser to AllegroServe it is sent as string using the encoding application/x-www-form-urlencoded. We provide the following functions to convert between the encoding and the query list:
(form-urlencoded-to-query string &key external-format)
Decodes the string and returns the query list. The default value for external-format is the value of *default-aserve-external-format*.
(query-to-form-urlencoded query &key external-format)
Encodes the query and returns a string. The default value for external-format is the value of *default-aserve-external-format*.
Examples:
user(4): (query-to-form-urlencoded '(("first name" . "joe")
("last name" . "smith")))
"first+name=joe&last+name=smith"
user(5): (form-urlencoded-to-query "first+name=joe&last+name=smith")
(("first name" . "joe") ("last name" . "smith"))
user(6): (query-to-form-urlencoded
`(("last name" . ,(coerce '(#\hiragana_letter_ta
#\hiragana_letter_na
#\hiragana_letter_ka)
'string)))
:external-format :euc)
"last+name=%a4%bf%a4%ca%a4%ab"
user(7): (query-to-form-urlencoded
`(("last name" . ,(coerce '(#\hiragana_letter_ta
#\hiragana_letter_na
#\hiragana_letter_ka)
'string)))
:external-format :shiftjis)
"last+name=%82%bd%82%c8%82%a9"
user(8): (coerce
(cdr
(assoc "last name"
(form-urlencoded-to-query "last+name=%82%bd%82%c8%82%a9"
:external-format :shiftjis)
:test #'equalp))
'list)
(#\hiragana_letter_ta #\hiragana_letter_na #\hiragana_letter_ka)
You may want to restrict certain entities to be accessible from only certain machines or people. You can put the test for authorization in the entity response function using one of the following functions, or you can have the check done automatically by storing an authorizer object in the entity.
(get-basic-authorization request)
This function retrieves the Basic authorization information associated with this request, if any. The two returned values are the name and password, both strings. If there is no Basic authorization information with this request, nil is returned.
(set-basic-authorization request realm)
This adds a header line that requests Basic authorization in the given realm (a string). This should be called between with-http-response and with-http-body and only for response of type 401 (i.e. *response-unauthorized*). The realm is an identifier, unique on this site, for the set of pages for which access should be authorized by a certain name and password.
This example manually tests for basic authorization where the name is foo and the password is bar.
(publish :path "/secret"
:content-type "text/html"
:function
#'(lambda (req ent)
(multiple-value-bind (name password) (get-basic-authorization req)
(if* (and (equal name "foo") (equal password "bar"))
then (with-http-response (req ent)
(with-http-body (req ent)
(html (:head (:title "Secret page"))
(:body "You made it to the secret page"))))
else ; this will cause browser to put up a name/password dialog
(with-http-response (req ent :response *response-unauthorized*)
(set-basic-authorization req "secretserver")
(with-http-body (req ent)))))))
If an entity has an associated authorizer object, then before that entity's response function is run the authorizer is tested to see if it will accept or deny the current request. AllegroServe supplies three interesting subclasses of authorizer and users are free to add their own subclasses to support their own authorization needs.
The protocol followed during authorization is this:
This subclass of authorizer is useful if you want to protect an entity using the basic authorization scheme that asks for a name and a password. When you create this class of object you should supply values for the two slots:
| Slot Name | initarg | what |
| allowed | :allowed | list of conses, each cons having the form ("name" . "password") where any of the listed name password pairs will allow access to this page. |
| realm | :realm | A string which names the protection space for the given name and password. The realm will appear in the dialog box the browser displays when asking for a name and password. |
An example of it's use is the following where we allow access only if the user enters a name of joe and a password of eoj or a name of fred and a password of derf.
(publish :path "/foo"
:content-type "text/html"
:authorizer (make-instance 'password-authorizer
:allowed '(("joe" . "eoj")
("fred" . "derf"))
:realm "SecretAuth")
:function
#'(lambda (req ent)
(with-http-response (req ent)
(with-http-body (req ent)
(html (:head (:title "Secret page"))
(:body "You made it to the secret page"))))))
This authorizer class checks the IP address of the request to see if it is permitted access to the entity. The authorizer can specify a sequence of patterns and for each pattern a command of :accept (permit the access) or :deny (forbid the access). The first pattern that matches determines if the request is accepted or denied. If the pattern list is empty or if no pattern matches, then the request is accepted.
The single slot of an object of class location-authorizer is
| Slot Name | initarg | what |
| patterns | :patterns | a list of patterns and commands, where the syntax of a pattern-command is described below. |
A pattern can be
The example of using a location-authorizer only permits connections coming in via the loopback network (which occurs if you specify http://localhost/whatever) or if they come from one particular machine (tiger.franz.com). Note that we end the pattern list with :deny so that anything not matching the preceding patterns will be denied.
(publish :path "/local-secret-auth"
:content-type "text/html"
:authorizer (make-instance 'location-authorizer
:patterns '((:accept "127.0" 8)
(:accept "tiger.franz.com")
:deny))
:function
#'(lambda (req ent)
(with-http-response (req ent)
(with-http-body (req ent)
(html (:head (:title "Secret page"))
(:body (:b "Congratulations. ")
"You made it to the secret page"))))))
This authorizer contains a function provided by the user which is used to test if the request is authorized. The function take three arguments, the http-request object, the entity and the authorizer object. It must return one of the four value that the authorize function returns, namely t, nil :deny or :done.
A function-authorizer is created as follows
(make-instance 'function-authorizer
:function #'(lambda (req ent auth)
t ; always authorize
))
The function slot can be set using (setf function-authorizer-function) if you wish to change it after the authorizer has been created.
Cookies are name value pairs that a web server can direct a web browser to save and then pass back to the web server under certain circumstances. Some users configure their web browsers to reject cookies, thus you are advised against building a site that depends on cookies to work.
Each cookie has these components:
(set-cookie-header request &key name value expires domain path secure encode-value external-format)
This function should be called between the calls to with-http-response and with-http-body. It can be called more than once. Each call will cause one Set-Cookie directive to be sent to the web browser. The name and value arguments should be given (and they should be strings). They will be automatically encoded using the same encoding used in urls (we call it uriencoding). The purpose of this encoding is to convert characters that are either unprintable or those that have a special meaning into a printable string. The web browser doesn't care about the name and value, it just stores them and sends them back to the web server. If you use the get-cookie-values function to retrieve the cookie name and value pairs, then it will automatically decode the uriencoding.
You can disable the encoding of the value by specifying a nil value to
encode-value. This should only be necessary if you are
working with buggy http client applications.
If the path argument isn't given, it will default to "/" which
will allow this cookie to match all requests.
If the domain argument isn't given then it will default to the host to
which this request was sent. If you wish to specify this you are only allowed to
specify a subsequence of the host to which this request was sent (i.e the name of the
machine running the webserver). The domain should have at
least two periods in it (i.e. ".foo.com").
expires can be a lisp universal time or it can be the symbol :never
meaning this should never expire. If expires isn't given or is nil
then this cookie will expire when the user quits their web browser.
secure should be true or false. Any non-nil value is interpreted as
true. The default value is false.
The external-format is used to convert bytes to characters. It
defaults to the value of *default-aserve-external-format*.
(get-cookie-values request &key external-format)
Return the cookie name and value pairs from the header of the request. Each name value pair will be in a cons whose car is the name and whose cdr is the value. The names and values will be decoded (in other words the decoding done by set-cookie-header will be undone). The external-format is used to convert bytes to characters. It defaults to the value of *default-aserve-external-format*.
These special variables contain information about AllegroServe or help control AllegroServe:
*aserve-version* - a list of three values: (major-version minor-version sub-minor-version) which is usually printed with periods separating the values (i.e. X.Y.Z).
*default-aserve-external-format* - a symbol or external format object which is the default value for those AllegroServe functions that take an external-format argument. http requests are normally run in separate lisp threads and those threads bind *default-aserve-external-format* to the value of the external-format argument to the start function. Thus changing the value of *default-aserve-external-format* in one thread will not affect its value in other threads. You should decide the default external format before you start AllegroServe running.
*http-response-timeout* - the default value for the timeout argument to with-http-response. [in future versions of AllegroServe we'll treat this value like *default-aserve-external-format* and bind it in each worker thread]
*mime-types* - a hash table where the keys are the file types (e.g. "jpg") and the values are the MIME types (e.g. "image/jpeg").
We'll describe here the steps AllegroServe goes through from the time it receives a request until a response to that request has been sent back to the browser. We want the protocol to be open so that users can extend AllegroServe's behavior to suit their needs. However given that AllegroServe is a new program and will be undergoing extensive review from its users, we expect that the protocol will change. It shouldn't lose any of its current extensibility but the names and argument lists of generic functions may change.
When a client connects to the port on which AllegroServe is listening, AllegroServe passes that connected socket to a free worker thread which then wakes up and calls the internal function net.aserve::process-connection. If there are no free worker threads then AllegroServe waits for one to be available.
In each worker thread the variable *wserver* is bound to the wserver object that holds all the information about the webserver on which the connection was made (remember that one AllegroServe process can be running more than one webserver). process-connection reads the request from the socket (but doesn't read past the header lines). If the request can't be read within *read-request-timeout* seconds (currently 20) then the request is rejected. The request is stored in an object of class http-request. Next process-connection calls handle-request to do all the work of the request and then log-request to log the action of the request. Finally if the response to the request indicated that the connection was to be kept open rather than being closed after the response, then process-connection loops back to the top to read the next request.
(handle-request (req http-request)) [generic function]
This generic function must locate the entity to handle this request and then cause it to respond to the request. If there is no matching entity then handle-request must send a response back to the client itself. handle-request uses locators to find the entity (more on this below), and then if an entity is found and that entity has an authorizer, it calls authorize to see if this request is allowed to access the selected entity. If the entity passes the authorization then process-entity is called to cause the entity to respond to the request. process-entity returns true if it processed the entity, and nil if did not in which case the search continues for an entity. If there is no entity to respond then failed-request is called to send back a failure message.
A locator is an object used to map requests into entities. The value of (wserver-locators *wserver*) is a list of locator objects. handle-request calls
(standard-locator (req http-request) (loc locator)) [generic function]
on each successive locator in that list until one returns an entity object. AllegroServe has two built-in locator classes, locator-exact and locator-prefix, that are subclasses of locator. When you call publish or publish-file you are adding the entity to locator of class locator-exact found in the wserver-locators list. When you call publish-directory you are adding to the locator of class locator-prefix. Users are free to define new locator classes. Locators should define the standard-locator method as well as
(unpublish-locator (loc locator)) [generic function]
which if called should remove all published entities from the locator.
Let's return to handle-request. It has called standard-locator and found an entity. Next it checks to see if the entity has an authorizer value and if so calls
(authorize (auth authorizer) (req http-request) (ent entity)) [generic function]
The return value will be one of
If there is no authorizer for this entity then we just call process-entity. If there is no entity, then we call failed-request.
(failed-request (req http-request)) [generic function]
send back a response to the effect that the url request doesn't exist on this server.
(denied-request (req http-request)) [generic function]
send back a response to the effect that access to the requested url was denied.
(process-entity (req http-request) (ent entity)) [generic function]
Send back a response appropriate to the given entity. The macros with-http-response and with-http-body should be used in the code that sends the response.
AllegroServe has a set of functions that perform http client-side actions. These functions are useful in generating computed pages that reflect the contents of other pages. We also use the client-side http functions to test AllegroServe.
The client-side functions described in this section are exported from the net.aserve.client package.
The function do-http-request sends a request and retrieves the whole response. This is the most convenient function to use to retrieve a web page.
If you need more control over the process you can use the functions: make-http-request, read-client-response-headers and client-request-read-sequence.
(do-http-request uri
&key method protocol accept
content content-type query format cookies
redirect redirect-methods basic-authorization
keep-alive headers proxy user-agent external-format ssl
skip-body)
Sends a request to uri and returns four values:
The uri can be a uri object or a string. The scheme of the uri must be nil or "http". The keyword arguments to do-http-request are
| Name | default | description |
|---|---|---|
| method | :get | The type of request to make. Other possible values are :put, :post and :head. :head is useful if you just want to see if the link works without downloading the data. |
| protocol | :http/1.1 | The other possible value is :http/1.0. Modern web servers will return the response body in chunks if told to use the :http/1.1 protocol. Buggy web servers may do chunking incorrectly (even Apache has bugs in this regard but we've worked around them). If you have trouble talking to a web server you should try specifying the :http/1.0 protocol to see if that works. |
| accept | "*/*" | A string listing of MIME types that are acceptable as a response to this request. The type listed can be simple such as "text/html" or more complex like "text/html, audio/*" The default is to accept anything which is expressed "*/*". |
| content | nil | If the method is :put or :post then the request should include something to be sent to the web server. The value of this argument is either a string or a vector of type (unsigned-byte 8) which will be sent to the web server. It may also be a list of strings or vectors. See the query argument for a more convenient way to :post data to a form. |
| content-type | nil | A string which is to be the value of the Content-Type header field, describing the format of the value of the content argument. This is only needed for :put and :post requests. |
| query | nil | This is a query alist of the form suitable for query-to-form-urlencoded. If the method is a :get then the value of this argument is urlencoded and made the query string of the uri being accessed. If the method is :post then the query string is urlencoded and made the content of the request. Also the content-type is set to application/x-www-form-urlencoded. |
| format | :text | The body of the response is returned as a string if the value is :text or as an array of type (unsigned-byte 8) if the value is :binary. When the body is a string the external-format argument is important. |
| cookies | nil | If you wish the request to include applicable cookies and for returned cookies to be saved, then a cookie-jar object should be passed as the value of this argument. |
| redirect | 5 | If the response is a redirect (code 301, 302, 303), and the method is one given by the value of redirect-methods then if this argument is true (and, if an integer, positive), do-http-request will call itself to access the page to which the redirection is pointed. If redirect is an integer then in the recursive call the value passed for redirect will be one less than the current value. This prevents infinite recursion due to redirection loops. |
| redirect-methods | (:get :head) | List of http methods which will be redirected if redirect is true. |
| basic-authorization | nil | If given, it is a cons whose car is the name and whose cdr is the password to be used to get authorization to access this page. |
| keep-alive | nil | If true then the web server will be told to keep the connection alive. Since do-http-request closes the connection after the request this option currently does no more than allow us to experiment with how a web server responds to a keep-alive request. |
| headers | nil | an alist of conses ("header-name" . "header-value") for additional headers to send with the request. |
| proxy | nil | the name and optionally the port number of a proxying web server through which this request should be made. The form is of the argument is "www.machine.com" or "www.machine.com:8000" if the web server is listening on port 8000 rather than 80. Proxying web servers are often used when clients are behind firewalls that prevent direct access to the internet. Another use is to centralize the page cache for a group of clients. |
| user-agent | nil | If given it specifies the value of the User-Agent header to be sent with the request. Some sites respond differently based on the user-agent they believe has made the request. The lack of a User-Agent header may cause a server to ignore a request since it believes that it is being probed by a robot. The value of user-agent can be a string or one of the keywords :aserve, :netscape or :ie in which case an appropriate user agent string is sent. |
| external-format | the value of *default-aserve-external-format* | This determines the socket stream's external format. |
| ssl | nil | If true then the connection is made using the Secure Sockets Layer protocol. If the uri uses the https scheme then ssl is assumed to be true and the ssl argument need not be specified. |
| skip-body | nil | If the value is a fucntion (satisifies functionp) then the value is funcalled passing the client-request object as an argument. At this point the client-request object contains the information on the headers of the response. The function should return true if the body of the response should be skipped and nil returned as the first value from do-http-request. If skip-body is not a function then if its value is true then reading the body is skipped and nil returned in its place. |
For example:
user(5): (do-http-request "http://www.franz.com")
"<HTML>
<HEAD>
<TITLE>Franz Inc: Allegro Common Lisp and Common Lisp Products</TITLE>
<BASE FONTFACE=\"helvetica, arial\" FONTSIZE=\"1\">
.....
"
200
(("content-type" . "text/html") ("transfer-encoding" . "chunked")
("server" . "Apache/1.3.9 (Unix) PHP/3.0.14")
("date" . "Mon, 24 Apr 2000 11:00:51 GMT"))
It's easy to use do-http-request to fill in form objects on a page. If the form has input elements named width and height then you can send a request that specifies that information in this way:
(do-http-request "http://www.foo.com/myform.html"
:query '(("width" . 23) ("height" . 45)))
The above assumes that the method on the form is "GET". If the method is "POST" then a similar call will work:
(do-http-request "http://www.foo.com/myform.html" :method :post
:query '(("width" . 23) ("height" . 45)))
Before we describe the lower level client request functions we will describe two classes of objects used in that interface.
A client-request object includes the information about the request and the response.
The public fields of a client-request that are filled in after a call to make-http-client-request are:
| Accessor | Description |
|---|---|
| client-request-uri | uri object corresponding to this request |
| client-request-socket | socket object open to the web server denoted by the uri |
| client-request-cookies | the cookie-jar object (if any) passed in with this request. |
After read-client-response-headers is called, the following fields of the client-request objects are set:
| Accessor | Description |
|---|---|
| client-request-response-code | the integer that is the response code for this request. The most common codes are 200 for Success and 404 for Not Found. |
| client-request-headers | an alist of header values in the response. Each entry is a cons of the form ("header-name" . "header-value"). The header names are all lower case. |
| client-request-protocol | A keyword symbol naming the protocol that the web server returned (which may be different that the protocol given in the request). A typical return value is :http/1.1 |
| client-request-response-comment | A string giving a textual version of the response code. The string is arbitrary and you should not depend on all web servers returning the same string for any given response code. |
A cookie-jar is a respository for cookies. Cookies are stored in a jar when a response from a client request includes Set-Cookie headers. Cookies from a jar are sent along with a request when they are applicable to the given request. We won't describe the rules for cookie applicability here, you need only know that if you use our client functions to access a site that uses cookies to implement persistence, then you should create a cookie-jar object and pass that same object in with each request. More information on cookies can be found here.
A cookie-jar is created with (make-instance 'cookie-jar).
(cookie-jar-items cookie-jar)
returns an alist of the cookies in the jar where each item has the form:
(hostname cookie-item ...)
The hostname is a string which is matched against the suffix of the name of the host in the request (that is, a hostname of ".foo.com" matches "a.foo.com" and "b.foo.com". ). The hostname should have at least two periods in it. The following cookie-item objects in the list all apply to that hostname. A cookie-item is a defstruct object and has these fields
| Accessor | Description |
|---|---|
| cookie-item-path | A string that must be the prefix of the path of the request for it to match. The prefix "/" matches all paths. |
| cookie-item-name | The name of the cookie. A string. |
| cookie-item-value | The value of the cookie. A string. |
| cookie-item-expires | A string holding the time the cookie expires [in a future release we may make this a universal time] |
| cookie-item-secure | true if this cookie should only be sent over a secure connection. |
(make-http-client-request
uri &key method protocol keep-alive
accept cookies headers proxy
basic-authorization query
content content-type content-length
user-agent external-format ssl)
This function connects to the web server indicated by the uri and sends the request. The arguments are the same as those for do-http-request and are documented there. There is one additional argument: content-length. This argument can be used to set the content-length header value in the request. After setting the content-length the caller of make-http-client-request would then be responsible for sending that many bytes of data to the socket to serve as the body of the request. If content-length is given, then a value for content should not be given.
If make-http-client-request succeeds in contacting the web server and sending a request, a client-request object is returned. If make-http-client-request fails, then an error is signalled.
The returned client-request object contains an open socket to a web server, thus you must ensure that client-request object isn't discarded before client-request-close is called on it to close the socket and reclaim that resource.
After calling make-http-client-request the program will send the body of the request (if any), and then it will call read-client-response-headers to partially read the web server's response to the request.
The default value for external-format is the value of *default-aserve-external-format*
(read-client-response-headers client-request)
This function reads the response code and response headers from the web server. After the function returns the program can use the client-request accessors not