W3Olista only runs in multi-tasking Unix environments. I don't know yet
whether this includes Windows NT, but certainly excludes DOS
On this machine, the following conditions must be met:
- The closer your operating system is to Posix, the better.
Certain assumptions are made by the software, if you know about
your system, you can verify them below.
- You must have an Ansi-compatible C compiler installed. The
program doesn't compile with pre-Ansi compilers. A bad example is
the default SunOS 4.x compiler (the one that comes with
SunWorks is OK). If you don't have an Ansi compiler yet,
hurry and install the GNU C compiler, GCC, which works
- Your Web server must produce logfiles in the Common logfile
format, which is probably the default on your system. If it isn't,
you can still write your own
function, if you know how to.
of the Common format.
These are some assumtions that must be met to run the program. If you
don't know too much of C and Unix, they'll probably be of no interest
- The '/' must be your directory separator.
- popen() and pclose() must be present.
- time_t is an integer value; the substraction of two time_t
values must yield the time difference in seconds, and adding or
substracting integer values (seconds) must be allowed and result
in another correct time_t value.
- The include file dirent.h is needed, and the functions
must comply to the Posix standard. Most notably, the field
d_name of the dirent structure must be a null-terminated
- The include files unistd.h, string.h and
stdlib.h are present.
If you have daily or monthly logfiles, each logfile has, of course,
a different filename that reflects the period of time of logging.
W3Olista uses this information from the filename to check if the
data inside a file is needed for a report, and based on this information,
it may decide not to read the file. For example, if you want statistics
of Yesterday, it would be quite useless to scan last month's files. This
procedure saves an enormous amount of time.
In order for this to work, you must use the logfile pattern to describe
how each filename is constructed from a date, so that W3Olista can
reverse-engineer the date from a filename. This is done similar to
the C function strftime(), where you can give date formatting
directives to fill a string with date information. You construct your
logfile pattern as follows:
Ordinary characters in your pattern must exactly match the same position in
the filename. Formatting specifiers begin with a percent sign '%', followed
by one of the following characters:
You can also use the dot '.' to step over changing arbitrary information
in the filename, like the day of the week or other goodies. This way the
dot has two meanings (as the literal dot, and as a wildcard character), but
this shouldn't be a problem.
- The abbreviated english month name, exactly three
characters long. The first character may or may not be uppercase.
- The full english month name. The first character may
or may not be uppercase.
- The two-digit day of the month, ranging from 00 to 31.
- The two-digit month of the year, ranging from 01 to 12.
Note: Month information as one digit is accepted if followed by
a non-digit character, although use of this policy is not encouraged.
Better, rename your files.
- Two digits for the number of years since 1900. If this
number is lower than 60, a year after 2000 is assumed.
- The year as four digits.
- The percent sign.
The compression of logfiles is automatically detected upon its suffix. You
must not give this suffix to the pattern. See the below examples, they
explain much better how this is done.
A typical logfile on our site has the name
httpd.zeus.access.Apr0195 (zeus is the host name of our Web server).
Now the correct logfile pattern for this is httpd.zeus.access.%b%d%y,
upon which the program can correctly detect that this logfile is from the
first of April in 1995. Here are more examples:
Logfile Name Logfile Pattern Used Information
access.04.01.1995 access.%d.%m.%Y April 1, 1995
access.March1995 access.%B%Y March 1 to March 31 1995
access.SunApr011995 access....%d%m%Y April 1, 1995
access.0395.Z access.%m%y March 1995, compressed
access.0395.gz access.%m%y March 1995, gzip'ped
Multiple Logfile Patterns
On our site, we have the policy that daily logs are kept for the current
month. At the end of the month, the daily logs are pasted together and
compressed with gzip. These different filename formats can be used by giving
multiple filename patterns separated by a semicolon.
Thus, on our site, I use
httpd.zeus.access.%b%d%y;httpd.zeus.access.%m.%y. The first pattern
reads the daily logs for the current month, and the second pattern reads the
compressed monthly files.
Note that these different filename patterns must not overlap. Imagine the
following scenario: Similar to above, daily logs for the current month are
kept. But instead of pasting them together at the end of the month, the
daily logs are added at the end of each day. In this case, the data for the
current month, except today, exist twofold, once in the daily file and once
in the compressed monthly log. W3Olista cannot detect this 'mistake' and
would doubly report all the data.
Custom Headers and Footers must contain several required elements of HTML,
so that the final document appears stylishly correct.
The Header should at least contain, for good style:
Besides, you can mention the purpose of the page and some introducing
- The tags <html><head>
- A title as in <title>This is a title</title>
- the end-marker of the HTML header, and the beginning tag of
the body </head><body>
- a caption <h1>Statistics from our site</h1>
Defines a filename to print as a footer below the report. The filename
should be given as absolute filename. The file should at least contain
It is good style to also mention the page's maintainer.
- The end marker of the document </body> </html>
Splitlog is a simple shell script that reads in a big logfile and creates
the logs for individual days. There are two cases when you want to use this
program, first, if you want to speed up report processing, and second, if
you can't get timezones to work on your system. Let me explain these two
- If you keep daily logfiles, W3Olista can select which files to use,
and which not to use. For example, if you want statistics for the
last week, then it's pointless to let the program browse through
last year's data. With daily logs, only exact seven files are read.
- The program can either report dates and times in local time or
corrected to GMT. If you want them to be reported in local time,
daily logfiles have to begin at midnight and run through 23:59:59.
If the logfile has an offset to midnight, e.g. it runs from 02:00:00
to 01:59:59 on our site, the times must be corrected to GMT (since
2 o'clock local time is midnight GMT here). Now, this time correction
requires that your operating system supports time zones, and that they
are correctly setup. You can detect this correct setup by running
a report on yesterday, printing a detailed date listing (split up by
the hour). If each hour from 00 to 23 gets reported, then you're fine.
If some times don't get covered, then the time conversion failed.
Splitlog will then help you to reorganize the logs.
Usage of Splitlog
splitlog prefix logfiles [logfiles ...]
- A filename prefix to be given to the
created files. A string of MMMDDYYY will automatically
be appended to this pattern.
- A single or multiple logfiles to be
read in and split up.
splitlog access. big_logfile
reads in the data from big_logfile and produces files like
access.Apr0195 and so on.
Correcting the Setup of W3Olista
After splitting up the logfile, you must patch the configuration of the
- Set the new logfile pattern in the Makefile.
- With the files produced by splitlog, you must configure the
program to use localtime. To do so, comment out the GMT
setting in the Makefile.
- Recompile the program with the
make new command.
This ensures that all files are correctly rebuilt.
If your server does not use the Common logfile format for
logging access, you have to write your own scanning function. If you
know C, then it should be quite easy and straightforward.
fileio.c. You must rewrite the function
scanstr, which is responsible for splitting up an input
string into separate parts. It gets one line of input,
and must fill the following output variables.
- hostname the fully qualified hostname (or IP address).
May not contain any whitespaces. May not be longer than 256 chars.
- filename the full filename of the request. CGI input
(after a '?') must be truncated. Must contain an initial slash,
and may not contain any whitespaces. Must be shorter than 512 chars.
- when The date and time of the request. if GMT is not
defined, you must return the date and time exactly as they are in the
logfile. If GMT is defined, then this date and time must be
corrected to GMT. The output string must be in the internal format
"YYY MMM DD HH mm ss". The month must be the english three-character
abbreviation. All other values are numerical. It is recommended that
days, hours, minutes and seconds are given as two digits each (values
lower than ten with an additional zero). Minutes and seconds are not
required and may be omitted.
- status Integer value. The status of the request. As for now, all
requests that produced a status other than decimal 200 are ignored.
So if there are no status codes in your logs, set this status to 200.
- size Integer value. The number of bytes that were sent to the
The function returns 0 on failure (unexpected end of string or wrong
input format) or 1 on success. If you have successfully written your
own function, please send it to me!
<fp -AT- fpx.de>
Back to the Homepage
Last modified: Wed Apr 12 11:23:35 1995