Alpha_1 Programmer's Manual
Alpha_1 Global Makes
Alpha1 uses a single source tree, multiple binary tree arrangement for
compiling on multiple machine types. The source tree changes daily and
the system is recompiled nightly on numerous platforms. This
system-wide, multiple platform recompilation is known collectively as
the "global makes." This document describes the basic tools for
managing the global makes and information about how to monitor and
maintain the process.
Table of Contents
There are a number of sources of information concerning the daily
changes and status of the system. First of all, the script
cvs-chgs can be used to inquire about recent source code
changes. Various options exist. Use "cvs-chgs -H" for help. (Use
a1chgs to find out about changes before switching to CVS
[3/31/95].) See Using CVS with
Alpha_1 for details on cvs-chgs.
The check-makes script reports the status of the global
makes. For each machine, the starting and ending times of the last
global make are reported (or just the starting time if a global make
is still running).
There is a directory containing all of the binary and debug
compiled Alpha_1 trees. This directory is
/res/alpha1/sys. Knowing this is useful for wild-card
searches across all machine types, to remove old binary files for example.
The global makes are run by the cron program which allows
commands to be scheduled at particular times for particular users. The
alpha1 account (UID) is used to do this so the makes run as
user alpha1. The setup files for cron are found on
~alpha1/crontabs. There is a file for each machine where cron
is used and a log file for that machine for error messages from
cron.
Currently (7/1/99), there are cron entries on Kzin for cvs-update and the SGI
"asgin32" makeall for MIPS R5k and R10k processors, as well as makealls
running on Spline (SGI "asgin32mips3" for R4k's), Joker (SGI "aindy" mips2
architecture), gemini (Sun asparc), and sweep (HP ahp800). See "man crontab"
for details.
Before any compilation occurs, the source tree has to be updated from
the central CVS repository. The is done with a cron entry that runs
the script cvs-update. This is currently run on lily at
midnight. This script maintains the file permissions on the source tree
as well. The output of the script is appended to
~alpha1/crontabs/cvs-update.log.
The global makes are controlled by files on $a$amk. The
makeall script is what is know as the "global make." This
script reads a file which specifies which directories to visit and
which makefile target is to be made. See the script for details. A
specific set of directories is specified for each machine type.
A number of actions under the global make are NOT machine specific and
apply to the source tree. For example some source files (.h and OH_*.C
files) are generated from .str files. For this reason, one global make
is designated the "master" and it's make actions include the machine
independent ones. Currently real is the master and it's
global make starts half an hour earlier (12:30 am) than the rest (1:00
am) since the other makes are dependent on these actions.
The global make produces three output files all on the directory
$b$amk. The makescript.log file is the output of the
makeall script itself (which directories were visited etc.). The
makeall.log file contains the output of the make
commands. Finally, at the end of the makeall, the file
make-errors.log is written. This file contains the compilation
errors for the just-completed global. These are extracted from the
makeall.log file. The make-errors.log file is overwritten for every
global make, but the other log files accumulate.
The makeall.log and makescript.log files are compressed and backed up
once a week (also by a cron command) so there are from one to two
weeks of log information available for reference. Tag files
and grep databases are also recomputed once a week.
Global makes can take more than 24 hours on some machines for full
recompiles. Makeall has a mechanism to prevent multiple global makes
in this case. If any temporary makeall files from a previous makeall
are found (on /tmp), the makeall will not run.
The daily responsibilities associated with the global makes include:
- Make sure the makes are running on all systems.
- Check the error logs for compilation errors.
- Monitor the daily source changes (checkin's) so problems can be
pinpointed.
- Fix errors, or notify the person who likely caused the
problem. Put in at least a little effort into the problem
first. Determine if it is specific to a particular machine for
example. It is often easier to fix the problem yourself.
- Improve this documentation!
The global make maintainers are provided the following information
daily via email:
- A summary of the last 24 hours of source changes as of the
beginning of the global makes (see cvs-chgs above).
- A summary of the global make status as of 8:30 am.
- A summary of compilation errors across all platforms as of 8:30 am.
(A concatenation of all make-errors.log files. See above.)
Most of the time, the global makes are finished by early a.m. so the
assumption is that the maintainer will check the status first thing in
the morning and address any problems. Remember there are a least 10-20
people depending on the system in one form or another every day.
The makeall.log file contains the output of all the a1make commands
for the global make for a given machine (see above). If the summary
file (make-errors.log) does not provide enough information to begin
tracking down a problem, you will need to look at the makeall.log
file. Alpha1 provides some help for doing this in emacs (see
$a$aem/make-error.el which is automatically loaded into your emacs by
alpha1.el). To look at the most recent makeall errors, do the
following:
- Visit $b$amk/makeall.log
- Go the end of the file and search backwards for ++++++. This is
the beginning of the latest makeall.
- Use M-? to find the next error in the file. There are various
other search functions defined in make-error.el.
- Makeall not running
- Sometimes a machine or makeall crashes and file are left on /tmp.
The next makeall thinks there is a previous makeall running and will
not run. Look in the crontab log files for this and remove the /tmp
files.
- 0 sized libraries
- Sometimes the time daemons get out of sync which really screws up
make. This seems to happen on the SUNS most often. The symptom is 0
sized libraries (.a files). You will see something like "unexpected
EOF" messages from the linker in the make error logs. From the
machine where the make runs, use
"rsh a1fs date; date" to check the time. (Should be within a couple of
seconds.) If this is the case, a message to unixsys@cs is in order.
Occasionally you have to kill off a global make. As uid alpha1, you
need to kill the processes, but there are lots of them. For example,
here is a makeall that was "stuck" running the no_space script:
7 speedy> ps -fu alpha1
UID PID PPID C STIME TTY TIME COMMAND
alpha1 18551 18383 2 Dec 19 ? 0:00 /res/alpha1/sys/ahpux800-bin/sbin/no_space -f /res/a
alpha1 18552 18551 2 Dec 19 ? 0:00 grep -s Filesystem
alpha1 18366 18364 2 Dec 19 ? 0:02 /res/alpha1/src/a1src/src/a1make/makeall -f /res/al
alpha1 18364 151 2 Dec 19 ? 0:00 sh -c /res/alpha1/src/a1src/src/a1make/makeall > /h
alpha1 18553 18552 2 Dec 19 ? 0:00 bdf /res/alpha1/src/a1src/emacs
alpha1 18383 18366 2 Dec 19 ? 0:00 csh -fx /tmp/makeall18366
alpha1 18554 18552 2 Dec 19 ? 0:00 head -1
alpha1 19728 19727 1 09:53:03 ttys0 0:01 -csh
alpha1 19752 19728 7 09:54:17 ttys0 0:00 ps -fu alpha1
The best strategy for killing the processes is top-down. First kill
the global make csh and the makeall script (pids 18366 and 18383
above). Then kill the current a1make or other higher level
process. Eventually, all the processes should be gone.
After killing all makeall processes, remove any leftover /tmp files.
Look for names /tmp/makeall* and /tmp/a1make*. If you do not remove
these, the next global make will not run thinking a makeall is still
in progress.
Currently, one RA and the senior staff monitor the nightly
makes. Global make duty is a weekly assignment. The RA "on duty"
receives the following mail each day:
- A summary of the last 24 hours of source changes (at midnight).
- A summary of the global make status (8:30 am).
- A summary of compilation errors for all platforms (8:30 am). (This is a
concatenation of all make-errors.log files.)
Here is the current rotation of global make maintainers:
- Bruce Gooch
- Greg Coombe (on duty)
- Matt Kaplan
Alpha_1 Programmer's Manual Home Page
Alpha_1 Programmer's Manual. Version 95.06.
Copyright © 1995, University of Utah
alpha1@gr.utah.edu