I finally had some time to upgrade my FreeBSD servers to 7.1. This followed the upgrade process I originally outlined after my first attempt. However, I've learnt a few things since then...
First, there's no need for the cvsup port, as a similar
capability is part of the base system, called csup, which
operates identically (for all practical purposes). So the upgrade
steps now are:
csup -g -L 2 /usr/local/etc/cvsup/src-supfile
cd /usr/src
./usr.sbin/mergemaster/mergemaster.sh -p
make buildworld
make buildkernel KERNCONF=CRIMSON
make installkernel KERNCONF=CRIMSON
[reboot to single-user ]
fsck -p
mount -u /
mount -a -t ufs
swapon -a
cd /usr/src
make installworld
mergemaster -U # replaces unaltered files automatically
# get rid of old, unused, files
make -DBATCH_DELETE_OLD_FILES delete-old
make -DBATCH_DELETE_OLD_FILES delete-old-libs
# Enable networking so that package downloads will work
/etc/rc.d/hostname start
/etc/rc.d/netif start
/etc/rc.d/routing start
# get ports tree up-to-date
csup -g -L 2 /usr/local/etc/cvsup/ports-supfile
# replace all packages
sh /home/mark/bin/pkg_upgrade.sh
[Reboot into multi-user (normality has now been resumed...)]
Some warnings about the above process:
make delete-old with the flag
-DBATCH_DELETE_OLD_FILES unless you are sure you are removing
all the old releases packages.
/home/mark/bin/pkg_upgrade.sh is described in
the journal entry for upgrading
to FreeBSD 6.3. While it does its best, there are somethings
it cannot cope with. In the 7.1 upgrade, a number of ports that
were available as packages in 6.3 were not available in 7.1, for
example cyrus-sasl2-saslauthd and py-libxml2. I
had to add these manually seeing the download errors in the log
file.
So far, FreeBSD 7.1 has been working just fine. However, I have
noticed that the memory usage, as shown by top, is much
more than it used to be on FreeBSD 6.3. This is very noticable on
crimson, as it only has 64MB. Here's a typical top display:
last pid: 71173; load averages: 0.05, 0.03, 0.01 up 17+19:50:57 17:01:35 47 processes: 1 running, 46 sleeping CPU: 0.8% user, 0.0% nice, 0.8% system, 0.8% interrupt, 97.7% idle Mem: 21M Active, 11M Inact, 21M Wired, 2060K Cache, 14M Buf, 3012K Free Swap: 128M Total, 19M Used, 109M Free, 14% Inuse PID USERNAME THR PRI NICE SIZE RES STATE TIME WCPU COMMAND 71173 mark 1 96 0 3488K 1428K RUN 0:00 1.24% top 17349 root 1 96 0 4664K 1172K select 5:45 0.00% ntpd 17418 root 1 96 0 6836K 1780K select 4:19 0.00% sendmail 17395 root 1 96 0 6852K 2172K select 3:53 0.00% httpd 17166 root 1 96 0 3176K 740K select 0:43 0.00% syslogd 17428 root 1 8 0 3204K 740K nanslp 0:37 0.00% cron 17722 bind 4 96 0 26464K 8056K select 0:29 0.00% named 17241 root 1 96 0 3204K 832K select 0:08 0.00% rpcbind 17295 root 1 4 0 3120K 500K - 0:05 0.00% nfsd 17422 smmsp 1 20 0 5948K 1396K pause 0:05 0.00% sendmail 17382 dhcpd 1 96 0 3120K 1268K select 0:05 0.00% dhcpd 71024 mark 1 96 0 8352K 3012K select 0:02 0.00% sshd 71026 mark 1 20 0 4448K 2052K pause 0:01 0.00% tcsh 17294 root 1 4 0 3120K 592K accept 0:01 0.00% nfsd 17413 root 1 96 0 5700K 1268K select 0:01 0.00% sshd 71021 root 1 4 0 8352K 3028K sbwait 0:01 0.00% sshd 66257 www 1 4 0 6852K 2488K accept 0:01 0.00% httpd
You can see that 19M of swap is in use. With 6.3, the amount of swap was normally around 2-3M. However, this does not seem to cause any performance problems, and may be just down to how the new malloc works in FreeBSD 7.1, or possibly because the new version of bind needs 26M!
As part of the upgrade to FreeBSD 7.1, I re-worked the kernel
configuration files for crimson and chrome to use the
config "include file" facility, along with nooptions,
nodevice etc. This allows a configuration file to be based on
GENERIC, but removes elements which are not required. I was in two
minds about doing this, as the level of removal from the GENERIC
config file for both crimson and chrome is large, but it's done now.
One tool I would have liked, but could not find, was something that would output the final configuration file, with elements added and removed. I couldn't find anything that did this, so I wrote one in python. Using this tool, the CRIMSON effective configuration file looks like:
ident CRIMSON # machine i386 # # cpu I486_CPU cpu I586_CPU # option UFS_ACL option KBD_INSTALL_CDEV option KSE option _KPOSIX_PRIORITY_SCHEDULING option STOP_NMI option PREEMPTION option MD_ROOT option MSDOSFS option NFSLOCKD option SOFTUPDATES option ATA_STATIC_ID option PSEUDOFS option SYSVSEM option SCSI_DELAY=5000 option STACK option INET option SCHED_ULE option COMPAT_FREEBSD6 option NFS_ROOT option UFS_GJOURNAL option COMPAT_FREEBSD5 option COMPAT_FREEBSD4 option ISAPNP option GEOM_BSD option SYSVMSG option SYSVSHM option PROCFS option CD9660 option GEOM_PART_GPT option COMPAT_43TTY option FFS option NFSCLIENT option UFS_DIRHASH option SCTP option GEOM_MBR option GEOM_LABEL option ADAPTIVE_GIANT option NFSSERVER # makeoption DEBUG=-g # device ppc device eisa device random device pty device atkbd device pci device psm device bpf device md device atadisk device uart_ns8250 device rl device mem device ata device uart device ppbus device loop device isa device kbdmux device ppi device ether device atkbdc device splash device atapifd device fdc device miibus device io device atapicd device sis device plip device vga device sio device npx device lpt device sc
Here's the python code:
"""
NAME
config.py - Displays effective version of FreeBSD kernel config
SYNOPSIS
python config.py [-n] config_file
-n - do not attempt to process the DEFAULT configuration file
DESCRIPTION
config.py will process a FreeBSD kernel configuration file and
output the effective configuration generated, taking account of
values provided in any include files, and the impact of the
various 'no..' keywords which are used to remove previously defined
values.
NOTES
MODIFICATION HISTORY
Mnemonic Rel Date Who
config 1.0 20090216 mpw
Created.
"""
import sys
import getopt
import os
import shlex
class Lex(shlex.shlex):
def sourcehook(self,filename):
f = open(filename)
return (filename,f)
class Config:
def __init__(self):
self.lex = None
self.elements = ((("ident",),dict(),1),
(("machine",),dict(),1),
(("maxusers",),dict(),1),
(("cpu","nocpu"),dict(),0),
(("option","options","nooption","nooptions"),dict(),0),
(("makeoption","makeoptions","nomakeoption",
"nomakeoptions"),dict(),0),
(("device","nodevice","nodevices"),dict(),0))
return
def process_element(self,tok,target):
for e in self.elements:
if tok in e[0]:
if e[2] == 1: e[1].clear() # only one element value allowed
if tok[0:2] == "no":
if target in e[1]: del e[1][target]
else:
e[1][target] = 0
return
def build(self,filename,process_default):
try:
f = open(filename)
except IOError:
return 1
self.lex = Lex(f)
# extend word token characters for makeoptions
self.lex.wordchars = self.lex.wordchars+"-="
self.lex.source = "include" # keyword for include file
self.lex.dir = os.path.dirname(filename)
self.lex.commenters = "#" # comment character
if process_default:
try:
fd = open("DEFAULTS")
except IOError:
return 2
self.lex.push_source(fd)
tok = "start"
while len(tok) != 0:
tok = self.lex.get_token()
target = self.lex.get_token()
self.process_element(tok,target)
return 0
def display(self):
for e in self.elements:
for k in e[1].keys():
print e[0][0],k
print "#"
return
if __name__ == '__main__':
process_default = True
try:
opts,args = getopt.getopt(sys.argv[1:],'-n')
for o,v in opts:
if o == '-n': process_default = False
except getopt.GetoptError,e:
print "config.py: illegal argument:",e.opt
sys.exit(1)
if len(args) == 0:
print >>sys.stderr,"config.py: usage: python [-n] config.py "\
"<CONFIG_FILE>"
sys.exit(1)
config = Config()
result = config.build(args[0],process_default)
if result == 0:
config.display()
elif result == 1:
print >>sys.stderr,"config.py: Unable to open",args[0]
elif result == 2:
print >>sys.stderr,"config.py: Unable to open DEFAULT config file"\
" (consider -n)"
