Upgrading to FreeBSD 9.1

The upgrade from FreeBSD 8.3 to 9.1 was not the same path I'd become used to. The source tree is no longer offered via cvs but now only by svn. svn has been used by the FreeBSD developers since 2008, so I guess I shouldn't have been too surprised that cvs was being retired.

N.B. This decision has now been changed (announced in the 20120-10-10 mail re FreebSD-9.1-RC2 availability). 9.1 will continue to make source updates available through cvsup. No commitments to cvsup after the 9.1 release.

Replacing csup with svn

Subversion is not part of base (and appears not likely to be in the near future), so it is installed from ports:

  pkg_add -r subversion

This also adds neon and sqlite.

The existing csup managed /usr/src must be removed and then the new subversion repository can be retrieved by:

  svn co svn://svn.freebsd.org/base/releng/9.1 /usr/src

Once you have the full src tree, it can be updated to reflect changes in the releng/9.1 branch:

  svn up /usr/src

To change to tracking a newer release, the svn switch should be used:

  svn switch svn://svn.freebsd.org/base/releng/9.2

cvs for ports is also being retired (in 2013). It is possible to handle ports through the same svn mechanism, i.e.

  rm -rf /usr/ports
  svn co svn://svn.freebsd.org/ports/head /usr/ports

Updating is similar:

  svn up /usr/ports

However, for ports I decided to go the portsnap route

Portsnap

Using portsnap is easy; first, the current port snap file must be fetched:

  portsnap fetch

Then, for the first time, the full tree is extracted:

  portsnap extract

For subsequent updates (following a portsnap fetch), use:

  portsnap update

Installing FreeBSD 9.1

I'd decided not to bother with Clang (the proposed new C compiler for FreeBSD, replacing GCC), by putting WITHOUT_CLANG=true" in /etc/src.conf. This is because it adds about an hour to the make buildworld time. I successfully installed 9.1 on both crimson and chrome (using crimson's /usr/src and /usr/obj directories).

When I came to install on topaz, I forgot to make the modification to /etc/src.conf. Therefore, make installworld died when it couldn't find the Clang binaries. I fixed up /etc/src.conf and re-ran make installworld, expecting that to run to a successful conclusion. Hopes were dashed. The installworld failed with:

  /tmp/install.xtvF5dw3/libdialog.so.7: Undefined symbol "_nc_wacs"

during the tzsetup in /usr/src/share/zoneinfo. I wasn't the only one to stumble across this (see this post from Kevin Oberman). However, the patch suggested in the mail thread didn't work for me, so I followed Kevin's lead and just commented out the tzsetup invocation from /usr/src/share/zoneinfo/Makefile. I ran the tzsetup -r command manually without problem after installworld had finished.

I'm not sure why I wasn't bitten by this during the first two installs. I can only assume the failure to install Clang must have had something to do with it.

Revised ports installation

This is the revised recipe for a FreeBSD upgrade:

  cd /usr/src
  ./usr.sbin/mergemaster/mergemaster.sh -p
  make buildworld
  make buildkernel # using GENERIC now
  make installkernel 
  [reboot to single-user ]
  fsck -p
  mount -u /
  mount -a -t ufs
  swapon -a
  cd /usr/src
  make installworld
  mergemaster -U # replaces unaltered files automatically
  # get rid of old, unused, files
  make -DBATCH_DELETE_OLD_FILES delete-old
  make -DBATCH_DELETE_OLD_FILES delete-old-libs
  # Enable networking so that package downloads will work
  /etc/rc.d/hostname start
  /etc/rc.d/netif start
  /etc/rc.d/routing start
  # get ports tree up-to-date
  portsnap fetch && portsnap update
  # remove and re-install all packages 
  sh /home/mark/bin/pkg_install.sh
  [Reboot into multi-user (normality has now been resumed...)]

The pkg_install.sh script is my new attempt for a simple method of re-installing ports following a major OS upgrade. My original scheme did not work too well, so this is something a whole lot simpler:

  #!/bin/sh
  # NAME
  #
  #   pkg_install.sh - (Re)install all standard hydrus packages on FreeBSD
  #
  # SYNOPSIS
  #
  #   sh pkg_install.sh [-f] [-v] {remote|local}
  #
  #   Command args:
  #
  #   -f     fallback mode.  If a remote package cannot be found or fails to
  #                          install, attempt to build from the port. Fallback
  #                          requires that the ports INDEX file is available.
  #   -v     verbose         Display package utility and make messages on tty
  #
  # DESCRIPTION
  #
  #   Installs the standard hydrus packages on FreeBSD, normally
  #   following a major revision upgrade. Packages are used where possible.  
  #
  #   For an initial install, specify remote, which will download
  #   packages from the FreeBSD infrastucture.  Packages downloaded are
  #   kept in a local directory (default /usr/ports/pacakges).  Where
  #   ports are built, packages are also created locally (including
  #   dependencies).
  #
  #   Specifying local will cause the local package repository to be
  #   used for the installation.
  #
  #   The output from the package utilities and the ports make mechanisms
  #   are output into a log file (/tmp/pkg_install-$$.log).  If the -v
  #   switch is given, the messages are also displayed on the terminal.
  #
  # NOTES
  #
  #   Fallback mode is experimental.
  #
  #   samba: /etc/make.conf flags set to not build for CUPS and LDAP
  #   py-libxml2: Needed for www tools (XML processing etc)
  #   ispell: /etc/make.conf flags set for BRITISH dictionary only
  #   cyrus-sals2-saslauthd: Needed for authorised mail access from outside LAN
  #   py-setuptools, py-bsddb3: These are needed for spambayes
  #   spambayes should be reinstalled with "python setup.py install"
  #   
  #   Updated for 8.3 [2012-04-22]
  #   Added subversion now cvs is no longer used by FreeBSD [2012-08-27]
  #

  # environment variables for package/ports management
  export PORTSDIR="/usr/ports"
  export PACKAGES="$PORTSDIR/packages" # package directory for make package
  export PKGDIR=$PACKAGES              # downloaded package directory for pkg_add
  export PKG_PATH="$PACKAGES"          # local package location

  # define list of standard packages
  PKG_LIST="apache22 cmucl cyrus-sasl-saslauthd emacs-nox11 gmake \
  isc-dhcp41-server lftp portaudit portmaster procmail python27 \
  py27-bsddb3 py27-libxml2 py27-setuptools qpopper rsync subversion wol"

  # build from ports required due to local make.conf flags
  PORT_LIST="net/samba35 textproc/ispell"

  user=`whoami`
  if [ "$user" != "root" ] ; then
      echo "$0: must be run as root."
      exit 1
  fi

  fallback=0
  sink=/dev/null
  logfile=/tmp/pkg_install_$$.log

  while [ $# -gt 1 ]; do
     case $1 in
         -f)
             fallback=1
             ;;
         -v)
             sink=/dev/tty
             ;;
         *)
             echo "$0: unrecognised switch: $1."
             exit 1
             ;;
     esac;
     shift
  done

  trap "exit" SIGINT SIGHUP SIGTERM

  echo "Logging package utility output into ${logfile} ..."

  if [ ${fallback} = "1" -a "$1" = "remote" ]; then
      echo "$0: fetching INDEX to support port fallback ..."
      (cd ${PORTSDIR};make fetchindex)
  fi

  if [ "$1" = "remote" ]; then
      # fetch packages from FreeBSD servers
      pkg_delete -af 2>&1 | tee ${logfile} >${sink}
      rm -rf ${PACKAGES}/*
      for pkg in $PKG_LIST; do
          echo -n "Adding ${pkg} ... "
          pkg_add -K -r $pkg 2>&1 | tee ${logfile} >${sink}
          if [ $? != "0" -a ${fallback} = "1" ]; then
              echo -n "[FAILED: "
              # add failed package to ports list
              portp=`cd ${PORTSDIR};\
                     make quicksearch name=${pkg}|grep -m 1 Path|cut -f 2`
              port=${portp#${PORTSDIR}/}
              echo "Added ${port} to PORT_LIST]" 
              PORT_LIST="${PORT_LIST} ${port}"
          else
              echo "[OK]"
          fi
      done
      # PKGDIR interferes in some arcane way with port building
      # (fails to find pkg-descr)
      unset PKGDIR
      # PKG_PATH causes problems with pkg_info and 'make missing'
      unset PKG_PATH
      for port in $PORT_LIST; do
          cd ${PORTSDIR}/${port}
          echo -n "Building ${port} ..."
          # make package-recursive fails if a dependency was not created from a
          # port (i.e. was installed from a package) so use the ports
          # listed by the 'missing' target to manually build packages
          deps=`make missing`
          (make clean && make && make install && make package) 2>&1 | \
              tee ${logfile} >${sink}
          if [ "$?" = "0" ]; then
              echo "[OK]"
              # now make packges of the built dependencies
              for dep in $deps; do
                  cd ${PORTSDIR}/${dep}
                  make package
              done
          else
              echo "[FAILED]"
          fi
      done
  elif [ "$1" = "local" ]; then
      echo "Deleting existing packages ..."
      pkg_delete -af 2>&1 | tee ${logfile} >${sink}
      # build using local package repository
      # install every package, so no need to worry about dependencies
      for dir in ${PACKAGES} ${PACKAGES}/All; do
          cd ${dir}
          for pkg in *.tbz; do
              echo -n "Installing ${pkg} ... "
              pkg_add -fi ${pkg} 2>&1 | tee ${logfile} >${sink}
              if [ $? = "0" ]; then
                  echo "[OK]"
              else
                  echo "[FAILED]"
              fi
          done
      done
  else
      echo "$0: please specify remote or local."
      exit 1
  fi

Replace the contents of PKG_LIST and PORT_LIST to reflect the needs of your installation.