LiveUpgrade Troubleshooting
Solaris™ LiveUpgrade is a nice feature of Solaris, which lets one upgrade the currently running version of Solaris to a new realease or patch it without the need to bring the system into single user mode for a more or less long time. Also it allows one to switch back to the previous state, when something went wrong, without the neeed to backuot patches from the "male"functioning boot environment.
The main steps for LiveUpgrade are
create a snapshot aka alternative boot environment (ABE) from the current running system
applying the changes (upgrade or patches) to the ABE instead of the currently running system
activate the ABE
boot into the new ABE by 'init 6'
However, especially when zones are used, LiveUpgrade has often issues which require you to manually fix the problems until they get fixed in the main release. This page tries to show most common problems and how to resolve them.
If one of the lu* commands fails, the best thing to do is to find out what the command in question actually does. The follwing receipt shows a way how to get all information one usually needs to find the cause of the problem and finally to resolve it.
+ tcsh
setenv SNVR b103
setenv NBE snv_$SNVR
setenv CD /net/install/pool1/install/`uname -p`/Solaris_11_$SNVR
setenv JUMPDIR /net/install/pool1/install/jumpstart
setenv RPOOL `df / | grep -v ^Filesystem | cut -f1 -d/`
setenv ERR err.${NBE}.log
setenv OUT out.${NBE}.log
setenv DBG dbg.${NBE}.log
setenv TRC trc.${NBE}.log
setenv LU_DEBUG_OVERRIDE 11
echo 'proc:::exec-success { \
printf("%d\t%s\n",timestamp,curpsinfo->pr_psargs); \
}' >/tmp/exe.d
setenv DTRACE "dtrace -s /tmp/exe.d -o $TRC -c"
setenv GAWK "gawk -f /local/misc/etc/tsortd.gawk"
cd /var/tmp
rm -f $TRC $DBG $OUT $ERR
$DTRACE "lucreate -n $NBE -p $RPOOL -l $ERR -o $OUT" |& tee $DBG
$GAWK $TRC >lucreate.${TRC}
rm $TRC; mv $DBG lucreate.${DBG}; mv $OUT lucreate.${OUT}; mv $ERR lucreate.${ERR}
$DTRACE "lumount -l $ERR -o $OUT $NBE /mnt" |& tee $DBG
$GAWK $TRC >lumount.${TRC}
rm $TRC; mv $DBG lumount.${DBG}; mv $OUT lumount.${OUT}; mv $ERR lumount.${ERR}
$DTRACE "luumount -l $ERR -o $OUT $NBE" |& tee $DBG
$GAWK $TRC >luumount.${TRC}
rm $TRC; mv $DBG luumount.${DBG}; mv $OUT luumount.${OUT}; mv $ERR luumount.${ERR}
$DTRACE "luupgrade -u -n $NBE -s $CD -j /var/tmp/profile.$SNVR -l $ERR -o $OUT" |& tee $DBG
$GAWK $TRC >luupgrade.${TRC}
rm -f $TRC
mv $DBG luupgrade.${DBG}; mv $OUT luupgrade.${OUT}; mv $ERR luupgrade.${ERR}
$DTRACE "luactivate -l $ERR -o $OUT $NBE" |& tee $DBG
$GAWK $TRC >luactivate.${TRC}
rm $TRC; mv $DBG luactivate.${DBG}; mv $OUT luactivate.${OUT}; mv $ERR luactivate.${ERR}
$DTRACE "ludelete -l $ERR -o $OUT $NBE" |& tee $DBG
$GAWK $TRC >ludelete.${TRC}
rm $TRC; mv $DBG ludelete.${DBG}; mv $OUT ludelete.${OUT}; mv $ERR ludelete.${ERR}
Rule of thumb: When luactivate or ludelete fails, usually it is sufficient to look for problems wrt. lumount. If lumount fails, all mentioned LU commands will fail as well OR the machine will not be able to boot into the new BE OR will boot, but go into single user mode because of problems mounting one or more slices/ZFSs!
Always make sure, that the currently installed LU packages SUNWluu SUNWlur SUNWlucfg have at least the version of the target boot environment. E.g. if you are running snv_b98 and wanna upgrade to snv_b103, make sure, that you have installed the LU packages from snv_b103 in the currently running snv_b98.
pkgrm SUNWluu SUNWlur SUNWlucfg pkgadd -d $CD/Solaris_11/Product SUNWlucfg SUNWlur SUNWluu # Solaris gpatch -p0 -d / -b -z .orig < /local/misc/etc/lu-5.10.patch # Nevada gpatch -p0 -d / -b -z .orig < /local/misc/etc/lu-5.11.patch
Also check, whether all required patches are installed. Especially 121430-36 (sparc) / 121431-37 (x86) and not earlier are highly recommended: allmost all problems especially wrt. zones should vanish with these patches. However, don't forget to check Solaris™ Live Upgrade Software: Minimum Patch Requirements!
checkpatches.sh -p 119081-25 124628-05 ...
One common problem is, that lucreate properly clones a zone by cloning its zonepath ZFS, however it doesn't set the zonepath of the cloned zone to the cloned ZFS.
# zfs list | grep zones rpool/zones 285M 219G 21K /rpool/zones rpool/zones/sdev 284M 219G 283M /rpool/zones/sdev rpool/zones/sdev@snv_b103 394K - 283M - rpool/zones/sdev-snv_b103 754K 219G 283M /rpool/zones/sdev-snv_b103 # zfs set mountpoint=/mnt rpool/ROOT/snv_b103 # zfs mount rpool/ROOT/snv_b103 # zonecfg -R /mnt -z sdev info zonepath zonepath: /mnt/rpool/zones/sdev
The error you will probably see in this case using 'lumount snv_b103 /mnt' is:
ERROR: unable to mount zones: cannot mount '/rpool/zones/sdev': directory is not empty ... zone 'sdev': zone root /rpool/zones/sdev/root already in use by zone sdev zoneadm: zone 'sdev': call to zoneadmd failed ERROR: unable to mount zone <sdev> in </.alt.snv_b103> ...
To fix that problem permanently, you can apply the lu patch mentioned above.
Otherwise, fix it manually. E.g.
zfs set mountpoint=/mnt rpool/ROOT/snv_b103 zfs mount rpool/ROOT/snv_b103 zfs mount rpool/ROOT/snv_b103/var zonecfg -R /mnt -z sdev set -F zonepath=/mnt/rpool/zones/sdev-snv_b103 umount /mnt/var umount /mnt
When lucreate creates a BE, it stores the none-zone filesystems in a file called /etc/lu/ICF.$NUM, whereby $NUM is the ID of the coresponding BE (see /etc/lutab). This file will be used by luumount, to mount all those filesystems before the filesystems of the zones get mounted.
Unfortunately, when a BE gets created for the first time (initial BE), existing filesystems are recorded in the wrong order, which leads to hidden mountpoints, when lumount is called.
snv_101:-:/dev/zvol/dsk/rpool/swap:swap:4192256 snv_101:/:rpool/ROOT/snv_101:zfs:0 snv_101:/export/home:rpool/export/home:zfs:0 snv_101:/export:rpool/export:zfs:0 snv_101:/rpool/zones:rpool/zones:zfs:0 snv_101:/rpool:rpool:zfs:0 snv_101:/var:rpool/ROOT/snv_101/var:zfs:0
So in this example lumount mounts rpool/zones to /rpool/zones first (which contains the directory aka mountpoint sdev for the zone sdev). After that, it mounts rpool on /rpool, which contains the empty directory zones (mountpoint for rpool/zones). Finally it tries to mount the zonepathes, e.g. rpool/zones/sdev on /rpool/zones/sdev, but this obviously fails, since this mountpoint is hidden by /rpool.
ERROR: unable to mount zones: zoneadm: /mnt/rpool/zones/sdev: No such file or directory could not verify zonepath /mnt/rpool/zones/sdev because of the above errors. zoneadm: zone sdev failed to verify ERROR: unable to mount zone <sdev> in </mnt> ...
To fix the problem, just edit the /etc/lu/ICF.$NUM file and bring the entries into the correct order.
snv_101:-:/dev/zvol/dsk/rpool/swap:swap:4192256 snv_101:/:rpool/ROOT/snv_101:zfs:0 snv_101:/export:rpool/export:zfs:0 snv_101:/export/home:rpool/export/home:zfs:0 snv_101:/rpool:rpool:zfs:0 snv_101:/rpool/zones:rpool/zones:zfs:0 snv_101:/var:rpool/ROOT/snv_101/var:zfs:0
Unfortunately luupgrade recreates ICF when it thinks, it is necessary and thus it may fail to upgrade zones. That means the script in charge needs to be fixed permanently by applying lu patch mentioned above.
If you got caught by this bug, remember to clean up the ZFS mountpoints! Otherwise the system will not come up when booting in the BE, because 'zfs mount -a' will fail dueto none-empty directories.
If you have split the /var from the / zfs, lucreate may fail with an error message like this
ERROR: cannot mount '/.alt.tmp.b-3F.mnt/var': directory is not empty ERROR: cannot mount mount point </.alt.tmp.b-3F.mnt/var> device <rpool/ROOT/test/var> ERROR: failed to mount file system <rpool/ROOT/test/var> on </.alt.tmp.b-3F.mnt/var> ERROR: unmounting partially mounted boot environment file systems ERROR: cannot mount boot environment by icf file </etc/lu/ICF.3> ERROR: Unable to mount ABE <test> ERROR: Unable to clone the existing file systems from boot environment <s10u8_20100821> to create boot environment <test> ERROR: Cannot make file systems for boot environment.
This happens, if the current / filesystem contains a /var directory, which is not empty. Usually it contains a tmp/ subdirectory or other stuff, however one can not see this, since the /var is mounted over it.
Unfortunately we can't simply mount -F lofs / /mnt, since Solaris "mounts" /var to /mnt/var as well (unlike Linux mount --bind ...). So the easiest way is to luactivate a working BE, boot into it and fix the bogus root filesystem of the BE you came from. E.g.:
zfs set mountpoint=/mnt rpool/ROOT/buggyBE zfs mount rpool/ROOT/buggyBE rm -rf /mnt/var/* ls -al /mnt/var zfs umount /mnt zfs set mountpoint=/ rpool/ROOT/buggyBE
Finally luactivate the buggyBE, boot into it and delete the incomplete BE and destroy all ZFS left over from the previously failed lucreate. E.g.:
ludelete test # delete the remaining BE ZFS, e.g. zfs list -t all |grep test zfs destroy rpool/ROOT/test/var zfs destroy rpool/ROOT/test zfs destroy rpool/ROOT/`lucurr`/var@test zfs destroy rpool/ROOT/`lucurr`@test # and do not forget the cloned zone filesystems/snapshots
If one calls e.g. 'lumount snv_101 /mnt' (current BE is e.g. snv_b103), lumount mounts first the none-zone ZFSs and than the zonepath ZFSs using /usr/lib/lu/lumount_zones. Unfortunately lumount_zones checks only, whether the corresponding ZFS is already mounted, but NOT its current mountpoint. So if rpool/zones/sdev is already mounted (the default since usually canmount property of rpool/zones is on as well) e.g. to /rpool/zones/sdev, it does not get mounted on /mnt/rpool/zones/sdev as a loopback filesystem and you'll get usually the follwoing error:
ERROR: unable to mount zones: /mnt/rpool/zones/sdev must not be group readable. /mnt/rpool/zones/sdev must not be group executable. /mnt/rpool/zones/sdev must not be world readable. /mnt/rpool/zones/sdev must not be world executable. could not verify zonepath /mnt/rpool/zones/sdev because of the above errors. zoneadm: zone sdev failed to verify ERROR: unable to mount zone <sdev> in </mnt> ...
NOTE: The "not ... readable|executable" error message is a little bit misleading. This is because the mountpoints for the zones have 0755 permission.
zfs mount rpool/ROOT/snv_101 mount -o ro,nosub,nodevices -F lofs /rpool /mnt/rpool mount -o ro,nosub,nodevices -F lofs /rpool/zones /mnt/rpool/zones ls -al /mnt/rpool/zones total 12 drwxr-xr-x 4 root root 4 Dec 13 21:19 . drwxr-xr-x 5 root root 5 Dec 13 19:07 .. drwxr-xr-x 2 root root 2 Dec 13 21:19 sdev drwxr-xr-x 2 root root 2 Dec 13 20:59 sdev-snv_b103
or a little bit shorter
umount /rpool/zones/sdev ls -al /rpool/zones/ total 12 drwxr-xr-x 4 root root 4 Dec 13 21:19 . drwxr-xr-x 5 root root 5 Dec 13 19:07 .. drwxr-xr-x 2 root root 2 Dec 13 21:19 sdev drwx------ 5 root root 5 Dec 13 20:59 sdev-snv_b103
So if one 'chmod 0700 /rpool/zones/sdev' and mounts rpool/zones/sdev again, one gets a slightly more descriptive error message, which indicates, that the ZFS for the zonepath is not mounted on the required mountpoint:
ERROR: unable to mount zones: zone 'sdev': can't stat /mnt/rpool/zones/sdev/root: No such file or directory zoneadm: zone 'sdev': call to zoneadmd failed ERROR: unable to mount zone <sdev> in </mnt> ...
To fix the problem, simply unmount the zonepath in question.
umount /rpool/zones/sdev # OR ls -1d /$RPOOL/zones/* | xargs -L 1 umount
This usually happens on luupgrade of x86 machines and is an aftereffect of the lumount_zones bug described above. The error message looks like this:
luupgrade: The Solaris upgrade of the boot environment <snv_b103> is partially complete. luupgrade: Installing failsafe luupgrade: ERROR: Unable to mount boot environment <snv_b103> ...
Since this is the last more or less important step in luupgrade, one can "ignore" this message and fix the problem manually. E.g.:
zfs set mountpoint=/mnt rpool/ROOT/snv_b103 zfs mount rpool/ROOT/snv_b103 # if `isainfo -k` = "amd64" AND `uname -r` = "5.11" cp $CD/boot/amd64/x86.miniroot /mnt/boot/x86.miniroot-safe # else cp $CD/boot/x86.miniroot /mnt/boot/x86.miniroot-safe umount /mnt
On x86 we encountered the situation, where lucreate was able to create a new BE, however it did not clone the zonepath ZFSs. The reason for this misbehavior was, that the current BE wasn't setup properly - luactivate and friends did not set the mountpoint for its root path to / but kept the temporary mountpoint instead.
# zfs list -r rpool rpool 20.5G 113G 29.5K /rpool rpool/ROOT 4.47G 113G 18K legacy rpool/ROOT/zfs1008BE 4.47G 113G 3.17G /.alt.zfs1008BE rpool/ROOT/zfs1008BE/var 1.30G 113G 1.30G /.alt.zfs1008BE/var rpool/dump 8.02G 113G 8.02G - rpool/swap 8.01G 121G 16K - # df -h /var Filesystem size used avail capacity Mounted on rpool/ROOT/zfs1008BE 134G 3.2G 113G 3% /
So GRUB managed to find the right root ZFS and managed to mount it on / ignoring the ZFS mountpoint property. However, lucreate uses the mountpoint property value of the current BE's root ZFS to determine all none-global zones in it and their corresponding zonepath. And since there is no zone index file (e.g. /.alt.zfs1008BE/etc/zones/index) lucreate wrongly assumes, that there are no zones to clone.
Thus the missing zone clones are the reason, why ludelete of the created zone fails: The new BE aka / clone has the unmodified copy of the /etc/zones/index for the zones in the current BE and thus tries to recursively the zonepath ZFSs of the current BE (e.g. zfs destroy -r pool1/zones/edub-zfs1008BE). Hopefully, all zones are running, otherwise the zonepath ZFSs will be lost! Hint number 1:
Never delete a BE with uncloned zones without making sure, that all zones in the current BE are up and running, i.e. the zonepath ZFSs are busy. Otherwise you will loose them!
To fix the problem, boot into the current BE's failsafe archive and fix its mountpoint property.
SunOS Release 5.10 Version Generic_137138-09 32-bit Copyright 1983-2008 Sun Microsystems, Inc. All rights reserved. Use is subject to license terms. Booting to milestone "milestone/single-user:default". Configuring devices. Searching for installed OS instances... fdisk: Cannot stat device /dev/rdsk/c0t600C0FF000000000099C790E. mount: I/O error mount: Cannot mount /dev/dsk/c2t0d0s2 Failed to mount /dev/dsk/c2t0d0s2 read-only: skipping. mount: I/O error mount: Cannot mount /dev/dsk/c2t1d0s2 Failed to mount /dev/dsk/c2t1d0s2 read-only: skipping. ROOT/zfs1008BE was found on rpool. Do you wish to have it mounted read-write on /a? [y,n,?] y mounting rpool on /a cannot mount '/a/.alt.zfs1008BE': failed to create mountpoint Unable to mount rpool/ROOT/zfs1008BE as root #
# zfs set mountpoint=/ rpool/ROOT/zfs1008BE # init 6
On some servers lu commands like lumount, luumount, lucreate, luactivate, etc. seem to hang/never gets finished. Usually this happens on servers with a lot of filesystems and/or zones. Against the latter we can't do anything right now, however one should let lu* commands ignore all filesystems, which are not required for the purpose of live upgrade/patching, so that the lu commands do not try to [un]mount per loopback aka lofs into the appropriate BE/zones. E.g. excluding ~2200 ZFS on a X4600 M2 (4xDualCore Opteron 8222 3GHz) saves about 40 min per lucreate, lumount, luactivate, etc. (or about 1s per ignored filesystem) ...
To accomplish that, one needs to apply the lu patch mentioned above, create the file /etc/lu/fs2ignore.regex and add the regular expressions (one per line) which match those filesystems, which should be ignored by lu* commands.
E.g. to exclude all user home directories or better ZFSs, whomes mountpoint is below /export/home, the following command is sufficient:
echo '/export/home' > /etc/lu/fs2ignore.regex # check whether this is, what you want pfexec /sbin/sh /usr/lib/lu/ludefine -v /etc/vfstab -n tmpBE # and make sure, no pathes inherited to zones are hidden - if the following # command lists anything on stdout, chances for getting into trouble are very high /usr/bin/egrep -f /etc/lu/fs2ignore.regex /etc/zones/*
For more information about valid regular expressions see the man page for /usr/bin/egrep, which is the command used with the option -v to filter out the filesystems from mount -p and zfs list.
Always make sure, that /etc/lu/fs2ignore.regex matches the filesystems you wanna have ignored, and nothing else. Otherwise LU commands may fail or even destroy valueable data!
The latest bug introduced with 121430-36/121431-37. This happens, because the appropriate script uses the environment variable BOOT_MENU_FILE very often, but is unset. So the simple workaround is to set this variable to its intended value, before invoking luactivate:
setenv BOOT_MENU_FILE menu.lst
It may happen, that patchadd moans about not having enough space in /var/run to copy overlay objects (e.g. kernel patch 139555-08). On recent Solaris versions /var/run is a tmpfs. So if you have not enough RAM or e.g. a line like the following one in your /etc/vfstab, patchadd can fail:
swap - /tmp tmpfs - yes size=512M
Since Solaris seems to allocated per default the half of the given size parameter per default, patchadd sees even on a almost clean /tmp only ~ 256M or even less, depending on what your system is currently doing. E.g.:
swap 262M 136K 262M 1% /tmp swap 262M 48K 262M 1% /var/run swap 264M 1.3M 262M 1% /etc/svc/volatile # bootenvironment to be patched rpool0/ROOT/s10u6_20090517 33G 4.7G 18G 22% /mnt swap 262M 8K 262M 1% /mnt/var/run swap 262M 0K 262M 0% /mnt/tmp
Obviously all tmpfs filesystems try to share the same "memory partition".
So to fix the problem, one may change the size parameter for /tmp at least to 2x the size patchadd needs and reboot (in contrast to other OSs remounting it is not supported on Solaris). However, rebooting is often not an option for production systems, so one may try to force the OS to allocate the max. tmpfs size by creating an appropriate sized file and removing it right before starting patchadd. Note: In contrast what one may expect, Solaris does not satisfy immediately your "more space request" and thus your used "create sized file" procedure may fail several times 'til /tmp gets its max size as specified in /etc/vfstab (IMHO that's a bug). Also note, that the OS tries to resize it down, as soon as possible. So you need to be fast, when starting patchadd after you got an appriopriate sized /tmp aka /var/run .
cd /var/tmp unzip 139555-08.zip # repeat until you got, what you want dd if=/dev/zero of=/tmp/test.dd bs=1024k count=512 df -h /tmp # than hurry up with patching, e.g. rm /tmp/test.dd ; /usr/sbin/patchadd -R /mnt /var/tmp/139555-08
If you have packages installed, which don't obey the restrictions of pkginfo(4), e.g. having a description with more than 1023 characters, luupgrade may fail with the error message ERROR: Failure reading non-global zone module: zonename. The only way to fix it is to uninstall the given package and re-install it after the zone has been upgraded. To find the problematic package, one may use something like this:
nawk '/DESC=/ { print length($0)-5 , FILENAME }' \
`find /var/sadm/pkg -name pkginfo | fgrep -v /save/` | sort -nr
Copyright (C) 2008 Jens Elkner (jel+lu@cs.uni-magdeburg.de)