Solaris 11: Minimize your zones

Because the zone setup has been changed between Solaris 10/Solaris Express, I decided to dig a little bit deeper and provide a more convinient, efficient way to setup minimized zones wrt. the installed software.

Why to minimize your zones?

Well, this questions is easy to answer with some simple keywords:

1. stability

2. maintainance

3. downtime

4. resource consumption

5. security

Yes, we often hear SUN/Oracle developers say something like "don't minimize to optimize" or "it is not worth to ...". They are plain wrong and very often it seems, that they just never do phase 1 of software development, or we hear bullshit like "our customers never asked" - sure, if one always asks the big cash cows/dead meat only, one will probably get always the same answer: "it is working, leave everything as is" - they just prefer to never touch a running system. However, SMEs (some people say, there where innovation actually happens) need to use their equipment/resources to full capacity, use it in a very flexible manner, or even in a more or less volatile environment, i.e. need to adjust quickly to new trends/challenges/technologies. But as we can see, old/fat cows may die, young generation takes over, and finally people will use, what fits their needs (no matter, what developers with a very limited view may say), as the success of e.g. Linux (bottom up) shows ...

So without going into details, not only IMHO a healthy minimization is not only healthy for yourself, but for your system as a whole, company and the whole environment (green IT ;-)) as well and not doing it is a big mistake/another cornerstone to send the used OS into the grave.

What/How to minimize?

The obvious answer is: don't install anything/remove everything you'll never use. Unfortunately this is not that easy to accomplish without deep knowledge of the OS/SW, and even if one has that knowledge, one would get a so called "unsupported" system when all overhead gets removed. So for Solaris 11 we have to live with what the Image Packaging System (IPS) provides, and how the package designers assemble the related packages including their dependencies.

In Solaris 11 non-global zones are now installed similar to global zones (i.e. using AI under the hood), sparse zones are history. Basically zoneadm install [-m manifest] [-c profile | dir] just creates the appropriate ZFSs according to the zone configuration and installs the software as specified in the given manifest. If no manifest is given, /usr/share/auto_install/manifest/zone_default.xml will be used. Last but not least the given profile/all profiles from the given directory are applied to the zone, when the zone gets started for the first time (see svccfg apply ...). If no profile is given, /usr/share/auto_install/sc_profiles/enable_sci.xml will be used, which just configures the system to popup the system configuration tool (SCI) dialog at the zone's console.

Knowing this, it is trivial to determine, where to start minimizing the software, which gets installed into the zone: the manifest is the key. So looking at the default manifest, we see, that the meta-package pkg:/solaris/group/system/solaris-small-server will be used to install the zone's software. Since there seems to be no other smaller package, which can be used, one may use this one as a starting point. Unfortunately there is no tool, which displays all the package wrt. their dependencies (e.g. like a tree) and thus lets one instantly decide, what to remove, one needs to experiment a little bit:

• change the facets (Do it in the manifest! In the zone you would not see the effect unless you reinstall all packages, which is per default not possible.)

• use pkg list inside the zone to determine, which packages finally got installed

• use pkg uninstall ... inside the zone to remove all the stuff, which seems to be real overhead and also to get notified, when a [cyclic] dependency got hit (package can not be uninstalled).

• finally adjust the manifest you will use in the future as the default one for zone installs

What we got, using this approach for Solaris 11 SRU2 is shown in the table below:

As we can see, the main knob to reduce the size of a zone without an impact on usebility is to specify the locales, one really needs. For small servers which just host a DNS server or firewall/router, usually one does not need any locale stuff (LC_*="C") and thus can save about 35/100 MB wrt. the default manifest (A/B MB means, A MB on a ZFS with default compression enabled, B MB with compression disabled). Even if one installs de and en locales, one still gets an advantage of about 25/75 MB wrt. the default.

But if one thinks a little bit further, one realizes soon, that installing alle the manuals, documentation and developer stuff in every zone is a lot of overhead as well - usually not needed: In most cases one has a desktop, developer zone, or other server, where this stuff is already available and possibly easier to read. If not, one can think about installing just a "doc zone", where people can read the man pages/docs, or even share the related pathes via NFS or mount them via loopback into the zones. Since one may save about 100/210 MB wrt. the default (having de+en locales installed), such a separate "doc zone" makes, if you host more than 2 zones, just wrt. to space savings!

However, space savings is not all what one gets. As we know, Patches - now called "Support Repository Update" (SRU) - are published on a more or less regular base. Since the sparse description of them often does not allow one to judge, whether it is really needed, a good admin has to install these updates. So having less installed on the system, one may get faster updates, because less dependencies/packages are involved (especially calculating the so called update "Plan" is pretty slow with IPS) and some people might even consider power/cpu/memory savings as a little plus. Furthermore, because updates often include locale and some doc updates, one may save some space again when new BEs gets created ...

So what is left, is the software itself. Why should one reduce the set of software, which gets installed into a zone? The answer is simple again: The less software is installed, the less is the probability, that one needs to patch a zone. And if one does not need to patch anything, one also does not need to reboot the zone! Furthermore, if a certain software is not installed, one does not need to monitor security alerts wrt. this software - monitoring is important, since often it takes some time, until a fix/patch is available, and that's why one probably needs to apply workarounds and evaluate the impact on other software/the system at a whole to keep the system stable and secure. Also we can often see, that just because a software is installed, the corresponding daemons are run, driver gets loaded even if nobody is using them.

One example is the IPv6 autoconfig daemon in.ndpd. As soon as it gets discovered, that the system has an IPv6 capable interface, this daemon gets run. So to avoid this, one may either uninstall the network/routing package or if IPv4 routing is needed, apply a little hack, because there is no way anymore, to setup IPv4 only interfaces.

Another example of overhead are the tcp-wrappers. This is ancient stuff! Today we use firewall software like ipfilter to control the access from and to a system/port and modern software comes with its own authentication/authorization facilities (e.g. PAM modules, SSO plugins, SASL, Radius, GSS-API stuff).

Of course, there are much more: E.g. the Sun Workshop Compilers Bundled libC aka C++ libs. The history has shown, that this one gets patched very frequently. But when we look, which software actually needs it, we see: it is just the NIS stuff. And who is using NIS today ... ? But, yes, here we can also see an example of pretty dumb package assembly: If one wants to setup the zone as an ldap client, one needs to install the network/nis package because corresponding ldap binaries and svc manifests are merged into this package. And since the NIS stuff needs the mentioned c++-runtime libs, ...

Anyway, to see what we consider right now (SRU 2) as minimal/overhead, just have a look at this manifest. The win wrt. to space savings is not that much (about 32/60 MB), but as said, here this is only a secondary target.

Finally it should be said, that wrt. downtime, the need to update/reboot the system also depends on the global zone, because non-gloabl zones must be on par wrt. the software and version used in the global zone. So minimizing non-global zones only may not provide, what one actually tries to accomplish!

What else?

When one thinks about minimizing zones one also starts to think about minimizing the effort, to setup these zones. Looking a little bit closer, usually all required information to fully setup a zone is already available on the system: Per default/convention:

• the hostname of the zone is equal to the name of the zone itself

• the DNS and possible NIS/LDAP domain name is equal to the ones of the hosting global zone

• the IP address related to the hostname is in the hosts database (/etc/inet/hosts or DNS or LDAP/NIS)

• the default route is the same as in the global zone

• the VNIC(s) used for the exclusive-IP stack of the zone (also called "Physical Point of Attachment" - PPA), have the same name as zone

• the base sysconfig (root, admin account; timezone, keymap, terminal type, static network, and enabled services) is for all zones the same/similar

So what is needed is just a script, which collects/deduces all the required information, than prepares the manifest and system config profiles used by zoneadm install .., installs and boots the zone and finally runs the post install customization of the zone. So I wrote a zinstall.kshlib, which does exactly what has been mentioned before.

To get started quickly, just download the archive, extract the contents to a directory of your choice and run zinstall.kshlib to get an overview about available options.

cd /tmp
curl -O http://iws.cs.uni-magdeburg.de/~elkner/s11/zinstall.tgz
gtar vxzf zintall.tgz ; cd zinstall
# to create the script, which can be used to setup a zone called 'myzone' use
etc/zinstall.kshlib -s > myzone-zonesetup.ksh
# OR: to create the same script but with annotations
etc/zinstall.kshlib -S > myzone-zonesetup.ksh
# make it executable
chmod 774 myzone-zonesetup.ksh

# now you may adjust the script by adding the VAR=value pairs for all
# inappropriate default settings. To get an overview, what one can change,
# simply use:
./myzone-zonesetup.ksh -v
# To see, where the variables you are interested in get created/changed,
# you may just insert 'listVars $LINENO' at the position you like. E.g.:
gsed -i -e '/createTempdir/ a\listVars $LINENO' myzone-zonesetup.ksh
# Or one may add other packages
gsed -i -e '/prepareManifest/ a\addPkg install /service/network/dns/bind' \
	myzone-zonesetup.ksh

# finally create, install and reboot the zone
pfexec ./myzone-zonesetup.ksh -i

To see the complete help and variables used by the script, use as usual the option -h. Also to get an idea of a little bit more complex setup, you may have a look at the www-zonesetup.ksh script we use to setup our httpd zones.

Copyright (C) 2011-2013 Jens Elkner (jel+s11@cs.uni-magdeburg.de)