Why use Solaris 8 zones?

First of all, just a few words about the niche for this product. Many companies are rather conservative about their Solaris upgrades. Most systems are still running Solaris 8, if not something older. Quite often this is also dictated by third-party software dependencies – products which were bought and configured for Solaris 8, which are now so tightly integrated that there isn’t an easy way to migrated them into Solaris 10. Such systems are doomed for slow but very expensive death. Expensive, because with every year the hardware support for  servers capable of running Solaris 8 raises again and again.

That’s where the Solaris 8 zones come in. It’s very easy, really: you create a flar-copy fo your existing physical server under Solaris 8, then create a Solaris 8 zone, import your flar-archive and get a virtual copy of your Solaris 8 environment, with all your processes, programs and startup scripts.

To make things easier, it’s even possible to configure your hostid in Solaris 8 zone to match the one of the physical Solaris 8 system, this way no programs running in the zone will even guess that they’ve been virtualized.

Who knows, maybe I’ll tell you more about this technology some other day, but for now – just the simplest list of actions and commands for your S8MA proof of concept.

Preparation: Solaris 10u4 server an S8MA packages

1. Find and prepare a sparc box with Solaris 10u4. It is important to have the latest Solaris 10 update. Preparations are usually limited to applying a kernel patch, 127111-01 in my case.

2. Download the Solaris 8 Migration Assitant (current version is 1.0) from this location: Solaris 8 Migration Assistant. The 3 packages in archive are dead easy to install using standard pkgadd.

Here are the packages you’ll get:

Make a snapshot of your old Solaris 8 server

You need to create a flar-archive of your Solaris 8 system.

Log onto your Solaris 8 box, and run the command. In this and all the following examples, solaris8 is nothing but an arbitrary name I’ve chosen for my zone. You might as well call it anything you like.

solaris# flarcreate -S -n solaris8 solaris8.flar

Set up a Solaris 8 zone

Here’s how you do it:

solaris# zonecfg -z solaris8
solaris8: No such zone configured
Use 'create' to begin configuring a new zone.
zonecfg:solaris8-system> create -t SUNWsolaris8
zonecfg:solaris8> set zonepath=/export/solaris8
zonecfg:solaris8> add net
zonecfg:solaris8:net> set address=172.21.7.155/24
zonecfg:solaris8:net> set physical=ce0
zonecfg:solaris8:net> end
zonecfg:solaris8> commit
zonecfg:solaris8> exit

Naturally, your IP and network device name will be different. As of now, our zone is fully configured, but not yet installed.

Install Solaris 8 zone using flar-archive

solaris# zoneadm -z solaris8 install -u -a /export/solaris8.flar
Log File: /var/tmp/solaris8.install.13597.log
Source: /export/solaris8.flar
Installing: This may take several minutes…
Postprocessing: This may take several minutes…
WARNING: zone did not finish booting.

In my case the Solaris 8 zone got stuck on sys-unconfig, and so I had to connect to the virtual console of the zone to help it move on:

Here’s how you connect to a zone’s console:

solaris# zlogin -C solaris8

That’s it! The rest was easy – just a few minutes of configuring the network parameters and DNS/NIS settings. Finally, I was able to ssh into the new zone and run uname:

solaris8 #uname -a
SunOS solaris8 5.8 Generic_Virtual sun4u sparc SUNW,Sun-Fire-V490

I liked Solaris 8 Migration Assistant very much. It’s an incredibly quick and easy way to have a whole bunch of Solaris 8 systems virtualized and running under on one of the most advanced servers with the most advanced OS – Solaris 10u4.

In this entry I’ll try and explain how to set up an install server to allow you do a network install of Solaris with PXE.

Solaris PXE/DHCP server howto

The procedure described below describes steps necessary for PXE booting and installing a client with Grub bootloader, which is the default one in Solaris since Nevada b14. Obviously, the very first requirement is that you absolutely must have a PXE-enabled network card installed in the client machine you’re about to install.

Here is the full list of server-side components you need to have in order to be able to PXE-boot and install a client machine over network:

1. DHCP server setup and configured
2. Network install OS image
3. Jumstart configuration files for a given client

I’ll go through this list first, and then will give a list of actions you have to take to configure and PXE-boot a client:

1. DHCP server configuration

1.1. Setup
Start off by doing this:

solaris# dhcpconfig -D -r SUNWbinfiles -p /var/dhcp

This configures your DHCP server: it creates an empty resources table for its use in /var/dhcp, and creates an /etc/inet/dhcpsvc.conf file with a contents like this:

DAEMON_ENABLED=TRUE
RESOURCE=SUNWbinfiles
RUN_MODE=server
PATH=/var/dhcp
CONVER=1

Your default DHCP daemon (/usr/lib/inet/in.dhcpd) is started at this stage, andthe above shown dhcpsvc.conf file ensures your dhcp server gets started up automatically upon reboot from /etc/init.d/dhcp script.

1.2. Networks

Here you have to look at your topology and decide, which networks are going to be served by your DHCP server. If you have a DHCP server configured by someone else, here’s how you look at the list of networks served:

solaris# pntadm -L

It returns you a list, on our server it looks like this:

192.1.3.0
192.1.4.0
192.1.1.0
192.1.2.0
192.1.9.0

To create a table for another network, do this:

solaris# pntadm -C 192.1.10.0

Make sure you have/etc/netmasks updated, if your network is subnetted. Update your /etc/inet/dhcpsvc.conf file to include a line with all the DHCP active interfaces. On our server it looks like this:

INTERFACES=hme1,qfe0,qfe1,qfe3

2. Network install OS image setup

Follow the instructions from this page: Creating an install server with CD media

In short: create a directory for your netinstall image, and use setup_install_server and add_instll_server commands to copy necessary files from Solaris_10.1/Tools directory of your install media (CDs or DVD).

Share your netinstall image out.

Checklist:

• make sure the netinstall directory is shared out to your clients
• if you have more than one network, make sure all your networks have access to this directory.

Share command should return you something like this for your install directory:

-               /export/install2   ro=@192.1.1:@192.1.2:@192.1.3:@192.1.4:@192.1.9	""

3. JumpStart configuration

Follow the instructions from this page: Creating a profile for networked system.

On our server, the JumpStart directory is /export/install/config. Then each machine has a directory there, like /export/install/config/perf-123.

Okay. Assuming you’ve done everything above, you’ve got a DHCP server configured, netinstall image ready and JumpStart files created for your client machine.

Now we’re ready to reinstall your client. Here’s what to do:

1. Generic add_install_client

Go to your netinstall image directory, and go to Solaris_10.1/Tools dir, where you have add_install_script. On our server, for Nevada b14 it’s this directory:

/export/install2/nv/snv_14/Solaris_10.1/Tools

Now, do this:

solaris# ./add_install_client -d SUNW.i86pc i86pc

It’s a very important commant. It creates, if necessary, /tftpboot directory, or updates it with generic files for the netinstall OS image we’re using. This command will also enable TFTP server on your machine. At last, it will create a /tftboot/pxegrub.I86PC.Solaris_10.1-1 file, which will be used to PXE boot each client you want to install with this build.

2. Client-specific add_install_client

Then, for our client, we have to do this:

solaris# ./add_install_client -d -e $ETHER 
-c $SERVER:/export/install/config/$SYSNAME 
-p $SERVER:/export/install/config/$SYSNAME 
-s $SERVER:$SOLARIS_PATH 
-b "console=$TTY" i86pc 

It’s a line from our script which automates the installation, and it shows clearly what is being passed to the add_install_client command: $ETHER is an MAC-address of the client box. Bear in mind, that for most MAC-addresses you have to add “01″ to the beginning of the address. Also, you have to get rid of the colons usually found in MAC address. So, for instance, if your client has this entry in /etc/ethers file on your server:

00:09:3d:00:68:f3 perf-123

The MAC-address you’ll pass to the add_install_client will be this: 0100093D0068F3.

Other parameters are:

• $SERVER - IP address of your server’s interface on the subnet your client is on
• $SYSNAME - a hostname of the client
• $SOLARIS_PATH - is a full path to the netinstall image you’re using
• $TTY- is an input-output device used during pxegrub boot. You can specify ttya/ttyb here to get an output on a console, connected to your client. You may also not specify this parameter at all for the default device to be used (screen/keyboard).

So, for our client perf-123 the final line looks like this:

solaris# ./add_install_client -d -e 0100093D0068F3
-c 192.1.1.254:/export/install/config/perf-123 
-p 192.1.1.254:/export/install/config/perf-123 
-s 192.1.1.254:/export/install2/nv/snv_14 
-b "console=ttya" i86pc

This line creates three files in your /tftpboot directory:

• menu.lst.0100093D0068F3 - grub menu for your client’s PXE boot
• rm.0100093D0068F3 - a standard shell script for cleaning up /tftpboot files for your client
• 0100093D0068F3 - a symbolic link to a build’s pxegrub file in /tftpboot

For perf-123, this link looks like this:

solaris# ls -al /tftpboot/0100093D0068F3
lrwxrwxrwx   1 root     other         28 May 23 17:27 
/tftpboot/0100093D0068F3 -> pxegrub.I86PC.Solaris_10.1-1

If you check now, the grub menu file will look like this:

solaris# more /tftpboot/menu.lst.0100093D0068F3 
default=0
timeout=30
title Solaris_10.1 Jumpstart
        kernel /I86PC.Solaris_10.1-1/multiboot kernel/unix -B  
console=ttya,install_config=192.1.1.254:/export/install/config/perf-123,  
sysid_config=192.1.1.254:/export/install/config/perf-123,  
install_media=192.1.1.254:/export/install2/nv/snv_14
        module /I86PC.Solaris_10.1-1/x86.miniroot

Unfortunatelly, at this early stage, we have to manually update this file so that the machine not only boots off the network, but starts JumpStart install automatically. To do this, you have to pass a “- install dhcp” as a boot argument to Solaris kernel. So we update one of the menu file lines to look like this:

kernel /I86PC.Solaris_10.1-1/multiboot kernel/unix - install dhcp  -B  
console=ttya,install_config=192.1.1.254:/export/install/config/perf-123,  
sysid_config=192.1.1.254:/export/install/config/perf-123,  
install_media=192.1.1.254:/export/install2/nv/snv_14

I’m sure later there will be a command line option, so I’ll update this how-to accordingly.

As you can see, most of the add_install_clients parameters end up in this file. A menu file is also a good way to do basic sanity checks, cause it shows you where your client will try to mount a netinstall image from, and where JumpStart will expect to have your client’s configuration files to be found.

4. DHCP – IP address and a macro for your client system

First, add a fixed IP for your client, which will be issued to it by your server upon DHCP request:

solaris# pntadm -f 01 -e 12/31/2010 -A $IPADDR -i $ETHER -m $SYSNAME -s $SERVER $NETWORK.0

Most of the parameters are self-explanatory, and you know most of them from add_install_client command. So I’ll mention only the new ones:

• $IPADDR is the IP address you want DHCP server to assign to your client
• $NETWORK is the network your client belongs to. Obviously, $IPADDR should be in this network as well.
• -f 01 means that this IP lease is permanent
• -e 12/31/2010sets the expiration date for this lease (we sure have got some time before 2010)

Now, what this line does apart from assigning an IP address to our client, is that it also assigns the client a DHCP macro. By following the macro specified, your DHCP server will give the client all the additional info the client needs to PXE boot.

Here’s the line:

solaris# dhtadm -A -m $SYSNAME -d ":BootSrvA=$SERVER:BootFile=$ETHER:"

This line tells the client to try and PXE boot off a specified server with IP $SERVER, and suggests that the bootfile will be $ETHER. This file, as you remember, essentially points to a pxegrub file for the netinstall build.

That’s it! Make sure your BIOS points to a PXE-enabled network card on your client, reboot the box, and you should see it downloading the Solaris miniroot onto the client and then booting Solaris. As always, snoop is your friend with network troubleshooting.

CSW software repository

CSW (Community SoftWare) available through openCSW.org is wonderful repository with hundreds of software packages for Solaris 8, Solaris 9 and Solaris 10. Armed with the mighty pkg-get tool, this is definitely one of the easiest and quicked ways to get new software installed without having to compile anything to meet dependencies.

Today I’ll just give you a very brief intro, with a promise to revisit CSW in more detail later.

1. Packages storage
Create a directory where all the downloaded packages will be stored. It will be better if this directory is on a filesystem with plenty of space – it’s hard to make any space usage predictions, but I believe a couple of available gigabytes would be useful.

WARNING: this directory is merely a storage for the packages. The software installed from these packages will reside in a different directory structure – under /opt/csw (read on).

solaris# mkdir /export/dist/csw
solaris# cd /export/dist/csw

2. Using pkg-get command

This command line tool is all you’re going to ever need to download, install or update your CSW packages. Before you do anything else, you have to download the package with this tool and get it installed. I’m not showing the standard output commands will give you. Here I’m only giving you the commands you may use.

solaris# /usr/sfw/bin/wget http://www.opencsw.org/pkg_get.pkg
solaris# pkgadd -d ./pkg_get.pkg

At this stage you’ll have a /opt/csw directory created, where all the installed software will be put. pkg-get is the first installed software, and the full path to this command will be /opt/csw/bin/pkg-get

3. Configuring pkg-get

To change configuration parameters of pkg-get you’re going to edit the /opt/csw/etc/pkg-get.conf file. There are parameters you may want to change.

URL parameter in pkg-get

url – this specifies what mirror of the openCSW repository we want to use.

For best results, visit the openCSW mirrors page and select the mirror closest to you geographically.

In the pkg-get.conf file you have to find this:

# default site, in USA:
url=http://ibiblio.org/pub/packages/solaris/opencsw/current

and now edit the line to include your new mirror server, and not the default ibiblio.org. The closest one to me is heanet.ie:

# default site, in USA:
url=http://ftp.heanet.ie/pub/opencsw/current

WARNING: updating mirrors is a responsibility of their personnel, and not the opencsw.org maintainers. Because of this, some of the official mirrors might not have all the latest changes of the main repository ibiblio.org. So, if something doesn’t work, try changing this url parameter back.

UPDATED: for instance, heanet.ie didn’t seem to have the Solaris 11 support – which is essentially what all the latest Solaris Express builds are. Because of his, pkg-get couldn’t find the 5.11 catalog file for my Solaris version, so I had to get back to ibiblio.org, even though my access to it can some be really slow.

PKGADDFLAGS=-G – this line has to be uncommented if you don’t want pkg-get trying to install new packages into any Solaris zones but the global one

PKGGET_DOWNLOAD

This parameter has to be changed to reflect the location of packages storage directory:

PKGGET_DOWNLOAD_DIR=/export/dist/csw

4. CSWcommon package

This package has all the core files and directories pkg-get needs to begin with. This is also the necessary package to be installed as most of other openCSW packages depend on it.

solaris# /opt/csw/bin/pkg-get -i common

That’s it! Now you’re ready to install any CSW software packages!

Quite often it happens that the programs we run don’t work as they should, showing us errors like “referenced symbol not found” or this:

ld.so.1: ./thunderbird-bin: fatal: libmozjs.so: open failed: No such file or directory

So in order for us to find out what exact librabries are being used by a given binary, or where it expects these libraries to be found, and what libraries are not found at all, we have to take the following steps:

1. Determine what libraries are needed

First we have to see what libraries the ld linker is going to use for our file. For instance, this is how you would find our what libraries are used by the main executable of the Thunderbird mail client:

solaris$ ldd thunderbird-bin
libmozjs.so =>   (file not found)
libgtk-x11-2.0.so.0 =>   /usr/lib/libgtk-x11-2.0.so.0
libgdk-x11-2.0.so.0 =>   /usr/lib/libgdk-x11-2.0.so.0
libatk-1.0.so.0 =>       /usr/lib/libatk-1.0.so.0
libgdk_pixbuf-2.0.so.0 =>        /usr/lib/libgdk_pixbuf-2.0.so.0
libm.so.2 =>     /lib/libm.so.2
...

As always in my examples, the command output is abridged, but you get the idea. It’s clearly seen that one of the files – the libmozjs.so library – isn’t found. This means, there is no such file in either of the standard library paths or in the current working directory.

2. Confirm where dynamic libraries are expected

Now it’s about time we find our where the linker expects this libmozjs.so to be found. To do this, we’re going to use the truss command. All we’re interested in at this stage are the stat and open system calls: stat is used to verify whether some file exists, and open is used to open this file for the later access.

bash-3.00$ truss -f -t stat,open ./thunderbird-bin
13070:  stat("/export/soft/thunderbird/thunderbird-bin", 0xFFBFFAD0) = 0
13070:  open("/var/ld/ld.config", O_RDONLY)             Err#2 ENOENT
13070:  stat("/export/soft/thunderbird/libc.so.1", 0xFFBFF600) Err#2 ENOENT
13070:  stat("/export/soft/thunderbird/../libc.so.1", 0xFFBFF600) Err#2 ENOENT
13070:  stat("/usr/sfw/lib/libc.so.1", 0xFFBFF600)      Err#2 ENOENT
13070:  stat("/opt/sfw/lib/libc.so.1", 0xFFBFF600)      Err#2 ENOENT
13070:  stat("/usr/local/lib/libc.so.1", 0xFFBFF600)    Err#2 ENOENT
13070:  stat("/usr/openwin/lib/libc.so.1", 0xFFBFF600)  Err#2 ENOENT
13070:  stat("/lib/libc.so.1", 0xFFBFF600)              = 0
13070:  open("/lib/libc.so.1", O_RDONLY)                = 3
13070:  stat("/export/soft/thunderbird/libxpcom.so", 0xFFBFF418) = 0
13070:  open("/export/soft/thunderbird/libxpcom.so", O_RDONLY) = 3
13070:  stat("/export/soft/thunderbird/libmozjs.so", 0xFFBFF398) Err#2 ENOENT
13070:  stat("/export/soft/thunderbird/../libmozjs.so", 0xFFBFF398) Err#2 ENOENT
13070:  stat("/usr/sfw/lib/libmozjs.so", 0xFFBFF398)    Err#2 ENOENT
13070:  stat("/opt/sfw/lib/libmozjs.so", 0xFFBFF398)    Err#2 ENOENT
13070:  stat("/usr/local/lib/libmozjs.so", 0xFFBFF398)  Err#2 ENOENT
13070:  stat("/usr/openwin/lib/libmozjs.so", 0xFFBFF398) Err#2 ENOENT
13070:  stat("/lib/libmozjs.so", 0xFFBFF398)            Err#2 ENOENT
13070:  stat("/usr/lib/libmozjs.so", 0xFFBFF398)        Err#2 ENOENT
ld.so.1: ./thunderbird-bin: fatal: libmozjs.so: open failed: No such file or directory
13070:  stat("/export/soft/thunderbird/libgtk-x11-2.0.so.0", 0xFFBFF398) Err#2 ENOENT
...

From this output you can see very well how linker goes through one standard libraries directory to another in a cyclic search for ever library our binary needs. And for our libmozjs.so we can see that logically this library would be found in the current directory where I’m running the thunderbird-bin from, /export/soft/thunderbird. There is no such file there, simply because I’ve renamed it for this demo.

And this is how the same ldd command would look when all the files are in place:

bash-3.00$ ldd thunderbird-bin
libmozjs.so =>   /export/soft/thunderbird/libmozjs.so
libgtk-x11-2.0.so.0 =>   /usr/lib/libgtk-x11-2.0.so.0
libgdk-x11-2.0.so.0 =>   /usr/lib/libgdk-x11-2.0.so.0
libatk-1.0.so.0 =>       /usr/lib/libatk-1.0.so.0
libgdk_pixbuf-2.0.so.0 =>        /usr/lib/libgdk_pixbuf-2.0.so.0
libm.so.2 =>     /lib/libm.so.2
...

So you can be assured now that our library is found and loaded from the /export/soft/thunderbird/libmozjs.so location.

Thus, by using this simple yet very powerful combination of ldd and truss commands, you can stop guessing and find out exactly which libraries are loaded in a particular order and where they are loaded from.

3. Resolve the referenced symbols errors

For the “referenced symbol not found” errors, we have to take one more step. These errors happen mostly because the linker finds the library our binary needs, but this library somehow doesn’t have the functions we’re interested in. So, once we know the exact library file after following the steps 1 and 2 of this manual, it’s time for us to look at this particular file and see what functions are available. Many libraries have thousands of functions accessible, so it would make sense to use grep if you know the function you’re looking for.

For instance, the following command shows all the functions of the libgdk-x11-2.0 library, which have gdk_threads in their names:

solaris$ nm /usr/lib/libgdk-x11-2.0.so.0 | grep gdk_threads
[3632] | 86512| 68|FUNC |GLOB |0 |11 |gdk_threads_enter
[219] | 86648| 104|FUNC |LOCL |0 |11 |gdk_threads_impl_lock
[220] | 86752| 104|FUNC |LOCL |0 |11 |gdk_threads_impl_unlock
[4329] | 86856| 212|FUNC |GLOB |0 |11 |gdk_threads_init
[3507] | 86580| 68|FUNC |GLOB |0 |11 |gdk_threads_leave
[4095] | 532720| 4|OBJT |GLOB |0 |17 |gdk_threads_lock
[3579] | 532716| 4|OBJT |GLOB |0 |17 |gdk_threads_mutex
[4391] | 87068| 144|FUNC |GLOB |0 |11 |gdk_threads_set_lock_functions
[4443] | 532724| 4|OBJT |GLOB |0 |17 |gdk_threads_unlock

As you understand, if you get a “referenced symbol not found” error, this is quite often because an incorrect (incompatible) version of some library is used – it could be very old or just too new a version of the library, so the functions your binary is looking for are simply not there anymore or called differently.

Well, that’s all for today. If you have any additional info to add to this short manual, you’re very welcome to leave comments. I’ll be happy to hear your opinion.

X11 forwarding with SSH

X11 Forwarding with ssh is a wonderful feature which allows you get windows of a remotely started applications shown on your own desktop. For Windows, there are lots of pretty good albeit expensive products like Citrix, GoGlobal and XWin32, which allow you access your remote Unix desktop sessions. But in reality, if you don’t need any sessions but only want to remotely start an application and get a window from it on your desktop, you won’t need any of these expensive products – the easiest will be to use X11 forwarding and Xming.

Xming

Xming is an X Windows port for Microsoft Windows. Essentially it’s an X-server which starts transparently on top of your MS Windows desktop. It allows you to redirect graphical output of applications you run on remote Unix servers and therefore see these applications windows on your MS Windows desktop.

Xming is very simple and easy to use. All you have to do is download its distribution archive from the project’s page on SourceForge: XMing @ SourceForge.

After you install it, I recommend you have a quickstart panel shortcut created for it, and have your command line altered to something like this:

"C:Program FilesXmingXming.exe" :0 -clipboard -multiwindow

Configure SSHd for X11 forwarding

Now that you have Xming installed, start it and it’s time to take care of the ssh side of things. We have to alter the ssh daemon config file: /etc/ssh/sshd_config. Just ensure that it has the following:

# X11 tunneling options
X11Forwarding yes
X11DisplayOffset 10
X11UseLocalhost yes

If you already have similar parameters, don’t do anything. But if you had to change the config file, it’s time to restart ssh:

solaris# svcadm restart svc:/network/ssh

All that’s left now is to download the wonderful free Putty client, that is if you’re not using it yet. You can get it here:
Putty: a free telnet/ssh client.

I won’t go into all the Putty configuration details, I’ll only mention that for X11 forwarding, you have to do the following: in the main configuration window of Putty (you get it when you start Putty), select Connection section, then SSH, then X11, and make sure you tick the Enable X11 Forwarding option.

That’s it, you’re good to go! Good luck!