Solaris 10 Installation on VMware ESX Server 2.5.x

If you have problems with the integrity of the install DVD ISOs and don't want you download everything again, you can use the following checksums to find out, which file is defective:

For Solaris 10 GA (3/05): chunks-GA.md5
For Solaris 10 Update 1 (1/06): chunks-u1.md5

You now have the following files:

sol-10-GA-x86-ESX.iso or --> ISO image of bootable Solaris install DVD

sol-10-u1-ga-x86-ESX.iso for VMware ESX Server 2.5.x

vmware-solaris-tools.iso --> ISO image of VMware Tools (vmware-guestd

and vmware-toolbox) installation CD

Create a VM on your ESX Server:
- Select Linux as Guest Operating System
- Don't define more than one CPU. The installation on virtual SMP systems isn't possible, because of an incompatibility of the APIC handling between Solaris 10 and VMware ESX 2.5.x
- Define at least 512 MB Memory (better 1 GB)
- Set SCSI Adapter to LSI Logic
- Define a Hardisk of at least 2 GB
  Use at least 8 GB for a complete installation
- Set Network Adapter type to vlance
- Define the CDROM to connect at power on and
  use ISO image sol-10-GA-x86-ESX.iso for Solaris 10 GA (3/05)
  or sol-10-u1-ga-x86-ESX.iso for Solaris 10 Update 1 (1/06)
  
  Note: In some cases it has been reported, that the installation stalls after displaying the "Solaris Interactive" message, if the ISO image is stored on a NFS or CIFS/SMB share. If you experience this problem, please copy the ISO image into your /vmfs tree or on a local disk of your ESX server.
Power on the VM. The boot process needs some time, so be patient.

The system will boot without any attention to the “Solaris Interactive” installation process and come up with the graphic installer. You can as usual interrupt before the installer comes up to define your preferred video, keyboard and mouse settings. The following settings work fine:

Install Solaris as usual.
If you installed Xorg and Xsun X-Windows support, on the first reboot after install the system will default to Xorg, which is not working very well. Just login using line mode, call kdmconfig to change it back to Xsun and use the same settings as during install. If kdmconfig then comes up at a later reboot, tell it not to check again and the X-Windows system will work perfect.
Install VMware Tools:
- Define the CDROM to use ISO image vmware-solaris-tools.iso
- Login as root
- Mount the CDROM and change directory to it:
  
  If you don’t have vold installed, you will find the CDROM at /dev/dsk/c0t0d0s2. So issue for example
  
  # mount –rF hsfs /dev/dsk/c0t0d0s2 /mnt
  # cd /mnt
  #
  
  If you have vold installed issue:
  
  # eject cdrom
  # volcheck
  # cd /cdrom/vmware-solaris-tools
  #
- Run the installer and reboot:
  
  # ./vmware-tools-install
  # reboot
  #
Have fun with your Solaris 10 VM!

Some Remarks:

Of course anything mentioned here is completely unsupported by VMware and Sun. So use at your own risk!
Each virtual disk attached to a Solaris VM must have a corresponding entry in /kernel/drv/sd.conf of the form

name="sd" parent="mpt" target=x lun=0
allow-bus-device-reset=0x0
sd_max_xfer_size=0x40000;

where x is the SCSI target address of the disk. The sd.conf installed using the above procedure contains this entry for one disk at target address 0. You will be able to use additional disks only after adding the corresponding entries and doing a reconfigure reboot.

If you installed an Update 1 (1/06) system using an ISO image downloaded before Jan/25/2006, I strongly recommend removing all entries from /kernel/drv/lsimega.conf and /kernel/drv/st.conf, as these might cause unwanted effects on the SCSI bus. Later downloads already contain empty lsimega.conf and st.conf.
The Solaris pcn driver for the PCnet Lance Ethernet interface has severe problems (lost packets, hangs) driving the virtual PCnet interface of VMware ESX. Installing Solaris 10 using the procedure above automatically replaces the pcn driver with the ae driver, which is available from Nexenta OS (OpenSolaris + GNU, www.gnusolaris.org). This driver works stable under VMware ESX. Because of the replacement, your network interface is listed as ae0 instead of pcn0.
The implementation of VMware Tools is done using the VMware Tools binaries for Linux through a modified version of the Linux emulation "lxrun". The vmware-guestd daemon for Linux uses the ifconfig command to get configuration information from the network interface. The interface name as well as the output format of ifconfig are different between Linux and Solaris, which is compensated with a simple script (/opt/vmware/linux/sbin/ifconfig) using the interface name ae0. If you are using a different interface, you need to replace ae0 in this script with the name of your virtual network interface (for example pcn0 or, on VMware WS, e1000g0).
The CDROM image vmware-solaris-tools.iso corresponds to ESX Server 2.5.1. On newer ESX Versions you will probably get the “need upgrade” warning. This really doesn’t matter, because only a small part of VMware Tools functionality is usable on Solaris. But if you want, you can upgrade easily:

Put your VM into “Install VMware Tools” mode
mount the CD
unzip and untar the Linux(!) distribution as usual and enter

# cd vmware-tools-distrib/sbin
# cp -p vmware-guestd vmware-checkvm /opt/vmware/linux/sbin
# cd ../bin
# cp -p vmware-toolbox /opt/vmware/linux/bin
# reboot
#

Installation of VMware Tools provides partial vmware-guestd and vmware-toolbox functionality. vmware-guestd gets started automatically during boot through /etc/init.d/vmware-tools and the corresponding links in the /etc/rcX.d directories. To start vmware-toolbox just call it as usual by entering /usr/bin/vmware-toolbox&. The following functionality is available:

- enter and leave console window without CTRL-ALT
- Copy and paste to and from console window
- Connect and disconnect removable devices through the “Devices” tab.

Note: If you want to run vmware-toolbox to a remote X-Server tunneled through an ssh session, you need to make localhost known as 127.0.0.1 in /opt/vmware/linux/etc/hosts or change lxrun_pathmap to map this to /etc/hosts.
The implementation of VMware Tools through the Linux emulation lxrun also works for other VMware platforms (Workstation, Player, etc.). See http://wotho2.ethz.ch/vmware/tools for details.
You can update Solaris 10 to a current level using a Sun Subscription. If you do so, you need to be be aware of the following facts:

Patch 120041-02 installs an incompatible version of ata.bef, which will hang Configuration Assistant. This is a Solaris bug, which also occurs on some real hardware, so it’s not VMware related. Just back this patch out, it has no dependencies.

Update Jan/10/2006: With patch 118844-27 Sun finally managed to release the grub based boot process and abandoned that dreadful Configuration Assistant; ata.bef has been deleted and patch 120041-02 is obsoleted. So this problem is solved on all systems with patch 118844-27 or newer.
Solaris kernel patch 118844-18 through 118844-23 installs an incompatible version of /platform/i86pc/kernel/unix, which will fail on boot during memory initialization because of an invalid value of “physmax”. This bug is documented in Sun's Bug ID 6342422. It also occurs on some real hardware and thus is not VMware related. Do not install any of these revisions of kernel patch 118844. Older or newer revisions are safe to install.

The only publicly released revision of the defective kernel patch is 118844-20. If you really need to install this revision you can use script ./ESX_patches/mpt_1.26/physmax_patch from http://wotho.ethz.ch/ESX_solaris/Solaris_10_patches_for_VMware_ESX.tar to patch /platform/i86pc/kernel/unix.

Update Jan/10/2006: Patch 118844-27 contains the fix for bug 6342422 (from 118844-24 which wasn’t published). The physmax bug is solved on all systems with patch 118844-24 or newer; physmax_patch ist no longer needed, do not try to install it on systems with patch 118844-24 or newer.
Solaris 10 as installed with the procedure outlined here uses a modified version of Sun’s driver for the LSI Logic SCSI adapter (/kernel/drv/mpt). The Solaris 10 GA (3/05) distribution contains mpt version 1.24 and thus the modified version is also based on 1.24 (check for example using “modinfo | grep mpt”). Patch 119851-02 updates the mpt driver to Sun’s unmodified mpt version 1.26, patch 119851-08 updates it to version 1.34, patch 119851-16 updates it to version 1.38. The updated versions will make your system unbootable, if you use them as distributed by Sun.

To be able to boot with mpt 1.26, 1.34 or 1.38 the same changes need to be applied as to 1.24. This can be done using script ./ESX_patches/mpt_x.xx/mpt_patch from http://wotho.ethz.ch/ESX_solaris/Solaris_10_patches_for_VMware_ESX.tar, where x.xx is the mpt version (1.26, 1.34 or 1.38) to patch. To run the script, the xxd utility (can be found on the Software Companion CD/DVD) needs to be installed and in your PATH. To be sure to patch the correct version of mpt and get the correct result, it is advisable, to have the md5sum utility available and check the MD5 checksum of the mpt binary before and after running the mpt_patch script. Use for example the following procedure:

# tar xf Solaris_10_patches_for_VMware_ESX.tar
# cd ESX_patches/mpt_x.xx
# cp -p /kernel/drv/mpt .
# md5sum -c md5_orig
mpt: OK
# ./mpt_patch
patching mpt
done!
# md5sum -c md5_patched
mpt: OK
# cp -p mpt /kernel/drv
#

Caution: Never apply the mpt_patch script to other versions of mpt than indicated above, because this will definitely result in an unbootable system. And, of course, make sure to run the mpt_patch script after applying the corresponding Sun patch (119851-vv) but before rebooting, because the mpt binaries distributed by Sun will render your system unbootable. If you did reboot before running mpt_patch, you need to boot another system (for example the failsave/miniroot system from my Solaris 10 Update 1 (1/06) media) to be able to access your disk and then run mpt_patch.

In mpt version 1.34 Sun introduced an undocumented switch through which mpt can be forced not to download firmware to the LSI Logic SCSI adapter. Because VMware’s virtual LSI Logic SCSI adapter (vmxlsilogic) doesn’t support firmware download the mpt_patch scripts for versions 1.24 and 1.26 implemented code that suppresses the download. This code is not contained in mpt_patch for version 1.34 and 1.38. To suppress the firmware download with mpt version 1.34 or 1.38 you need to add the following line to /etc/system

set mpt:mpt_disable_firmware_download = 1

If you don’t add this line, your system will NOT boot with mpt 1.34 or 1.38.

The Solaris 10 Update 1 (1/06) distribution already contains mpt version 1.34. Installing my modified Solaris 10 Update 1 (1/06) distribution following the instructions outlined above installs the modified mpt 1.34 and adds the “mpt_disable_firmware_download” switch automatically to /etc/system.

Solaris 10 Update 1 (1/06) and Solaris 10 GA (3/05) systems patched to level 118844-27 or higher use the GNU/grub boot loader instead of the old Configuration Assistant. Obviously VMware’s LBA support doesn’t work reliable together with Sun’s implementation of the grub based boot process:

During shutdown processing Solaris automatically creates a new boot_archive (ramdisk) if there is the slightest chance that any of it’s contents might have changed since the last reboot. If you don’t use a boot partition and your virtual disk is big and full enough you’ve a good chance to end up with an unbootable system after a normal shutdown, because grub cannot address the newly created boot_archive any more.

Although Sun doesn’t, I strongly recommend using a boot partition to circumvent this problem. Or, at least keep a working copy (test it!) of the boot_archive, that you can use, in case a newly created one isn’t addressable any more.

Changes:

Jan/10/2006: Updates to the “Remarks” section due to publication of patch 118844-27 from Sun

Jan/12/2006: Added vmware-toolbox support to the VMware Tools ISO

Jan/12/2006: Added VMware Tools ISO for VMware Workstation 5.0.0 build-13124

Jan/14/2006: Added remark on VMware Tools dependancy on the network interface name ae0

Jan/15/2006: Added remark on pcn versus ae driver for the virtual PCnet Lance Ethernet interface

Jan/19/2006: Added support for Solaris 10 Update 1 (1/06)

Jan/23/2006: Added remark on sd.conf, st.conf and lsimega.conf
Jan/25/2006: Uploaded new Solaris 10 Update 1 (1/06) ISO with empty lsimega.conf and st.conf

Feb/10/2006: Added VMware Tools ISO for VMware Workstation 5.5.1 build-19175

Mar/06/2006: Moved information concerning VMware Tools for other VMware platforms (Workstation, etc.) to a separate website

Mar/10/2006: added checksum for each chunk of the install DVD ISOs

Mar/10/2006: added information on unsupported virtual SMP to the virtual machine definition instructions

May/08/2006: added support for mpt version 1.38 from Solaris 10 patch 119851-16