Storage Migration – Solaris with VxVM

Storage Migration – Solaris with VxVM in detailed steps

click here


Two node cluster storage migration from EMC to hitachi

1) Consider the data copying is done in the background from SAN side.
1. Sun Explorer
2. EMC Grab output
3. Emc database files
[/etc/power.custom ; /etc/emcp_devicesDB.dat
4. Output of “inq –no_dots”
5. Output of “cldev show –v “
6. Cldev list –v |grep voor1 ( scdidadm –l)

Actual migration
1) Shutdown the application in production
2) Offline all the resource groups.
3) Add the quorum server quorum device
follow –
4) Check the quorum configuration.
In cluster nodes run
# clq status
# clq list -v
5) Remove the emc quorum disk from cluster – once you confirm there is quorum added from qurum server.
6) Check the quorum server configuration
#clq status
# clq list -v
This will show the minimum quorum devices needed and available.
7) Reboot one node to check that new quorum device is working fine.

8) Disable EMC powerpath to enable native Multipathing on both the servers.
#svcadm disable /system/emcpower/powerstartup
# sysadm disable /system/emcpower/powershift

9) We have to configure native Multipathing on one server and reboot ,
10) Repeat step 9 on node2 if step9 CHECK is successful.
11) All the devices are offline, cluster is running on quorum server and native mulstipathing is enabled and emc powerpath is disabled.
At this stage SAN can remove EMC luns and assign HDS Lun.

12) San team will un-present luns
Below commands will refresh the DID list and remove the disk from OS control
#cldevice refresh
#devfsadm -Cv

13) SAN team will present only quorum disk via Hitachi
After SAN team presents the LUN, we have to scan the disks.
#cfgadm -c configure c2/c3
#devfsadm –Cv
#cldevice populate
At this point we shall see the quorum disk from Hitachi LUNS and Global device ID will be generated.

14) Now we need to add the quorum disk .

#cldev list or scdidadm -L

15) We need to remove the quorum server device now.
( )
# clsetup
[Select Quorum>Remove a quorum device]
[Answer the questions when prompted.] remove mint04
Quit the clsetup Quorum Menu and Main Menu.]
[Verify that the quorum device is removed:]
# clquorum list -v
Next clean up the cluster quorum server configuration
clquorumserver clear -c clustername -I clusterID quorumserver [-y]
#clqs show
16) SAN team present other LUNS
After SAN team presents the LUN, we have to scan the disks.
#cfgadm -c configure c2/c3
#devfsadm –Cv
#cldevice populate

At this point we shall see the other Hitachi LUNS and Global device ID will be generated.

17) Now we shall online the cluster RG’s one by one in below order
# clrg online <res group>

 If you have to copy data from OS side then follow same procedure to add the hitachi disks including quorum server.
Mirror the disk.
remove EMC disk
boot from Hitachi disk
repopulate disk
configure quorum on hitachi disk and then remove quorum server.
Make sure you are configuring Quorum disk only after removing EMC disk and repopulating device.

I have successfully migrated Two node cluster from EMC to hitachi using above procedure.

Credit:  vikrant Raut

Recover Root Password from ZFS File System

ok>boot cdrom -s
#zpool import

# zpool import rpool

#zfs list

# zfs get mounted rpool/ROOT/s10s_u10_08a

# zfs set mountpoint=/mnt rpool/ROOT/s10s_u10_08a

# zfs mount rpool/ROOT/s10s_u8wos_08a

#zfs list

Now you can change the password from /mnt/etc/shadow file by either using vi editor or

# cd /

# zfs umount rpool/ROOT/s10s_u10_08a

# zfs set mountpoint=/ rpool/ROOT/s10s_u10_08a
#init 6

more clink here

ILOM 3.0 Sparc Servers

First let’s start with how to access ILOM:

1. if NetMgmt port configured, using ssh you can login to the ILOM

2.If you configure SerialMgmt Port, then when you connect to serial port, you will access ILOM

3. if you connect to console thorught netmgmt or serial mgmt ports and you connect Operating System. Use #. (Hash symbol with . (dot)) which will take you to the ILOM.

From ILOM to go back to the Host Operating System then start the console service ->start /SP/console

-> start /SP/console  ==>ILOM CLI prompt
Are you sure you want to start /SP/console (y/n)? y

Serial console started.  To stop, type #.

bash-3.2#   –> OS prompt
Serial console stopped.

->  ==>ILOM CLI prompt

Notes on ILOM CLI follow hierarchial Architecture

ILOM Target types:

/SP – configuring ILOM service processor (SP)

/SYS – inventory/environmental and hardware management

/HOST – monitoring and managing host operating system.

Blade Platforms:

/CMM – Blade platform this replace SP, used to configure ILOM Chassis Monitoring Module

/CH – Blade platform this replace /SYS and provides inventory, environmentals and hardware management

ILOM CLI commands:


ILOM CLI Syntax:

command [options] [target] [properties

ex.,: set /SP/services/https port=80 servicestate=enabled

User Management ILOM CLI:

Roles in CLI: Admin|Operator (a), User Management (u), Console (c), Reset and Host control (r) and Read Only (o)

1. Add a local user

create /SP/users/user1 password=password  role=a|u|c|r|o|s

2.  Delete a local user

delete /SP/users/user1

 3. Change a local user’s properties

set /SP/users/user1  role=operator

 4. Display information about all local users

show -display [targets|properties|all ] -level all /SP/users

 5. Display information aboutLDAP settings

show /SP/clients/ldap

6. Change LDAP settings

set /SP/clients/ldap binddn=proxyuser bindpw=proxyuserpassword defaultrole=a|u|c|r|o|s address=ipaddress

 Set ILOM clock to sync with NTP server

set /SP/clients/ntp/server/1 address=ntpIPAddress

To configure IP on NetMgmt port set values to the /SP/network properties

        commitpending = (Cannot show property)
        dhcp_server_ip = (none)
        ipaddress = (none)
        ipdiscovery = (none)
        ipgateway = (none)
        ipnetmask = (none)
        macaddress = xx:xxx:xx:xx:xx:xx
        managementport = /SYS/MB/SP/NETMGMT
        outofbandmacaddress = xx:xx:xx:xx:xx:xx
        pendingipaddress = (none) –>IPV4 address
        pendingipdiscovery = (none)
        pendingipgateway = (none)  –> Gateway
        pendingipnetmask = (none) –> Netmask
        pendingmanagementport = /SYS/MB/SP/NETMGMT
        sidebandmacaddress = xx:xx:xx:xx:xx:xx
        state = disabled

you can configure above values using set command

set /SP/network pendingipaddress=

set /SP/network pendingipnetmask=

set /SP/network pendingipgateway=

Configuring SNMP and email alert:

set /SP/alertmgmt/rules/1…15 type=snmtrap snmp_version=3 comunity_or_username=username destination=ipaddress level=down|critical|major|minor

set /SP/alertmgmt/rules/1…15 type=email level=down|critical|major|minor

To enable web access to ILOM, can http

set /SP/services/http port=80 secureredirect=enabled|disabled servicestate=enabled|disabled

set /SP/services/https port=xxx servicestate=enabled|disabled

HOST system commands (*warning: if your host is up and running, HOST commands can cause outage)

To start the host system ->start /SYS

To stop the host system -> stop /SYS

Force shutdown (required when host hung)->stop -f /SYS

Reset the host system  (required when host hung) ->reset /SYS

start a sessions to connect to host console -> /SP/console

stop the console session -> /SP/console

Filtering output options:

Users with admin roles -> show /SP/users -level all role==”a*”

SNMP trap alerts ->show /SP/alertmgmt -level all type==”snmptrap”

List services which are disabled -> show /SP/services -level all servicestate==disabled

List memory moduels with some part number -> show /SYS -level all type==”DIMM” fru_part_number==07014642

list all hard disks -> show /SYS -level all type==”Hard Disk”

How to backup the ILOM configuration

cd /SP/config

set passphrase=passphrase

set dump_uri=transfer_method://username:password@ipaddress_or_hostname/directorypath/filename

set dump_uri=scp://adminuser:userpswd@x.x.x.x/Backup/Lab9/SP-backup.config

How to Restore the ILOM configuration

Note: you have to login ILOM with user which has Admin,User Management, Console, Reset, Host control and Read only (a,u,c,r,o) roles.

cd /SP/config

set passphrase=passphrase

set load_uri=transfer_method://username:password@ipaddress_or_hostname/directorypath/filename

How to update ILOM Firmware

Pre requisite: Shutdown the host running on the hardware whose ILOM firmware you are upgrading

1.Login to ILOM CLI with user which has Admin role

2. ->version

3. Download the zip file packge and copy to TFTP server that is accessible from network

4.load -source <supported_protocol>://<server ip>/<path_to_firmware_image/<>

Do you want to preserve the configuration (y/n)?y

Once firmware loades it automaticall reboots to complete the firmware update

ILOM CLI is based on DMTF CLP (Distributed Management Task Force specification Server Management Command-Line Specification, Version 11.0a.8 Draft


Rename a guest domain (Solaris LDOM)

Disclaimer: I test following procedure to rename the guest domain. It worked for me, but it’s may not be the right procedure. Do not try on your prod environment

Rename the Guest domain:
Bring down the guest domain OS to OK prompt

bash-3.2# ldm list old-ldg1
old-ldg1 active     -t—-  5000    16    8G       6.2%  23h 17m

#ldm stop-domain old-ldg1
LDom old-ldg1 stopped

bash-3.2# ldm list old-ldg1
old-ldg1 bound      ——  5000    16    8G

Then run unbind command, it will change to inactive state

bash-3.2# ldm unbind old-ldg1
bash-3.2# ldm list old-ldg1
old-ldg1 inactive   ——          16    8G

Save the guest domain configuration to a xml

bash-3.2# ldm list-constraints -x old-ldg1 > /var/tmp/old-ldg1.xml

Create copy of the xmls

cp -p /var/tmp/old-ldg1.xml /var/tmp/new-ldg1.xml

bash-3.2# grep -i old-ldg1 /var/tmp/old-ldg1.xml
<Content xsi:type=”ovf:VirtualSystem_Type” ovf:id=”old-ldg1″>
<gprop:GenericProperty key=”vol_name”>old-ldg1_sys</gprop:GenericProperty>
<gprop:GenericProperty key=”vol_name”>old-ldg1_sys</gprop:GenericProperty>
<gprop:GenericProperty key=”block_dev”>/dev/zvol/dsk/root/vdsk_old-ldg1_sys</gprop:GenericProperty>
bash-3.2# cp /var/tmp/old-ldg1.xml /var/tmp/new-ldg1.xml
bash-3.2#perl -pi -e ‘s/old-ldg1/new-ldg1/g’ /var/tmp/new-ldg1.xml
bash-3.2# grep -i new-ldg1 /var/tmp/new-ldg1.xml
<Content xsi:type=”ovf:VirtualSystem_Type” ovf:id=”new-ldg1″>
<gprop:GenericProperty key=”vol_name”>new-ldg1_sys</gprop:GenericProperty>
<gprop:GenericProperty key=”vol_name”>new-ldg1_sys</gprop:GenericProperty>
<gprop:GenericProperty key=”block_dev”>/dev/zvol/dsk/root/vdsk_new-ldg1_sys</gprop:GenericProperty>

For ldm 3.x version please remove existing domain configuration to clear uuid. Thanks to Neeraj and Raphael for their comments

bash-3.2# ldm remove-domain old-ldg1

bash-3.2# ldm add-domain -i /var/tmp/new-ldg1.xml
bash-3.2# ldm list
primary          active     -n-cv-  UART    8     1G       1.1%  1d 3h 24m
new-ldg1 inactive   ——          16    8G
bash-3.2# ldm bind new-ldg1
bash-3.2# ldm start new-ldg1
LDom new-ldg1 started
bash-3.2# ldm list
primary          active     -n-cv-  UART    8     1G       4.8%  1d 3h 25m
new-ldg1 active     -t—-  5000    16    8G       3.5%  2s

Verify connecting to new guest domain console:

bash-3.2# telnet localhost 5000
Connected to localhost.
Escape character is ‘^]’.

Connecting to console “new-ldg1” in group “new-ldg1” ….
Press ~? for control options ..

{0} ok
telnet> quit
Connection to localhost closed.

Installing Oracle VM for Sparc 2.2 (LDOM) and Configuration

I have installed Oracle VM Server for Sparc 2.2 on Sparc T4-1 server

Pre requisites:

supported Platforms:

OS: Solaris 10 8/11 or Solaris 11 SRU 7 (Support Repository Update)

For Sparc T4-1 required firmware patch:  148822-01

148822-01 Synopsis: Obsoleted by: 148822-02 FIRMWARE: SPARC T4-1 – Sun System Firmware 8.2.0.a
Date: May/22/2012

We have uptodate firmware i.e.: 148822-04

Synopsis: Obsoleted by: 148822-04 FIRMWARE: SPARC T4-1 – Sun System Firmware 8.2.1.b
Date: Aug/31/2012


  1. Download and install the package “”
  2. cd /var/tmp/ldom (Assuming copied .zip file in this folder)
  3. unzip
  4. cd OVM_Server_SPARC-2_2/Install
  5. ./install-ldm

Welcome to the Oracle VM Server for SPARC installer.

You are about to install the LDoms Manager package, SUNWldm, that will enable
you to create, destroy and control other domains on your system. You will
also be given the option of running the Oracle VM Server for SPARC Configuration
Assistant (ldmconfig) to setup the control domain and create guest domains.

If the Solaris Security Toolkit (SST) is installed, you will be prompted to
optionally harden your control domain.

Installing LDoms packages.
pkgadd -n -d “./Product” -a pkg_admin SUNWldm.v
Copyright (c) 2010, Oracle and/or its affiliates. All rights reserved.

Installation of <SUNWldm> was successful.
pkgadd -n -d “./Product” -a pkg_admin SUNWldmp2v
Copyright (c) 2010, Oracle and/or its affiliates. All rights reserved.

Installation of <SUNWldmp2v> was successful.

Oracle VM Server for SPARC Configuration

Once installed, you may configure your system for a basic LDoms
deployment.  If you select “y” for the following question, the Oracle
VM Server for SPARC Configuration Assistant (tty) will be launched
following a successful installation of the packages.

(You may launch the Configuration Assistant at a later time with the
command:  /usr/sbin/ldmconfig, or use the GUI Configuration Assistant
which is bundled in the Oracle VM Server for SPARC zip file – see
README.GUI for more details)

Select an option for configuration:

y) Yes, launch the Configuration Assistant after install
n) No thanks, I will configure the system manually later

Enter y or n [y]:n

Verify installed packages:

bash-3.2# pkginfo -l SUNWldmp2v
   PKGINST:  SUNWldmp2v
      NAME:  LDoms P2V tool
  CATEGORY:  application
      ARCH:  sparc
   BASEDIR:  /
    VENDOR:  Oracle Corporation
      DESC:  Oracle VM Server for SPARC Physical-to-Virtual (P2V) Migration Tool
    PSTAMP:  scapen-on10-020120419141017
  INSTDATE:  Dec 21 2012 15:20
   HOTLINE:  Please contact your local service provider
    STATUS:  completely installed
     FILES:       19 installed pathnames
                   6 shared pathnames
                  12 directories
                   3 executables
                 264 blocks used (approx)

bash-3.2# pkginfo -l SUNWldm
      NAME:  LDoms Manager software
  CATEGORY:  application
      ARCH:  sparc.sun4v
   BASEDIR:  /
    VENDOR:  Oracle Corporation
      DESC:  Oracle VM Server for SPARC – Virtualization for SPARC T-Series
    PSTAMP:  scapen-on10-020120419141012
  INSTDATE:  Dec 21 2012 15:20
   HOTLINE:  Please contact your local service provider
    STATUS:  completely installed
     FILES:       79 installed pathnames
                  18 shared pathnames
                  23 directories
                  22 executables
                5851 blocks used (approx)

Verify the ldms

bash-3.2# svcs|grep ldmd
online         15:20:55 svc:/ldoms/ldmd:default

bash-3.2# ldm list
primary          active     -n-c–  UART    64    32256M   0.2%  1d 4h 38m

Save and Restore ldm configuration:

bash-3.2# tar -cvpf /var/tmp/autosave.tar var/opt/SUNWldm/autosave-*
a var/opt/SUNWldm/autosave-_Default_SP_Config/ 0K
a var/opt/SUNWldm/autosave-_Default_SP_Config/md-primary.bin 61K
a var/opt/SUNWldm/autosave-_Default_SP_Config/hv-md.bin 22K
a var/opt/SUNWldm/autosave-_Default_SP_Config/timestamp 1K
a var/opt/SUNWldm/autosave-_Default_SP_Config/sp-pri.bin 88K
a var/opt/SUNWldm/autosave-_Default_SP_Config/sp-timestamp 1K

to restore

cd /

tar -xvpf /var/tmp/autosave.tar

Saving and Restorig LDOM constraint Database
backup and restore the xml file: /var/opt/SUNWldm/ldom-db.xml

create default services after installing the LDOM manager:

3 defaults services need to be created on primary ldom:

virtual control concentrator service (vcc) service for use by the virtual network terminal server
daemon (vntsd) and as a concentrator for all logical domain consoles.

virtual disk server (vds) to allowimporting virtual disks into a logical domain

virtual switch service (vsw) to enable networking between virtual network (vnet)
devices in logical domains

VCC # ldm add-vcc port-range=5000-5100 vcc_primary primary
VDS # ldm add-vds vds0_primary primary
VSW # ldm add-vsw net-dev=nxge0 vsw0_primary primary

MAC address to vsw can be manually set using below command: only on Solaris 10:

primary# ldm add-vsw mac-addr=2:04:4f:fb:9f:0d net-dev=nxge0 primary-vsw0 primary

list services:
#ldm list-services primary
    NAME             LDOM             PORT-RANGE
    vcc_primary primary          5000-5100

    NAME             LDOM             MAC               NET-DEV   ID   DEVICE     LINKPROP   DEFAULT-VLAN-ID PVID VID                  MTU   MODE   INTER-VNET-LINK
    vsw0_primary primary          00:14:4f:fb:af:0d nxge0     0    switch@0              1               1                         1500         on     

    NAME             LDOM             VOLUME         OPTIONS          MPGROUP        DEVICE
    vds0_primary primary
bash-3.2# ldm set-vcpu 8 primary
bash-3.2# ldm list-constraints primary

bash-3.2# ldm start-reconf primary
Initiating a delayed reconfiguration operation on the primary domain.
All configuration changes for other domains are disabled until the primary
domain reboots, at which time the new configuration for the primary domain
will also take effect.

bash-3.2# ldm set-memory 1G primary
Notice: The primary domain is in the process of a delayed reconfiguration.
Any changes made to the primary domain will only take effect after it reboots.

bash-3.2# ldm add-config initial
bash-3.2# ldm list-config
initial [current]

bash-3.2# /usr/sbin/shutdown -y -g0 -i6

How to stop all domains:

1. power down all domains OS to OK prompt
2. primary# ldm stop-domain -a
3. primary# ldm unbind-domain ldom

 To remove all guest domains, run above 3 steps and below one step
4. ldm remove-domain -a

 How to upgrade LODM Manager:

1. svcadm disable ldmd
2. pkgrm SUNWldm
3. Instll the new package (copied to /var/tmp)
cd /var/tmp;pkgadd -Gd. SUNWldm.v
4. ldm list (verify)

 How to disable logical domain manager and remove the package?
1. svcadm disable ldmd
2. pkgrm SUNWldm; pkgrm SUNWldmp2v


Oracle®VMServer for SPARC 2.2 Administration Guide : E23807.pdf