LDOM Oracle VM for SPARC Tutorial 1

LDOM

SUN LDOMS aka oracle VM for SPARC is a complete virtual machines that runs
an independent operating systems and contains its own virtual CPUs,Storage,Memory,
Cryptographic and console.Unlike Solaris zones,LDOMS can run its own operating
system with different kernel patch levels. You can also install Solaris 8,Solaris 9 ,Solaris
10 and Solaris 11 on LDOMS. LDOM also provides the OBP for each logical domains but
we wont get that in zones. LDOM also allow to move the resources across the logical
domains or virtual hosts exclusively. LDOM also allows you to configure zones under
that.

Prerequisite:
Oracle/Sun T-series Server (T5140 Firmware:7.3.3 )

Solaris 10 or Solaris 11 (To be directly installed on T-series server)

LDOM Packages (Ex: 2.1)
Server Hardware Console Access
Contents:
Basic Concepts of LDOM

Installation of LDOM Software

Configuring the Service Domains
Configuring the Control Domains
Saving the SP-Profile
Basic Concepts of LDOM or Oracle VM for SPARC:
LDOM is a hardware based virtualization which works on top the hypervisor. To
understand better, you should know the type of terms used in LDOM or oracle VM for
SPARC.

Control Domain is the place where you are going to install the LDOM software and
managing the complete logical domain environment.It used to configure the
resources and guest domains
Service domain provides the various virtual services to guest domains.Virtaul services
can be virtual disks,network switches or virtual consoles.Mostly we will be
configuring the control domain as service domain too.
I/O Domains have a direct ownership of PCI Bus.You need to directly map the I/O
devices to guest domain to improve the application performance in some cases.
 There is an another domain called root domain and this domain has direct
ownership of PCI devcies. So it also called I/O domain.
Guest domains doesnt performing any of the above mentioned roles.It jsut runs the
operating system instances.
Control domain and service domain can be run together.But the user application
should not be installed on this domain in a order to protect domain stability and
performance.

LDOM
Basic concept
Installation of LDOM software:
Download the LDOM packages from the oracle support website. For your
information ,you cant download without having the oracle support login credentials.
Once you have downloaded the packages ,just copy the packages to the Solaris host
which you have already installed on T- series server.

1.The below mentioned packages needs to be installed .

-rwxr-xr-x 1 root root 3564851 Jan 16 23:26 OracleVM-Server-for-SPARC-2.0.0_(64-

bit).zip

drwxrwxrwx 5 root root 512 Jan 16 23:26 SUNWldomr.v

drwxrwxrwx 5 root root 512 Jan 16 23:26 SUNWldomu.v

2.Install the SUNWldomr and SUNWldomu packages first.

-bash-3.2# pkgadd -d . SUNWldom*

Processing package instance from </>

Solaris Logical Domains (Root)(sparc.sun4v) 11.10.0,REV=2006.10.04.00.26

Copyright (c) 2011, Oracle and/or its affiliates. All rights reserved.

This appears to be an attempt to install the same architecture and

version of a package which is already installed. This installation

will attempt to overwrite this package.

Using </> as the package base directory.

## Processing package information.

## Processing system information.

42 package pathnames are already properly installed.

## Verifying package dependencies.

## Verifying disk space requirements.

## Checking for conflicts with packages already installed.

## Checking for setuid/setgid programs.

This package contains scripts which will be executed with super-user

permission during the process of installing this package.

Do you want to continue with the installation of [y,n,?] y

Installing Solaris Logical Domains (Root) as

## Executing preinstall script.

## Installing part 1 of 1.

[ verifying class ]

## Executing postinstall script.

Installation of was successful.

Processing package instance from </>

Solaris Logical Domains (Usr)(sparc.sun4v) 11.10.0,REV=2006.08.08.12.13

Copyright (c) 2011, Oracle and/or its affiliates. All rights reserved.

This appears to be an attempt to install the same architecture and

version of a package which is already installed. This installation

will attempt to overwrite this package.

Using </> as the package base directory.

## Processing package information.

## Processing system information.

11 package pathnames are already properly installed.

## Verifying package dependencies.

## Verifying disk space requirements.

## Checking for conflicts with packages already installed.

## Checking for setuid/setgid programs.

This package contains scripts which will be executed with super-user

permission during the process of installing this package.

Do you want to continue with the installation of [y,n,?] y

Installing Solaris Logical Domains (Usr) as

## Installing part 1 of 1.

Installation of was successful.

-bash-3.2#

3.Install the LDOM software . Unzip the package and continue the installation like
below
-bash-3.2# cd OVM_Server_SPARC-2_0

-bash-3.2# ls -lrt

total 12

-r--r--r-- 1 root root 3368 Sep 17 2010 README

drwxr-xr-x 2 root root 512 Sep 18 2010 Install

drwxr-xr-x 7 root root 512 Dec 16 2010 Product

-bash-3.2# cd Install/

-bash-3.2# ls -lrt

total 56

-rwxr-xr-x 1 root root 28471 Sep 18 2010 install-ldm

-bash-3.2# ./install-ldm

Welcome to the Oracle VM Server for SPARC installer.

You are about to install the LDoms Manager package, SUNWldm, that will enable

you to create, destroy and control other domains on your system. You will

also be given the option of running the Oracle VM Server for SPARC Configuration

Assistant (ldmconfig) to setup the control domain and create guest domains.

If the Solaris Security Toolkit (SST) is installed, you will be prompted to

optionally harden your control domain.

Installing LDoms packages.

pkgadd -n -d "/OVM_Server_SPARC-2_0/Product" -a pkg_admin SUNWldm.v

Copyright (c) 2010, Oracle and/or its affiliates. All rights reserved.

Feb 17 10:37:54 sol10-10 sendmail[2631]: [ID 702911 mail.alert] unable to qualify my own
domain name (sol10-10) -- using short name

Installation of was successful.

pkgadd -n -d "/OVM_Server_SPARC-2_0/Product" -a pkg_admin SUNWldmp2v

Copyright (c) 2010, Oracle and/or its affiliates. All rights reserved.

Feb 17 10:38:54 sol10-10 sendmail[2705]: [ID 702911 mail.alert] unable to qualify my own
domain name (sol10-10) -- using short name

Installation of was successful.

Oracle VM Server for SPARC Configuration

Once installed, you may configure your system for a basic LDoms

deployment. If you select "y" for the following question, the Oracle

VM Server for SPARC Configuration Assistant (tty) will be launched

following a successful installation of the packages.

(You may launch the Configuration Assistant at a later time with the

command: /usr/sbin/ldmconfig, or use the GUI Configuration Assistant

which is bundled in the Oracle VM Server for SPARC zip file - see

README.GUI for more details)

Select an option for configuration:

y) Yes, launch the Configuration Assistant after install

n) No thanks, I will configure the system manually later

Enter y or n [y]: n

Verifying that all packages are fully installed. OK.

You have new mail in /var/mail//root

-bash-3.2#

Configuring the Service Domains:

Login to the Solaris host where you have installed the LDOM or oracle VM software.

1.List the configured services.By default it will be nothing.

-bash-3.2# ldm list-services

2.Create a new virtual disk service.This service is responsible to provide the virtual
disks to guest domains.

-bash-3.2# ldm add-vdiskserver primary-vds0 primary

-bash-3.2# ldm list-services

VDS
 NAME LDOM VOLUME OPTIONS MPGROUP DEVICE

primary-vds0 primary

-bash-3.2#

3.Create a new virtual switch. By creating this , we can create N-number of VNIC and
share to the guest domains.

-bash-3.2# dladm show-dev

nxge0 link: up speed: 1000 Mbps duplex: full

nxge1 link: up speed: 1000 Mbps duplex: full

nxge2 link: up speed: 1000 Mbps duplex: full

nxge3 link: up speed: 1000 Mbps duplex: full

-bash-3.2# ifconfig -a

lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1

inet 127.0.0.1 netmask ff000000

nxge0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2

inet 192.168.2.11 netmask ff000000 broadcast 192.168.2.255

ether 0:21:98:57:cd:16

-bash-3.2# ldm add-vswitch net-dev=nxge1 primary-vsw0 primary

-bash-3.2#

-bash-3.2# ldm list-services

VSW

NAME LDOM MAC NET-DEV ID DEVICE LINKPROP

DEFAULT-VLAN-ID PVID VID MTU MODE
 primary-vsw0 primary 00:14:4f:f9:05:b1 nxge1 0 switch@0
1 1 1500

VDS

NAME LDOM VOLUME OPTIONS MPGROUP DEVICE

primary-vds0 primary

-bash-3.2#

Here i have configured virtual switch using nxge1 . You can also use nxge0 configure
the virtual swtich but you need to remove the IP from nxge0 and need to configure the
IP over virtual switch.

-bash-3.2# ifconfig -a

lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1

inet 127.0.0.1 netmask ff000000

vsw0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 3

inet 198.168.2.11 netmask ff000000 broadcast 192.168.2.255

ether 0:14:4f:98:83:f3

-bash-3.2#

4.Create a virtual service for terminal consoles.

-bash-3.2# ldm add-vconscon port-range=5000-5100 primary-vc0 primary

-bash-3.2# ldm list-services

VCC

NAME LDOM PORT-RANGE

 primary-vc0 primary 5000-5100

VDS

NAME LDOM VOLUME OPTIONS MPGROUP DEVICE

primary-vds0 primary

-bash-3.2#

VSW

NAME LDOM MAC NET-DEV ID DEVICE LINKPROP

DEFAULT-VLAN-ID PVID VID MTU MODE

primary-vsw0 primary 00:14:4f:f9:05:b1 nxge1 0 switch@0

1 1 1500

Now you have successfully configured the default services or service domain.

(Creating the spconfig is must after making any changes on the service.Otherwise you
will loose the changes after the system power cycle.)

Configuring the control Domains:

You need to login to Solaris host where you have installed the ldom software to
configure the control domain.

1.List the logical domain devices using ldm list-device -a command.You can see all the
resource has been allocated to the primary domain.

bash-3.2# ldm list-devices -a

CORE

ID %FREE CPUSET

0 0 (0, 1, 2, 3, 4, 5, 6, 7)
 1 0 (8, 9, 10, 11, 12, 13, 14, 15)

2 0 (16, 17, 18, 19, 20, 21, 22, 23)

3 0 (24, 25, 26, 27, 28, 29, 30, 31)

4 0 (64, 65, 66, 67, 68, 69, 70, 71)

5 0 (72, 73, 74, 75, 76, 77, 78, 79)

6 0 (80, 81, 82, 83, 84, 85, 86, 87)

7 0 (88, 89, 90, 91, 92, 93, 94, 95)

VCPU

PID %FREE PM

0 0 no

1 0 no

2 0 no

3 0 no

4 0 no

5 0 no

6 0 no

7 0 no

8 0 no

9 0 no

10 0 no

11 0 no
12 0 no

13 0 no

14 0 no

15 0 no

16 0 no

17 0 no

18 0 no

19 0 no

20 0 no

21 0 no

22 0 no

23 0 no

24 0 no

25 0 no

26 0 no

27 0 no

28 0 no

29 0 no

30 0 no

31 0 no

64 0 no

65 0 no
66 0 no

67 0 no

68 0 no

69 0 no

70 0 no

71 0 no

72 0 no

73 0 no

74 0 no

75 0 no

76 0 no

77 0 no

78 0 no

79 0 no

80 0 no

81 0 no

82 0 no

83 0 no

84 0 no

85 0 no

86 0 no

87 0 no
 88 0 no

89 0 no

90 0 no

91 0 no

92 0 no

93 0 no

94 0 no

95 0 no

MAU

ID CPUSET BOUND

0 (0, 1, 2, 3, 4, 5, 6, 7) primary

1 (8, 9, 10, 11, 12, 13, 14, 15) primary

2 (16, 17, 18, 19, 20, 21, 22, 23) primary

3 (24, 25, 26, 27, 28, 29, 30, 31) primary

4 (64, 65, 66, 67, 68, 69, 70, 71) primary

5 (72, 73, 74, 75, 76, 77, 78, 79) primary

6 (80, 81, 82, 83, 84, 85, 86, 87) primary

7 (88, 89, 90, 91, 92, 93, 94, 95) primary

MEMORY

PA SIZE BOUND
 0x0 512K _sys_

0x80000 1536K _sys_

0x200000 94M _sys_

0x6000000 32M _sys_

0x8000000 96M _sys_

0xe000000 32544M primary

IO

DEVICE PSEUDONYM BOUND OPTIONS

pci@400 pci_0 yes

pci@500 pci_1 yes

bash-3.2#

2.List what are the resources assigned to the primary domain or control domains.

bash-3.2# ldm list-bindings primary

NAME STATE FLAGS CONS VCPU MEMORY UTIL UPTIME

primary active -n-c-- SP 64 32544M 0.1% 9m

UUID

f063b0a8-8fb0-4e61-cd5b-91826d2c1550

MAC
 00:21:28:57:cd:16

HOSTID

0x8557cd16

CONTROL

failure-policy=ignore

DEPENDENCY

master=

CORE

CID CPUSET

0 (0, 1, 2, 3, 4, 5, 6, 7)

1 (8, 9, 10, 11, 12, 13, 14, 15)

2 (16, 17, 18, 19, 20, 21, 22, 23)

3 (24, 25, 26, 27, 28, 29, 30, 31)

4 (64, 65, 66, 67, 68, 69, 70, 71)

5 (72, 73, 74, 75, 76, 77, 78, 79)

6 (80, 81, 82, 83, 84, 85, 86, 87)

7 (88, 89, 90, 91, 92, 93, 94, 95)

VCPU

VID PID CID UTIL STRAND

0 0 0 1.5% 100%

1 1 0 0.0% 100%

2 2 0 0.0% 100%

3 3 0 0.0% 100%

4 4 0 0.0% 100%

5 5 0 0.0% 100%

6 6 0 0.0% 100%

7 7 0 0.0% 100%

8 8 1 0.0% 100%

9 9 1 0.1% 100%

10 10 1 0.0% 100%

11 11 1 0.1% 100%

12 12 1 0.1% 100%

13 13 1 3.0% 100%

14 14 1 0.0% 100%

15 15 1 0.1% 100%

16 16 2 0.1% 100%

17 17 2 0.2% 100%

18 18 2 0.1% 100%

19 19 2 0.0% 100%
20 20 2 0.1% 100%

21 21 2 0.0% 100%

22 22 2 0.1% 100%

23 23 2 0.0% 100%

24 24 3 0.0% 100%

25 25 3 0.0% 100%

26 26 3 0.0% 100%

27 27 3 0.0% 100%

28 28 3 0.2% 100%

29 29 3 0.1% 100%

30 30 3 0.1% 100%

31 31 3 0.0% 100%

64 64 4 0.0% 100%

65 65 4 0.0% 100%

66 66 4 0.0% 100%

67 67 4 0.0% 100%

68 68 4 0.0% 100%

69 69 4 0.0% 100%

70 70 4 0.0% 100%

71 71 4 0.0% 100%

72 72 5 0.0% 100%

73 73 5 0.0% 100%
74 74 5 0.0% 100%

75 75 5 0.0% 100%

76 76 5 0.0% 100%

77 77 5 0.0% 100%

78 78 5 0.0% 100%

79 79 5 0.0% 100%

80 80 6 0.0% 100%

81 81 6 0.0% 100%

82 82 6 0.0% 100%

83 83 6 0.0% 100%

84 84 6 0.0% 100%

85 85 6 0.0% 100%

86 86 6 0.0% 100%

87 87 6 0.0% 100%

88 88 7 0.0% 100%

89 89 7 0.0% 100%

90 90 7 0.0% 100%

91 91 7 0.0% 100%

92 92 7 0.0% 100%

93 93 7 0.0% 100%

94 94 7 0.0% 100%

95 95 7 0.0% 100%
MAU

ID CPUSET

0 (0, 1, 2, 3, 4, 5, 6, 7)

1 (8, 9, 10, 11, 12, 13, 14, 15)

2 (16, 17, 18, 19, 20, 21, 22, 23)

3 (24, 25, 26, 27, 28, 29, 30, 31)

4 (64, 65, 66, 67, 68, 69, 70, 71)

5 (72, 73, 74, 75, 76, 77, 78, 79)

6 (80, 81, 82, 83, 84, 85, 86, 87)

7 (88, 89, 90, 91, 92, 93, 94, 95)

MEMORY

RA PA SIZE

0xe000000 0xe000000 32544M

IO

DEVICE PSEUDONYM OPTIONS

pci@400 pci_0

pci@500 pci_1

VCONS
 NAME SERVICE PORT

SP

bash-3.2#

3.Set the resource control for the control domain aka primary domain.So the resource
will be freed up for the logical domains / Guest domains.

bash-3.2# ldm set-mau 1 primary

bash-3.2# ldm set-vcpu 8 primary

bash-3.2# ldm set-memory 4G primary

A ldm set-mem 4G command would remove 28448MB, which is not a

multiple of 256MB. Instead, run ldm set-mem 4128M primary

to ensure a 256MB alignment.

bash-3.2# ldm set-memory 3872M primary

mau Security Encryption (LDM commands will be sent in encrypted manner.)

vpcu Virtual CPU cores
memory Physical memory
4.Now list the devices again to verify the new settings.

bash-3.2# ldm ls-devices -a

CORE

ID %FREE CPUSET

0 0 (0, 1, 2, 3, 4, 5, 6, 7)

1 100 (8, 9, 10, 11, 12, 13, 14, 15) ----------->(1-7) Free CPU cores

2 100 (16, 17, 18, 19, 20, 21, 22, 23)

 3 100 (24, 25, 26, 27, 28, 29, 30, 31)

4 100 (64, 65, 66, 67, 68, 69, 70, 71)

5 100 (72, 73, 74, 75, 76, 77, 78, 79)

6 100 (80, 81, 82, 83, 84, 85, 86, 87)

7 100 (88, 89, 90, 91, 92, 93, 94, 95)

VCPU

PID %FREE PM

0 0 no ----------These (0-7) CPU is not available since its assigned to

primary domain

1 0 no

2 0 no

3 0 no

4 0 no

5 0 no

6 0 no

7 0 no

8 100 ---

9 100 ---

10 100 ---

11 100 ---

12 100 ---

13 100 ---
14 100 ---

15 100 ---

16 100 ---

17 100 ---

18 100 ---

19 100 ---

20 100 ---

21 100 ---

22 100 ---

23 100 ---

24 100 ---

25 100 ---

26 100 ---

27 100 ---

28 100 ---

29 100 ---

30 100 ---

31 100 ---

64 100 ---

65 100 ---

66 100 ---

67 100 ---
68 100 ---

69 100 ---

70 100 ---

71 100 ---

72 100 ---

73 100 ---

74 100 ---

75 100 ---

76 100 ---

77 100 ---

78 100 ---

79 100 ---

80 100 ---

81 100 ---

82 100 ---

83 100 ---

84 100 ---

85 100 ---

86 100 ---

87 100 ---

88 100 ---

89 100 ---
 90 100 ---

91 100 ---

92 100 ---

93 100 ---

94 100 ---

95 100 ---

MAU

ID CPUSET BOUND

0 (0, 1, 2, 3, 4, 5, 6, 7) primary

1 (8, 9, 10, 11, 12, 13, 14, 15)

2 (16, 17, 18, 19, 20, 21, 22, 23)

3 (24, 25, 26, 27, 28, 29, 30, 31)

4 (64, 65, 66, 67, 68, 69, 70, 71)

5 (72, 73, 74, 75, 76, 77, 78, 79)

6 (80, 81, 82, 83, 84, 85, 86, 87)

7 (88, 89, 90, 91, 92, 93, 94, 95)

MEMORY

PA SIZE BOUND

0x0 512K _sys_

0x80000 1536K _sys_

 0x200000 94M _sys_

0x6000000 32M _sys_

0x8000000 96M _sys_

0xe000000 32M primary

0x10000000 28G -----------------------------Free Memory

0x710000000 3840M primary -----------Allocated to Primary or control

Domain

IO

DEVICE PSEUDONYM BOUND OPTIONS

pci@400 pci_0 yes

pci@500 pci_1 yes

bash-3.2#

You have successfully configured the control domain.

Note:Here you havent saved the configuration.These are all in-memory

configuration.After the power cycle, you will loose the configuration if its not saved in
sp config profile.

Saving the sp-config file:

You have to save the configuration after every ldom settings that you made. For an
example, if you didnt save, ldom configuration will be flushed after server power cycle.

1.Create a new profile the current configuration.

bash-3.2# ldm add-spconfig unixarena

bash-3.2# ldm ls-spconfig

factory-default

unixarena [current]

bash-3.2#

In the above command output, you can see that currents settings are saved in
unixarena and this settings will come up after the system reboot and power cycle.

For an example , if i do any small changes on ldom ,it will not update in the spconfig
file.

bash-3.2# ldm set-vcpu 12 primary

bash-3.2# ldm ls-spconfig

factory-default

unixarena [next poweron]

bash-3.2#

Here we just increased the number vpcus to 12 . The previous number of vpcu was 8.
Here sp-config file unixarena will be activated on next power cycle. Which mean you
will loose the cpu setting you just done. In a order to save the configuration, just re-
create the sp-config like below.

bash-3.2# ldm ls-spconfig

factory-default

unixarena [next poweron]

bash-3.2# ldm remove-spconfig unixarena

bash-3.2# ldm ls-spconfig

factory-default [next poweron]

bash-3.2# ldm add-spconfig unixarena_new

bash-3.2# ldm ls-spconfig

factory-default

unixarena_new [current]

bash-3.2#

Checking the OBP,Hypervisor and LDOM version:

-bash-3.2# ldm -V

Logical Domain Manager (v 2.0)

Hypervisor control protocol v 1.4

Using Hypervisor MD v 1.1

System PROM:

Hypervisor v. 1.7.4. @(#)Hypervisor 1.7.4.a 2009/09/21 08:25\015

OpenBoot v. 4.30.4 @(#)OBP 4.30.4 2009/08/19 07:25

-bash-3.2#

Hope this article is first step to learn LDOM for Solaris beginners.In this article we have
seen the basic concepts of oracle VM for SPARC (LDOM), packages
installation,configuration the service domain,configuring the control domain or primary
domain and saving the sp-configuration.

I will try to complete the tutorial by adding another 6 to 8 articles. Hope you will like it .
Please share it in social media to reach maximum number of people. Thank you for
visiting UnixArena.