HP Data Protector collecting debug files.

Technical white paper

Version: 1.03

Table of contents
Summary ...................................................................................................................................... 3
Change the default location to store debug files ................................................................................ 3
Windows host ........................................................................................................................... 3
Set the variable OmniTmp in the registry ......................................................................................... 3
Set the variable OB2DBGDIR in the omnirc file ................................................................................ 3
Redirect the files using a path prefix : ............................................................................................. 4
Linux and Unix host .................................................................................................................... 4
Methods to start creating debugs and where to define the debug options. ............................................ 4
From the GUI ............................................................................................................................. 4
Scheduled debugging................................................................................................................. 4
From the command line............................................................................................................... 5
Enabling the trace file ................................................................................................................. 5
For Windows ............................................................................................................................... 5
For Unix/Linux ............................................................................................................................. 6
The omnirc ................................................................................................................................ 6
The cell server omnirc file .............................................................................................................. 6
The client omnirc file ..................................................................................................................... 6
Debug options ............................................................................................................................... 6
Range....................................................................................................................................... 7
Circular debugs ......................................................................................................................... 7
Timestamp in seconds and milliseconds ........................................................................................ 8
Debug files in Unicode format ..................................................................................................... 8
Postfix ....................................................................................................................................... 8
Program and hostname ............................................................................................................... 8
Debug definition changes and re-activation. .................................................................................. 9
Special debugs .......................................................................................................................... 9
Inet debugs ............................................................................................................................... 9
On Windows host: ....................................................................................................................... 9
On UNIX host ............................................................................................................................ 10
On Linux host ............................................................................................................................. 10
CRS and MMD debugs ............................................................................................................. 10
On Unix/Linux CS ...................................................................................................................... 10
For cluster configurations ............................................................................................................. 11
Omnitrig debugs ...................................................................................................................... 11
On Windows ............................................................................................................................. 11
On Unix/Linux ........................................................................................................................... 11
What debug files are needed? ...................................................................................................... 11
General debugs ....................................................................................................................... 11
Veagent debug log files ............................................................................................................ 11
VMware VDDK log files ............................................................................................................ 12
VMware log files...................................................................................................................... 12
VMware ESX(i) log files ............................................................................................................ 13
vCenter Server trivia log files ..................................................................................................... 14
How to collect debugs for MC/Serviceguard integration .............................................................. 14
Collecting CRS only debugs......................................................................................................... 14
Collecting CRS and MMD debugs ................................................................................................ 15
Collecting all debugs from cluster nodes excluding CLI and GUI debugs ........................................... 15
Disabling debugs ....................................................................................................................... 15
Collecting omniinet debugs.......................................................................................................... 15
Media Agent debugs ................................................................................................................ 15
Information required for Cell Manager and RDS problems. ........................................................... 17
The following information should be obtained ................................................................................ 17

See Appendix A for a list of helpful data for RDS problems where DP runs on HP-UX.......................... 17
Appendix A Data list for RDS problems on HP-UX........................................................................... 17
Appendix B Data list for RDS problems on Windows ...................................................................... 18
Information required for Disk and Media Agent Issues .................................................................. 19
Problem Description .................................................................................................................... 19
Collect Debugs........................................................................................................................... 20
Collect Matching Session Report. ................................................................................................. 20
Collect Supporting Data. ............................................................................................................. 20
Run the vbda as standalone command .......................................................................................... 20

Summary
Each Data Protector program can produce debug files. They contain messages that can be used in the
reconstruction of the program execution. The debug trace files are meant to inform the support
engineering, how and why something happened?

Where can I store debug files

Methods to start creating debug files
What kind of options are available when creating debug files
What specific debug files need to be created for further troubleshooting

Change the default location to store debug files

Sometimes the space in the default repository is insufficient and debug files need to be redirected to
another repository.

Windows host
By default the debug files are stored under the Omniback tmp directory.
C:\ProgramData\OmniBack\tmp (for Windows vista, 2008, 7)
C:\Program Files\OmniBack\tmp (older Windows versions)
Set the variable OmniTmp in the registry
One important note: the destination directory must exist and the permissions to the full path
need to be correct for the process writing the debugs.
Commands to query, add or delete the registry entry:
reg add HKEY_LOCAL_MACHINE\SOFTWARE\HewlettPackard\OpenView\OmniBackII\Common /v "OmniTmp" /d "C:\temp\debug"
reg query HKEY_LOCAL_MACHINE\SOFTWARE\HewlettPackard\OpenView\OmniBackII\Common /v "OmniTmp"
HKEY_LOCAL_MACHINE\SOFTWARE\Hewlett-Packard\OpenView\OmniBackII\Common
OmniTmp
REG_SZ
C:\temp\debug
reg delete HKEY_LOCAL_MACHINE\SOFTWARE\HewlettPackard\OpenView\OmniBackII\Common /v "OmniTmp"
Set the variable OB2DBGDIR in the omnirc file
for Windows 2008 and vista omnirc is now in C:\ProgramData\OmniBack
for older Windows versions (2003) omnirc is C:\Program Files\OmniBack

Set in the omnirc file

OB2DBGDIR=C:\temp\debug

Redirect the files using a path prefix :

omnib -datalist test -debug 1-500 c:\temp\output.txt

Linux and Unix host

For Linux, HP-UX debug files are stored by default in the root directory /tmp
Destination is specified in the debug option, as prefix to the debug file name:
-debug "1-99 104-140" /xyz/DBG.txt
Or with a .omnirc variable
OB2DBGDIR=/mount/depot

Methods to start creating debugs and where to define the

debug options.
From the GUI

Note that only range and postfix can be defined. Those generic debug files may be created on several
host.

Scheduled debugging
Specific backup specifications can run in debug mode when scheduled.
To enable debugs open the corresponding file in the server config schedule folder.
For barlist open Barschedules, select the type of barlist.
e.g. on W2003:
C:\Program Files\OmniBack\Config Server\Barschedules\MSSQL

on Windows 2008 and later:

C:\ProgramData\OmniBack\Config\Server\Barschedules\VEAgent
on linux/unix:
/etc/opt/omni/server/barschedules/oracle8
Look for the barlist specification, copy it first and edit the file. Add debug 1-500 full.txt as first line.
-debug 1-500 full.txt
-incr
-starting 8 5 2012
-every
-day
-at 14:30
For other backups go to the Schedules folder, and copy and edit the corresponding backup
specification file. Add debug 1-500 full.txt as first line.
To stop the debugging, remove the entry or restore the backup.

From the command line

Add option debug [range] postfix
omnib -debug 1-99 debug.txt -filesystem miki:/ -tree ...

Enabling the trace file

The trace file is located under <Data_Protector_home>\Config\Server\Options
Typical configurations are available in the trace file. Read also the header for more details.
For Windows
#--------------------------------------------------------------------------# Filename: <Data_Protector_home>\Config\Server\Options\trace
#
# $Revision: 12151 $
#
# This file is a template for Data Protector debugging. It contains many
# different debugging options for collecting debug files for different
# Data Protector modules.
#
# How to use it?
#
# 1. Uncomment line(s) containing the debug options (make sure only one
#
of each options is uncommented!):
#
#
ranges=
#
postfix=
#
select=
#
# 2. Save changed trace file (this file)
#
# 3. Make sure no Data Protector session is running
#
# 4. Restart the crs daemon to apply debug options:
#
#
Settings / Control Panel / Services / Data Protector CRS
[Stop]
#
Settings / Control Panel / Services / Data Protector CRS
[Start]
#
# 5. To turn the debugging off, set all debugging options to empty string:
#
#
ranges=

#
postfix=
#
select=
#
#
and restart the crs daemon the same way as in 4.
#---------------------------------------------------------------------------

For Unix/Linux
#--------------------------------------------------------------------------# Filename: /etc/opt/omni/server/options/trace
#
# $Revision: 12151 $
#
# This file is a template for Data Protector debugging. It contains many
# different debugging options for collecting debug files for different
# Data Protector modules.
#
# How to use it?
#
# 1. Uncomment line(s) containing the debug options (make sure only one
#
of each options is uncommented!):
#
#
ranges=
#
postfix=
#
select=
#
# 2. Save changed trace file (this file)
#
# 3. Make sure no Data Protector session is running
#
# 4. Restart the crs daemon to apply debug options:
#
#
/opt/omni/lbin/crs -redebug
#
# 5. To turn the debugging off, set all debugging options to empty string:
#
#
ranges=
#
postfix=
#
select=
#
#
and restart the crs daemon the same way as in 4.
#
#---------------------------------------------------------------------------

The omnirc
The cell server omnirc file
Run debugs on the cell server or on a specific client
OB2DBG=1-500 MA.txt 'BMA@dpsupport.hp.com,UMA@dpsupport.hp.com'
The client omnirc file
When a program starts it will always verify is a omnirc variable is set locally.
Run debugs only locally on the client
OB2DBG=1-500 bma.txt 'BMA,UMA'
Note: always restart the dataprotector inet service omniInet when removing the OB2DBG variable.

Debug options
The options are always a combination of

Range and options

Postfix
Module and host selection
Re-Activate the change

Almost all Data Protector commands can be started with an additional -debug
parameter that has the following syntax:
-debug 1-500[,C:n][,T:s][,U] XYZ [host]
where:

Range

1-500 is the debug range. Specify an extended range when instructed.

Specify optional parameters as a part of the range parameter, separated by commas.

Examples:
When setting a large range the debug files will be large. Make sure there is enough space in the
debug files repository.
The range can be split. The separator can be a , or a space within a double quoted string.
Valid ranges are:
-debug "1-99 104-140" debug.txt
-debug "1-99,104-140" debug.txt
-debug 1-99,104-140 debug.txt
Range 11, range 30 and 31, range 60 up to 63 and range 131.
OB2DBG=11,30-32,60-64,131,C:1000000 scsi.txt 'BMA'
One single range, 240
OB2DBG=240 cell_info.txt 'VEPA_BAR'

Circular debugs

C:n limits the size of debug files to n kilobytes. The minimum value is 4 (4kB) and the
default value is 1024 (1 MB).

Examples:
Select a range and make it circular
11,30-32,60-64,131,C:1000000
OB2DBG=1-99,C:2000000 MA.txt
'BMA@dpsupport.hp.com,UMA@dpsupport.hp.com'
ranges=1-500,C:100000
here the circular debug is for 100000 kBytes
Note: One Gigabyte is 1024*1024=1048576 kBytes.
In the trace file

ranges=1-500,C:2000000
postfix=BMA.txt
select=BMA@system.domain
Note: Only use circular debugs if you have limited space.

Timestamp in seconds and milliseconds

T:s is the timestamp resolution, accepted values are 0,1 and 1000, where the default
value is 1, 1000 means the resolution is one millisecond and 0 means timestamps are
turned off.

Examples:
OB2DBG=11,30-32,60-64,131,C:1000000,T:1000 scsi.txt 'BMA'
The timestamp will have a 3 digit postfix with milliseconds. 2012-04-09 10:39:03.054

Debug files in Unicode format

U is the Unicode flag. If it is specified, the debug files on Windows are written in the
Unicode format.

Examples:
OB2DBG=11,30-32,60-64,131,C:1000000,T:1000,U scsi.txt 'BMA'

Postfix

XYZ is the debug postfix, for example My_debug.txt.

Note: the postfix can be used to redirect the debug files to another directory. The destination directory
must exist and the permissions to the full path need to be correct for the process writing the debugs.

Program and hostname

host is a list of clients where debugging is turned on.

Use this option to run the debugging only on the clients specified. Delimit multiple
clients by spaces. Enclose the list in quotes, for example:
"computer1.company.com computer2.company.com". (single or double quotes can be
used)

Examples:
In the Cell Server omnirc, create BMA debug files on all MA gent
OB2DBG=11,30-32,60-64,131 bma.txt 'BMA'
In the Cell Server omnirc, create BMA and UMA debug files on dpsupport.hp.com only
OB2DBG=1-99,C:2000000 MA.txt
'BMA@dpsupport.hp.com,UMA@dpsupport.hp.com'
In the client omnirc,e.g. create only VEPA debugs on the backup host

OB2DBG=1-199,240 VEPA.txt
VEPA_BAR,VEPALIB_VMWARE_EXECUTION_THREAD,VEPALIB_VMWARE,VEPALIB_VMWARE_THR
EAD

Debug definition changes and re-activation.

When changing the OB2DBG options, restart DP services on the Cell server or omniinet services on the
client.
Also when removing the entry in the omnirc file restart DP services on the Cell server or omniinet
services on the client.
The client omniinet service will inherit the settings from the cell server, changing a setting on a CS will
not remove it from the client even if the setting was removed from the preference debug tab. unless the
omniinet is restarted on the client. So it is always safe to clean up the CS and agent when trace
activities are terminated.

Special debugs
Inet debugs
On Windows host:
Check for the omniInet service

Stop the service and enter start parameters debug 1-500 inet.txt
Select start

Now the omniIinet service is restarted in debug mode.

Note this is a non-persistent setting and is removed with the next omniinet startup.
On UNIX host
Edit the /etc/inetd.conf file:
Modify inet with the debug option
omni stream tcp6 nowait root /opt/omni/lbin/inet inet log
/var/opt/omni/log/inet.log -debug 1-500 inet.txt
Save the file and run the /etc/inetd -c command to apply the changes.
Note that this change is persistent.
On Linux host
Go to
/etc/xinetd.d
Edit omni
server_args = inet -log /var/opt/omni/log/inet.log -debug 1-500
inet.txt
Reload services
/etc/init.d/xinetd reload
Restart services
/etc/init.d/xinetd restart
Note that this change is persistent.

CRS and MMD debugs

Media Management Daemon (MMD) is started as a part of the CRS. To create debug files for Media
Management, the CRS has to be started in debug mode.
On Windows CS
Launch the Windows Service Control Manager and restart the Data Protector CRS service with the
following startup parameters:-debug 1-500 crs.txt

On Unix/Linux CS
For HP-UX, first terminate the CRS using the command:
/opt/omni/lbin/crs -shutdown

Restart the CRS with the debug option:

/opt/omni/lbin/crs -debug "1-500" crs.txt
Note the range is here an example.
For cluster configurations
Update the trace file for CRS debugging and restart the CRS service as explained in the header of the
trace file.
On Windows Server 2008 and later:
Data_Protector_program_data\Config\server\options\Trace
On other Windows systems:
Data_Protector_home\Config\server\options\Trace
HP-UX/Linux
/etc/opt/omni/server/options/trace

Omnitrig debugs
On Windows
Omnitrig is running as a part of CRS service. For testing and support purpose it can be started manually
as any ordinary command from the command line:
Omnitrig -debug 1-500 DBG
On Unix/Linux
Omnitrig is the following line into the crontab:
0,15,30,45 * * * * /opt/omni/sbin/omnitrig
If the line is changed into something like this:
0,15,30,45 * * * * /opt/omni/sbin/omnitrig -debug 1-500 omnitrig.txt
The following sequence of commands will do:
$ crontab -l > FILE
$ vi FILE
$ crontab FILE

What debug files are needed?

General debugs
In most cases general debugs are 1-500 range. The full debugs can be large, specific with DA and MA
debug files.

Veagent debug log files

If a problem is related to the VEAgent backup host the following general setting is recommended.
From the GUI Preferences Debug tab
-debug 1-199
If the problem is within the VEAgent, the unwanted BMA debugs will be very large. In order to limit the
debugs the following is recommended.
Start the inet debugs on the VEAgent backup host, add startup parameter -debug 1-500
inet.txt and restart omniInet.

11

Create a VEAgent backup host omnirc file or add the following line:
OB2DBG=1-199,240 VM.txt
"VEPA_BAR,VEPALIB_VMWARE_EXECUTION_THREAD,VEPALIB_VMWARE,VEPALIB_VMWARE_THR
EAD"
The range 1-199 is sufficient; the 240 range is adding the omni_cell content in the vepa_bar debug file.
If networking details are needed use the following range 0-199,240-270

VMware VDDK log files

When the vmware integration fails the vddk log files could reveal more information on the root case.
To enable, on the VEagent backup host, go to
C:\ProgramData\OmniBack\Config\client
Now edit the file vepa_vddk.config and change the LogLevel to the highest, 6.
vixDiskLib.transport.LogLevel="6"
The log file is dumped in the session report.
vixDiskLib.transport.LogLevel = <logLevel>
0: Quiet
1: Panic
2: Error
3: Warning
4: Informational
5: Verbose
6: Trivia

VMware log files

The Management agent (hostd), VirtualCenter Agent Service (vpxa), and VirtualCenter (vpxd) logs are
automatically rotated and maintained to manage their growth. Information in the logs can be lost if the
logs are rotated too quickly.
Note: Rotation means the turn over of files. For example, if you set the maximum number of log files to
10, after every 10 log files, the numbering restarts at 0.
Each logs are found at these locations (note that the log files are physical on datastore):

hostd
o
o

In ESX/ESXi 4.x hosts, hostd logs are located at /var/log/vmware/

In ESXi 5.0 hosts, hostd logs are located at /var/log/

vpxa
o
o

In ESX/ESXi 4.x hosts, vpxa logs are located at /var/log/vmware/vpx/

In ESXi 5.0 hosts, vpxa logs are located at /var/log/

vpxd logs are located in %ALLUSERSPROFILE%\Application Data\VMware\VMware

VirtualCenter\Logs, which translates to:
o C:\Documents and Settings\All Users\Application Data\VMware\VirtualCenter\logs
in Windows 2003

C:\ProgramData\VMware\VMware VirtualCenter\Logs in Windows 2008

This article provides steps to increase the size and number of the hostd, vpxa, and vpxd logs so that
additional data is saved. This data may be useful for troubleshooting purposes.
http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1004795

VMware ESX(i) log files

The esx(i) host has log files of all activities performed, e.g. snapshot creation, removal etc.
The log files are text files starting with hostd and are zipped once full. hostd.log is the active log file.
The files are located physical on datastore (e.g. /var/log ->/scratch/log->/vmfs/volumes/4e265cdb6b91f4b2-bc38-e4115b13545a/log)
/vmfs/volumes # find . | grep hostd
./4e2090d4-3fd60a10-eb80-e4115bb0ff04/log/hostd.log
./4e2090d4-3fd60a10-eb80-e4115bb0ff04/log/hostd.5.gz
./4e2090d4-3fd60a10-eb80-e4115bb0ff04/log/hostd.4.gz
./4e2090d4-3fd60a10-eb80-e4115bb0ff04/log/hostd.3.gz
./4e2090d4-3fd60a10-eb80-e4115bb0ff04/log/hostd.2.gz
./4e2090d4-3fd60a10-eb80-e4115bb0ff04/log/hostd.1.gz
./4e2090d4-3fd60a10-eb80-e4115bb0ff04/log/hostd.0.gz
./4e2090d4-3fd60a10-eb80-e4115bb0ff04/log/hostd.9.gz
./4e2090d4-3fd60a10-eb80-e4115bb0ff04/log/hostd.8.gz
./4e2090d4-3fd60a10-eb80-e4115bb0ff04/log/hostd.7.gz
./4e2090d4-3fd60a10-eb80-e4115bb0ff04/log/hostd.6.gz
Tip: Best is to copy/move the files into an browsable datastore, then its easy to download and to
remove the files afterwards.
E.G.
/vmfs/volumes/4e2090d4-3fd60a10-eb80-e4115bb0ff04/log # cp hostd.*
/vmfs/volumes/esx10_int

2012-02-21T17:44:26.883Z [3A4F5B90 verbose 'vm:/vmfs/volumes/4e2151a2f70c1db0-3535-e4115bb0ff00/g53/g53.vmx' opID=31603181-91] Create snapshot

request received: _DP_VEPA_SNAP_, Created by Data Protector A.07.00 on
10/02/12 09:38:41, false, false
2012-02-21T17:44:26.883Z [3A4F5B90 info 'vm:/vmfs/volumes/4e2151a2f70c1db0-3535-e4115bb0ff00/g53/g53.vmx' opID=31603181-91] State Transition
(VM_STATE_ON -> VM_STATE_CREATE_SNAPSHOT)
.
2012-02-21T04:50:28.004Z [39EB8B90 verbose 'vm:/vmfs/volumes/4e2151a2f70c1db0-3535-e4115bb0ff00/g53/g53.vmx' opID=b341f405-e5] Remove snapshot
request received: _DP_VEPA_SNAP_, 0

13

2012-02-21T04:50:28.004Z [39EB8B90 info 'vm:/vmfs/volumes/4e2151a2f70c1db0-3535-e4115bb0ff00/g53/g53.vmx' opID=b341f405-e5] State Transition

(VM_STATE_ON -> VM_STATE_REMOVE_SNAPSHOT)
2012-02-21T04:50:28.013Z [FFA23AC0 info 'Libs' opID=b341f405-e5] SNAPSHOT:
SnapshotDelete '/vmfs/volumes/4e2151a2-f70c1db0-3535e4115bb0ff00/g53/g53.vmx' : 9
2012-02-21T04:50:28.057Z [FFA23AC0 verbose 'vm:/vmfs/volumes/4e2151a2f70c1db0-3535-e4115bb0ff00/g53/g53.vmx' opID=b341f405-e5] Consolidate disks
after snapshot removal..
2012-02-21T04:50:28.057Z [FFA23AC0 verbose 'vm:/vmfs/volumes/4e2151a2f70c1db0-3535-e4115bb0ff00/g53/g53.vmx' opID=b341f405-e5] Consolidate disks
job number 11534487...
2012-02-21T04:50:29.691Z [390B6B90 verbose 'Default'] Job cb 11534487
received
2012-02-21T04:50:29.691Z [39EB8B90 verbose 'vm:/vmfs/volumes/4e2151a2f70c1db0-3535-e4115bb0ff00/g53/g53.vmx'] Done disk consolidation.
ter.

vCenter Server trivia log files

Caution: Enabling trivia or verbose logging for a longer duration might cause performance
degradation on vCenter Server. Only enable trivia or verbose logging for troubleshooting purposes.
VMware highly recommends reverting to default login (info level) immediately after the troubleshooting
is complete. VMware does not recommend trivia/verbose logging to be enabled in a production
environment. Perform the steps in this article only after consulting VMware technical support.
For more details:
http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1001584

Enabling trivia level logging is normally done by clicking:

Administration > vCenter Server Settings > Logging Options > Trivia for vCenter
Server 4.x/5.x

How to collect debugs for MC/Serviceguard integration

Note that these instructions are applicable when Data-Protector cell manager is running as a virtual
package. If the Data-Protector client is a virtual package, then it is no different from any other client.
Collecting CRS only debugs
On the node where DP package is running (use cmviewcl -v), edit
/etc/opt/omni/server/options/trace
Example:
ranges=1-99 104-500
postfix=/tmpdir/DBG
select=CRS
Get CRS to reread the trace file
# /opt/omni/lbin/crs -redebug
Debugs will be generated under /tmpdir/*DBG

Collecting CRS and MMD debugs

On the node where DP package is running (use cmviewcl -v), edit
/etc/opt/omni/server/options/trace
Example:
ranges=1-99 104-500
postfix=/tmpdir/DBG
select=CRS MMD
Restart Data-Protector daemons
# cmhaltpkg <package_hostname>
# cmrunpkg [-n <node_name>] <package_hostname>
Debugs will be generated under /tmpdir/*DBG
Collecting all debugs from cluster nodes excluding CLI and GUI debugs
On the node where DP package is running (use cmviewcl -v), edit
/etc/opt/omni/server/options/trace
Example:
ranges=1-99 104-500
postfix=/tmpdir/DBG
select=sgnode1.abc.com sgnode2.abc.com
IF you don't need MMD and KMS debug, just run the redebug
# /opt/omni/lbin/crs -redebug
(does generate MMD
or KMS debugs)
If you need MMD and KMS debugs, restart Data-Protector daemons
# cmhaltpkg <package_hostname>
# cmrunpkg [-n <node_name>]<package_hostname>
Debugs will be generated under /tmpdir/*DBG
Disabling debugs
Edit trace file and delete or comment out the statements included previously
If you enable debugging using redebug option then do the same again.
# /opt/omni/lbin/crs -redebug
If you enable debugging by restarting Data-Protector daemons then do the same again.
# cmhaltpkg <package_hostname>
# cmrunpkg [-n <node_name>]<package_hostname>
Collecting omniinet debugs
Edit omni entry in /etc/inetd.conf
Example
omni stream tcp nowait root /opt/omni/lbin/inet inet -log
/var/opt/omni/log/inet.log -debug 1-99,104-500 /tmpdir/DBG
Get inetd to reread its new configuration
inetd -c
Debugs will be generated under /tmpdir/*DBG

Media Agent debugs

For Media Agent specific problems, the following general guidelines can be applied:
1-200 for most common cases

15

1-300 for SHMIPC

1-350 for consolidation
1-505 for memory tracking
Note that ranges 220,221 are used for timing information. These ranges are hardcoded and hence not
seen in the list of range definitions below.
Include these ranges to get device timing information.

Here's a list of range definitions:

#define DBG_MA_OPTIONS
#define DBG_MA_NETINIT

2
3

#define DBG_MA_LIBDEV
#define DBG_MA_LIBSCSI
#define DBG_MA_AUX

61
62
63

#define
#define
#define
#define
#define

DBG_MA_WORK
DBG_MA_WORK_LOW
DBG_MA_CATALOG
DBG_MA_LIBPOL
DBG_MA_LIBMH

31
131
32
33
34

#define
#define
#define
#define

DBG_MA_STATES
11
DBG_MA_CONSOLIDATE 12
DBG_MA_CONSOLIDATE_LOW 112
DBG_MA_CONSOLIDATE_LOW_LOW 212

#define DBG_MA_CONMSG
#define DBG_MA_NETMSG
#define DBG_MA_NETEVENT

21
22
23

#define DBG_MA_LIBCTRL

51

#define DBG_MA_UTIL
71
#define DBG_MA_UTIL_SIMULATE 72
#define DBG_MA_UTIL_SIMULATE_LOW 172
#define DBG_MA_SCSITAB_ERR
#define DBG_MA_SCSITAB

78
207

#define
#define
#define
#define

47
147
247
347

DBG_FTS
DBG_FTS_LOW
DBG_FTS_LOWLOW
DBG_FTS_HASHTABLES

Information required for Cell Manager and RDS problems.

The following information should be obtained
An exact, and detailed, description of what the problem is. Supply a copy of any error messages.
When did the problem start?
Details any change to the backup environment that may have caused the problem.
Can the problem be reproduced at will? If the problem can be reproduced, include details of the
exact steps that cause the problem.
Explain how the customer expects DP to work when it is OK.
If possible, run an omnidbcheck -extended` and supply the text output. If, for time reasons,
an omnidbcheck -extended` is not possible, run `omnidbcheck` and supply the text output.
If the customer has found a workaround for the problem, let us know what that is.
If the problem is seen, or caused, using the GUI, include whether the GUI runs on the DP Cell
Manager or another system. If another system, where possible, see if the problem also exists when
using a GUI running on the Cell Manager. If the problem exists on a GUI running on another
system, include details of that system (hostname, DP version and patch level, Operating System).
Confirm if the IDB backup runs OK, or not. If they have problems with their IDB backup confirm
when they last had a successful IDB backup.
See Appendix A for a list of helpful data for RDS problems where DP runs on HPUX.
See Appendix B for a list of helpful data for RDS problems where DP runs on Windows.
Regarding the collection DP debug files for RDS crashing problems.
If RDS is crashing intermittently, then usually DP debugs file collection is not required.
If a specific operation is causing RDS to crash/hang, then debugs of that operation should be
collected.
Regarding the collection of the RDS debug file (RDSdebug.log).
It is possible to put RDS in a Raw Logging mode by use of the rdmserver setting
LogRawCoreAPI=1.
This is not usually required in the initial data for an RDS problem. If it is required it will be
requested by the Data Protector lab.
Appendix A Data list for RDS problems on HP-UX
Minimum data
All /var/opt/omni/server/db40/datafiles/catalog/RDS* log files
grep -v ^# /etc/opt/omni/server/options/global | grep -v "^ *$" >
globalactivelines.txt
omnicheck -patches > patchDP_check.txt
tail -n 5000000 /var/opt/omni/log/debug.log > debug_tail.log
Further Usual Data
The data listed in the "Minimum data" section +...
All /var/opt/omni/server/log/Check_* log files
/var/opt/omni/server/db40/datafiles/catalog/rdmserver.ini file
/var/opt/omni/server/log/HealthCheck.log file
/opt/omni/.omnirc file
grep -v ^# /opt/omni/.omnirc | grep -v "^ *$" >
omnircactivelines.txt
/etc/opt/omni/server/options/global file
/etc/opt/omni/server/options/trace file

grep -v ^# /etc/opt/omni/server/options/trace | grep -v "^ *$" >

traceactivelines.txt
crontab -l > crontab_op.txt
what /stand/vmunix > what_vmunix.txt
uname -a > uname_a.txt
model > model_out.txt
swlist -l fileset | grep -i OV DP > patchDP_swlist.txt
swlist -l product > swlist_prod.txt
swlist -l fileset > swlist_file.txt
swlist -l fileset -a state > swlist_a.txt
tail -n 5000000 /var/opt/omni/log/inet.log > inet_tail.log
omnicc -ver > omnicc_ver.txt
sysdef > sysdef_out.txt
kctune | egrep -i "maxdsiz|semmnu|maxssiz" > kctune_siz.txt
kctune > kctunelist.txt
bdf > bdf.txt
du -skx /var/opt/omni/server/db40 > db40_du_skx.txt
omnidbutil -info > omnidbutil_info.txt
omnidbutil -extendinfo > omnidbutil_extendinfo.txt
omnidbutil -list_large_directories 100000 > /list_dirs_100000.txt
***
/etc/opt/omni/server/cell/cell_info file
/etc/opt/omni/client/cell_server file
ls -l /var/opt/omni/server/log > ls_varoptomniserverlog.txt
ls -l /var/opt/omni/log > ls_varoptomnilog.txt
ls -l /var/opt/omni/tmp/ > var_opt_omni_tmp.txt
ps -ef | grep omni > ps_ef_omni.txt
ps -ef > ps_ef.txt
tail -n 5000000 /var/adm/syslog/syslog.log > syslog_tail.log
omnirpt -report db_purge_preview > purge_preview.txt
***
/opt/omni/sbin/utilns/get_info > get_info_op.txt
***
Copy of the contents of dirs...
/etc/opt/omni/server/rptgroups
/etc/opt/omni/server/rptschedules
If a core file exists in either /var/opt/omni/log or var/opt/omni/server/log
file core text output
If DP 6.2x please also collect
omnicc -encryption -status all > omnicc_encry_status_all.txt
Extra Data that may reduce the chance that we will ask for more.
The data listed in the "Minimum data" and "Further Usual Data" sections above.
tail -n 1000 /var/opt/omni/server/log/media.log > medialog_tail.txt
omnidb -sess -last 20 > omnidb_sess_last_20.txt
omnidownload -list_libraries > list_libs.txt
omnidownload -list_devices > list_devs.txt
omnidownload -list_devices -detail > list_devs_det.txt
omnidownload -list_libraries -detail > list_libs_det.txt
omnicellinfo -schinfo > omnicellinfo_schinfo.txt`
omnidbutil -list_dcdirs > list_dcdirs.txt`
omnidbutil -list_large_directories 500000 -top 50 -detail >
list_dirs_500000.txt
***
/var/opt/omni/server/db40/logfiles/rlog/obrindex.dat file
omnicc -debug 20 SETTINGS.txt
- makes a file in /tmp ending in SETTINGS.txt
*** Note: These commands can put a load on DP, best run when cell is quiet.
Appendix B Data list for RDS problems on Windows
Note: The PATH values list are default PATHs for DP on Windows 2008.

Minimum data
All RDS* files from C:\ProgramData\OmniBack\db40\datafiles\catlog\
C:\ProgramData\OmniBack\Config\Server\Options\global
omnicheck -patches > patchDP_check.txt
C:\ProgramData\OmniBack\log\debug.log
Further Usual Data
The data listed in the "Minimum data" section +...
C:\ProgramData\OmniBack\log\server\log\Check_* log files
C:\ProgramData\OmniBack\db40\datafiles\catlog\rdmserver.ini
C:\ProgramData\OmniBack\log\server\log\HealthCheck.log file
C:\ProgramData\OmniBack\omnirc file (if it exists)
C:\ProgramData\OmniBack\Config\Server\Options\trace file
C:\ProgramData\OmniBack\log\inet_tail.log file
omnicc -ver > omnicc_ver.txt
omnidbutil -info > omnidbutil_info.txt
omnidbutil -extendinfo > omnidbutil_extendinfo.txt
omnidbutil -list_large_directories 100000 > /list_dirs_100000.txt
***
C:\ProgramData\OmniBack\Config\Server\cell\cell_info file
omnirpt -report db_purge_preview > purge_preview.txt
***
/opt/omni/sbin/utilns/get_info > get_info_op.txt
***
Copy of the contents of dirs...
C:\ProgramData\OmniBack\Config\Server\rptgroups
C:\ProgramData\OmniBack\Config\Server\rptschedules
dir listing of directory
C:\ProgramData\OmniBack\tmp
dir listing of directory
C:\ProgramData\OmniBack\DB40
If DP 6.2x please also collect
omnicc -encryption -status all > omnicc_encry_status_all.txt
If the server is Windows 2003, the boot.ini file.
The System and Event log files in text format.
Extra Data that may reduce the chance that we will ask for more.
The data listed in the "Minimum data" and "Further Usual Data" sections above.
omnidb -sess -last 20 > omnidb_sess_last_20.txt
omnidownload -list_libraries > list_libs.txt
omnidownload -list_devices > list_devs.txt
omnidownload -list_devices -detail > list_devs_det.txt
omnidownload -list_libraries -detail > list_libs_det.txt
omnicellinfo -schinfo > omnicellinfo_schinfo.txt`
omnidbutil -list_dcdirs > list_dcdirs.txt`
omnidbutil -list_large_directories 500000 -top 50 -detail >
list_dirs_500000.txt
***
If a process is crashing, sometimes it will be necessary to enable Dr. Watson logging and
supply Dr. Watson logs of the failing process.
*** Note: These commands can put a load on DP, best run when cell is quiet.

Information required for Disk and Media Agent Issues

Problem Description
Clearly describe the specific issue to be addressed in the case. The problem description in the
case needs to only describe a single problem. The description will need to include the following:

List the exact steps the customer performed which produced the undesired result. For
example, specify the exact command, what buttons were selected in the GUI, and what
was entered in any of the GUI fields.

Describe exactly what you and the customer observed.

Describe exactly what you and the customer expected.

Specify when the problem first occurred.

Specify what has changed in the environment that could have caused the problem.

Describe, if possible, what you or the customer has determined as a workaround.

Collect Debugs

Verify the latest Disk Agent, Media Agent and Core binaries have been properly installed
on the problematic client. A patch list is welcome.

Isolate the problem to as small of an area as possible. For example, if multiple hosts are
involved in the issue, isolate the issue to a single host. If multiple volumes are involved,
isolate the issue to a single volume. If multiple files are involved, isolate the issue to a
single file.

Reproduce the issue using the following debug ranges.

1-500 debugs for normal sessions
1-999 If the Disk Agent appears to be hanging, core-dumping, or abending (Netware).
These can be circular debugs.

Collect the debugs logs from the Cell Manager, Media Agent client(s) and problematic
Disk Agent client(s).

Verify the debug files are complete. The set will need to include disk agent, media agent,
Session Manager (BSM, RSM, CSM & MSM), debug.log from each host and complete
session report.

Verify the debugs were created for the correct debug range.

Verify the Disk Agent debugs are from the latest binary.

Package the files in either in WinZip format or a standard format on HP-UX.

Collect Matching Session Report.

From the command line use the command omnidb and direct the output of the session
report which matches the debugs to a file named session.out. This will be an ascii text file.
Note: Do not convert this file to a different file format.

Collect session.out.

Verify session.out has captured the issue which the WTEC case is opened for.

Verify timestamps in the session report match the timestamps of the debugs logs.

Use session.out to verify that debugs for every host reported has been accounted for.

Collect Supporting Data.

In many cases it will become necessary to collect the following supporting information.

Windows System: drwtsn32.log, System & Application Event Logs Note: Windows
Event logs must be saved as text files only. Do not supply .EVT formatted event logs.

Novell Netware: Config .txt file and Abend log file if the issue is for a Novell Netware
abend.

Run the vbda as standalone command

It is often interesting to run the vbda as a command. E.g. for HP-UX/ Linux
# /opt/omni/lbin/vbda -vol /apps/all/dir -trees /apps/all/dir/admin out /dev/null -profile -debug 1-500,T:1000 /alternate/debug.txt
Here debugs are created under the alternate directory.

Here are examples on Windows besides volumes as C:,D:,E:.. we have also /CONFIGURATION.
In quick mode, without reading data:
vbda -vol /CONFIGURATION -profile -preview
In physical reading mode:
vbda -vol /CONFIGURATION -profile -out nul
In debug mode:
vbda -vol /CONFIGURATION -profile -out nul -debug 1-200 out.txt
In debug with reporting:
vbda -vol /CONFIGURATION -profile -report 0 -out nul -debug 1-200
out.txt

Share with colleagues