Académique Documents
Professionnel Documents
Culture Documents
Connectivity
Understanding “How To” manage connectivity issues for Control-M V9 Implementations
Jim Gingras
SOUTHEASTERN CONTROL-M USER GROUP |
HTTPS://COMMUNITIES.BMC.COM/GROUPS/SOUTH-EAST-USER-GROUP
TABLE OF CONTENTS
TABLE OF CONTENTS .............................................................................................................................................1
1 OVERVIEW .......................................................................................................................................................2
1.1 Basic Control-M Application Architecture ...........................................................................................3
1.2 Basic Connectivity Testing and Validation ..........................................................................................4
2 ENTERPRISE MANAGER CONNECTIVITY .............................................................................................................7
2.1 Summary.............................................................................................................................................9
3 AGENT CONNECTIVITY................................................................................................................................... 10
3.1 Summary.......................................................................................................................................... 13
4 DESKTOP CLIENT CONNECTIVITY ................................................................................................................... 14
4.1 Summary.......................................................................................................................................... 18
5 AGENTLESS CONNECTIVITY ........................................................................................................................... 19
5.1 Summary.......................................................................................................................................... 21
APPENDICES ......................................................................................................................................................... 22
6 CONTROL-M CONNECTIVITY UTILITIES ........................................................................................................... 22
7 V9 PORTS .................................................................................................................................................... 24
8 PRE-INSTALLATION PORT CONSIDERATIONS ................................................................................................... 25
9 POST INSTALLATION CONNECTIVITY ISSUES ................................................................................................... 26
10 ARCHIVAL CONNECTIVITY ........................................................................................................................... 28
11 SELF SERVICE .......................................................................................................................................... 29
12 ADVANCED TROUBLESHOOTING FOR CONTROL-M V9 CONNECTIVITY ISSUES ................................................ 30
Control-M V9 Connectivity 1 of 30
1 OVERVIEW
Today’s modern IT architectures still take advantage of standard TCP/IP fundamentals that are
foundational to application and solution architectures. Whether those solutions are running on servers,
virtual machines, or in the cloud. Whether they are being created on conventional hardware and
infrastructure or in hybrid converged environments where all devices are virtual as well as their
connections or whether the entire environments are being created with tools like Terraform or Dockers.
They all rely on the standard Domain Naming Standards running on top of some type of TCP/IP stack
that needs to route packets to their destinations.
Control-M V9 Connectivity 2 of 30
1.1 Basic Control-M Application Architecture
A key consideration when determining the installation and implementation requirements for Control-M is
that it IS an Enterprise Class Infrastructure application. In other words, this application will be operating
and executing workloads across your entire IT Enterprise. Whether that IT Enterprise is internal,
external or hybridized with the cloud. Control-M requires TCP/IP communications across all the IT
Domains you intend to run your enterprise workload.
The diagram above shows the basic Control-M application architecture. It is a three-tiered functional
architecture that can be implemented on one server or many depending on the size of the enterprise
workload:
Enterprise Manager (EM) – Responsible for user interfaces and coordinating workload across
Control-M servers
Control-M Server (CTM) – Responsible for workload execution and management, it is the work
load engine that gets the workload done.
Control-M Agents – Responsible for execution of the workload. Control Modules are installed
on an agent.
All these components use TCP/IP ports to communicate. Control-M is a mature product that allows
customized TCP/IP port definitions, but network and infrastructure SMEs must allow these
communications for Control-M to both install and execute in and organizations environment.
Control-M V9 Connectivity 3 of 30
F IGURE 4: ADVANCED CONTROL-M IMPLEMENTATION WITH TCP/IP PORTS
The diagram above shows a very advanced implementation where Terraform is being used to create
and manage Control-M environments in the AWS cloud. However, no matter where the Control-M
environment will ultimately reside, the same TCP/IP Ports must be open and, in some cases, allow bi-
directional communication.
Connectivity testing and validation can be broken into three groups coinciding with the three-tiered
architecture. If we consider agentless technology and SSL, SLS, there are other connectivity
considerations in each tier.
Control-M V9 Connectivity 4 of 30
In general use standard TCP/IP tools to help troubleshoot your connectivity issues. These include, but
are not limited to:
These three basic utilities can be used on any platform to help determine the status and connectivity of
the Control-M application.
Use PING1 (an icmp ping request) to determine if the components can recognize each other on
the network. This will also test whether the Domain Name Service is resolving the IP and Node
Names correctly. (Note: ICMP PING requests must be enabled on your network for this
command to execute).
Use Telnet to connect to specific ports on the nodes (at the IP Addresses) where Control-M
components are installed.
Use Netstat to determine how ports are configured on a specific Server where Control-M is
running.
In addition, you may also use other utilities to control the amount of output and give you specific
information such as GREP and FiNDSTR. (e.g. netstat <EM-Server> port | grep <port number>, to get
the status of a Control-M component on that server and that port.
1 https://en.wikipedia.org/wiki/Internet_Control_Message_Protocol
2 Official BMC V9 Control-M Ports Diagram
Control-M V9 Connectivity 5 of 30
EM Component Default Ports Description
Range of 20 ports for Client
CORBA Naming Service 13075 + 20 Connectivity, bi-directional with
reverse name lookup in the DNS
Oracle – 1521
Postgres - 5432
Control-M V9 Connectivity 6 of 30
2 ENTERPRISE M ANAGER C ONNECTIVITY
Because the Enterprise Manager is the first part of a Control-M installation, troubleshooting connectivity
issues generally starts with that component. The Enterprise Manager requires the following ports to be
open and, depending on the component, bi-directional. From the diagram above3, these are:
Oracle – 1521
Postgres - 5432
The CORBA4 Naming Service will be covered under the Desktop Client Communications, and the
Agent ports will be covered under the Agent Connectivity.
The main components are the Control-M Enterprise Manager (EM) to the Control-M Server (CTM) and
the Web Server ports, if you are implementing any of the web-based interfaces for Control-M.
Control-M uses the gateway process on the EM to manage communications with the Control-M servers.
Control-M V9 Connectivity 7 of 30
F IGURE 5: ARCHITECTURE
There are two database instances, one for the EM dB and one for the CTM dB. This diagram shows
some of the process communication for the EM.
Control-M V9 Connectivity 8 of 30
Training Video: Connect with Control-M: Troubleshooting the Enterprise Manager Gateway -
https://www.youtube.com/watch?v=UWIdW8PHOKU
2.1 Summary
In keeping with our previously defined troubleshooting process we will:
PING the EM and CONTROL-M servers from each other to make sure DNS can resolve the
names and IP addresses correctly.
Use Telnet from the EM to CTM and from CTM to EM to make sure you can connect.
Use NETSTAT to port 2370 on the Control-M Server to make sure a listener process from
Control-M is up and running and can connect and from the EM server to the Control-M Server.
Use NETSTAT to port 2369 on the Control-M Server to make sure a listener process from
Control-M is up and running and can connect and can connect and from the EM server to the
Control-M Server.
There are other tools within Control-M as discussed in the video, but these are basic tests with basic
TCP/IP tools that need to work before Control-M can connect.
Control-M V9 Connectivity 9 of 30
3 AGENT CONNECTIVITY
users
VPN
13075 + 20
Internet
Workload
Servers
Control-M
Agents
engine
Web Svr:
18080
Control-M
Control-M Agent
Enterprise Mgr. Firewall
20Gb 20Gb
Oracle Oracle
EM DB CTM DB
Oracle DB Oracle DB
instance instance
F IGURE 7: BASIC CONTROL-M PORTS FOR ORACLE D B FOCUS ON SERVER TO AGENT CONNECTIVITY
Control-M Agent Connectivity occurs between the Control-M Server and the Control-M Agents in the
environment. This is generally the most over-arching concern because these agents are installed on
servers across your enterprise wherever jobs in your enterprise workload will be executed, whether in
the cloud, on the internal or external network.
Control-M V9 Connectivity 10 of 30
F IGURE 8: AGENT ARCHITECTURE
For agents this, by default, these communications take place over ports 7005 and 7006. The diagram
shows the processes that run as the agent for Control-M.
Control-M V9 Connectivity 11 of 30
If you are using a persistent connection for agents the is also a router component included in the agent
processes to keep communications going.
The Control-M Configuration Manager (CCM) is the preferred method of managing the status of the
Control-M server and its agents. Unfortunately, if we are having connectivity issues the CCM is not
always available. In those cases, Control-M comes with a number of utilities to assist in determining
the status of the communications between the server and the agents under its control.
These include:
Utility Description
Control-M/Server
Control-M/Agent
Control-M V9 Connectivity 12 of 30
F IGURE 10: SUMMARY
The training video provides generous detail and examples of how to use these utilities to troubleshoot
connectivity issues with Control-M and its agents:
Connect with Control-M: Agent Connectivity - https://www.youtube.com/watch?v=KnI9SQOfbQ4
3.1 Summary
In keeping with our previously defined troubleshooting process we will:
PING the Agent Server(s) and CONTROL-M servers from each other to make sure DNS can
resolve the names and IP addresses correctly.
Use Telnet from the Agent Server(s) to CTM and from CTM to EM to make sure you can
connect.
Use NETSTAT to port 7006 on the Control-M Server to make sure a listener process from
Agent-M is up and running and can connect to the Control-M Server.
Use NETSTAT to port 7005 on the Agent Server(s) to make sure a listener process from
Control-M Server is up and running and can connect.
There are other tools within Control-M as discussed in the video, but these are basic tests with basic
TCP/IP tools that need to work before Control-M can connect.
Control-M V9 Connectivity 13 of 30
4 DESKTOP CLIENT CONNECTIVITY
users
VPN
13075 + 20
Internet
Workload
Servers
Control-M
Agents
engine
Web Svr:
18080
Control-M
Control-M Agent
Enterprise Mgr. Firewall
20Gb 20Gb
Oracle Oracle
EM DB CTM DB
Oracle DB Oracle DB
instance instance
F IGURE 11: BASIC CONTROL -M PORTS FOR ORACLE DB FOCUS ON EM TO DESKTOP CLIENT
CONNECTIVITY
Control-M V9 Connectivity 14 of 30
F IGURE 12: CONTROL-M DESKTOP CLIENT ARCHITECTURE
Control-M Naming Services use CORBA5 to facilitate communications with Desktop Clients. The
Desktop Clients communicate with EM components and to provide administrative and workload
management capabilities. The main component that communicates with the Desktop Clients is the GUI
Server on the EM.
Control-M recommends at least a range of 20 ports for the communications between the EM and the
Desktop clients. The actual number required will depend on the number of Desktop Clients running in
your environment.
CORBA requires some specific DNS characteristics and Control-M resolves component locations using
server names, not just IP address. This requires reverse DNS lookup6 or reverse DNS resolution on
the network.
BMC provides two utilities to assist in managing CORBA ports, ORBCONFIGURE and ORBADMIN.
ORBCONFIGURE is a GUI interface that enables users to visualize how CORBA ports are being
configured and is generally more “friendly” to use. (It requires X server on Linux/UNIX Operating
systems). ORBADMIN is a command line utility that is good for scripting or power users. Both utilities
are able to configure CORBA ports for Control-M.
5
CORBA® is the acronym for Common Object Request Broker Architecture™, OMG®'s open, vendor-independent
architecture and infrastructure that computer applications use to work together over networks.
6 https://en.wikipedia.org/wiki/Reverse_DNS_lookup
Control-M V9 Connectivity 15 of 30
The following process should be used to avoid confusion when configuring CORBA ports with either
tool.
The diagram shows the ORBCONFIGURE GUI as it is running. It also shows the files on the
Enterprise Manager that are accessed by the ORBCONFIGURE and ORBADMIN utilities.
7
https://documents.bmc.com/supportu/ctrlm9/help/Main_help/en-US/index.htm#89782.htm
Control-M V9 Connectivity 16 of 30
`
F IGURE 14: TROUBLESHOOTING REVIEW
The figure above shows a sample workflow where connectivity for Naming Services is checked on both
the Desktop client and EM server.
The figure shows some Knowledge Articles that may provide additional information about the scenario
you are seeing in your environment.
Control-M V9 Connectivity 17 of 30
Additionally, the following video shows generous detail on troubleshooting Desktop Client Connectivity
issues:
Connect with Control-M: Client Connectivity - https://www.youtube.com/watch?v=AyeOG50OJTc
4.1 Summary
In keeping with our previously defined troubleshooting process we will:
PING the Remote Desktop Client(s) and EM servers from each other to make sure DNS can
resolve the names and IP addresses correctly.
Use Telnet from the Remote Desktop Client(s) to the EM to make sure you can connect.
Use ORBCONFIGURE and ORBADMIN to ensure CORBA is configured
There are other tools within Control-M as discussed in the video, but these are basic tests with basic
TCP/IP tools that need to work before Control-M can connect.
Control-M V9 Connectivity 18 of 30
5 AGENTLESS CONNECTIVI TY
If you are going to use the agentless functionality in Control-M, then this section applies to you.
Figure 19 shows a high-level architecture for agentless configuration in Control-M. Two considerations:
Control-M V9 Connectivity 19 of 30
F IGURE 17: AGENTLESS CONSIDERATIONS
If you are experiencing connectivity issues with agentless functionality check the troubleshooting list to
help identify the issue and remediate it.
Control-M V9 Connectivity 20 of 30
F IGURE 19: KNOWLEDGE ARTICLES
5.1 Summary
In keeping with our previously defined troubleshooting process we will:
PING the Agentless servers from the Control-M Agent Server(s) and vice versa to make sure
DNS can resolve the names and IP addresses correctly.
Use Telnet from the Remote Agentless Server(s) to the Control-M Agent Server to make sure
you can connect.
There are other tools within Control-M as discussed in the video, but these are basic tests with basic
TCP/IP tools that need to work before Control-M can connect.
Control-M V9 Connectivity 21 of 30
APPENDICES
6 CONTROL-M CONNECTIVITY UTILITIES
The communication, startup, and troubleshooting utilities are used to set up communication between Control-M
components, startup/shut down Control-M components and entities and determine if communication between the
components is occurring effectively. Various troubleshooting utilities are also included here.
By including a utility command in the command line of a job processing definition, you can run the utility at a
predetermined time or under a predetermined set of conditions without being present.
Some of the parameter names changed for Control-M version 8.0.00 and above, terminology from previous
versions is still supported. For a complete list of the parameter names, see Abbreviations and conventions.
Utility Description
Control-M/Server
orbadmin Manages the Naming Service process and the CORBA configuration file.
ctmspdiag Prints or erases diagnostics from stored procedures and set or show
diagnostic request status of stored procedures.
init_prflag Resets sleep times and trace levels for Control-M/Server processes.
Control-M V9 Connectivity 22 of 30
shut_ca Shuts down the Control-M/Server Configuration Agent.
ctmchangeshdir Changes the shared directory path that is used for the PostgreSQL
replication in a high availability environment.
Control-M/Agent
Control-M V9 Connectivity 23 of 30
7 V9 PORTS
Control-M V9 Connectivity 24 of 30
8 PRE-INSTALLATION PORT CONSIDERATIONS
As security becomes more and more significant when building and managing services and the
applications that make them up, ALL port considerations must be taken into account.
Many software vendors, including BMC, take advantage of what is called the Echo Protocol.8 For ICMP
messages the Echo protocol communicates over port 7, but can also communicate over port 9.
This creates a requirement to open one or both ports for the installation to complete successfully.
When managing applications and services “in the cloud”, this becomes a requirement.
8 https://en.wikipedia.org/wiki/Echo_Protocol
Control-M V9 Connectivity 25 of 30
9 POST INSTALLATION CONNECTIVITY ISSUES
BMC Control-M recognizes the vulnerability of requiring a more static IP configuration for Control-M to
function efficiently and consistently. This is not a problem or “bug” just a requirement for workload
automation because of its’ an automation engine that automates workflows across the entire enterprise
infrastructure, whether local or remote, on premise or in the cloud, or both.
Control-M includes a GUI based tool that automates changes to the Control-M connectivity
configuration based on the environment it is running in.
9 http://win-c191afttulu:18080/help/Main_Help/en-US/#Tools_Troubleshooting.htm
Control-M V9 Connectivity 26 of 30
The Troubleshooting Connectivity dialogue box appears. The selections from the troubleshooting form
are not intuitive but designed to let you test them individually and find out which form of connectivity
definitions work best in your environment.
An example of when this is helpful is when you are working remotely or have changed how you connect
to Control-M.
Control-M V9 Connectivity 27 of 30
10 ARCHIVAL CONNECTIVITY
The Archival Add-On Control Module for Control-M also uses TCP/IP ports to pass log and output files
to the Archival repository or database. Depending on how Archival is installed, for instance, if it is on a
separate server (which is the case most of the time). There are other ports that must be opened to for
Archival traffic. The ports can be customized in a file. See info from knowledge articles below.
“Control-M Workload Archiving is a Java application and it implements the CORBA communication with JacOrb 3rd party.
JacORB has a configuration file named jacorb.properties and default settings are used.
When Control-M Workload Archiving starts and sets up its CORBA communications, it publishes itself via hostname in a
default way and using a default port range.
The host and/or ports published to CORBA are blocked by the firewall and need to opened. The ports need to be configured
for a specific range.”10
11Ensure that the following components are allowed through the firewall:
10 https://communities.bmc.com/docs/DOC-62441
11 https://communities.bmc.com/docs/DOC-49473
Control-M V9 Connectivity 28 of 30
11 SELF SERVICE
Self Service is generally installed over HTTPS. If this is the case, then the web server must also have
port 1443 open for the HTTPS traffic.
There are some limitations in terms of the number of jobs to display. There are plans to address this12.
https://bmcsites.force.com/casemgmt/sc_KnowledgeArticle?sfdcid=kA014000000doe1CAA&type=FAQ
https://bmcsites.force.com/casemgmt/sc_KnowledgeArticle?sfdcid=kA014000000dnrOCAQ&type=FAQ
12 https://communities.bmc.com/docs/DOC-77960
Control-M V9 Connectivity 29 of 30
12 ADVANCED TROUBLESHOOTING FOR CONTROL -M V9
CONNECTIVITY ISSUES
Do to the broad connectivity requirements of an enterprise class tool, and because of the extreme
requirements of security there are situations that require the ability to examine network traffic and a
lower level to determine what is being sent and what is being blocked. Fortunately, there are tools
designed to help with these scenarios.
TCPDUMP13- Tcpdump prints out a description of the contents of packets on a network interface that
match the boolean expression; the description is preceded by a time stamp, printed, by default, as
hours, minutes, seconds, and fractions of a second since midnight. It can also be run with the -w flag,
which causes it to save the packet data to a file for later analysis, and/or with the -r flag, which causes it
to read from a saved packet file rather than to read packets from a network interface. It can also be run
with the -V flag, which causes it to read a list of saved packet files. In all cases, only packets that match
expression will be processed by tcpdump.
If you are installing in the AMAZON Cloud, there are also the AMAZON Logs14
If you are installing in the Microsoft Azure Cloud, there are Azure log analytics.15
All these tools require a good bit of knowledge and patience to identify and diagnose information
specific to TCP and the respective network configuration.
Basically, you need a timeframe, and specific TCP/IP information about the source and destination on
the network to use them.
13 http://www.tcpdump.org/tcpdump_man.html ; http://www.ijsett.com/images/Paper11(5).pdf
14 https://aws.amazon.com/about-aws/whats-new/2014/07/10/introducing-amazon-cloudwatch-logs/
15 https://docs.microsoft.com/en-us/azure/app-service/web-sites-enable-diagnostic-log
Control-M V9 Connectivity 30 of 30