SMTP Best Practices, Configurations and Troubleshooting on Lotus Domino.

9th June 2011

Chandra Shekhar Anumandla - Lotus Technical Support Engineer Presenter Seema Janjirkar Lotus Technical Advisor Presenter Hansraj Mali - Lotus Technical Advisor Focussing on Notes/Domino, LotusLive Ranjit Rai - Lotus Technical Advisor Focussing on entire Notes/Domino Soumitra Limaye Lotus Support Facilitator for Open Mics

Agenda
Overview Of SMTP Mail Routing & Components. Best Practices for SMTP Mail Routing. Troubleshooting of SMTP Mail Routing. Case Studies

Q/A

Overview of SMTP Mail Routing.

The primary purpose of SMTP is to transfer emails between mail servers. In order to send email, the client sends the message to an outgoing mail server, which in turn contacts the destination mail server for delivery. 1. On Domino The sending server checks the recipient's address, which is in the format localpart@domain, and looks up the domain in the Domain Name System (DNS). 2. DNS returns the Mail Exchanger (MX) record for the domain, indicating the IP address of the servers in the domain that accept mail over SMTP. 3. The sending server connects to the destination server over TCP/IP, establishes an SMTP connection on port 25, transfers the message, and closes the connection. Inbound case, Based on our MX Record we receive SMTP message on SMTP Domino server.

Overview of SMTP Mail Routing & Components.

Relay Host

A relay host is a server within (or outside) your organization that routes messages outside your local Internet domain. A relay host can be a Domino server or another type of server, for example, a Microsoft Exchange Server or a UNIX Server running SendMail. Although a Domino server can be used as a relay, most common SMTP implementations do not relay to Domino.

Overview of SMTP Mail Routing & Components.

Domain Documents:

Global Domain Document.

A Global Domain document is used to determine how to convert Notes addresses to Internet addresses and how to convert Internet addresses to Notes addresses. Specifies the primary internet domain(s) and aliases . All internet domains and aliases defined in any global domain document are local internet domains

Overview of SMTP Mail Routing & Components.

Domain Documents:

Foreign SMTP Domain Document.

Foreign SMTP domain documents that define the next domain for sending SMTP mail addressed to a given set of destination addresses. To work Foreign SMTP Domain document, Need SMTP connection documents. The Foreign SMTP Domain document will be used by all non-SMTP enabled R5 servers and by 4.x servers (in a mixed release environment).

Note: When there is relay mentioned and also have Foreign SMTP Domain Document ( FSDD) available then FSDD will take precedence.

Overview of SMTP Mail Routing & Components.

Domain Documents:

SMTP Connection Document.

SMTP Connection documents specifying the server that processes outbound SMTP mail for each Foreign SMTP domain document.

Overview of SMTP Mail Routing & Components.

Smart Host

A smart host is a directory server to which SMTP-routed messages are sent when the message recipient cannot be found in the IBM Lotus Domino Directory or other secondary directories configured on the server. Typically, a smart host is used in organizations that employ multiple mail systems within a single Internet domain. Users on these systems may not be in the Domino Directory. For example, if some users are on a UNIX sendmail system but their inbound messages are routed through the Domino mail system, you can set up a smart host to ensure proper address resolution. After you set up a smart host, when Domino receives a message, if the domain part of the recipient's address matches the local Internet domain or one of the alternate Internet domain aliases defined in the Global Domain document, the Router looks up the address against all configured directories. If the address is not found, the Router then uses SMTP to forward the message to the configured smart host. Domino sends all messages addressed to unknown recipients in the local Internet domain to the configured smart host. You cannot configure Domino to send to the smart host only messages addressed to recipients in some subset of the internal domains and domain aliases defined in the Global domain document. Note Domino does not send messages addressed to unknown IBM Lotus Notes addresses to the smart host. You must have DNS set up correctly to use a smart host.

Overview of SMTP Mail Routing & Components.

Inbound:

Outbound:

http://www.ibm.com/support/docview.wss?uid=swg21089344

Overview of SMTP Mail Routing & Components.

SMTP Authentication

SMTP Authentication (SMTP-AUTH) is generally a security improvement over unauthenticated SMTP; however, it can also introduce a weakness. If authenticated users are allowed to submit messages from IP addresses, and unauthenticated users are not, then an attacker who manages to get the credentials of one user's account is then able to use the authenticated server as an open mail relay.

It can add another layer of security to send email. it provides mobile users who switch hosts with the ability to use the same mail server without needing to reconfigure their mail client settings each time. Servers that support SMTP-AUTH can usually be configured to require clients to use this extension, ensuring that the true identity of the sender is known The SMTP-AUTH extension also allows one mail server to indicate to another that the sender has been authenticated when relaying mail.

Best Practices for SMTP Mail Routing.

When we should create more mail.box?

Product Development recommends that when the percentage of access conflicts consistently exceeds 2%, another mailbox should be created. This does not take into account peak mail routing time periods, just the average time under normal mail routing conditions. To measure this percentage, you can use two statistics: --Mail.Mailbox.Accesses and Mail.Mailbox.AccessConflicts. These figures can be obtained by issuing a "show stat mail" command on the Domino Server console. Use those figures in the following formula: -- (Mail.Mailbox.AccessConflicts / Mail.Mailbox.Accesses) x 100 > 2. Reference : http://www.ibm.com/support/docview.wss?uid=swg21148438

Best Practices for SMTP Mail Routing.

In large user base company with multiple servers, It is common practice to have separate inbound and outbound routing because it simplifies troubleshooting under normal circumstances. Single servers at each of these points would represent the simplest configuration. OS Anti-virus should not be enabled on Domino's data directory. The Anti-Virus designed Domino mail server should be disabled on the mail.box. Transaction logging recommended to be disabled on the mail.box. DAOS-enabling the mail.box is optional. It has no storage advantage, but enabling DAOS optimizes performance by not needing to copy bytes..

Best Practices for SMTP Mail Routing.

Open relay

An open mail relay is an SMTP server configured in such a way that it allows anyone on the Internet to send E-mail through it, not just mail destined to or originating from known users (see figure).

Best Practices for SMTP Mail Routing.

Determining whether Lotus Domino is open relay or closed relay

Best Practices for SMTP Mail Routing.

Making Lotus Domino a closed-relay server.

To protect SMTP servers from unauthorized relaying, Lotus Domino provides inbound relay controls used to define the hosts to which and from which a server can relay messages. The Domino SMTP listener denies requests to relay messages to or from unauthorized hosts.

Setting inbound relay controls : 1. Make sure you already have a Configuration Settings document for the SMTP server to be configured 2 In configuration document Select the Router/SMTP > Restrictions and Controls > SMTP Inbound Controls tabs.

Best Practices for SMTP Mail Routing.

Domino SMTP Relay Enforcement.
Inbound relay enforcement controls:

Select the Router/SMTP > Restrictions and Controls > SMTP Inbound Controls tabs. In the Inbound Relay Enforcement section (see figure 6) make sure the field Exceptions for authenticated users is set to Allow all authenticated users to relay.

NOTE: The Perform Anti-relay enforcement for these connecting hosts field has the following three options, so make sure you select the proper one: External hosts (default) . The server applies the inbound relay controls only to hosts that connect to it from outside the local Internet domain. Hosts in the local Internet domain are exempt from anti-relay restrictions. The local Internet domain is defined by either a Global Domain document, if one exists, or as the Internet domain of the host server. All connecting hosts. The server applies the inbound relay controls to all hosts attempting to relay mail to external Internet domains. None. The server ignores the settings in the inbound relay controls. All hosts can always relay.

Troubleshooting SMTP Mail Routing.

Probable causes from Domino for slow mail routing.

Large Bcc Group in a mail document and that try to expand on Domino server. Mass Mailing. Limited transfer and delivery threads. Not Support version of Anti virus software for current domino server version. Anti-virus scan for mail.box. Less number of mail.boxes. DNS issue with Domino server.

Troubleshooting SMTP Mail Routing

Debug Parameters:
Output of SMTPClientDebug=1
Example: "SMTP Protocol returned Permanent Error" when trying to send messages via SMTP The text in Red is the indicates an attempt to authenticate and the SMTP's rejection of the authentication as invalid
=================================

02/25/2010 07:58:29 AM SMTPClient: Starting to transfer 1 messages to outgoing.verizon.net for user CN=John Smith/ O=Acme 02/25/2010 07:58:29 AM [0238:0006-048C:wrepl] SMTPClient: Attempting to Connect: Host outgoing.verizon.net, Port 25, SSL Port 0, Connecting Domain outgoing.verizon.net 02/25/2010 07:58:30 AM [0238:0006-048C:wrepl] SMTPClient: Connection successful 02/25/2010 07:58:30 AM [0238:0006-048C:wrepl] SMTPClient: ReceiveResponse: 220 vms048pub.verizon.net -- Server ESMTP (Sun Java System Messaging Server 6.2 HotFix 0.04 (built Dec 24 2004)) 02/25/2010 07:58:30 AM [0238:0006-048C:wrepl] SMTPClient: CommandEHLO: EHLO outgoing.verizon.net 02/25/2010 07:58:30 AM [0238:0006-048C:wrepl] SMTPClient: ReceiveResponse: 250-vms048pub.verizon.net 02/25/2010 07:58:30 AM [0238:0006-048C:wrepl] SMTPClient: ReceiveResponse: 250-8BITMIME 02/25/2010 07:58:30 AM [0238:0006-048C:wrepl] SMTPClient: ReceiveResponse: 250-PIPELINING 02/25/2010 07:58:30 AM [0238:0006-048C:wrepl] SMTPClient: ReceiveResponse: 250-DSN 02/25/2010 07:58:30 AM [0238:0006-048C:wrepl] SMTPClient: ReceiveResponse: 250-ENHANCEDSTATUSCODES 02/25/2010 07:58:30 AM [0238:0006-048C:wrepl] SMTPClient: ReceiveResponse: 250-HELP 02/25/2010 07:58:30 AM [0238:0006-048C:wrepl] SMTPClient: ReceiveResponse: 250-XLOOP 3ED6E1E76A4AE7ABA7D00699A10F262B 02/25/2010 07:58:30 AM [0238:0006-048C:wrepl] SMTPClient: ReceiveResponse: 250-AUTH PLAIN LOGIN 02/25/2010 07:58:30 AM [0238:0006-048C:wrepl] SMTPClient: ReceiveResponse: 250-AUTH=LOGIN 02/25/2010 07:58:30 AM [0238:0006-048C:wrepl] SMTPClient: ReceiveResponse: 250-ETRN 02/25/2010 07:58:30 AM [0238:0006-048C:wrepl] SMTPClient: ReceiveResponse: 250-NO-SOLICITING 02/25/2010 07:58:30 AM [0238:0006-048C:wrepl] SMTPClient: ReceiveResponse: 250 SIZE 8388608 02/25/2010 07:58:30 AM [0238:0006-048C:wrepl] SMTPClient: CommandAUTH: AUTH LOGIN bGF0b3J0dWU 02/25/2010 07:58:30 AM [0238:0006-048C:wrepl] SMTPClient: ReceiveResponse: 501 5.5.0 Invalid input (Invalid authentication protocol). 02/25/2010 07:58:30 AM [0238:0006-048C:wrepl] SMTPClient: Attempting to Disconnect: 02/25/2010 07:58:30 AM [0238:0006-048C:wrepl] SMTPClient: CommandQUIT: 02/25/2010 07:58:30 AM [0238:0006-048C:wrepl] SMTPClient: ReceiveResponse: 221 Closing connection. Good bye. 02/25/2010 07:58:30 AM [0238:0006-048C:wrepl] SMTPClient: Connection terminated successfully

Troubleshooting SMTP Mail Routing.

Debug Parameters :
Output of SMTPDebugIO=3 Example: incoming smtp connections being disconnected with 0 messages received. 02D8:0008-1678] 02/08/2010 03:51:27.95 PM SMTP CITask RecvErrorHandler> ERROR: (206.191.0.234) Receive failed (did not timeout), error = 0A02h (Remote system no longer responding) [02D8:0008-1678] 02/08/2010 03:51:27.97 PM SMTP CITask RecvErrorHandler> Running default handler, error = 0A02h (Remote system no longer responding) [02D8:0008-1678] 02/08/2010 03:51:28 PM SMTP Server: queue1.magma.ca (206.191.0.234) disconnected. 0 message[s] received.

debug_threadid=1 Example: log_mailrouting=40:

[06B0:00080854] 08/22/2010 11:28:09 SMTP Server: tom_main (9.161.148.68) connected [06B0:00080854] 08/22/2010 11:28:44 SMTP Server: Originator: <dummy@test.net> [06B0:00080854] 08/22/2010 11:29:03 SMTP Server: Recipient: <user@abc.de> [06B0:00080854] 08/22/2010 11:29:07 SMTP Server: Message 00399921 (MessageID: ) received from tom_main (9.161.148.68) size 205 bytes [06B0:00080854] 08/22/2010 11:29:10 SMTP Server: tom_main (9.161.148.68) disconnected. 1 message [s] received SMTPSaveImportErrors=2: [06B0:0008059C] 08/22/2010 12:11:50 SMTP Server: tom_main (9.161.148.68) connected [06B0:0008059C] 08/22/2010 12:12:01 SMTP Server [06B0:0008059C] RFC822 message inbound stream saved to E:\WINNT\TEMP\notes671E62\st993548.TMP [06B0:0008059C] 08/22/2010 12:12:01 SMTP Server: Message 003D8679 (MessageID: ) received.

Troubleshooting SMTP Mail Routing.

Debug Parameters:

Any issue with DNS. Debug_TCP_Resolver=1

If you are getting network error code in smtpclientdebug then only collect Network packet with help of Wireshark Example : Error: 'Server is not responding' sending SMTP mail to external domains.. From the debug log, it is clear that there is a problem with the DNS service. There are long delays before DNS responds and DNS is timing out at certain periods. 0974:0002-0DEC] 27-03-2010 17:07:44,55 TCPEndp_Resolver> RES_SEND: Results: -1 [0974:0002-0DEC] 27-03-2010 17:07:44,55 TCPEndp_Resolver> Request Failed [0974:0002-0DEC] 27-03-2010 17:07:44,55 TCPEndp_Resolver> Unknown Error 0974:0002-0DEC] 27-03-2010 17:07:44,55 TCPEndp_Resolver> Exit status = 105Eh [0974:0002-0DEC] 27-03-2010 17:07:44,55 cmd_SendTranPvdrMsg> exit hEndp: 110C0002h wMsg: 1009h iError = 105Eh [0974:0002-0DEC] res_send failure DNS resolver error, domain xxxxxx.COM Refrence: http://www.ibm.com/support/docview.wss?uid=swg21312913

Troubleshooting SMTP Mail Routing.

SMTP Performance/slow/hang issues: Debug Parameters:

Debug parameter when we collect NSD:

DEBUG_CAPTURE_TIMEOUT=1 DEBUG_SHOW_TIMEOUT=1 DEBUG_THREADID=1

Server Commands:

Show Server Tell Router Show Queue Show Task Show Task Time Show Stat Mail Route Trace

Troubleshooting SMTP Mail Routing.

Network Tools (Wireshark ):

Outbound SMTP sessions may fail consistently or intermittently. Debug and log analysis reveals the connection was broken abruptly with a status code: 2055 or 2562. Examples: SMTPClient: Data Send Failed XXXXXX bytes, Status: 2562 SMTPClient: Connection broken after an error sending DATA command SMTPClient: Connection terminated with status: 2055 Router: No messages transferred to <acme> (host acme) via SMTP: Server not responding.

In the below example, the Domino server (Source 192.168.220.128) is trying to connect (option SYN) to an SMTP server on port 25. The example shows packets containing the options [RST, ACK] sent from the destination server for the Domino server. This means that the destination server is not reachable, or has actively reset the connection.

Case Study 1
Domino stopped routing all the emails.
In Lotus Domino, you notice that your outbound SMTP server not routing all the emails but held in the mail.box without any failure status.

Tell Router Show Queue output shows no errors but none of the threads busy or used.

Example of the console output from issuing TELL ROUTER SHOW: Msgs State Via Destination

14 Busy(1) SMTP MYDOMAIN.COM (Push) Transfer Threads: Max = 25; Total = 0; Inactive = 0; Max Concurrent = 9 Delivery Threads: Max = 25; Total = 0; Inactive = 0

Solution : In this case, as no error coming, we can suspect the 3rd party softwares like Anti-Virus Disabling Anti-Virus resolved in a couple cases. To disable, Anti Virus You must remove the antivirus task from notes.ini parameter EXTMGR = <AV Task>and restart the server.

Case Study 2
Mail being delivered to the wrong recipient
If mail is being delivered to the wrong person, check the: 1. Person document of the recipient (see Step 3). 2. Configuration document, Router/SMTP tab, Basics tab (see Step 4). 3. Mail file name for the recipient; if its wrong, correct the same. [DONE] 4. Address lookup field; if its set to Fullname then local part, change it to Fullname only (see figure 9). Also, make sure this setting is done on the first server that receives the mail (Incoming SMTP server). Figure 9. Address lookup field

Case Study 3
Problems with inbound SMTP mail
1. Issue a show tasks command at the Domino console; if you:

a. See SMTP Server: Listening for requests on port 25, go to Step 3. b. Do not see SMTP Server, go to Step 4. c. See SMTP Server listening for requests on a different port than 25, go to Step 5. 2. Gather the DFR and any relevant errors seen in the console log or Log.nsf. Call Lotus Support. [DONE] 3. Verify an MX record exists in DNS for your Internet Domain. (There are Web sites can be used to help with this, for example, www.dnsstuff.com.) If an MX record: a. Does not exist, contact your DNS Admin or ISP to obtain an MX record. [DONE] b. Exists and points to a non-Domino server, go to Step 6. c. Exists and points to a Domino server, go to Step 7. 4. Go to the Basics tab of the Server document and locate the field SMTP Listener Task. Verify it is set to ENABLED and then issue a load smtp command at the server console (see Step 1). 5. Third-party software may be configured to listen on port 25 and then transfer mail to Lotus Domino over another port (usually 26). Check whether you are running a third-party SMTP listener on port 25: a. If Yes, contact vendor support for further troubleshooting as all inbound SMTP will be first directed to port 25. [DONE] b. If No, reset the Domino Inbound SMTP port back to 25 in the Server document, Ports, Internet Ports, Mail tab (see Step 1). 6. A non-Domino gateway sits between the outside world and the Domino environment: a. Contact vendor support for non-Domino server to troubleshoot why it cannot receive SMTP mail. [DONE] b. The gateway is receiving SMTP mail but unable to transfer to Domino, in which case go to Step 7. 7. Use Telnet to test connectivity to Lotus Domino over port 25 from outside the firewall. To do this, find a workstation outside the firewall; if its a Windows platform, open a command prompt and type telnet <MX_hostname_or_IPaddress> 25.

Case Study 3
Problems with inbound SMTP mail Cont..

If you are: Unable to receive SMTP mail from all Internet domains, go to Step 1. Able to receive SMTP mail from some Internet domains or addresses but not others, go to Step 2.

If Telnet connectivity: a. Is successful, but you get a 220 Domino banner, go to Step 8. b. Is successful, but you get a non-Domino banner, go back to Step 5. c. Fails, go to Step 9.

8. Step through a complete SMTP conversation via Telnet. Figure 6 illustrates a successful SMTP conversation with a Domino server. For assistance regarding this step, call Lotus Support. Successful conversation with Lotus Domino :

Case Study 3
Problems with inbound SMTP mail Cont..
If the SMTP conversation over Telnet: a. Was successful, the message was accepted for delivery, and the recipient has received the message in the mail file, then Lotus Domino is working as expected and can accept SMTP mail. [DONE] b. Appears to be successful, and the Domino server claims the message was accepted for delivery, but the recipient never got the message in the mail file, go back to Step 3 of Section 5.1. c. Fails and you get a 4XX or 5XX error message during the exchange of commands, gather the error message and call Lotus Support. [DONE] 9. Find a workstation inside the firewall and repeat the same Telnet connectivity test: a. If Telnet connectivity is successful, the inbound SMTP problem is likely caused by the firewall blocking inbound traffic over port 25. [DONE] b. If Telnet connectivity fails, go to Step 10. 10. Telnet directly from the physical server to itself. If Lotus Domino is running on a Windows platform, type telnet localhost 25 at the command prompt: a. If Telnet connectivity is successful, go to Step 8. b. If Telnet connectivity fails, call Lotus Support. [DONE]

Resources
Troubleshooting IBM Lotus Domino 8 mail routing issues Understanding SMTP authentication and securing your IBM Lotus Domino 8 server from spam How to Configure a Domino Server to Send Internet Mail Determining the number of mailboxes required for a server Error: 'Server is not responding' sending SMTP mail to external domains

Questions ?