#LinuxCBT Moni-Zab Edition#

Features of Zabbix:
1. Infrastructure Monitoring
a. Provided by: Zabbix Server
b. Data collection provided by: Zabbix Server (Default=MySQL) || Zabbix Proxy
Package (Scales better by offloading data-collection)
b1. Pollers - interrogate devices for details - pull mechanism - fetches data
from targets (monitored HOSTs)
b2. Trappers - Receive data from: Senders, Agents, Proxies, Child Nodes
Note: Whether push || pull - retrieved data are collected by trappers and stored in
back-end DB: i.e. SQLite, MySQL, PostGreSQL, etc.
c. Users standard protocols: ICMP, TCP, UDP, etc. (i.e. MySQL, SSH, HTTP, HTTPS)

2. Important Concepts in Zabbix Monitoring

a. 'Items' - define a single metric to obtain from HOST: i.e. CPU Load, Swap
usage, Total available memory, currently-logged-in users
b. 'Templates' - cover virtually ALL Apps: i.e. ICMP, HTTP, MySQL, SSH, etc.
b1. Templates include various 'Items', 'Triggers', 'Actions'
c. Triggers define conditions that invoke actions
c1. Triggers evaluate incoming data based on thresholds
d. Actions define the task to carry-out when triggers have fired:
d1. Message generation: E-Mail, Jabber, SMS
d2. Execute remote commands

3. Host resources: Disk | CPU | RAM usage - Zabbix Agents (Multi-Platform)

a. Additional metrics per HOST: i.e. /etc/passwd changes (Auditing), SWAP
monitoring, Restarts, etc.

4. Web scenarios monitoring - Click-Path

5. Modular
a. Zabbix Server (for particular DBs: i.e. MySQL, PostGreSQL) - Main monitoring
engine
b. Zabbix Proxy - Optional Data Collection Offload - ultimately propagates data
back to Zabbix Server -> DB
c. Zabbix Front-End - PHP Web Interface
d. Zabbix Agent (Multi-Platform) - Installed on Targets (HOSTs)
d1. NOTE: Default security = IP-based security
d2. NOTE: Zabbix communicates via: TCP: 10050
d3. NOTE: Default Agent behavior accepts Zabbix Server requests from localhost,
restricting access to potentially unknown, rogue, Zabbix Servers
d4. NOTE: Add L3 security (Firewall) to protect access

6. Reporting - Extensive
a. Zabbix Status, Availability Report, Top 100 Triggers, Bar Reports, Screens,
Graphs, etc.

7. Various Notifications via: 'Media Types': E-mail, SMS, Jabber, Custom Script
8. Maintenance Window Definition - Schedule Downtime of HOSTs
NOTE: Typically applied to groups or hosts: i.e. 'Linux Servers', 'ALL', 'Zabbix
Servers', 'linuxcbtbuild1'
9. SLA Monitoring - Maintenance Window Definition will NOT impact SLAs

10. Multiple User Classes:

a. GUEST (Monitoring) (No access, by Default, to Host Groups)
b. Zabbix Admin (Monitoring & Configuration) permissions - no access by default -
Assign DevOps, Ops, and possibly Junior IT here
NOTE: No access to host groups by default, must be explicilty assigned by Zabbix
Super Admin
 c. Zabbix Super Admin (Everything)
Note: Permissions to Host Groups are granted to User Groups ONLY, NOT individual
users

11. IT Services High-Level View - Helps (Executives) to track SLAs of services

NOTE: This is an Admin-defined hierarchy of important IT Services (abstraction) -
reduces monitored items to essentials for executive use

#Installation & Configuration - Zabbix Server MySQL && Front-End Web Interface #
Features:
1. Monitoring Engine - 'zabbix-server-mysql...'
2. Web Front-End - 'zabbix-frontend-php...'

Tasks:
1. Obtain packages from: zabbix.com
a. NOTE: Packages are grouped by platform: i386, amd64, except Web Front-end,
which is platform-agnostic
b. 'wget http://repo.zabbix.com/zabbix/2.2/ubuntu/pool/main/z/zabbix/zabbix-
server-mysql_2.2.3-1+precise_amd64.deb'
c. 'wget http://repo.zabbix.com/zabbix/2.2/ubuntu/pool/main/z/zabbix/zabbix-
frontend-php_2.2.3-1+precise_all.deb'

2. Install Pre-Requisites:
a. 'aptitude install libiksemel3 libopenipmi0 libssh2-1 fping dbconfig-common
php5 libodbc1' - support the Zabbix packages

3. Install Server
a. 'dpkg -i zabbix-server-mysql_2.2.3-1+precise_amd64.deb' - auto-generates MySQL
password for user = 'zabbix'
b. 'dpkg -i zabbix-frontend-php_2.2.3-1+precise_all.deb'

4. Server Footprint
a. 'dpkg -L zabbix-server-mysql'
a1. '/etc/zabbix' - config container
a2. '/etc/zabbix/zabbix_server.conf' - primary config - also contains MySQL
credentials in clear-text - use on agents
a3. '/etc/zabbix/apache.conf' - applies to ALL virtual hosts and default host on
Apache instance. Move to VHost if desired.
a4. '/var/log/zabbix/zabbix_server.log' - primary log file

a5. 'ps -ef | grep zabbix' - all spawned instances revealed

Note: 'zabbix_server' is spawned as many times as indicated by various sections of:
'/etc/zabbix/zabbix_server.conf' file

a6. 'less /var/log/zabbix/zabbix_server.log' - look for failures and successes

5. Front-end Footprint
a. 'dpkg -L zabbix-frontend-php'

6. Ensure that 'date.timezone = ?' - set to something prior to usage - Default PHP
implemenation is unset
a. 'grep timezone /etc/php5/apache2/php.ini' - ensure that a timezone is
specified

7. Complete Setup of Web-Front-End

a. 'http://192.168.75.103/zabbix'
 a1. Obtain 'zabbix' MySQL password from: '/etc/zabbix/zabbix_server.conf'
a2. '/usr/share/zabbix/conf/zabbix.conf.php"' - for future Web Front-End changes,
especially regarding DB access

# Exploration of Web Interface and Initial Setup #

Features:
1. Full operation of Zabbix via Web
2. '/zabbix' - alias
3. DB Credentials: '/etc/zabbix/zabbix_server.conf'
4. '/usr/share/zabbix/conf/zabbix.conf.php'
5. NOTE: Web interface supports SSL

Tasks:
1. Login as Super Admin: 'admin' with Default password: 'zabbix'
2. Disable GUEST access
a. 'Administration -> Users -> Toggle 'Enabled' for Guests Group'
3. Change 'admin' password
a. 'Administration -> Users -> Admin... -> Change Password'

4. Configure Media Type - E-Mail - Configure

a. Admnistration -> Media Types -> E-Mail - set appropriate settings for the
delivery of E-Mail
b. Ensure that Super User is enabled for Media type E-Mail
b1. 'Administration -> Users -> Admin -> Media -> Add'
Note: No Media Types are associated by default, however, this is NOT a problem
initially, as no HOSTs are initially monitored
Note: Default associated media type configuration covers 24/7 for ALL severities.
This can be tweaked as needed.
Note: Multiple instances of the same media type can be associated per user to
govern varying criteria: i.e. time-of-day, and severity

c. Ensure SMTP configuration reflects indicated domain information:

'/etc/postfix/main.cf'

5. Monitor Initial HOST (Zabbix Server)

a. 'Configure -> Host -> Zabbix Server -> Toggle 'Not monitored'
b. Install Zabbix Agent on Zabbix server
c. '/etc/zabbix/zabbix_agentd.conf' - agent config file, which restricts polling
to localhost

# HOST Monitoring - PROD #

Features:
1. Recurringly monitors (every 2-3 minutes) HOSTs
2. Covers PROD services
3. Templates encompass a variety of common services, with triggers and actions

Tasks:
1. Monitor HOST
a. Download the agent
b. install & confirm
c. Add to Zabbix
c1. Configuration -> HOST -> Add - associate various templates
d. Troubleshoot - problems with NTP and MySQL
e. Removed NTP
f. Troubleshoot MySQL
NOTE: MySQL polling requires credentials in: $HOME of 'zabbix' user:
'/var/lib/zabbix'
g. 'mkdir /var/lib/zabbix && chown zabbix.zabbix /var/lib/zabbix && chmod 750
/var/lib/zabbix'
/var/lib/zabbix/.my.cnf
[client]
user=zabbix
password=t1G7hX5du20Mfff

2. Repeat process on CentOS - 192.168.75.120

# Problem with CentOS 6.5 regarding SELinux - Inability to access MySQL Socket #
Checks:
1. Ability to connect to MySQL from $SHELL as the user 'zabbix'
2. '/var/log/audit/audit.log' - 'avc' messages

Workarounds:
1. Set SELinux to 'Permissive'
a. 'setenforce 0'
b. Add 'linuxcbtcent2' to list of monitored HOSTs and then set SELinux to
Permissive

2. Rebuild Zabbix SELinux Policy

a. '/etc/init.d/zabbix-agent stop'
b. 'semodule -r zabbix'
c. '/etc/init.d/zabbix-agent start'

Note: Post SELinux Policy update, the option to rebuild 'zabbix' policy exists once
Note: Post-rebuild, Zabbix agent will be able to acces MySQL

# Notifications #
Features:
1. Actions invoked by Triggers
2. Logged via: Administration -> Notifications

Tasks:
1. Debug Zabbix Server's inability to contact agent on: 'linuxcbtcent2'
a. 'ps -ef | grep zabbix' - returns list of processes
b. 'netstat -ntlp | grep 10050' - returns socket
c.'grep -i server /etc/zabbix/zabbix_agentd.conf' - restricted to: 'localhost' -
this is the problem
NOTE: 'linuxcbtcent2' failed to generate a notification because it was NOT
completely, initially monitored. Limited data for SSH was returned

2. Monitor HTTP on HOSTs and Disable services ad-hoc

NOTE: Use 'mass-update' option
a. 'Configuration - Hosts - Select ALL Hosts - Assign 'HTTP | HTTPS ' templates'

NOTE: Ensure that 'Actions' 'Report Problems to Zabbix Administrators' is Enabled

(Default = Disabled)

Notification Components (Basic)

1. Action(s) is/are enabled to deliver messages to Zabbix Users | Groups
2. User | Group should have configured 'Media Type'
3. At least 1 Media Type should be properly configured: i.e. E-Mail
# Maintenance Window(s) #
Features:
1. SLA Exceptions for: i.e. patches, updates, re-builds, hardware issues, etc.
a. If system is impacted (unavailable) during maintenance window, SLAs are
unaffected
2. Reduction in False-Positives - by NOT generating messages | executing remote
commands during this period
3. Ability to define recurring schedule for maintenance: i.e. monthly

Tasks:
1. Setup basic Maintenance window for today for 2-hours
a. Configuration -> Maintenance - New Window with recurring schedule
b. Fail HTTP services and ensure that messages are NOT generated

2. Reduce Maintenance window to effectively bring PROD systems OUT of maintenance

a. Confirm the generation of messages due to out-of-maintenance window

# Define 'devops' User #

Features:
1. Reduced Privilege-set: Zabbix Administrator as opposed to: Zabbix Super
Administrator
2. No Default access to HOSTs
3. Reduce the load on IT

Tasks:
1. Create 'devops' user -> operations@linuxcbtmon1.linuxcbt.internal
a. Add 'Media'
b. Assign permissions to HOST Group

# Reports #
Features:
1. Zabbix Status - Server
2. Availability for SLAs
3. Top Triggers - 100
4. Customizable Bar Reports