Mysql Crash Recovery

Contents
[hide]
1
Mysql
Crash
Recove
ry
2
Overvie
w of
Recove
ry
Process
3
Details
of
Recove
ry
Process
3
.
1
1
.
C
o
n
f
i
g
u
r
e
m
y
s
q
l
t
o
s
t
a
r

Mysql Crash Recovery

In the event of database corruption it may be necessary to manually perform database recovery. See
Bug 15797 for an example of an issue with mysql that will require database recovery. In that example,
a warning message like the following appeared in the mysql error log:
InnoDB:
InnoDB:
InnoDB:
InnoDB:
InnoDB:

Serious error! InnoDB is trying to free page 716

though it is already marked as free in the tablespace!
The tablespace free space info is corrupt.
You may need to dump your InnoDB tables and recreate the whole
database!

Before beginning a full database recovery, check to see if the corruption may be limited to a single
mboxgroup or a single user within an mboxgroup. This type of corruption frequently lets the server run
normally for extended periods of time, with crashes occurring only when an affected user attempts to
access certain mailbox items. If this is the case, it may be possible to dump, drop and recover only the
affected entries without disrupting the database as a whole. Please see the instructions in the Mysql
Crash Recovery (alternate method) article.

Overview of Recovery Process

1.
2.
3.
4.
5.
6.

Configure mysql to start in recovery mode

Generate SQL dumps of all relevant databases
Remove all existing (and possibly corrupt) databases
Re-create all databases
Repopulate the databases with the data from the SQL dumps
Test databases and start all ZCS services

Details of Recovery Process

1. Configure mysql to start in recovery mode
1. Edit the file /opt/zimbra/conf/my.cnf and add a line like innodb_force_recovery = 1 under the
[mysqld] section (Note that it may be necessary to increase the recovery level depending on the
extent of the database corruption, as shown at the end of the database dump step)
2. Save the file and re-start mysqld
mysql.server start

2. Generate SQL dumps of all databases

1. Load some mysql configuration into shell variables (i.e. $mysql_socket and
$mysql_root_password; note that you will use these again in step 3)
2. Make a list of the existing databases
3. Create a directory to hold the SQL dumps
4. Generate the SQL dumps from the database list
source ~/bin/zmshutil ; zmsetvars

mysql --batch --skip-column-names -e "show databases" | grep -e mbox -e zimbra

> /tmp/mysql.db.list
mkdir /tmp/mysql.sql
for db in `cat /tmp/mysql.db.list`; do
~/mysql/bin/mysqldump $db -S $mysql_socket -u root
--password=$mysql_root_password > /tmp/mysql.sql/$db.sql
echo "Dumped $db"
done

Note: If you encounter any mysql errors while dumping the databases, start over by re-editing
/opt/zimbra/conf/my.cnf, incrementing the value for innodb_force_recovery by one, and
restarting mysqld. The maximum recovery level is 6. Please see MySQL's Forcing InnoDB Recovery
guide for more information.
Note: An error of "bash: /tmp/mysql.sql/$db.sql: ambiguous redirect" probably indicates your
using an apostrophe or single quote ' rather than a tick ` -- which is one the same key as the tilde
~.
Note: Do not reboot the machine, as some Operating Systems will remove all contents in the /tmp
directory during the reboot sequence, i.e. your /tmp/mysql.sql will be removed.

3. Remove all existing (and possibly corrupt) databases

Note that we drop the zimbra database last because the mboxgroup* databases depend on it
for db in `cat /tmp/mysql.db.list |grep mbox`
do
mysql -u root --password=$mysql_root_password -e "drop database $db"
echo -e "Dropped $db"
done
mysql -u root --password=$mysql_root_password -e "drop database zimbra"

Remove existing InnoDB tablespace and log files

rm -rf /opt/zimbra/db/data/ib*

Note: First, use with caution - this shouldn't need to be used often. Issue came about because of
some rsync issues. Can't dump db's because of 'connection' issues at this point? One could move the
/opt/zimbra/db/data directory - mv /opt/zimbra/db/data /opt/zimbra/db/data-old and then make the db mkdir /opt/zimbra/db/data w/ ownership of zimbra:zimbra . Remove the innodb_force_recovery line
from /opt/zimbra/conf/my.cnf . Then recreate a default mysql db by running
/opt/zimbra/libexec/zmmyinit --sql_root_pw $mysql_root_password and then attempt this steps over
again to confirm you can drop them. Also note that you may have to reset the zimbra password
manually in mysql, then set it again in Zimbra with the instructions from this page:
http://wiki.zimbra.com/wiki/Resetting_LDAP_%26_MySQL_Passwords

4. Re-create all databases

1. Run mysql in non-recovery mode
1. Remove the innodb_force_recovery line from /opt/zimbra/conf/my.cnf

2. Save the file and restart mysqld

2. Re-create the databases from the database list
mysql.server restart
for db in `cat /tmp/mysql.db.list`
do
mysql -e "create database $db character set utf8"
echo "Created $db"
done

5. Repopulate the databases with the data from the SQL dumps
Import the data from the SQL dumps. Note that we import the zimbra database first because the
mboxgroup databases depend on it
mysql zimbra < /tmp/mysql.sql/zimbra.sql
for sql in /tmp/mysql.sql/mbox*
do
mysql `basename $sql .sql` < $sql
echo -e "Updated `basename $sql .sql` \n"
done

6. Test databases and start all ZCS services

Note that this is an example query. If you know of any particular databases that were corrupt, you may
want to construct other queries to verify normal access to the data.
mysql zimbra -e "select * from mailbox order by id desc limit 1"

Once you are satisfied that the databases are restored intact, start the rest of the zimbra services.
zmcontrol start

Check /opt/zimbra/log/mysql_error.log and /opt/zimbra/log/mailbox.log for database errors.

Help
/opt/zimbra/mysql/bin/mysqlcheck --defaults-file=/opt/zimbra/conf/my.cnf -S
/opt/zimbra/db/mysql.sock -A -C -s -u root --password=mypass
/opt/zimbra_2/mysql-standard-5.5.28-pc-linux-gnu-i686-glibc23/bin/mysqldump
--defaults-file=/opt/zimbra/conf/my.cnf -S /opt/zimbra/db/mysql.sock
-u root --password=__RbC5aZmURqMyJBRlcGg_JfS

mboxgroup63
/opt/zimbra/mysql/bin/mysqlcheck --defaults-file=/opt/zimbra/conf/my.cnf -S
/opt/zimbra/db/mysql.sock -A -C -s -u root --password=__RbC5aZmURqMyJBRlcGg_JfS --repair
--databases mboxgroup63
/opt/zimbra/mysql/bin/mysqldump --defaults-file=/opt/zimbra/conf/my.cnf -S
/opt/zimbra/db/mysql.sock --databases mboxgroup63 -u root
--password=__RbC5aZmURqMyJBRlcGg_JfS > /tmp/zimbra_mboxgroup63_dump.sql
/opt/zimbra/mysql/bin/mysqldump --defaults-file=/opt/zimbra/conf/my.cnf -S
/opt/zimbra/db/mysql.sock --all-databases -u root --password=__RbC5aZmURqMyJBRlcGg_JfS >
/tmp/zimbra_dump_20140122213535_dump.sql
cat /tmp/zimbra_mboxgroup63_dump.sql
/opt/zimbra/mysql/bin/mysqlcheck --defaults-file=/opt/zimbra/conf/my.cnf -S
/opt/zimbra/db/mysql.sock -u root --password=__RbC5aZmURqMyJBRlcGg_JfS --repair --databases
mboxgroup63
mysql -u root --password=__RbC5aZmURqMyJBRlcGg_JfS -e "drop database mboxgroup63"
mysql -u root --password=__RbC5aZmURqMyJBRlcGg_JfS < /tmp/zimbra_mboxgroup63_dump.sql
@/tmp/zimbra_mboxgroup63_dump.sql
drop database mboxgroup63;