Improved EBS snapshots on Amazon EC2

So I’ve been working on a system where the MySQL instance on EC2 sporadically locks up (mysqld is zombied) during a snapshot process =)

Essentially the EC2 snapshot is triggered like this:

system("xfs_freeze -f /vol")                        and die;
system("ec2-create-snapshot $vol -K $key -C $crt ") and die;
system("xfs_freeze -u /vol")                        and die;

This method is based on advice from here

Notice I am doing an xfs_freeze of the entire volume on which the mysql data sits. It is intended to be used with volume managers and hardware RAID devices that support the creation of snapshots such as EBS.

The -f flag requests the specified XFS filesystem to be frozen from new modifications. When this is selected, all ongoing transactions in the filesystem are allowed to complete, new write system calls are halted, other calls which modify the filesystem are halted, and all dirty data, metadata, and log information are written to disk. Any process attempting to write to the frozen filesystem will block waiting for the filesystem to be unfrozen.

Note that even after freezing, the on-disk filesystem can contain information on files that are still in the process of unlinking. These files will not be unlinked until the filesystem is unfrozen or a clean mount of the snapshot is complete.

The -u flag is used to un-freeze the filesystem and allow operations to continue. Any filesystem modifications that were blocked by the freeze are unblocked and allowed to complete.

EC2 recommends this for ec2-create-snapshot:

When taking a snapshot of a file system, we recommend unmounting it first. This ensures the file system metadata is in a consistent state, that the ‘mounted indicator’ is cleared, and that all applications using that file system are stopped and in a consistent state. Some file systems, such as xfs, can freeze and unfreeze activity so a snapshot can be made without unmounting.

Clearly we don’t want to unmount the whole drive, so freezing the XFS volume is our best bet.

On a couple of occassions, we’ve seen the whole mysql server crash when a backup was taking place. The first symptoms displayed in logs is a long semaphore wait. This message is repeated for different threads:

mysqld[26852]: InnoDB: Warning: a long semaphore wait:

mysqld[26852]: --Thread 46912644213072 has waited at ./../include/trx0sys.ic line 101 for 241.00 seconds the semaphore:

mysqld[26852]: X-lock on RW-latch at 0x2aaaaf01cc08 created in file buf0buf.c line 497

mysqld[26852]: a writer (thread id 46912644213072) has reserved it in mode  wait exclusive

After about 10 minutes of this behaviour the server crashes. This is because 101 threads have made 101 connections where the max_connections=100. This only occurs whenever we are doing a snapshot in the background which makes me thing its a deadlock condition on the underlying filesystem.

mysqld[26852]: InnoDB: We intentionally crash the server, because it appears to be hung.

mysqld[26852]: 091207 20:16:41InnoDB: Assertion failure in thread 46912625965392 in file srv0srv.c line 2097

mysqld[26852]: InnoDB: We intentionally generate a memory trap.

mysqld[26852]: InnoDB: Submit a detailed bug report to http://bugs.mysql.com.

mysqld[26852]: InnoDB: If you get repeated assertion failures or crashes, even

mysqld[26852]: InnoDB: immediately after the mysqld startup, there may be

mysqld[26852]: InnoDB: corruption in the InnoDB tablespace. Please refer to

mysqld[26852]: InnoDB: http://dev.mysql.com/doc/refman/5.0/en/forcing-recovery.html

mysqld[26852]: InnoDB: about forcing recovery.

mysqld[26852]: 091207 20:16:41 - mysqld got signal 11 ;

mysqld[26852]: This could be because you hit a bug. It is also possible that this binary

mysqld[26852]: or one of the libraries it was linked against is corrupt, improperly built,

mysqld[26852]: or misconfigured. This error can also be caused by malfunctioning hardware.

mysqld[26852]: We will try our best to scrape up some info that will hopefully help diagnose

mysqld[26852]: the problem, but since we have already crashed, something is definitely wrong

mysqld[26852]: and this may fail.

In panic mode, we restarted the whole server which was able to then recover from this error. But is there a more graceful recovery open to us?

Alestic have released a revised version of the ec2-create-snapshot called ec2-consistent-snapshot

Here are some of the ways in which the ec2-consistent-snapshot program has improved over the original:

Command line options for passing in AWS keys, MySQL access information, and more.
Can be run with or without a MySQL database on the file system. This lets you use the command to initiate snapshots for any EBS volume.
Can be used with or without XFS file systems, though if you don’t use XFS, you run the risk of not having a consistent file system on EBS volume restore.
Instead of using the painfully slow ec2-create-snapshot command written in Java, this Perl program accesses the EC2 API directly with orders of magnitude speed improvement.
A preliminary FLUSH is performed on the MySQL database before the FLUSH WITH READ LOCK. This preparation reduces the total time the tables are locked.
A preliminary sync is performed on the XFS file system before the xfs_freeze. This preparation reduces the total time the file system is locked.
The MySQL LOCK now has timeouts and retries around it. This prevents horrible blocking interactions between the database lock, long running queries, and normal transactions. The length of the timeout and the number of retries are configurable with command line options.
The MySQL FLUSH is done in such a way that the statement does not propagate through to slave databases, negatively impacting their performance and/or causing negative blocking interactions with long running queries.
Cleans up MySQL and XFS locks if it is interrupted, if a timeout happens, or if other errors occur. This prevents a number of serious locking issues when things go wrong with the environment or EC2 API.
Can snapshot EBS volumes in a region other than the default (e.g., eu-west-1).
Can initiate snapshots of multiple EBS volumes at the same time while everything is consistently locked. This has been used to create consistent snapshots of RAIDed EBS volumes.

Here’s hoping that this release fixes the problems with locking of the filesystem! Notice we don’t attempt to flush and lock since we are using the InnoDB engine.

Revised snapshot code looks like this:

# create snapshot
system("xfs_freeze -f /vol")                        and die;
system("ec2-consistent-snapshot --aws-access-key-id $key --aws-secret-access-key $secret --xfs-filesystem /vol $vol") and die;
system("xfs_freeze -u /vol")                        and die;