Planet HA

DRBD backported (or not) to 2.6.32 in EL6 ?

2010-07-20T12:54:32+00:00

As some of you already know it, DRBD is now (since kernel 2.6.33) part of the mainline/upstream kernel. Some were expecting RHEL6 to come with that kernel (used for Fedora 13). The latest RHEL6beta2 still comes with 2.6.32, which doesn’t include DRBD support. Of course we still don’t know what the ‘frozen’ RHEL6 kernel will be but on the other hand, we know that Red Hat quite often ‘backports’ modules from newer kernel into the RHEL kernel. What about DRBD ? At the time of writing this blog post, it seems still undecided, but you can follow the DRBD RFE on Upstream Bugzilla to get a clue, or even comment on it if you have a bugzilla account to make hear your voice. On the other hand, you can still be sure that even if DRBD isn’t part of EL6, CentOS will still ship it in the Extras repository, like for EL4/EL5 …

19 Jul 2010

2010-07-19T22:25:42+00:00

It has been a while since I took the chance to blog here; the time has been pretty packed with shipping SUSE Linux Enterprise 11 Service-Pack 1's High-Availability Extension (or SLE HA 11 SP1 for short ;-), and supporting the first deployments.

It is a good time to look back and review the very awesome new features that the community developed along with us, and that we are shipping as Enterprise-ready now.

A feature that I am personally very impressed by is the OCFS2 reflink feature; basically, OCFS2 cracked the hard nut of cluster-wide copy-on-write snapshots, which LVM2 has been trying to for years. This allows space-efficient and very fast provisioning of new VMs, snapshots for backup, cloning from templates, cloning from clones, etcetera; it really is amazing.

For those of you who prefer a visual, the team from NGN taped a video with me being interviewed by Sander at Novell's BrainShare in Amsterdam; this is my first video interview ever!

In case you would like an audio-only review, Ron and terry interviewed me for Novell Open Audio as well.

I hope you find them informative - if so, please spread them, and let me know your feedback.

English-language Linux Cluster stack training coming to Berlin

2010-07-16T07:44:03+00:00

We usually do our English-language Linux Cluster Stack training sessions in the U.S., with training in Europe being primarily held in German. So if you live in Europe, speak English, and want to hone your high availability skills on your own continent, learning from the top notch experts, here is your rare chance:

November 9-12, were are doing our DRBD Total training in one of Europe’s finest cities, Berlin. Berlin is easily reached by air or rail, and our training location is directly next to historical Allied Checkpoint Charlie on Friedrichstraße. Training is exclusively in English, and covers DRBD, the Heartbeat/Corosync/Pacemaker cluster stack, and a number of high availability applications.

I hear you say, “November? Then why are you telling me this now?” Well guess what, there is a 15% early bird discount off the standard price if you register before August 31. So register quickly and grab your chance! Our classes usually sell out fast.

By the way: if you do speak German, we also have a German-language training coming up in Vienna in September. Take a look!

High availability for MySQL on Amazon EC2 – Part 3 – Configuring the HA resources

2010-07-12T16:15:40+00:00

This post is the third of a series that started here.

From the previous of this series, we now have two working EC2 instances that are EBS based. The first instance is the monitor, usually an m1.small type instance and the second instance is hamysql, a large instance type. So far, we have configured Heartbeat for communication between the nodes. Now, it is time to configure the HA resources in Pacemaker. Here are the requirements for the HA resources:

On Monitor

Kill the other instance
Free the share resources
Start a new hamysql instance
Reconfigure the clients
Give away the resources

On hamysql

Run MySQL
Monitor MySQL
Get out of way if MySQL is not running

From the above requirements, the resource we will create will need to have an "affinity" or preference to run on hamysql. This is accomplished by location directives. I chose to put a weight of 1000 to hamysql and 1 for monitor. Also the resource is obviously not doing the same thing on both hosts so I decided to use a resource of type anything which is just a script. Same script name and path on each host but different content. To configure Pacemaker, assuming both nodes are online, just performs these steps:

Place holders for the resources scripts
On each host:

PLAIN TEXT

CODE:

root@monitor:~# touch /usr/local/bin/mysql
root@monitor:~# chmod u+x /usr/local/bin/mysql

We will come back in the next posts to these scripts. Then, on one of the host:

PLAIN TEXT

CODE:

root@monitor:~# crm_attribute -t crm_config -n stonith-enabled -v false
root@monitor:~# crm_attribute --attr-name symmetric-cluster --attr-value false
root@monitor:~# crm configure
crm(live)configure# primitive mysql ocf:heartbeat:anything \
params binfile="/usr/local/bin/mysql" pidfile="/var/run/heartbeat/mysql.pid" \
op start interval="0" timeout="300s"
crm(live)configure# location loc-1 mysql 1000: hamysql
crm(live)configure# location loc-2 mysql 1: monitor
crm(live)configure# commit

and that's it. We know have a resource which is just a shell script that will preferably run on hamysql but, if hamysql is not available, will be able to run on monitor. That's the basis we will expand on in the next posts.

Entry posted by Yves Trudeau | 2 comments

Add to: | | | |

Automating MySQL replication with Pacemaker

2010-07-12T06:54:32+00:00

A little while back we did a webinar on using Pacemaker to fully automate MySQL replication. We now have a recording of that webinar available from our web site.

Linux cluster stack Debian packages for lenny!

2010-07-09T06:00:48+00:00

Thanks to Martin Loschwitz, official Linux cluster stack Debian packages are now available for Debian lenny. Check out the the lenny-backports repository on backports.org (and your favorite local mirror).

When upgrading from Heartbeat 2.1.3, this webinar recording may come in handy. The webinar covers an upgrade to squeeze, but the upgrade procedure is identical if you’re staying on lenny.

LinuxTag presentation now available for download

2010-07-08T15:26:38+00:00

A live recording of my LinuxTag 2010 presentation entitled Storage Done Right: Building a Resilient, Distributed, Highly Available Open Source iSCSI SAN is now available from our web site. If you want to find out how to build a complete SAN from 100% open source, do take a look!

I do apologize for the less-than-optimal sound quality. I did the recording myself with my laptop mike, so unfortunately there’s quite a bit of clipping in the audio track. I hope my ramblings are still somewhat audible.

New webinar recordings available

2010-07-06T06:29:33+00:00

A couple of new webinar recordings are available from our web site:

Upgrading to Pacemaker on Debian squeeze tells you all there is to know about upgrading to the Pacemaker cluster manager on the upcoming Debian release, “squeeze”.
High Availability for Oracle databases shows you how to leverage the Linux cluster stack to deploy highly available Oracle databases.

Like our live webinars, the recordings are of course free of charge. Enjoy!

High availability for MySQL on Amazon EC2 – Part 2 – Setting up the initial instances

2010-06-29T15:23:45+00:00

This post is the second of a series that started here.

The first step to build the HA solution is to create two working instances, configure them to be EBS based and create a security group for them. A third instance, the client, will be discussed in part 7. Since this will be a proof of concept, I'll be using m1.small type instances while normally, the mysql host would be much larger. Using another type is trivial. I will assume you are using the command line api tools, on Ubuntu, install "ec2-api-tools". The use of these tools simplifies the expression of the command compared to the web based console.

Create the security group

The instances involved in the MySQL HA setup will need to be inside the same security group for networking purposes and the help identify them. To create a security simply run this command:

PLAIN TEXT

CODE:

yves@yves-laptop:~$ export EC2_CERT=cert-yves.pem
yves@yves-laptop:~$ export EC2_PRIVATE_KEY=pk-yves.pem
yves@yves-laptop:~$ ec2-add-group hamysql -d 'nodes for HA MySQL solution'
GROUP hamysql nodes for HA MySQL solution

From now, I'll always assume the EC2_CERT and EC2_PRIVATE_KEY environment variables are setup in your shell. Next, we need to authorize some communications for the security group. I'll authorize 3306/tcp (MySQL) from hamysql, 694/udp (Heartbeat) from hamysql and 22 (SSH) from everywhere. You can be more restrictive for SSH if you want to.

PLAIN TEXT

CODE:

yves@yves-laptop:~$ ec2-authorize hamysql -P tcp -p 3306 -o hamysql -u 834362721059
yves@yves-laptop:~$ ec2-authorize hamysql -P udp -p 694 -o hamysql -u 834362721059
yves@yves-laptop:~$ ec2-authorize hamysql -P tcp -p 22 -s 0.0.0.0/0
yves@yves-laptop:~$ ec2-describe-group hamysql
GROUP 834362721059 hamysql nodes for HA MySQL solution
PERMISSION 834362721059 hamysql ALLOWS tcp 3306 3306 FROM USER 834362721059 GRPNAME hamysql
PERMISSION 834362721059 hamysql ALLOWS udp 694 694 FROM USER 834362721059 GRPNAME hamysql
PERMISSION 834362721059 hamysql ALLOWS tcp 22 22 FROM CIDR 0.0.0.0/0

Launch the instances

Now we can start creating our instances. Since this is only a proof of concept, I'll built 2 m1.small instances, fell free to use other types. At the time I wrote this, the following AMI seems ok.

PLAIN TEXT

CODE:

yves@yves-laptop:~$ ec2-describe-images ami-1cdf3775
IMAGE ami-1cdf3775 099720109477/ubuntu-images-testing/ubuntu-lucid-daily-i386-server-20100618 099720109477 available public i386 machine aki-aca44cc5 instance-store

So, lauching 2 of these,

PLAIN TEXT

CODE:

yves@yves-laptop:~$ ec2-run-instances ami-1cdf3775 -n 2 -g hamysql -t m1.small -k yves-key
RESERVATION r-a29c31c9 834362721059 hamysql
INSTANCE i-a23a21c9 ami-1cdf3775 pending yves-key 0 m1.small 2010-06-18T20:11:14+0000 us-east-1c aki-aca44cc5 monitoring-disabled instance-store
INSTANCE i-a03a21cb ami-1cdf3775 pending yves-key 1 m1.small 2010-06-18T20:11:14+0000 us-east-1c aki-aca44cc5 monitoring-disabled
yves@yves-laptop:~$ ec2-describe-instances
RESERVATION r-a29c31c9 834362721059 hamysql
INSTANCE i-a23a21c9 ami-1cdf3775 ec2-174-129-89-188.compute-1.amazonaws.com domU-12-31-39-02-BD-C5.compute-1.internal running yves-key 0 m1.small 2010-06-18T20:11:14+0000 us-east-1c aki-aca44cc5 monitoring-disabled 174.129.89.188 10.248.194.51 instance-store
INSTANCE i-a03a21cb ami-1cdf3775 ec2-174-129-187-170.compute-1.amazonaws.com domU-12-31-39-03-A4-62.compute-1.internal running yves-key 1 m1.small 2010-06-18T20:11:14+0000 us-east-1c aki-aca44cc5 monitoring-disabled 174.129.187.170 10.249.167.144 instance-store

I don't know about you but I don't like multi-lines output so I wrote a small filter script to on one line the parameters I need separated by a delimiter.

PLAIN TEXT

CODE:

yves@yves-laptop:~$ cat filtre_instances.pl
#!/usr/bin/perl
$SecGroup = '';
$IPAdd = '';
$Instance_ID = '';
while (<STDIN>) {
chomp $_;
#print "Processing: $_\n";
@fields = split /\t/, $_;
if (/^RESERVATION/) {
$SecGroup = $fields[3];
}
if (/^INSTANCE/) {
$IPAdd = $fields[17];
$STORE = $fields[20];
$Instance_ID= $fields[1];
$AMI_ID= $fields[2];
$PUBDNS = $fields[3];
$STATUS = $fields[5];
$START = $fields[10];
print "$SecGroup|$IPAdd|$Instance_ID|$AMI_ID|$PUBDNS|$STATUS|$START|$STORE \n"
}
}

and now we have

PLAIN TEXT

CODE:

yves@yves-laptop:~$ ec2-describe-instances | ./filtre_instances.pl | grep hamysql
hamysql|10.248.194.51|i-a23a21c9|ami-1cdf3775|ec2-174-129-89-188.compute-1.amazonaws.com|running|2010-06-18T20:11:14+0000|instance-store
hamysql|10.249.167.144|i-a03a21cb|ami-1cdf3775|ec2-174-129-187-170.compute-1.amazonaws.com|running|2010-06-18T20:11:14+0000|instance-store

which is to my opinion easier to manipulate.

Configuring Heartbeat

Now, let's configure Heartbeat. The first thing to is to set up the hostname on both host. Heartbeat identifies the host on which it is running by its hostname so that's mandatory step.

First host:

PLAIN TEXT

CODE:

yves@yves-laptop:~$ ssh -i ~/.ssh/yves-key.pem ubuntu@ec2-174-129-89-188.compute-1.amazonaws.com
ubuntu@domU-12-31-39-07-C8-32:~$ sudo su -
ubuntu@domU-12-31-39-07-C8-32:~# hostname monitor

Second host:

PLAIN TEXT

CODE:

yves@yves-laptop:~$ ssh -i ~/.ssh/yves-key.pem ubuntu@ec2-174-129-89-188.compute-1.amazonaws.com
ubuntu@domU-12-31-38-04-E5-E4:~$ sudo su -
ubuntu@domU-12-31-38-04-E5-E4:~# hostname hamysql

We don't really need to set /etc/hostname since it is overwritten when the instance is started, even when using EBS based AMI. The next step is to install Heartbeat and Pacemaker on both host, with Ubuntu 10.04, it is very straightforward:

PLAIN TEXT

CODE:

root@monitor:~# apt-get install heartbeat pacemaker
and
root@hamysql:~# apt-get install heartbeat pacemaker

Then we can proceed and configure Heartbeat, Pacemaker will come later. Heartbeat needs 2 configuration files, /etc/ha.d/authkeys for cluster authentication and /etc/ha.d/ha.cf which is the configuration file per say. On both host, the chosen key in the authkeys file must be identical and good way to generate unique one is to run "date | md5sum" and grab a substring from the output.

PLAIN TEXT

CODE:

root@monitor:/etc/ha.d# cat authkeys
auth 1
1 sha1 c97f2bb4b5ae90f149dc314ed

Also don't forget to restrict the access rights on the file like:

PLAIN TEXT

CODE:

root@monitor:/etc/ha.d# chmod 600 authkeys

For the /etc/ha.d/ha.cf file, since EC2 does not support neither broadcast or multicast within the security group, we need to use unicast (ucast) so both files will not be identical. The ucast entry on one host will contain the IP address on the internal network of the other host. On the monitor host, we will have:

PLAIN TEXT

CODE:

root@monitor:/etc/ha.d# cat ha.cf
autojoin none
ucast eth0 10.249.167.144
warntime 5
deadtime 15
initdead 60
keepalive 2
crm respawn
node monitor
node hamysql

and on the hamysql host:

PLAIN TEXT

CODE:

root@hamysql:/etc/ha.d# cat ha.cf
autojoin none
ucast eth0 10.248.194.51
warntime 5
deadtime 15
initdead 60
keepalive 2
crm respawn
node monitor
node hamysql

Let's review briefly the configuration file. First we have setup "autojoin none" that means no host not listed explicitely in the configuration file can join the cluster so we know we have at most 2 members, "monitor" and "hamysql". Next is the ucast communication channel to reach the other node and the timing parameters. "warntime" is a soft timeout in second that logs the other node is later while "deadtime" is the hard limit after which heartbeat will the consider the other node dead and start actions to restore the service. The "initdead" is just a startup delay to allow host to fully boot before attempting actions and "crm respawn" starts the Pacemaker resources manager. Finally, we have the two "node" declarations" for the cluster members.

So we are done with the configure, time to try if it works. On both hosts, run:

PLAIN TEXT

CODE:

service heartbeat start

and if everything is right, after at most a minute, you should be able to see both heartbeat processes chatting over the network

PLAIN TEXT

CODE:

root@monitor:~# tcpdump -i eth0 port 694
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes
20:38:36.536302 IP domU-12-31-38-04-E5-E4.compute-1.internal.57802> domU-12-31-39-07-C8-32.compute-1.internal.694: UDP, length 211
20:38:36.928860 IP domU-12-31-39-07-C8-32.compute-1.internal.34058> domU-12-31-38-04-E5-E4.compute-1.internal.694: UDP, length 212
20:38:38.580245 IP domU-12-31-38-04-E5-E4.compute-1.internal.57802> domU-12-31-39-07-C8-32.compute-1.internal.694: UDP, length 211
20:38:38.938814 IP domU-12-31-39-07-C8-32.compute-1.internal.34058> domU-12-31-38-04-E5-E4.compute-1.internal.694: UDP, length 212

We can also use the "crm" tool to query the cluster status.

PLAIN TEXT

CODE:

root@monitor:~# crm status
============
Last updated: Tue Jun 29 13:56:04 2010
Stack: Heartbeat
Current DC: monitor (504f45ea-7aee-4fa5-b0ee-a5ac07975ce4) - partition with quorum
Version: 1.0.8-042548a451fce8400660f6031f4da6f0223dd5dd
2 Nodes configured, unknown expected votes
1 Resources configured.
============
Online: [ monitor hamysql ]

Install MySQL

For the sake of simplicity, we will just install the MySQL version in the Ubuntu repository by doing:

PLAIN TEXT

CODE:

root@hamysql:~# apt-get install mysql-server-5.1

The package install MySQL has an automatic startup script controlled by init (new to lucid). That's fine, I will surprise you but Pacemaker will not manager MySQL, just the host running it. I'll also skip the raid configuration of multiple EBS volumes since it is not the main purpose of this blog series.

EBS based AMI

Others have produce excellent article on how to create EBS based AMI, I will not reinvent the wheel. I followed this one: http://www.capsunlock.net/2009/12/create-ebs-boot-ami.html

Upcoming in part 3, the configuration of the HA resources.

Entry posted by Yves Trudeau | 2 comments

Add to: | | | |

Pacemaker 1.0.9 Released

2010-06-23T13:46:14+00:00

The latest installment of the Pacemaker 1.0 stable series is now ready for general consumption.

Coinciding with 1.0.9 is a new version of Corosync (1.2.5). Included in both are some important fixes that should resolve most of the startup issues people have been seeing. Also included in this release are the fixes for issues reported by Valgrind and Coverity.

As per our release calendar, the next 1.0 release is planned for mid-September and 1.1.3 will be available in late July.

I’d like to particularly thank Keisuke MORI for his help with this release. Keisuke-san has taken on the role of Patch Manager for 1.0, so it is because of his hard work that we have backports of all the bugfixes from 1.1 :-)

This change has enabled me to focus on 1.1 and, I hope, be slightly more responsive to bug reports and questions on the mailing list(s).

Pre-built packages for Pacemaker and it’s immediate dependancies are available immediately for openSUSE, SLES, Fedora, RHEL, CentOS from the ClusterLabs Build Area.

Regular updaters may also have noticed the expanded version scheme used by packages on clusterlabs.org. The build scripts now automatically bump the version numbers when rebuilding the stack. This usually occurs when new versions of corosync, cluster-glue or heartbeat come out.

Versions are now of the form: x.y.x-a.b

x.y.z is the upstream version (this is the only time the tarball is changed)
a indicates the number of spec file changes (ie. changes to dependancies)
b indicates how many times the package has been rebuilt with unchanged tarballs and spec files

So the following version: pacemaker-1.0.9-1.4 would mean the fourth rebuild of the initial spec file for the upstream version 1.0.9 of Pacemaker.

Debian users should check for updates Martin’s repo over the coming days and Ubuntu fans can visit LaunchPad for 8.04 and 9.10 packages.

The source tarball is also available directly from Mercurial.

General installation instructions are available at from the ClusterLabs wiki.

Release Statistics

Changesets	152
Diff	266 files changed, 14324 insertions(+), 3842 deletions(-)

Changes of note since Pacemaker-1.0.8

High: ais: Ensure the list of active processes sent to clients is always up-to-date
High: ais: Fix previous commit, actually return a result in get_process_list()
High: ais: Fix two more uses of getpwnam() in non-thread-safe locations
High: ais: Look for the correct conf variable for turning on file logging
High: ais: Need to find a better and thread-safe way to set core_uses_pid. Disable for now.
High: ais: Use the threadsafe version of getpwnam
High: cib: Also free query result for xpath operations that return more than one hit
High: cib: Fix the application of unversioned diffs
High: cib: Remove old developmental error logging
High: Core: Bug lf#2414 - Prevent use-after-free reported by valgrind when doing xpath based deletions
High: Core: Fix memory leak in replace_xml_child() reported by valgrind
High: Core: fix memory leaks exposed by valgrind
High: crmd: Bug 2401 - Improved detection of partially active peers
High: crmd: Bug lf#2379 - Ensure the cluster terminates when the PE is not available
High: crmd: Bug lf#2414 - Prevent use-after-free of the PE connection after it dies
High: crmd: Bug lf#2439 - cancel_op() can also return HA_RSCBUSY
High: crmd: Bug lf#2439 - Handle asynchronous notification of resource deletion events
High: crmd: Do not allow the target_rc to be misused by resource agents
High: crmd: Do not ignore action timeouts based on FSA state
High: crmd: Ensure we dont get stuck in S_PENDING if we loose an election to someone that never talks to us again
High: crmd: Fix memory leaks exposed by valgrind
High: crmd: Remove race condition that could lead to multiple instances of a clone being active on a machine
High: crmd: Send erase_status_tag() calls to the local CIB when the DC is fenced, since there is no DC to accept them
High: PE: Bug lf#1959 - Fail unmanaged resources should not prevent other services from shutting down
High: PE: Bug lf#2383 - Combine failcounts for all instances of an anonymous clone on a host
High: PE: Bug lf#2384 - Fix intra-set colocation and ordering
High: PE: Bug lf#2403 - Enforce mandatory promotion (colocation) constraints
High: PE: Bug lf#2412 - Correctly locate clone instances by their prefix
High: PE: Bug lf#2422 - Ordering dependencies on partially active groups not observed properly
High: PE: Bug lf#2424 - Use notify oepration definition if it exists in the configuration
High: PE: Bug lf#2433 - No services should be stopped until probes finish
High: PE: Do not be so quick to pull the trigger on nodes that are coming up
High: PE: Fix colocation for interleaved clones
High: PE: Fix colocation with partially active groups
High: PE: Fix memory leaks reported by valgrind
High: PE: Make the current data set a global variable so it does not need to be passed around everywhere
High: PE: Prevent endless loop when looking for operation definitions in the configuration
High: PE: Rewrite native_merge_weights() to avoid Fix use-after-free
High: Shell: always reload status if working with the cluster (bnc#590035)
High: Tools: crm_mon - fix memory leaks exposed by valgrind
Medium: ais: Correctly set logfile permissions in all cases
Medium: ais: create the final directory too for resource agents (bnc#603190)
Medium: ais: Make sure debug messages make it into the logfiles too
Medium: Build: Do not enable the -ansi compiler option by default, prevents use of strtoll()
Medium: cib: Bug lf#2352 - Changes to group order are not detected or broadcast to peers
Medium: cib: Correctly free the cib contents at signoff when in file-based mode
Medium: cib: xpath - Allow all hits to be deleted, allow the no_children option to return multiple hits
Medium: PE: Bug lf#2391 - Ensure important options (notify, unique, etc) are always exposed during resource operations
Medium: PE: Bug lf#2410 - Do not complain about missing agents during probes of a-symetric clusters
Medium: PE: Bug lf#2426 - stop-all-resources should not apply to stonith resources
Medium: PE: Bug lf#2435 - Support colocation sets with negative scores
Medium: PE: Check for use-of-NULL in dump_node_scores()
Medium: PE: Do not overwrite existing meta attributes (like timeout) for notify operations
Medium: PE: Ensure deallocated resources are stopped
Medium: PE: If there are no compatible peers when interleaving clones, ensure the instance is stopped
Medium: PE: Ignore colocation weights from clone instances
Medium: RA: SystemHealth: exit properly when the required software is not installed (bnc#587940)
Medium: Shell: do not error on missing resource agent with asymmetrical clusters (lf#2410)
Medium: Shell: do not verify empty configurations (bnc#602711)
Medium: shell: find hb_delnode in correct directory
Medium: Shell: observe op_defaults when verifying primitives (bnc#590033)
Medium: Shell: on no id match the first of property-like elements (lf#2420)
Medium: Shell: skip resource checks for property-like elements (lf#2420)
Medium: Shell: verify meta attributes and properties (bnc#589867)
Medium: Shell: verify only changed elements on commit (bnc#590033)
Medium: Tools: crm_mon: refresh screen on terminal resize (bnc#589811)

High availability for MySQL on Amazon EC2 – Part 1 – Intro

2010-06-17T19:56:47+00:00

Like many, I have been seduced by the power and flexibility of Amazon EC2. Being able to launch new instances at will depending on the load, is almost too good to be true. Amazon has also some drawbacks, availability is not guaranteed and discovery protocols relying on Ethernet broadcast or multicast cannot be used. That means, it is not easy to have a truly highly available solution for MySQL on EC2. If a MySQL instance fails, here are some challenges that we face:

Detect the failure
Kill the faulty instance
Free the shared resources (ex: EBS volumes)
Launch a new instances
Reconfigure the shared resources
Reconfigure instance monitoring
Reconfigure the database clients

Facing these challenges, a customer recently asked me if I could build a viable HA solution for MySQL on EC2. The goal is to have a cheap small instance monitor the availability of a large (any size) and taking measures to keep MySQL available. A few weeks later, I ended up with a solution that work and is decently elegant using Pacemaker and Heartbeat. The setup is fairly complex, being in the Amazon EC2 virtual world is not a simplification, far from. Because of the complexity, the story will be broken into multiple posts:

Part 1 - Intro (this post)
Part 2 - Setting up the initial instances
Part 3 - Configuring the HA resources
Part 4 - The instance restart script
Part 5 - The instance monitoring script
Part 6 - Publishing the MySQL server location
Part 7 - Pitfalls to avoid

Hopefully, I should be able to write those posts quickly but since consulting is my primary duty, I don't have much control over my workload.

Stay tuned!

Entry posted by Yves Trudeau | 8 comments

Add to: | | | |

DRBD source dropped from Debian unstable

2010-06-11T19:05:00+00:00

OK, not really... DRBD 8.3.7 is an official part of the Linux kernel since 2.6.33, and thanks to Dann Frazier we also have a backport of that patch in Debian's 2.6.32 kernel as well.

Until yesterday, the DRBD source package in Debian built a -utils and a -source package, the latter was now dropped. From now on, users no longer need to recompile the DRBD module after a kernel upgrade.

Speaking at LinuxTag 2010 this Friday

2010-06-09T09:58:15+00:00

Just as a quick reminder, as LinuxTag 2010 in Berlin is going into full swing: I am speaking on highly available, distributed, open source storage this Friday in room Europa I at 4pm CEST.

If you’re in Berlin, drop by! Admission is free.

Reminder: Pacemaker/Debian webinar, today 1400 UTC

2010-06-07T06:52:22+00:00

For those of you who haven’t yet registered, this is our reminder for today’s Clustering in Debian webinar at 1400 UTC. If you’re planning to run Pacemaker on the upcoming Debian squeeze release, don’t miss this!

Upcoming webinar: migrating to Pacemaker on Debian squeeze

2010-05-20T12:23:00+00:00

Martin Loschwitz, the longest-serving Debian Developer in our ranks, will present a walk-through of going from Linux clustering on Debian lenny (with Heartbeat 2.1.3, shudder) to squeeze (with Heartbeat 3 and Pacemaker). In this webinar you will learn everything you need to know about a painless migration to the new Linux cluster stack.

This webinar will be held on Monday, June 7, at 1400 UTC. You must provide a valid email address to receive a meeting key. Register below!

The widget should display the start time in your local timezone. In case Dimdim gets this wrong, 1400 UTC is the correct time.

In case you haven’t noticed, we no longer manage clusters with XML

2010-05-20T10:19:04+00:00

While Yves presents a valid approach for managing NDB with Pacemaker and the anything Linux-HA resource agent (a generic wrapper for any daemon based application), the XML configuration shown is likely to have people running away screaming. This is how we had to do things back in the Heartbeat 2 days, which meant that as a cluster admin you would permanently run around with a loaded cocked shotgun with the muzzle pressed firmly against your foot.

Those days are long gone. Please don’t do this anymore. You also no longer need to use the low-level tools such as cibadmin or crm_attribute.

We have the crm shell for those things now. So instead of

crm_attribute -t crm_config -n stonith-enabled -v false
crm_attribute --attr-name symmetric-cluster --attr-value false

we now simply write

crm configure property stonith-enabled=false symmetric-cluster=false

Can’t remember a property name or supported value? Interactive mode and tab completion to the rescue:

# crm configure
crm(live)configure# property sto<tab>
stonith-action=   stonith-enabled=  stonith-timeout=
crm(live)configure# property stonith-enabled=<tab>
stonith-enabled (boolean, [true]):
Failed nodes are STONITH'd
crm(live)configure# property stonith-enabled=false symmetric-cluster=false
crm(live)configure# commit

A bit more convenient, isn’t it? Now let’s get rid of dealing with XML. Instead of a horrific avalanche of tags and attributes that we then have to feed into the system with cibadmin, let’s do this!

# crm configure
crm(live)configure# primitive mgmd ocf:heartbeat:anything \
	params binfile="/usr/local/bin/fake_ndb_mgmd"
crm(live)configure# primitive ndbcluster ocf:heartbeat:anything \
	params binfile="/usr/local/bin/fake_ndb_cluster_start"
crm(live)configure# primitive ndbd ocf:heartbeat:anything \
	params binfile="/usr/local/bin/fake_ndbd"
crm(live)configure# clone ndbdclone ndbd \
	meta clone-max=2
crm(live)configure# location loc-1 mgmd inf: testvirtbox
crm(live)configure# location loc-2 ndbcluster inf: testvirtbox
crm(live)configure# location loc-3 ndbdclone -inf: testvirtbox
crm(live)configure# order ordered-set-1 inf: mgmd ndbdclone ndbcluster
crm(live)configure# commit

And of course, we get tab completion and online help there too. Give it a shot!

Pacemaker, please meet NDB Cluster or using Pacemaker/Heartbeat to start a NDB Cluster

2010-05-19T17:39:01+00:00

Customers have always asked me to make NDB Cluster starts automatically upon startup of the servers. For the ones who know NDB Cluster, it is tricky to make it starts automatically. I know at least 2 sets of scripts to manage NDB startup, ndb-initializer and from Johan configurator www.severalnines.com. If all the nodes come up at about the same time, it is not too bad but what if one the critical node takes much longer to start because of an fsck on a large ext3 partition. Then, a startup script becomes a nightmare. Finally, if the box on which the script is supposed to run didn't start at all. That's a lot of rules to handle.

Since all aspects of HA interest me, I was recently reading the Pacemaker documentation and I realized that Pacemaker has all the logic required to manage NDB Cluster startup. Okay it might seems weird to control a cluster by cluster but if you think about it, this is, I think, the best solution.

The Linux-HA project has split the old Heartbeat-2 project in 2 parts, the clustering and communication layer, Heartbeat and the resources manager, Pacemaker. A key new features that has been added to Pacemaker recently, a Clone resources set, that allows an optional startup if only one of 2 similar resources starts. I use this feature to start the data nodes. If after a major outage, only one of the physical host where the data nodes are located comes up, the cluster will start. The other features of Pacemaker that I need are resource location rsc_location and resource ordering rsc_order.

Let's start by the beginning. My NDB cluster is made of the following 3 nodes:

testvirtbox: ndb_mgmd (10.2.2.139)
test1: ndbd
test2: ndbd

For the sake of simplicity, I am not considering the SQL nodes but given the framework, extending to SQL nodes is trivial. Installing Pacemaker and Heartbeat is very easy on Lucid Lynx, just do the following:

PLAIN TEXT

CODE:

apt-get install heartbeat pacemaker

On other distributions, you might have to build from sources, look here for help.

There 2 minimal configuration files:

PLAIN TEXT

CODE:

root@test2:~# cat /etc/ha.d/authkeys
auth 1
1 sha1 yves
root@test2:~# cat /etc/ha.d/ha.cf
autojoin none
bcast eth0
warntime 5
deadtime 15
initdead 60
keepalive 2
node test1
node test2
node testvirtbox
crm respawn

And then, Heartbeat can be started on all nodes with /etc/init.d/heartbeat start.

Next, since Pacemaker is used to start resources and not to manage them, we don't need to define Stonith devices so (run on only one node):

PLAIN TEXT

CODE:

crm_attribute -t crm_config -n stonith-enabled -v false

A last before defining resources, since the Heartbeat cluster is asymmetrical, meaning resources will not be able to run anywhere, we must create an "Opt-In" cluster with (run on only one node):

PLAIN TEXT

CODE:

crm_attribute --attr-name symmetric-cluster --attr-value false

At this point, we have a running cluster controlling nothing. The trick with NDB Cluster is that Heartbeat is required to start the resources but not to stop them. In order to achieve this behavior, I created fake resource scripts that can be fully controlled by Heartbeat but allowing the one way behavior I wanted.

PLAIN TEXT

CODE:

root@testvirtbox:~# cat /usr/local/bin/fake_ndb_mgmd
#!/bin/bash
/usr/bin/nohup /usr/local/mysql/libexec/ndb_mgmd> /dev/null &
while [ 1 ]
do
/bin/sleep 60
done
root@testvirtbox:~# cat /usr/local/bin/fake_ndb_cluster_start
#!/bin/bash
#Give some time to the nodes to connect
/bin/sleep 15
/usr/local/mysql/bin/ndb_mgm -e 'all start'> /dev/null
while [ 1 ]
do
/bin/sleep 60
done
root@test1:~# cat /usr/local/bin/fake_ndbd
#!/bin/bash
#Give some time to ndb_mgmd to start
/bin/sleep 10
nohup /usr/local/mysql/libexec/ndbd -c 10.2.2.139> /dev/null &
while [ 1 ]
do
sleep 60
done

With Pacemaker it is not longer required to manipulate the cib in xml format but for this post, xml offers a compact way of presenting the configuration. The first things we need to define are the resources. A very handy resource type for us is the anything resource which allow an arbitrary script or binary to be run. The resources section will look like:

PLAIN TEXT

CODE:

<resources>
<primitive id="mgmd" class="ocf" type="anything" provider="heartbeat">
<instance_attributes id="params-mgmd">
<nvpair id="param-mgmd-binfile" name="binfile" value="/usr/local/bin/fake_ndb_mgmd"/>
<nvpair id="param-mgmd-pidnile" name="pidfile" value="/var/run/heartbeat/fake_ndb_mgmd.pid"/>
</instance_attributes>
</primitive>
<clone id="ndbdclone">
<meta_attributes id="ndbdclone-options">
<nvpair id="ndbdclone-option-1" name="globally-unique" value="false"/>
<nvpair id="ndbdclone-option-2" name="clone-max" value="2"/>
<nvpair id="ndbdclone-option-3" name="clone-node-max" value="1"/>
</meta_attributes>
<primitive id="ndbd" class="ocf" type="anything" provider="heartbeat">
<instance_attributes id="params-ndbd">
<nvpair id="param-ndbd-binfile" name="binfile" value="/usr/local/bin/fake_ndbd"/>
<nvpair id="param-ndbd-pidfile" name="pidfile" value="/var/run/heartbeat/fake_ndbd.pid"/>
</instance_attributes>
</primitive>
</clone>
<primitive id="ndbcluster" class="ocf" type="anything" provider="heartbeat">
<instance_attributes id="params-ndbcluster">
<nvpair id="param-ndbcluster-binfile" name="binfile" value="/usr/local/bin/fake_ndb_cluster_start"/>
<nvpair id="param-ndbcluster-pidfile" name="pidfile" value="/var/run/heartbeat/fake_ndb_cluster_start.pid"/>
</instance_attributes>
</primitive>
</resources>

Please note the ndbd resource is defined through the use of a clone set. The clone set will allow the cluster to start even if only one to the ndb node group is available. If you have multiple ndb node groups, you'll need one clone set per node group. The ndb_mgmd nodes or eventual SQL nodes could have been handled the same way although for SQL nodes, ndb_waiter is very handy. Once the resources are defined, we need to setup the constraints which cover mandatory locations and ordering.

PLAIN TEXT

CODE:

<constraints>
<rsc_location id="loc-1" rsc="mgmd" node="testvirtbox" score="INFINITY"/>
<rsc_location id="loc-2" rsc="ndbcluster" node="testvirtbox" score="INFINITY"/>
<rsc_location id="loc-3" rsc="ndbdclone" node="test1" score="INFINITY"/>
<rsc_location id="loc-4" rsc="ndbdclone" node="test2" score="INFINITY"/>
<rsc_order id="order-1">
<resource_set id="ordered-set-1" sequential="true">
<resource_ref id="mgmd"/>
<resource_ref id="ndbdclone"/>
<resource_ref id="ndbcluster"/>
</resource_set>
</rsc_order>
</constraints>

And... that's it. For my part, I configured Pacemaker by dumping the cib in xml format, editing and reloading. In term of commands, it means:

PLAIN TEXT

CODE:

cibadmin --query> local.xml
vi local.xml
cibadmin --replace --xml-file local.xml

Once NDB is started, you can even stop heartbeat, it is no longer required.

P.S.:

As suggested by Florian, here is the configuration in CLI format:

PLAIN TEXT

CODE:

root@testvirtbox:~# crm configure show
INFO: building help index
INFO: object order-1 cannot be represented in the CLI notation
node $id="27687295-f72c-49bd-b82d-25f32dbfe1e2" test2
node $id="3086852d-abb9-4bdb-93a1-9390e14c148c" test1
node $id="cad7f678-fc91-4f09-a39e-1dde6d5bcd30" testvirtbox
primitive mgmd ocf:heartbeat:anything \
params binfile="/usr/local/bin/fake_ndb_mgmd" pidfile="/var/run/heartbeat/fake_ndb_mgmd.pid"
primitive ndbcluster ocf:heartbeat:anything \
params binfile="/usr/local/bin/fake_ndb_cluster_start" pidfile="/var/run/heartbeat/fake_ndb_cluster_start.pid"
primitive ndbd1-IP ocf:heartbeat:anything \
params binfile="/usr/local/bin/fake_ndbd" pidfile="/var/run/heartbeat/fake_ndbd.pid"
clone ndbdclone ndbd1-IP \
meta globally-unique="false" clone-max="2" clone-node-max="1"
location loc-1 mgmd inf: testvirtbox
location loc-2 ndbcluster inf: testvirtbox
location loc-3 ndbdclone inf: test1
location loc-4 ndbdclone inf: test2
xml <rsc_order id="order-1"> \
<resource_set id="ordered-set-1" sequential="true"> \
<resource_ref id="mgmd"/> \
<resource_ref id="ndbdclone"/> \
<resource_ref id="ndbcluster"/> \
</resource_set> \
</rsc_order>

Entry posted by Yves Trudeau | 9 comments

Add to: | | | |

Novell announces SLE 11 SP 1

2010-05-19T12:00:52+00:00

Novell has announced Service Pack 1 for SUSE Linux Enterprise. LINBIT is a key contributor to this release, delivering DRBD and a number or cluster resource agents as part of Novell’s SLE High Availability Extension add-on.

This puts Novell firmly ahead of competitor Red Hat in terms of currently released and supported HA clustering solutions. In case you haven’t noticed, Red Hat is also being questioned by customers for not shipping DRBD at this time, and being ambiguous about its policy with regard to third-party support for kernel code. Novell, for their part, got that much right. Which entitles them to a bit of sarcasm, I suppose.

SLE 11 SP1 is coming on June 2.

Setting up DRBD in Pacemaker clusters on Fedora 13

2010-05-18T12:05:02+00:00

Andrew just published an update to his Clusters from Scratch documentation that contains a step-by-step guide for initially setting up DRBD based cluster resources on Fedora 13.

Yes, that’s the Fedora release with Linux 2.6.33, so no more installing DRBD kernel modules.

Linux cluster stack hits Debian squeeze

2010-05-16T10:52:27+00:00

The full Linux cluster stack, including Pacemaker, Heartbeat, Corosync and related packages, cleared all hurdles and migrated to the Debian testing repository this morning. This means that Pacemaker and both messaging layers it supports will be in the upcoming Debian release, codenamed squeeze.

Credit goes to the debian-ha maintainers crowd for making this possible: Martin Loschwitz, Simon Horman, Frederik Schüler, Anibal Monsalve Salazar, Guido Günther, Norbert Tretkowski, and others.

In terms of high availability support per distribution, this now has Debian, Ubuntu and SLES running roughly abreast.

Get real!

2010-05-13T18:04:31+00:00

So it seems Red Hat is a bit surprised (I’m not) by the number of subscribers on the “DRBD in RHEL 6, please” bug tripling in a couple of days. I guess they are also getting a few more tweets than usual about this.

But please, Red Hat’s EMEA Chief Supreme Open Source Evangelist saying “DRBD is limited with regard to integration with [Red Hat] Cluster”? Give me a break. I wrote the Red Hat Cluster Suite integration for DRBD. Lon Hohberger merged it almost a year ago. He also took care of the cluster.conf schema changes so DRBD properly integrates with Red Hat Cluster management tools. It’s been in all upstream Red Hat Cluster releases since. Lon also wrote the original cookbook for getting GFS2 running on Dual-Primary DRBD. Back in 2007!

The other cluster stack the RHEL 6 beta ships with, Pacemaker, has supported DRBD from the get-go.

That’s “limited integration”?

Want DRBD in RHEL 6? Make yourself heard!

2010-05-11T15:33:06+00:00

The folks over at Red Hat were kind enough to make public bug 585309, an enhancement request to include DRBD in RHEL 6. If you want DRBD on the last major platform that doesn’t include it yet, feel free to comment and make your voice heard!

A valid login on the Red Hat Bugzilla system is required to comment. Don’t have an account? Create one!

Debugging problems with row based replication

2010-05-07T01:27:48+00:00

MySQL 5.1 introduces row based binary logging. In fact, the default binary logging format in GA versions of MySQL 5.1 is 'MIXED' STATEMENT*; The binlog_format variable can still be changed per sessions which means it is possible that some of your binary log entries will be written in a row-based fashion instead of the actual statement which changed data, even when the global setting on the master is to write binary logs in statement mode. The row-based format does offer advantages particularly if triggers or stored procedures are used, or if non deterministic functions like RAND() are used in DML statements.

A statement based replication slave can get out of sync with the master fairly easily, especially if data is changed on the slave. It is possible for a statement to execute successfully on a slave even if the data is not 100% in sync, so MySQL doesn't know anything is wrong. This isn't the case with row-based replication. Only actual changes are written to the database with row-based logs. Take as an example a DELETE statement which does not modify any rows. No binary logging will be performed by this statement in row-based logs, but the DELETE will be written to a statement based binary log. Since row-based logs only contain information about changed rows, a replication slave which does not find a row referenced in a row-based log will immediately return an error: HA_ERR_KEY_NOT_FOUND.

This is probably best shown by example. First, lets create a test table on a MySQL master database and populate it with some data.

PLAIN TEXT

SQL:

CREATE TABLE `t1` (
`c1` int(11) NOT NULL AUTO_INCREMENT,
PRIMARY KEY (`c1`)
) ENGINE=InnoDB;
Query OK, 1 row affected (0.02 sec)

Insert a single "seed" row into the table:

PLAIN TEXT

SQL:

INSERT INTO repl.t1 VALUES ();
Query OK, 1 row affected (0.00 sec)

INSERT ... SELECT a few times to get some dummy data:

PLAIN TEXT

SQL:

INSERT INTO repl.t1 SELECT NULL FROM t1;
Query OK, 1 row affected (0.01 sec)
Records: 1 Duplicates: 0 Warnings: 0
INSERT INTO repl.t1 SELECT NULL FROM t1;
Query OK, 2 rows affected (0.01 sec)
Records: 2 Duplicates: 0 Warnings: 0
...
INSERT INTO repl.t1 SELECT NULL FROM t1;
Query OK, 32768 rows affected (0.62 sec)
Records: 32768 Duplicates: 0 Warnings: 0

Verify that we have a reasonable amount of data to play with:

PLAIN TEXT

SQL:

SELECT count(*) FROM t1;
+----------+
| count(*) |
+----------+
| 65536 |
+----------+
1 row IN SET (0.02 sec)

My test setup includes a MySQL slave. After the above setup completes, I am going to change data on the slave by truncating t1. After truncation:

PLAIN TEXT

SQL:

SELECT count(*)
FROM t1 AS empty_table_on_the_slave;
+----------+
| count(*) |
+----------+
| 0 |
+----------+
1 row IN SET (0.00 sec)

Now, lets make a modification on the master:

PLAIN TEXT

SQL:

UPDATE repl.t1 SET c1 = c1 + 100000 WHERE c1 = 1;
Query OK, 1 row affected (0.02 sec)
Rows matched: 1 Changed: 1 Warnings: 0

On the slave, SHOW SLAVE STATUS now reports an error (many fields omitted):

PLAIN TEXT

SQL:

SHOW SLAVE STATUS\G
Relay_Log_File: mysql_sandbox25162-relay-bin.000002
Relay_Log_Pos: 340718
Slave_SQL_Running: No
Last_Errno: 1032
Last_Error: Could NOT execute Update_rows event ON TABLE repl.t1; Can't find record in 't1', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log mysql-bin.000001, end_log_pos 340718

This error message is telling us that a row that was modified on the master can not be found on the slave:HA_ERR_KEY_NOT_FOUND. It doesn't tell us what row it couldn't find though. In our simple example, we know that all the rows are missing, and the one in particular that the server is trying to find is c1=1, but how can we find that out if we don't know the conditions under which replication went out of sync?

To determine exactly what row is having an issue, we can use the 'mysqlbinlog' utility in combination with the slave relay log. Examining the output of SHOW SLAVE STATUS, we can see that the slave is currently executing at relay log file: mysql_sandbox25162-relay-bin.000002 at position: 340718. MySQL 5.1 ships with a new version of mysqlbinlog that includes new options that make decoding RBR statements possible. These options are "--base64-output" and "-v". For debugging row based logs, I suggest '--base64-output=decode-rows -v". This omits any base64 output and instead replaces the output with semi-valid SQL statements. I say semi-valid because the statements do not include column names. Column names are not included in the binary log, only positions.

mysqlbinlog mysql_sandbox25162-relay-bin.000005 --base64-output=decode-rows -v > decoded.log

Examine the decoded binary log with the 'less' utility, and search forward for the value 340718:

PLAIN TEXT

SQL:

# at 340718
#100506 12:42:56 server id 1 end_log_pos 340637 Query thread_id=6 exec_time=0 error_code=0
SET TIMESTAMP=1273174976/*!*/;
BEGIN
/*!*/;
# at 340782
# at 340823
#100506 12:42:56 server id 1 end_log_pos 340678 Table_map: `repl`.`t1` mapped to number 15
#100506 12:42:56 server id 1 end_log_pos 340718 Update_rows: table id 15 flags: STMT_END_F
### UPDATE repl.t1
### WHERE
### @1=1
### SET
### @1=100001
# at 340863
#100506 12:42:56 server id 1 end_log_pos 340745 Xid = 51
COMMIT/*!*/;
# at 340890

As you can see, the RBR entry has been decoded as an update statement. The OLD value (the row that is missing) is represented in the WHERE clause. The binary log does not carry column names, so each column is represented by the column position prefixed with @.

There you have it. You can now find out exactly which row is missing and begin your investigation as to why it is missing. You may have to do a bit more leg work if many rows were modified during a transaction.

You should always set read_only = true on your MySQL slaves to prevent accidental changes in data from happening!

*edit*
The MySQL 5.1 default binary logging format changed back to STATEMENT in MySQL 5.1.29.

Entry posted by Justin Swanhart | 9 comments

Add to: | | | |

Feature Spotlight: Utilization

2010-05-04T08:00:00+00:00

New in 1.1 is the ability for Pacemaker to factor the system resources (RAM, CPU, etc) into its placement algorithms.

First, simply define the system resources provided by your nodes. We’ll use cores in this example, but you can literally use any name you care to dream up.

crm configure node pcmk-1 utilization cores=2
crm configure node pcmk-2 utilization cores=4

Then we tell the cluster how many cores are needed by each VM.

crm configure primitive kvm-small ocf:heartbeat:VirtualDomain utilization cores=1
crm configure primitive kvm-medium ocf:heartbeat:VirtualDomain utilization cores=2
crm configure primitive kvm-big ocf:heartbeat:VirtualDomain utilization cores=3

Finally, we tell Pacemaker how to use the information:

crm configure property placement-strategy=utilization

Now Pacemaker will ensure the load from your virtual machines will be distributed “evenly” throughout the cluster - without the need for convoluted sets of colocation constraints.

Download 1.1 today from: http://www.clusterlabs.org/rpm-next/

Limitations

This type of problem Pacemaker is dealing with here is known as the knapsack problem and falls into the NP-complete category of computer science problems - which is fancy way of saying “takes a really long time to solve”.

Clearly in a HA cluster, its not acceptable to spend minutes, let alone hours or days, finding an optional solution while services remain unavailable.

So instead of trying to solve the problem completely, Pacemaker uses a best effort algorithm for determining which node should host a particular service. This means it arrives at a “solution” much faster than traditional linear programming algorithms, but my do so at the price of leaving some services stopped.

In the contrived example above:

kvm-small would be allocated to pcmk-1
kvm-medium would be allocated to pcmk-2
kvm-large would remain inactive

Which is not ideal.

Strategies for Dealing with the Limitations

Ensure you have sufficient physical capacity It might sounds obvious, but if the physical capacity of your nodes is (close to) maxed out by the cluster under normal conditions, then failover isn’t going to go well. Even without the Utilization feature, you’ll start hitting timeouts and getting secondary “failures”.
Build some buffer into the capabilities advertised by the nodes Advertise slightly more resources than we physically have on the (usually valid) assumption that a VM will not use 100% of the configured number of cores/RAM/etc all the time. This practice is also known as “over commit”.
Specify service priorities If the cluster is going to sacrifice services, it should be the ones you care (comparatively) about the least.

Am I missing something?

2010-05-04T07:50:06+00:00

If you’re on software with a support contract, you should get support, right?

If the software you have support on has a documented feature that’s not working as documented, then you would expect to get a fix, a workaround, or even perhaps an explanation to the effect that you misunderstood the documentation in the first place, and the feature in fact does something else. Correct?

And if the feature in question is a customization facility, then you wouldn’t expect support on the customization itself, but you would expect the customization interface to work as documented, right?

So in summary, if you ran into an issue with a customization interface not working as the documentation says, you’d expect support to fix the interface, suggest a workaround, or provide additional documentation?

Well, I would.

SugarCRM appears to see things differently.

Pacemaker ships as part of Ubuntu 10.4 - Lucid Lynx

2010-05-03T09:10:00+00:00

Ubuntu LTS 10.04 now comes with full support for Pacemaker on Corosync and Heartbeat: http://fghaas.wordpress.com/2010/05/03/ubuntu-10-04-with-full-cluster-stack-support/

Kudos to everyone involved!

Ubuntu 10.04 with full cluster stack support

2010-05-03T06:52:11+00:00

Just in case you haven’t noticed, Ubuntu 10.04 “Lucid Lynx” has arrived. I upgraded my personal box over the weekend, and am happy to report that things went pretty flawlessly for me. So I’m now a happy lucid user.

What’s more important for the ever-growing cluster community is the fact that this LTS release comes with built-in support for the Pacemaker cluster stack — and indeed, it’s the first commercially-supported distribution that comes with support for both the Heartbeat and the Corosync cluster messaging layer. And it’s easy, too!

Here’s how I install the Pacemaker stack with Heartbeat (which I prefer) on an Ubuntu box:

aptitude install heartbeat pacemaker

And if you choose to go with Corosync, here’s how.

aptitude install corosync pacemaker

… and you’re good to go!

Credit for Ubuntu cluster stack packaging goes to Ante Karamatić (ivoks) and Andres Rodriguez (roaksoax), along with the rest of the ubuntu-ha team. Their work builds on Debian packaging by Martin Loschwitz, Guido Günther, Frederik Schüler and others.

MySQL HA , an alternative approach

2010-04-28T21:17:27+00:00

For those who've seen my presentation on MySQL HA, you already know that I often use a multimaster setup with a meta OCF resource that groups my favoured MySQL instance with the service ip , using a meta resource means that pacemaker monitors mysql, but it doesn't actually manage it. It's an approach that works for us.

One of the other approaches I will be looking at soon is the freshly released OCF resource that Florian announced last week.

Back in the days our approach meant we didn't have to use clone resources, which you might remember being pretty buggy in the v2 era, not wanting to use clons resources isn't really a valid reason anymore these days . I've also frequently mentioned the combination of using DRBD and MultiMaster replication, using this set of OCF resource makes that a lot more easy ..

Now all I need to do is find me some time to validate this setup.

Technorati Tags: cluster ha mysql

Share with Shareomatic!

Trackback URL for this post:

http://www.krisbuytaert.be/blog/trackback/1001

Why high-availability is hard with databases

2010-04-26T11:53:15+00:00

A lot of systems are relatively easy to make HA (highly available). You just slap them into a well-known HA framework such as Linux-HA and you’re done. But databases are different, especially replicated databases, especially replicated MySQL.

The reason has to do with some properties that hold for many systems, but not for most databases. Most systems that you want to make HA are relatively lightweight and interchangeable, with little to zero statefulness, easy to start, easy to stop, don’t care a lot about storage (or at least don’t write a lot of data; that’s usually delegated to the database), and there’s little or no harm done if you ruthlessly behead them. The classic example is a web server or even most application servers. Most of the time these things are all about CPU power and network bandwidth. If I were to compare them to a car, I’d say they are like matchbox cars: there are many of them, and they are cheap and easy to replace.

Databases are different. With or without replication, you’re looking at a system that is complex, stateful, heavyweight, and cares a lot about storage. It runs on bigger hardware with fast disks and a lot of memory. It’s usually disk-bound, and it does a lot of writes. It’s hard to start — it takes a long time to warm up and really get ready to serve production workloads (many minutes, hours, or even days). It tends to run with a lot of data in memory in a dirty state, so shutdown is slow, because a clean shutdown requires flushing a bunch of data to disk. If you yank its power plug or kill-dash-nine it, it’ll have to perform recovery on startup, which slows the startup process even more. If I were to compare a database server to a car, I wouldn’t even use a car as the analogy: I’d use one of those big-ass mining trucks. If your mining truck breaks down, you don’t just toss it in the trash and pull another off the shelf.

The problem with a lot of HA solutions is that they want to deal with inconsistencies or irregularities by killing the resource and replacing it in another location. This works fine with web servers, but not with database servers. Doing that will cause serious pain and downtime, defeating the point of HA. And when you add replication into the mix, it gets even worse. A system that wants to manage replication needs to deal with very complex conditions. A lot of replication failures are delicate matters that require skilled human intervention to solve. The HA solution must insulate the application from the misbehaving resource, but leave it running so the human can handle things.

This is not the way most applications are made HA. It’s different with databases, and it’s much harder.

MySQL master/slave support merged into Linux-HA

2010-04-21T05:27:07+00:00

MySQL replication support for the Pacemaker cluster manager (the stuff that we explained in this webinar) has made it into the Linux-HA resource agents default branch. If you are interested in testing — and you should! — please read the extended announcement. Feedback is extremely welcome on the linux-ha-dev mailing list.

We are expecting to release this as part of resource-agents 1.0.4, in late May/early June.