Planet HA

Today I removed Pacemaker from server:ha-clustering on the openSUSE build service.

I lost patience with the service some time ago and the project has been providing pre-built packages from cluster labs ever since (see our install page for more details).

It seems no-one else has had the time or patience to keep the build service updated since my departure so, after noticing their age and the fact that they no longer even build on the majority of targets, I made the decision to remove them.

Hopefully this will help avoid confusing those wanting the latest Pacemaker software.

The latest installment of the Pacemaker 1.0 stable series is now ready for general consumption.

In this release, we’ve made a number improvements to clone handling - particularly the way ordering constraints are processed - as well as some really nice improvements to the shell.

The next 1.0 release is anticipated to be in mid-March. We will be switching to a bi-monthly release schedule to begin focusing on development for the next stable series (more details soon). If you have feature requests, now is the time to voice them and/or provide patches :-)

Pre-built packages for Pacemaker and it’s immediate dependancies are currently building and will be available for openSUSE, SLES, Fedora, RHEL, CentOS from the ClusterLabs Build Area shortly.

Debian users should check for updates Martin’s repo over the coming days and Ubuntu fans can visit LaunchPad for 8.04 and 9.10 packages.

The source tarball is also available directly from Mercurial.

General installation instructions are available at from the ClusterLabs wiki.

Release Statistics

Changes of note since Pacemaker-1.0.6

• High: PE: Bug 2213 - Ensure groups process location constraints so that clone-node-max works for cloned groups
• High: PE: Bug lf#2153 - non-clones should not restart when clones stop/start on other nodes
• High: PE: Bug lf#2209 - Clone ordering should be able to prevent startup of dependant clones
• High: PE: Bug lf#2216 - Correctly identify the state of anonymous clones when deciding when to probe
• High: PE: Bug lf#2225 - Operations that require fencing should wait for ‘stonith_complete’ not ‘all_stopped’.
• High: PE: Bug lf#2225 - Prevent clone peers from stopping while another is instance is (potentially) being fenced
• High: PE: Correctly anti-colocate with a group
• High: PE: Correctly unpack ordering constraints for resource sets to avoid graph loops
• High: Tools: crm: load help from crm_cli.txt
• High: Tools: crm: resource sets (bnc#550923)
• High: Tools: crm: support for comments (LF 2221)
• High: Tools: crm: support for description attribute in resources/operations (bnc#548690)
• High: Tools: hb2openais: add EVMS2 CSM processing (and other changes) (bnc#548093)
• High: Tools: hb2openais: do not allow empty rules, clones, or groups (LF 2215)
• High: Tools: hb2openais: refuse to convert pure EVMS volumes
• High: cib: Ensure the loop for login message terminates
• High: cib: Finally fix reliability of receiving large messages over remote plaintext connections
• High: cib: Fix remote notifications
• High: cib: For remote connections, default to CRM_DAEMON_USER since thats the only one that the cib can validate the password for using PAM
• High: cib: Remote plaintext - Retry sending parts of the message that did not fit the first time
• High: crmd: Ensure batch-limit is correctly enforced
• High: crmd: Ensure we have the latest status after a transition abort
• High (bnc#547579,547582): Tools: crm: status section editing support
• High: shell: Add allow-migrate as allowed meta-attribute (bnc#539968)
• Medium: Build: Do not automatically add -L/lib, it could cause 64-bit arches to break
• Medium: PE: Bug lf#2206 - rsc_order constraints always use score at the top level
• Medium: PE: Only complain about target-role=master for non m/s resources
• Medium: PE: Prevent non-multistate resources from being promoted through target-role
• Medium: PE: Provide a default action for resource-set ordering
• Medium: PE: Silently fix requires=fencing for stonith resources so that it can be set in op_defaults
• Medium: Tools: Bug lf#2286 - Allow the shell to accept template parameters on the command line
• Medium: Tools: Bug lf#2307 - Provide a way to determin the nodeid of past cluster members
• Medium: Tools: crm: add update method to template apply (LF 2289)
• Medium: Tools: crm: direct RA interface for ocf class resource agents (LF 2270)
• Medium: Tools: crm: direct RA interface for stonith class resource agents (LF 2270)
• Medium: Tools: crm: do not add score which does not exist
• Medium: Tools: crm: do not consider warnings as errors (LF 2274)
• Medium: Tools: crm: do not remove sets which contain id-ref attribute (LF 2304)
• Medium: Tools: crm: drop empty attributes elements
• Medium: Tools: crm: exclude locations when testing for pathological constraints (LF 2300)
• Medium: Tools: crm: fix exit code on single shot commands
• Medium: Tools: crm: fix node delete (LF 2305)
• Medium: Tools: crm: implement -F (—force) option
• Medium: Tools: crm: rename status to cibstatus (LF 2236)
• Medium: Tools: crm: revisit configure commit
• Medium: Tools: crm: stay in crm if user specified level only (LF 2286)
• Medium: Tools: crm: verify changes on exit from the configure level
• Medium: ais: Some clients such as gfs_controld want a cluster name, allow one to be specified in corosync.conf
• Medium: cib: Clean up logic for receiving remote messages
• Medium: cib: Create valid notification control messages
• Medium: cib: Indicate where the remote connection came from
• Medium: cib: Send password prompt to stderr so that stdout can be redirected
• Medium: cts: Fix rsh handling when stdout is not required
• Medium: doc: Fill in the section on removing a node from an AIS-based cluster
• Medium: doc: Update the docs to reflect the 0.6/1.0 rolling upgrade problem
• Medium: doc: Use Publican for docbook based documentation
• Medium: fencing: stonithd: add metadata for stonithd instance attributes (and support in the shell)
• Medium: fencing: stonithd: ignore case when comparing host names (LF 2292)
• Medium: tools: Make crm_mon functional with remote connections
• Medium: xml: Add stopped as a supported role for operations
• Medium: xml: Bug bnc#552713 - Treat node unames as text fields not IDs
• Medium: xml: Bug lf#2215 - Create an always-true expression for empty rules when upgrading from 0.6

Ubuntu is looking to switch its supported cluster stack to Corosync+Pacemaker and has put out a “Call for testers”.
Check out the link if this is something you’re interested in.

Unless there are violent objections, I plan to officially stop supporting 0.6 at the end of February.

Since I’ve not seeing any bugs reported for some time, it seems that anyone still using 0.6 is happy with it for their workload.

Also, 1.0 has been out for over a year now and contains significant improvements over 0.6 including

• A unified shell that hides the XML scaffolding
• Migration thresholds that are easy to configure and understand
• Failures can be ignored after a specified period of time
• Ability to specify defaults for resource an operation parameters
• Man pages for all CLI tools
• Up-to-date online documentation

The online documentation has more details on whats new/different in Appendix C and detailed instructions for upgrading in Appendix E.

So you have this libvirt setup and you want to have a dhcp server on the virtual machines you are playing with , or you want to have all static IP's.

Libvirt uses dnsmasq to provide dhcp services etc and when you generate a config from the gui it will look like

<network>
  <name>piponet</name>
  <uuid>e87d3bf1-a2e7-96ca-e131-7ae51ac033f9</uuid>
  <bridge name='virbr2' stp='on' delay='0' />
  <ip address='192.168.100.1' netmask='255.255.255.0'>
    <dhcp>
      <range start='192.168.100.128' end='192.168.100.254' />
    </dhcp>
  </ip>
</network>

<network>
  <name>piponet</name>
  <uuid>e87d3bf1-a2e7-96ca-e131-7ae51ac033f9</uuid>
  <bridge name='virbr2' stp='on' delay='0' />
  <ip address='192.168.100.1' netmask='255.255.255.0'>
     </ip>
</network>

Technorati Tags: dhcp ha kvm libvirt xen

Share with Shareomatic!

/* Style Definitions */ table.MsoNormalTable {mso-style-name:"Table Normal"; mso-tstyle-rowband-size:0; mso-tstyle-colband-size:0; mso-style-noshow:yes; mso-style-priority:99; mso-style-qformat:yes; mso-style-parent:""; mso-padding-alt:0cm 5.4pt 0cm 5.4pt; mso-para-margin:0cm; mso-para-margin-bottom:.0001pt; mso-pagination:widow-orphan; font-size:10.0pt; font-family:"Times New Roman","serif";}

Who never encountered problems with a failing media, never accidently dropped a valuable file, corrupted his FAT(File Allocation Table) or simply got bad disk block sectors ? Of course restoring his backup in such cases would be the easiest way to recover precious data.. Unfortunately it is often in such a situation we saw that backup is a month old or our files have never been backuped. The following lines relate such stories and introduce some ways to recover data with open source softwares.

A nice sunny day

It was a nice sunny Saturday and I was invited at my girlfriend’s mother home for tea. During lunch she spoke about weird behavior with her PC under windows XP, error messages, impossible files to drop from the desktop and a lot of other weird stuff. I was quite proud to offer some help in order to solve those problems.

So I tried to switch on the computer by pressing the power button but the PC shutdown after 5 seconds! Furthermore, the power button remained pushed in each time I pressed on it due to a broken plastic inside the power switch. After 5 minutes I finally succeeded starting the PC and a wonderful Windows XP logo preceding  error messages about missing “.dll” files appeared. Finally after removing 2 out of 3 antivirus/spywares and some unused programs, the computer behavior became smoother, although still very slow so I decided to run a checkdisk on C:\ drive. As you know when one does such an operation on windows, one has to  reboot it, which I did. The checkdisk started, step 1 out of 5, 2 out of 5, when the following message appeared :  “unable to locate the file name attribute of index entry etc”, “not enough space to rebuild index”, etc..

When the chkdsk ended, the computer restarted, the windows xp logo appeared and the computer automatically switched off. Nothing concerning the power switch button this time, it was the disk. No more visible data on it. This nice sunny day became really cloudy when my girlfriend told me that her mother stored a lot of important documents concerning her work and many pictures. Of course she obviously hadn’t any backup as well as the majority of people storing precious data.

I decided to take the hard drive home to plug it in as ”slave” on my own PC. The disk was present but windows indicated 0 disk space available and 0 disk space used. Disappointed, I decided to contact a company specialized in data recovery to ask for a quote and received it a few days later with the amount of 1684 Euro for a complete data recovery without guarantee that the integrity of the data would be recovered. It indicated : “The reasons for the defective data are major structure damages.”

It was simply too expensive to recover maybe only unusable blocks on a drive.

Let’s save those data with open source softwares

I decided to look for softwares on the internet in order to resolve this problem. I was amazed to discover so many recovery tools e.g.: active partition recovery, ontrack easy recovery, active uneraser, lost & found, prosoft media tools, active undelete and many others.

Attempting to repair the filesystem directly on the defective disk would generate unnecessary disk activity and carries the risk of further damage to the drive. Therefore the first step in such a situation is to copy the hard drive onto another one.

I firstly thought about dd for this task and noticed a lot of better tools had been developed to copy data from one disk to another, like ddrescue, dd_rescue and dd_rhelp (wrapper script for dd_rescue). My choice went on ddrescue because it combines advantages from dd_rescue and dd_rhelp. Unlike dd, dd_rescue will not stop when it encounters errors. It is especially useful when you work with failing media. In addition it allows copying blocks backward, so if we have an error in the middle of a block, ddrescue will copy both data before and after the error inside this block.

Ubuntu distrib has been used to proceed to this recovery.Notice that it’s not needed to have any Linux installation, knoppix allows to boot on a live CD and all needed software to copy and recover data are available on it.

In the following case the partition /dev/sdc2 need to be duplicated. In order to proceed to the partition copy it’s mandatory to create a new partition of the same size as the defective one on a backup disk.  Fdisk will perfectly fullfill this mission.

The partition size can be calculated by substracting the end column(last cylinders) to the start column(first cylinder).  In the case below 10240-833 = 9507 cylinders.

Creation of a new partition - /dev/sdd1 - onto the new media - /dev/sdd - with fdisk is a straightforward process.

Then ddrescue can duplicate blocks on the second media. As shown bellow the average rate for duplicate is about 1Mb/s on a small configuration, meaning that about 28 hours are needed to copy a 100Gb partition and 12 days for 1Tb.

Once the failing media partition is fully copied, data recovery can start. One more time a variety of useful tools can be found on internet. The one used in this example is TestDisk.  TestDisk is a free open source software originaly created to recover lost partition or making non bootable disk bootable again.

TestDisk has a large set of functionalities such as : undeleting files from FAT,NTFS and ext2 filesystem, recovering/rebuilding NTFS boot sector, fixing FAT tables, copying files from deleted FAT, NTFS, ext2/ext3 partitions and many others. In addition TestDisk can be run on many operating systems such as Windows, Linux, macOS, SunOS, BSD.

Having a short look on TestDisk will present some interesting functionalities

After having selected the disk to analyze, TestDisk will ask you for the partition table type. In the current case “Intel” is used. Notice that it is also possible to use Mac, Sun or even Xbox partition.

After analyzing disk partitions, TestDisk shows the available partitions and also offers functionalities such as rebuilding boot sector.

As expected TestDisk recovered the backup boot sector

Sometimes the MFT (Master File Table) can be also corrupted. Microsoft Check Disk (chkdsk) can failed trying to repair the MFT. TestDisk offers the possibility to repair this MFT through the advanced menu after having selected the NTFS partition has shown above „Repair MFT“.

TestDisk provides plenty of other functionalities, adding a partition, changing is type or listing files inside a partition and copy those files to another location, etc…

Example below demonstrates the possibility to copy files from defected media to another place.

Normal 0 false false false EN-US X-NONE X-NONE MicrosoftInternetExplorer4 /* Style Definitions */ table.MsoNormalTable {mso-style-name:"Table Normal"; mso-tstyle-rowband-size:0; mso-tstyle-colband-size:0; mso-style-noshow:yes; mso-style-priority:99; mso-style-qformat:yes; mso-style-parent:""; mso-padding-alt:0cm 5.4pt 0cm 5.4pt; mso-para-margin:0cm; mso-para-margin-bottom:.0001pt; mso-pagination:widow-orphan; font-size:10.0pt; font-family:"Times New Roman","serif";}

But what about ext3 filesystem ?

Among the TestDisk limitations there is the impossibility to recover deleted files which stand on an ext3 partition.

Having a look on forum about ext3 file recovery will often discourage you. If you are not convice that file recovery on ext3 is not possible having a look on ext3 FAQ (http://batleth.sapienti-sat.org/projects/FAQs/ext3-faq.html ) will definitively remove any hope.

Hopefully it looks that this statement is too categorical. When a file is removed, data are not really overwritten. On ext3 filesystem the pointer that reference a file is simply removed meaning that the disk area can be overwritten if writes operation occur. Therefore the first thing to do after such a mistake is avoiding any additional write operation. The best way to achieve that is simply unmouting the filesystem.

Once the filesystem unmounted one can take time looking for recovery tools. I found two tools able to recover deleted files.  

• Foremost  is a Linux based program data for recovering deleted files and was origionally developed for the US Air Force Office of Special Investigations by Jesse Kornblum and Kris Kendall.

• ext3grep  is a simple tool developed by Carlo Wood and intended to aid anyone who accidentally deletes a file on an ext3 filesystem.

Both of them are intended to be run on disk images, meaning that it is mandatory to create a disk image of the partition where the removed files stand.

Normal 0 false false false EN-US X-NONE X-NONE MicrosoftInternetExplorer4 /* Style Definitions */ table.MsoNormalTable {mso-style-name:"Table Normal"; mso-tstyle-rowband-size:0; mso-tstyle-colband-size:0; mso-style-noshow:yes; mso-style-priority:99; mso-style-qformat:yes; mso-style-parent:""; mso-padding-alt:0cm 5.4pt 0cm 5.4pt; mso-para-margin:0cm; mso-para-margin-bottom:.0001pt; mso-pagination:widow-orphan; font-size:10.0pt; font-family:"Times New Roman","serif";}

Some doubts ?

A concrete exemple needing file restoration on ext3 partition could be removing all *.log files on a partition hosting redo log files from an  Oracle Database. Redo logs that would be named with *.log extension (which is by the way strongly not recommended especially for this reason) would be removed after such an operation. As you maybe know an Oracle Database cannot work without at least two redo log groups therefore such a delete would lead to a database crash.

The storage setup in the following example is composed by two groups of raid 5 with three disks each configured by mdadm. On the top a raid 0 (stripping) has been configured with LVM2. This configuration – RAID 50 - is illustrated in the figure below.

Normal 0 false false false EN-US X-NONE X-NONE MicrosoftInternetExplorer4 /* Style Definitions */ table.MsoNormalTable {mso-style-name:"Table Normal"; mso-tstyle-rowband-size:0; mso-tstyle-colband-size:0; mso-style-noshow:yes; mso-style-priority:99; mso-style-qformat:yes; mso-style-parent:""; mso-padding-alt:0cm 5.4pt 0cm 5.4pt; mso-para-margin:0cm; mso-para-margin-bottom:.0001pt; mso-pagination:widow-orphan; font-size:10.0pt; font-family:"Times New Roman","serif";}

The Oracle database version is 10.2.0.4 and the filesystem_io_option is set to “setall“.

After removing redo logs of the SOUK database, the following can be seen in the Oracle alert log.

In order to restore redo log files, the first step is to avoid any additional write on the filesystem. The most writes operations occur, the less chance we have to recover those files. That why it is needed to stop any processes that write on this specific filesystem.

The filesystem needs to be unmounted to avoid any write access.

Then an image copy of the filesystem can be done with “dd” command as shown bellow. “dd” perfectly fits our need in such a case.

Now we simply have to execute ext3grep with the filesystem image in parameter. Several options are provided allowing restoring from a specific date or a specific file or simply everything.

Restored files are copied in the “RESTORED_FILES” directory of the current path.  Now we only have to copy them in the correct directory.

Once redo logs restored and copied into the original path, database start can be done. However keep in consideration that committed transactions could be lost depending on your database and filesystem configuration!

 As we can see, dropping redo log files or any other file do not necessarily lead to definitely loosing transactions. Ext3grep can be used as an additional way to recover your database.

v\:* {behavior:url(#default#VML);} o\:* {behavior:url(#default#VML);} w\:* {behavior:url(#default#VML);} .shape {behavior:url(#default#VML);} Normal 0 false false false false EN-US X-NONE X-NONE MicrosoftInternetExplorer4 /* Style Definitions */ table.MsoNormalTable {mso-style-name:"Table Normal"; mso-tstyle-rowband-size:0; mso-tstyle-colband-size:0; mso-style-noshow:yes; mso-style-priority:99; mso-style-qformat:yes; mso-style-parent:""; mso-padding-alt:0cm 5.4pt 0cm 5.4pt; mso-para-margin:0cm; mso-para-margin-bottom:.0001pt; mso-pagination:widow-orphan; font-size:10.0pt; font-family:"Times New Roman","serif";}

Conclusion

Several situations can lead to media recovery, site disaster, block corruption, FAT corrupted, etc… In case of hardware errors the most secure way to solve it is often calling media recovery companies. Dealing with physical damages to a hard drive with such tools can destroy any last hope of a successful recovery. However if no physical damages are confirmed these tools can recover precious data.

The first thing to do when media error occurs is stopping immediately any activity on this media. It is only after stopping any activity on the media that we can take time to think about the way to proceed. As shown in this article the second step is generally copying the failing media on a trusted support. Working directly on the failing media could lead to definitively loosing valuable data. That’s why it is strongly recommended duplicating data before any other operation. Several tools provide duplication functionalities such as dd_rescue.

Once the failing media duplicated recovery can be done using the backup media. One more time a variety of tools can be used depending on the filesystem and data to recover.

Finally once data recovered it may worth doing a backup. Although those tools can get out of hopeless situations and add a way to recover files to usual methods, testing backup and restore processes at regular intervals is maybe the best way to avoid spending time in such recovery processes.

Gregory Steulet
Oracle Certified Professional 10G
MySQL Cluster 5.1 Certified
Avaloq Certified Professional

Trivadis SA
Rue Marterey 5
CH-1005 Lausanne
Tel: +41-21-321 47 00
Fax: +41-21-321 47 01
Internet:  www.trivadis.com
Mail: info@trivadis.com

Literature and Links…

http://www.cgsecurity.org/wiki/TestDisk

http://www.gnu.org/software/ddrescue/ddrescue.html

http://www.knoppix.org/

http://www.xs4all.nl/~carlo17/howto/undelete_ext3.html

http://foremost.sourceforge.net/

http://www.hiren.info/

http://www.krollontrack.com/

http://www.giis.co.in/

by @botchagalupe
on Virtualization, Open Source tools and DNS Problems

Technorati Tags: dnsproblem drupal ha heartbeat linux-ha mysql pacemaker puppet virtualization xen

Share with Shareomatic!

I’m pleased to report that the core Pacemaker documentation is now available in PDF, HTML (chunked and single page) and even TXT formats.

The old Pages.app sources have been replaced with DocBook which allows them to be:

• published in a variety of formats
• kept under version control
• included in the packages
• updated by anybody

Additionally, we’re using Publican to produce the final result so supporting multiple languages should be now possible. Let us know if you’re interested in doing some translation :-)

The primary location for Pacemaker documentation will remain http://www.clusterlabs.org/wiki/Documentation however there is also a index of the generated documentation at http://www.clusterlabs.org/doc/ which includes the date and version from which it was generated.

In the last 2 blog posts about High Availability for MySQL we have introduced definitions and provided a list of ( questions that you need to ask yourself before choosing a HA solution. In this new post, we will cover what is the most popular HA solution for MySQL, replication.

High Availability solution for MySQL: Replication

This HA solution is the easiest to implement and to manage. You basically need to setup MySQL replication between a master and one or more slaves. Upon failure of the master, one of the slaves is manually promoted to the master role and replication on the other slaves is re-adjusted to point to the new master. This solution works well with all the MySQL storage engines including MyISAM (NDB is a special discussed later) but it suffers from the limitation of MySQL replication. The main limitation, in term of HA, is the asynchronous design of MySQL replication which does not allow the master to be sure the slave has been updated before returning after a commit statement. There is a window in time where it is possible that a fully committed transaction has not been pushed to the slave(s) leading to data loss. Many large websites that are fine with some data loss rely on replication for HA and for read scaling.

In addition to hardware failure, the level of availability of this solution is affected by the availability of the MySQL replication link between the servers. Replication often break for various reasons and while replication is broken, there is no High-Availability. Also, the availability of this solution is affected by how much the slaves were behind the master when the outage occurred. So, if you want to have a good level of availability, you need a good monitoring and alerting system to quickly react to replication issue and you need a rather small write load so that the slaves do not lag behind the master too much. To maximize the level of availability, recovery should be automatic.

Apart of its simplicity, an HA solution based on replication as many interesting properties, no wonder it is so popular. First, if the application is well designed and has specific database handles for read and write operations, this HA solution can scales the read operations to a high level. Using the slaves for reads cause a second interesting side effect, the caches of the slaves are hot so failing over to a slave means no degraded performance associated with caches warm up. Finally, it is well known that with MySQL, altering a table means recreating the whole table and it is a blocking operations. Altering a large table may takes many hours. The trick here is to run the alter table on a slave and then, once done, we let the slave catch up with the master using the new table schema, we failover to the slave and repeat the alter table on the other server. Those online schema change are easier when a master to master topology is used.

The following figure summarize the simplest HA architecture using MySQL replication. All writes are going to the master while reads are spread between the master and the slave. Upon failure of the master, replication is stopped on the slave and all traffic is redirected to the slave which now handles reads and writes.

Automatic failover with replication

I already mentioned that for best HA levels, failover or recovery should be automatic. There are tools to manage automatic failover with replication like MMM, Flipper and Tungsten. Here, I will quickly describe the most popular one, MMM.

With MMM, you need to add a separate server, the Manager that, like the name imply, manages the availability of the MySQL service. A high availability solution based on MMM requires at the 2 MySQL servers configured in a Master to Master topology. Additional slaves can also be added. A MMM agent runs on all the MySQL servers and it is used to do OS level operations. The principle of operation of MMM is based on VIPs. There is one write VIP, where write operations are sent and as many read VIPs as the number of MySQL servers. For the write VIP, MMM monitors the state of the current master and, upon failure, try to kill all the connections to the failing server and transfer the write VIP to the other master. For the read VIPs, MMM monitors the state of the slaves and remove the read VIP of a slave if it has failed or is lagging behind the master by more than a defined threshold. One of the main limitation of MMM is its lack of fencing capability. It is important to stop all the connections to the failing master and if that server is not responding, maybe because of a network problem, a stonith device must be used to fence it. I am far from being an expert with MMM, other guys on my team are way better than me, but I heard that the MMM v1 code base had some deficiencies. MMM v2 is a complete rewrite that addresses some of the shortcomings of v1. Walter Heck from OpenQuery gave an excellent webinar on it recently.

The architecture of a highly available setup using MMM and Master-Master replication is presented on the figure below. Apart from the minimum requirement of two MySQL servers replicating each other, there is a third server, called the manager, that controls both MySQL server through an agent that is running on each server. The manager controls and monitors the state of the replication and assign virtual IPs for specific roles. There are one VIP where write operations are sent and two or more VIPs where read operations are sent. If replication on one of the MySQL servers lags behind too much, its read VIP will be moved to another server.

As a conclusion, replication can be used in many cases to build effective and scalable highly available solutions but it has some limitations. In my next blog post, I’ll present another HA solution build around Heartbeat and DRBD.

Entry posted by yves | 5 comments

Add to: | | | |

While browsing trough my enormous mailinglist backlog I ran into the following message from Gianluca Cecchi on the DRBD-user mailing list

guess I`ll have to give Lars a T-Shirt when we next meet ;)

From: Gianluca Cecchi
To: drbd-user@lists.linbit.com
Subject: [DRBD-user] notes on 8.3.2
 
 
- drbdadm create-md r0 segfaults when the command "hostname" on the
server contains the fully qualified domain name but you have put only
the hostname part in drbd.conf
Instead, the command "drbdadm dump" correctly gives you a warning in
this case (suggesting how to correct the error you made....):
 
suppose complete hostname is virtfed.domainname.com and you put
virtfed alone in drbd.conf
[root@virtfed ~]# drbdadm dump
WARN: no normal resources defined for this host (virtfed.domainname.com)!?
 
while
[root@virtfed ~]# drbdadm create-md r0
Segmentation fault

Guess I`ll have to give the Linbit crowd a T-Shirt when we next meet ;)

Technorati Tags: dnsproblem drbd ha linux-cluster

Share with Shareomatic!

The next installment of the Pacemaker 1.0 stable series is now ready for general consumption.

In addition to further polishing of the crm shell and CLI tools, this is the first release to support CoroSync (version 1.1.2 or greater is required).

The ”Pacemaker Explained” reference has also been converted to docbook and is included as part of the tarball (and pre-built packages if the relevant stylesheets are present at build time).

Pre-built packages for Pacemaker and it’s immediate dependancies will be available for openSUSE, SLES, Fedora, RHEL, CentOS from the OpenSUSE Build Service in the next couple of days depending in how overloaded it is.

Debian users should check for updates Martin’s repo over the coming days and Ubuntu fans can visit LaunchPad for 8.04 and 9.10 packages.

The source tarball is also available directly from Mercurial.

General installation instructions are available at from the ClusterLabs wiki.

Release Statistics

Project Administrivia

We may switch to a bi-monthly release cycle. If you have any thoughts on this (for or against), please get in touch.

Changes of note since Pacemaker-1.0.5

• High: cib: Correctly clean up when both plaintext and tls remote ports are requested
• High: ais: Avoid excessive load by checking for dead children every 1s (instead of 100ms)
• High: ais: Bug lf#2199 - Prevent expected-quorum-votes from being populated with garbage
• High: ais: Bug rh#525589 - Prevent shutdown deadlocks when running on CoroSync
• High: ais: Gracefully handle changes to the AIS nodeid
• High: ais: Prevent deadlock - dont try to release IPC message if the connection failed
• High: ais: Ubuntu needs a leading zero for directory modes
• High: cib: For validation errors, send back the full CIB so the client can display the errors
• High: cib: Prevent use-after-free for remote plaintext connections
• High: cib: Repair the ability to connect to the cluster from non-cluster machines
• High: Core: Bug lf#2169 - Allow dtd/schema validation to be disabled
• High: crmd: Bug bnc#527530 - Wait for the transition to complete before leaving S_TRANSITION_ENGINE
• High: crmd: Bug lf#2201 - Guard against possible cause of a segfault
• High: crmd: Prevent use-after-free with LOG_DEBUG_3
• High: Extras: Add sctp support to the controld RA
• High: PE: Bug bnc#515172 - Provide better defaults for lt(e) and gt(e) comparisions
• High: PE: Bug lf#2106 - Not all anonymous clone children are restarted after configuration change
• High: PE: Bug lf#2170 - stop-all-resources option had no effect
• High: PE: Bug lf#2171 - Prevent groups from starting if they depend on a complex resource which can’t
• High: PE: Bug lf#2197 - Allow master instances placemaker to be influenced by colocation constraints
• High: PE: Disable resource management if stonith-enabled=true and no stonith resources are defined
• High: PE: Don’t include master score if it would prevent allocation
• High: PE: Make sure promote/demote pseudo actions are created correctly
• High: PE: Prevent target-role from promoting more than master-max instances
• High: shell: Add allow-migrate as allowed meta-attribute (bnc#539968)
• High: tools: bnc#547579,547582 - crm: status section editing support
• High: Tools: crm: add semantic checks depending on the meta-data from resource agents
• High: Tools: crm: improve processing of group edit and constraints
• High: Tools: crm: improve the edit command
• High: Tools: pingd - Fix a number of critical bugs (patch via Kazunori INOUE)
• Med: xml: Mask the “symmetrical” attribute on rsc_colocation constraints (bnc#540672)
• Medium (bnc#520707): Tools: crm: new templates ocfs2 and clvm
• Medium (LF 2164): Tools: hb_report: expand the crm status command
• Medium (LF 2184): Tools: crm: extend ptest command
• Medium (LF 2185): Tools: crm: add resource promote/demote commands
• Medium (LF 2198): Tools: crm: add node fence command
• Medium: ais: Attempt to enable core file generation if it was disabled
• Medium: ais: Include version details in plugin name
• Medium: Build: Re-enable asciidoc documentation
• Medium: Build: Shell templates arent documentation
• Medium: cib: Remove delay for remote plaintext connections
• Medium: Core: Disable syslog for any process that doesn’t want its arguments logged
• Medium: crmd: Requery the resource metadata after every start operation
• Medium: cts: add —benchmark for scalability tests
• Medium: cts: Prepare for corosync testing
• Medium: Extra: Include SNMP MIB file for crm_mon (from Michael Schwartzkopff)
• Medium: PE: Bug lf#2178 - Indicate unmanaged clones
• Medium: PE: Bug lf#2180 - Include node information for all failed ops
• Medium: PE: Bug lf#2189 - Incorrect error message when unpacking simple ordering constraint
• Medium: PE: Correctly log resources that would like to start but can’t
• Medium: PE: Correctly log the state of orphaned clone instances
• Medium: PE: If no migrate_(from|to) action is defined, look for migrate instead
• Medium: PE: Only re-instate target-role if it is less than the calculated one
• Medium: PE: Provide details for the maintenance-mode option
• Medium: PE: Stop ptest from logging to syslog
• Medium: Tools: attrd_updater - Suppress all logging with —quiet
• Medium: Tools: crm: add extra flag to CibObject for invalid objects
• Medium: Tools: crm: do return cached resources dom node
• Medium: Tools: crm: expand template documentation
• Medium: Tools: crm: first child of a removed parent inherits constraints
• Medium: Tools: crm_attribute - Suppress all logging with —quiet
• Medium: Tools: crm_shadow - log diffs to stdout instead of stderr
• Medium: Tools: Use -q as the short form for —quiet (for consistency)

Again a tip on how to write your OpenAIS/Pacemaker configuration in a simpler fashion; this applies to SUSE Linux Enterprise 11 High-Availability Extension too, of course.

For the full cluster functionality with OpenAIS/OCFS2/cLVM2 and an OCFS2 mount on top, you need to configure DLM, O2CB, cLVM2 clones, one to start the LVM2 volume group, and Filesystem resources to mount the file system. Add in all the dependencies needed, and you end up with a configuration pretty much like this (shown in CRM shell syntax, which is already much more concise than the raw XML):

primitive clvm ocf:lvm2:clvmd
primitive dlm ocf:pacemaker:controld
primitive o2cb ocf:ocfs2:o2cb
primitive ocfs2-2 ocf:heartbeat:Filesystem \
        params device="/dev/cluster-vg/ocfs2"
directory="/ocfs2-2" fstype="ocfs2"
primitive vg1 ocf:heartbeat:LVM \
        params volgrpname="cluster-vg"
clone c-ocfs2-2 ocfs2-2 \
        meta target-role="Started" interleave="true"
clone clvm-clone clvm \
        meta target-role="Started" interleave="true"
ordered="true"
clone dlm-clone dlm \
        meta interleave="true" ordered="true"
target-role="Stopped"
clone o2cb-clone o2cb \
        meta target-role="Started" interleave="true"
ordered="true"
clone vg1-clone vg1 \
        meta target-role="Started" interleave="true"
ordered="true"
colocation colo-clvm inf: clvm-clone dlm-clone
colocation colo-o2cb inf: o2cb-clone dlm-clone
colocation colo-ocfs2-2 inf: c-ocfs2-2 o2cb-clone
colocation colo-ocfs2-2-vg1 inf: c-ocfs2-2 vg1-clone
colocation colo-vg1 inf: vg1-clone clvm-clone
order order-clvm inf: dlm-clone clvm-clone
order order-o2cb inf: dlm-clone o2cb-clone
order order-ocfs2-2 inf: o2cb-clone c-ocfs2-2
order order-ocfs2-2-vg1 inf: vg1-clone c-ocfs2-2
order order-vg1 inf: clvm-clone vg1-clone

However, there is a little known feature - you can actually clone a resource group:

primitive clvm ocf:lvm2:clvmd
primitive dlm ocf:pacemaker:controld
primitive o2cb ocf:ocfs2:o2cb
primitive ocfs2-2 ocf:heartbeat:Filesystem \
        params device="/dev/cluster-vg/ocfs2"
directory="/ocfs2-2" fstype="ocfs2"
primitive vg1 ocf:heartbeat:LVM \
        params volgrpname="cluster-vg"
group base-group dlm o2cb clvm vg1 ocfs2-2
clone base-clone base-group \
	meta interleave="true"

I think this speaks for itself; 20 lines of configuration reduced. You will also find that crm_mon output is much simpler and shorter, allowing you to see more of the cluster status in one go.

First time I heard about Galera on Percona Performance Conference 2009, Seppo Jaakola was presenting “Galera: Multi-Master Synchronous MySQL Replication Clusters”. It was impressed as I personally always wanted it for InnoDB, but we had it in plans at the bottom of the list, as this is very hard to implement properly.
The idea by itself is not new, I remember synchronous replication was announced for SolidDB on MySQL UC 2007, but later the product was killed by IBM.

So long time after PPC 2009 there was available version mysql-galera-0.6, which had serious flow, to setup a new node you had to take down whole cluster. And all this time Codership ( company that develops Galera) was working on 0.7 release that introduces node propagation keeping cluster online. You can play with 0.7pre release by yourself MySQL/Galera Release 0.7pre.

In current version propagation is done by mysqldump from one of nodes (”donor”). In next release Codership is going to support LVM snapshot and xtrabackup which will make the setup of new node even easier. The current annoyance I see is that if you shutdown one node for short period of time for quick maintenance, after start, the node has to load whole mysqldump, like it is new empty node. I hope Codership guys will address this also.
Another thing I miss for now is support of InnoDB-plugin, which as we know performs much better than standard InnoDB ®.

So what is so interesting about Galera. Couple things:

- High Availability. Any of N standby nodes are available immediately when main node fails. Galera is serious pretender to be included to the list, Yves put recently, http://www.mysqlperformanceblog.com/2009/10/16/finding-your-mysql-high-availability-solution-%e2%80%93-the-questions/. I am not sure how many nines it will provide , but efforts on test setup and deployment should be comparable with MMM setup.

- Scale Writes. Galera allows to write to any of N nodes and automatically propagate to other nodes. It sounds too ideal, and there is drawback – with increasing amount of nodes you write to, your transaction rollback rate may increase, especially if you working on the same dataset. You can find some results on Codership’s page, and I am going to run my own benchmarks also. Also from benchmark you can see that communication overhead maybe significant for short writes.

- Scale Reads. It can be done with regular replication, but with synchronous your “slaves-nodes” are in the same state, there is no “slave behind”. When you read from any slave, you read actual data. Although it also has serious drawback – our cluster is fast as fast the “weakest” node in the chain. So if one node gets overloaded and performance degrades, the same happens with whole cluster.

- Heterogeneous-database replication. It is not here yet, and I do not know what’s in Codership roadmap, but group manager protocol in Galera is database independent, and it’s only matter of database drivers. For InnoDB currently it is set of patches, and I see it is quite possible to make the same for Postgres. So MySQL-Postgres cluster setup is not so far ahead

On “Company page” Codership says their goal is “to promote and exploit the latest developments in computer science to produce fast and scalable synchronous replication solution that “just works” for databases and similar applications”, which I think they have success in. Implementing fast, scalable and working group communication and transaction manager is the art.

As for now I would not put 0.7 release into production yet, but you may seriously consider to play with it in test environment, and report bugs to Codership team, they are very responsive.
I am waiting for next releases and looking to make integration with XtraDB.

Entry posted by Vadim | 11 comments

Add to: | | | |

Funny how different experiences lead to different evaluations of tools. The MySQL HA solutions the MySQL Performanceblog list, are almost listed in the complete opposited order of what my impressions are.

Ok agreed, I should probably not put my MySQL NDB experiences from 2-3 years ago with multiple Query of deaths and more problems than you into account anymore , but back then went in the list Less stable than a single node. I've had NDB POC setups going down for much more than 05:16 minutes
Ndb comes with a lot of restrictions, there are

As for MySQL on DRBD, I've said this before , I love DRBD, but having to wait for a long InnoDB recovery after a failover just kills your uptime ,
I remember being called by a customer during Fred last holiday who was waiting over 20 minutes for recovery , twice, so putting the DRBD/San setup second would not be my preference. But agreed .. it's only listed at 99.9% meaning almost 9 hours of downtime per year are allowed.

On the other hand we've seen database uptime of MySQL MultiMaster setups with Heartbeat reaching better figures than 99.99% Heck I've seen single nodes achieve better than 99.99% :)

So what does this teach us ... there is no golden rule for HA, lots of situations are different, it's the preferences of the customer, the size of the database, the kind of application , and much
more .. you always need to think and evaluate the environment ...

Technorati Tags: cluster drbd ha linux-ha mysql nines

Share with Shareomatic!

After having reviewed the definition my the previous post (The definitions), the next step is to respond to some questions.

Do you need MySQL High-Availability?

That question is quite obvious but some times, it is skipped. It can also be formulated “What is the downtime cost of the service?”. In the cost, you need to include lost revenue from the service and you also need to consider less direct impact like loss of corporate image and other marketing costs. If your downtime cost is under $10/h, you can stop reading this document, you don’t need HA. For the others, let’s move on!

How to determine which MySQL High-Availability solution is best?

What is really tricky with MySQL is the number of possible HA solutions. From the simplest the most complex let’s list the most common ones:

- MySQL replication with manual failover
- Master-Master with MMM manager
- Heartbeat/SAN
- Heartbeat/DRBD
- NDB Cluster

These technologies are by far, not a one size fits all and many deployments use combination of solutions. I will not cover ScaleDB and Continuent because I know almost nothing of these solutions. There are many more questions you need to ask yourself before being able to pick the right one. Below, I listed the most common questions, I might have missed some.

1. What level of HA do you need?

Since all the technologies do not offer the same level of availability, this is a first important sorting factor. Here are estimates of the level of availability offered by the various solutions.

From the table, if your requirements are for 99.99%, you are restricted to NDB Cluster while if it is only 99% you have more options. I recall that the level of availability is hard to estimate and subject to debate. These are the usually accepted level of availability for these technologies.

2. Can you afford to lose data?

Obviously, if you are concerned about loss of data, you are most likely using the InnoDB storage engine, since MyISAM is not transactional and do not sync data to disk. Similarly, MySQL replication is an asynchronous process and although it is fairly fast at transferring data between the master and the slaves, there is a window of time where data loss is possible.

If you can afford to lose some data, you can consider “MySQL replication” and “Master-Master with MMM manager” otherwise, you can only consider the other three solutions.

3. Does your application use MyISAM only features?

There are some features like Full text indexes and GIS indexes that are supported only by MyISAM. The HA solutions that work well with MyISAM are “MySQL replication” and “Master-Master with MMM manager”. Depending on the application, the MyISAM Full text indexes might be replaced by another search engine like Sphinx in order to remove the restriction. There is no HA solution other than the ones based on replication that handles GIS indexes.

4. What is the write load?

The HA solutions we present are not equal in term of their write capacity. Due to the way replication is implemented, only one thread on the slave can handle the write operations. If the replication master is multi-cores servers and is heavily writing using multiple threads, the slaves will likely not be able to keep up. Replication is not the only technology that put a strain on the write capacity, DRBD, a shared storage emulator for Linux, also reduce by about 30% (very dependent on hardware) the write capacity of a database server. In term of write capacity here are you choices.

5. For what level of growth are you planning?

Since NDB Cluster is an integrated sharding environment, if you are planning for a growth that will need sharding (splitting the database over multiple servers), then you might need to take a serious at that solution. If not, then, apart from the write capacity, all the solutions are approximately equal.

6. How qualified is your staff or support company?

There is a quite direct relationship between the level of availability and the complexity of the solution. In order to reach the promised level of availability, the staff maintaining the HA setup, either internal or external, must have the required level of expertise. The required expertise level is summarized in the table below.

7. How deep are your pocket?

The last aspect that needs to be considered is the budget, complexity is expensive. We will consider two types of setup. The first one is a basic proof of concept of the technology with the hardware tested, the data imported and basic testing and documentation. A proof of concept setup is a good way to get used to a technology and experiment with it in a test environment. The other type of setup we will consider is a full production setup that includes extensive testing, fire drills, full documentation, monitoring, alerting, backups, migration to production and post migration monitoring. Of course, it is the safest way to migrate an HA solution to production. All the times here are estimates based on field experience, the values presented here are fairly typical and contains some buffers for unexpected problems. Although an HA solution can be built remotely through a KVM over IP and adequate remote power management, an on site intervention with physical access to the servers is the preferred way, especially for the most complex solutions.

Editor’s Note: We’ve gotten many questions about the time estimates mentioned here. The above estimates shouldn’t be used to compare against any specific situation. Time will vary greatly depending on your project. For example, “setting up replication” can be as simple as CHANGE MASTER TO, and can take as little as a few minutes in some circumstances. Yves’s estimate is for a project to create a replication slave for HA purposes, not for “setting up replication.” There is a big difference between an HA project and a DBA task. – Baron Schwartz

Entry posted by yves | 32 comments

Add to: | | | |

While upgrading a pretty recent Heartbeat cluster to OpenAis earlier today I ran into the following weird situation

Last updated: Fri Oct 16 08:50:03 2009
Stack: openais
Current DC: CO_NMS-1 - partition with quorum
Version: 1.0.5-462f1569a43740667daf7b0f6b521742e9eb8fa7
4 Nodes configured, 2 expected votes
1 Resources configured.
============
 
Online: [ CO_NMS-1 CO_NMS-2 ]
OFFLINE: [ co_nms-1 co_nms-2 ]

or

crm(live)node# show
co_nms-1(5c48ab4f-767f-e2dc-20ec-5969cddad152): normal
co_nms-2(922ff786-eca9-bed0-d79d-8222727a2c5b): normal
CO_NMS-1: normal
CO_NMS-2: normal

Whohoo.. OpenAIS must have realized I have upperase and lowercase cores :)

Funny to see .. but quickly solved..

Technorati Tags: ha heartbeat high availability openais

Share with Shareomatic!

Ronald Bradford wants to know what kind of Monitoring you use..
He specifically wants to know about Alerting tools

There's different cases , looking at it from a full infrastructure point my current favourite is Zabbix or good old Nagios,

But when looking at it from a debugging perspective you have MySQLAR or Hyperic, but those aren't in the alerting list.

However, when you are building HA clusters, you have custom scripts running either from mon or from pacemaker ..

Still .. Ronald probably wants more input :)

Technorati Tags: ha monitoring msql zabbix zenos

Share with Shareomatic!

As my first contribution to the MySQL Performance Blog, I joined Percona at the beginning September, I chose to cover the various high-availability (HA) options available for MySQL.  I have done dozen of MySQL HA related engagements while working for Sun/MySQL over the last couple of years using Heartbeat, DRBD and NDB cluster and I’ll probably be doing the same at Percona.  I have built my first DRBD based HA solution nearly 10 years ago.

There is quite a lot of confusion surrounding HA solutions for MySQL, I will try to present them objectively, my goal here been not to sell any specific technology but to help people choose the right one for their needs.  This post is first of a series,  I don’t yet know how many I will write in the series.

Before we start, it must be stated that high-availability is not only a matter of technical solutions, good management practices covering monitoring, alerting, security and documentation are also needed to insure a successful solution. In other words, no solution is fool proof, if a high-availability solution is running in recovery mode for months without nobody caring about it, the risk of a complete failure is much higher.

In order to all be on the same page, I will first give some definitions of the key terms.  I don’t pretend those definitions are perfect but let’s build on them.

High-Availability

Let’s first define what is meant by high-availability.  The most general definition would be that a high-availability setup is special  computer architecture designed to improve the availability of a computer service, like a MySQL database.  High-availability, HA for short, introduces a wealth of peculiar concepts, we will first review the main ones.

Uptime/Downtime

Uptime means the service is available even if degraded as long as it is above some defined performance threshold. Downtime means the opposite, either the service is completely down or unresponsive according to the defined performance threshold.  In many cases, people don’t define a performance threshold, it is basically the service monitoring frequency and timeout that fix it.

Level of Availability

The level of availability is basically the guaranteed percentage of uptime you will get over a year.  It has always been a subject of debate and it is something hard to evaluate since, most of the time, the samples are small and all the conditions of the deployments are not easily controlled. See the level of availability as the availability you, as the operator of the service, can promise in case of a worse case scenario. For example, 98% availability means a downtime of a little more than 7 days per year.  The cost is approximately an exponential function of the level of availability and has to be compared with the downtime cost. If an HA setup with a level of availability of 99% is fairly simple and affordable, moving to 99.9% and 99.99% can be much more expensive and complex. Also, you need to consider the environmental factors.  If your ISP cannot guarantee you a level of availability of more then 99.9% for the Internet access, it is useless to go beyond that no matter the importance of the application.

Single point of failure (SPOF)

Single points of failure are the things you are looking to remove when you build an HA solution.  Basically, they are the devices/things that if they are not available, the service is down.  A data center can be considered a SPOF at a high enough level of availability.  Usually the more SPOFs you consider, the higher the availability of your solution and the higher its cost.

Recovery/failover

Recovery (or failover) is the process by how a HA setup recovers from a failure. During the recovery time, the service is down.  With the most simple solutions, it can be a manual process but most of the time, it is automatic.  Also, there is a time associated with the recovery.  If a failure happened during the night and the operator is only available from 8am to 5pm then, you might have a recovery time of more than 12 hours.  The more complex solutions have automatic recovery and do not need human intervention.  Once again, although they are some exceptions, faster and automatic recovery usually means higher costs.

Cluster

A bunch of servers used for the same task.  In our case, dedicated to high availability of the MySQL database service.

With theses common definitions, we will then be able to move to the second step, the questions.

Entry posted by yves | 9 comments

Add to: | | | |

When your machine knows what you mean ..

[s3p-root@XMS-1 tomcat6]# crm configure 
crm(live)configure# bye
[s3p-root@XMS-1 tomcat6]# crm confiure 
crm(live)configure# bye
[s3p-root@XMS-1 tomcat6]# crm confiture 
crm(live)configure# bye
[s3p-root@XMS-1 tomcat6]# 

I'd better

apt-get install coffee

Technorati Tags: configure crm ha

Share with Shareomatic!

I spent some time looking into the state of the Pacemaker/Corosync integration today and I can only recommend Pacemaker users stay on the previous version of OpenAIS (aka. Whitetank).

In a nutshell, shutdown is utterly broken.

r2140 of Corosync removed the shutdown worker thread which allowed plugins such as Pacemaker to continue sending and receiving cluster messages.
Without it, Corosync waits for Pacemaker to finish and Pacemaker waits for the messages it tried to send to arrive and be acted upon. Needless to say no-one makes any progress.

Stay tuned, now that integration testing has started it shouldn’t take too long to get everything sorted out.

Update

Since writing this, the necessary testing has been done and Pacemaker is now supported on Corosync provided you have corosync >= 1.1.2 and pacemaker >= 1.0.6

Impressions from the first article (in its first day) and the first 24 hours of the GitHub migration, have caused us at Anchor to believe that;

1. GitHub is just as popular as we thought,
2. The migration was worth it, as things are running much faster (just check your twitter feeds, or better yet, check your GitHub source tree for no reason ); and,
3. People are interested in what has gone under the hood of the new GitHub (insert your favorite fast car here; otherwise lets say a roadster).

Taking these three things into account, this installment will discuss why things are so much faster post migration compared to prior.

I said ‘faster’ and not ‘fast’, because GitHub is now as fast as any website should be. So in comparison, yes, GitHub is fast now, however it is akin to riding your bicycle with half inflated tires: when fully inflated, suddenly your old bike is blazing fast. Now this is not to be critical of the former architecture which held its merits when GitHub was founded. GitHub had simply moved to a stage where a infrastructure architecture refresh was logical.

The main thing, in the large, that made this new architecture fast was that we were given a blank slate and large amounts of freedom to make an architecture that would do the job well.  This is an incredibly rare thing, and it no doubt took a lot of courage on Github’s part.  For that, we have to say “thankyou” to the Github team for letting us have that freedom.  I like to think that we’ve repaid that trust with a pretty awesome architecture that will serve them well for some time to come.

SCALE: When looking at the new architecture as a whole, the increased scale is immediately evident. GitHub now consumes far more hardware than ever before:

Old Infrastructure:

• 10 VMs
• 39 VCPUs
• 54GB RAM

New Infrastructure:

• 16 physical machines
• 128 physical cores
• 288GB RAM

Or for those who enjoy visual cues:

It is a credit to the old infrastructure and GitHub’s code that it ran so well on so little (in comparison). The first credit for increased performance is increased scale.

An important note regarding the hardware is that there is nothing special (or industry secretive) regarding it. The solution in its entirety is run from commodity hardware. No special black boxes doing scary things with packets and routes. No appliance servers. The solution architecture developed by Anchor can be used with any hardware vendor (insert: Dell, HP, IBM, SuperMicro, etc). Vendor neutrality provides GitHub with no encumbrance with either scaling up or out, a key issue when considering growth and future flexibility.

Note: The architectures flexibility allows for the user repository storage to be expanded with a mix of vendor hardware (should GitHub ever change hardware vendor). Furthermore, any component can be exchanged for another vendor’s hardware with no change to GitHubs architecture or software.

In a nutshell, the increased scale provides:

• More GitHub front-end servers to service your requests;
• More storage; and
• More I/O bandwidth when working with your repository data

HARDWARE PERFORMANCE: The speed specifications of the underlying components is important, in addition to how that hardware is utilised.

Storage I/O: A common factor in poor performance with any solution is an I/O bottleneck at the storage level.  This pain was GitHub’s. To alleviate this, not only is the storage now distributed across several servers (distributing the I/O), but it is now running on direct-attached 15,000 RPM SAS disks on battery-backed hardware RAID. Therefore, the second credit for increased performance is faster storage.

Direct access to hardware: Virtualisation is great. What isn’t great is when virtualisation is used as a universal solution. At Anchor we believe there is a place for virtualisation, and systems with massive I/O or CPU requirements is not that place. By moving resource heavy systems onto dedicated hardware, any contention for resources between individual VMs is removed. The third credit goes to less overhead.

ARCHITECTURE: Throwing hardware at a scaling problem is an easy solution, but without the right division of resources and the right software to properly use it, it’s not going to run real fast.

For GitHub, this was their innovative Git command proxying systems, which do an excellent job of taking requests from the frontends (where users connect with their web browser, git client, or SSH client) and shipping them to the fileservers.  The database structure, filesystem layout, and code efficiency also contribute to this.

Given that the software isn’t our speciality, there’s not a lot for us to say about this, but Github are planning a series of posts on their blog, and I’m quite sure it’ll be enlightening.

TO REVIEW: The factors involved in GitHub’s faster response on the new infrastructure include (but are not limited to):

• Increased Infrastructure (Scale)
• Faster Hardware ( Storage)
• No resource contention (More resources per server)
• Solid, scalable architecture (Awesomeness)

Keep an eye on this space, as we delve into technology specific posts regards what kinds of 11 herbs and spices Anchor used to realise the new GitHub architecture.

The first of a new series of step-by-step guides for Pacemaker.

This installment covers installation, the creation of an active/passive cluster and its conversion to active/active.

Technologies used include:

• Fedora 11 as the host operating system
• OpenAIS to provide messaging and membership services,
• Pacemaker to perform resource management,
• DRBD as a cost-effective alternative to shared storage,
• OCFS2 as the cluster filesystem (in active/active mode)
• The crm shell for displaying the configuration and making changes
• Apache as the example service.

The PDF is available from our Documentation page or directly via http://www.clusterlabs.org/mediawiki/images/9/9d/Clusters_from_Scratch_-_Apache_on_Fedora11.pdf

Future guides are anticipated to include MySQL, mail servers and asymmetrical clusters. Feedback and suggestions for additional topics are welcome.

I find it convenient to include current SCM data before my regular Bash prompt (reduces the chance of “accidents”). Perhaps someone else will find it useful too.

function prompt-pre-exec() {
    scm=""
    repo_root=$(hg root 2>/dev/null)
    if [ -e CVS ]; then
        scm=":: cvs ::"

    elif [ -e .svn ]; then
        scm=":: svn : ${prompt_hl}r$(svn info | grep Revision | sed s/.*:\ //)${prompt_n} ($(svn info | grep Date | sed s/.*\(\//)"

    elif [ -e .gitignore ]; then
        repo_branch=`git branch --no-color 2> /dev/null | sed -e '/^[^*]/d' -e 's/* \(.*\)/\1#/'`
        scm=`git show --pretty="format: : git : ${prompt_hl}${repo_branch}%h${prompt_n} : %an, %cr\n:: %s\n" | head -n 2`

    elif [ x != "x$repo_root" ]; then
        repo_cs=$(hg id -i)
        scm=`hg log --template " : hg : ${repo_root##*/} : ${prompt_hl}${repo_cs}${prompt_n} {tags} : {author|user}, {date|age} ago\n:: {desc|firstline|strip}\n" -r ${repo_cs%%+}`
    fi

    if [ "x$scm" != x ]; then
        # Trailing \n characters don't seem to expanded 
        scm="$scm
"
    fi
    export scm    
}

if [ x"$-" = "xhimBH" ]; then
  # Execute the following function before displaying the prompt
  export PROMPT_COMMAND='prompt-pre-exec'

  # Use \[ and \] to exclude the color code from the line wrapping calculations 
  export PS1='${scm}[\@] \u@\h \[${prompt_hl}\]\w #\[${prompt_n}\] '
fi

Then to add color, simply define prompt_hl and prompt_n. I use

export prompt_n="^[^E[00m^]"      # Default color
export prompt_hl="^[^E[01;32m^]"  # Highlight codes

To enter ^[ in emacs, type Ctrl-q then Ctrl-[. Likewise ^E is Ctrl-q Ctrl-e.