Showing posts from February, 2010

Use HAProxy 1.4 if you need MySQL health checks

I was using the latest version of HAProxy 1.3 and was load balancing backend MySQL servers while also checking their ports, so if one server went down it would be taken out of the load balancing pool. However, since the port checks in HAProxy happen at the TCP level, the MySQL instance which was being hit by the port checks wasn't happy, because it wasn't a proper MySQL connection. As a result, after a number of some checks, MySQL refused to allow clients to connect, with a message like this:

OperationalError: (1129, "Host 'myhost' is blocked because of many connection errors; unblock with 'mysqladmin flush-hosts'")

Solution: upgrade HAProxy to the newly released version 1.4 (at the date of this writing, the exact version is 1.4.0). Documentation is here.

For MySQL specific checks, you can specify 'option mysql-check' in a backend or 'listen' section of the configuration file. For example, I have something similar to this in my HAProxy c…

My top 10 PyCon takeaways

PyCon 2010 in Atlanta was a blast as always. While I still have things fresh on my mind, here are my top 10 takeaways from the conference, in no particular order.

1) Alternative Python implementations are getting increased attention

It seemed to me that PyPy, Unladen Swallow, IronPython and Jython got much more buzz this year. Maybe this is also due to the announcement that Unladen Swallow will be merged into Python 3.x. I recommend you watch Holger Krekel's talk on the topic of the diverse and healthy Python ecosystem, 'The Ring of Python'.

I was also glad to see that 2 core Jython developers, Frank Wierzbicki and Jim Baker, were hired by Sauce Labs.

2) Testing has gone mainstream

When Mark Shuttleworth mentions automated testing in his keynote as one of the most important ingredients of a sound software engineering process, you know that automated testing has arrived.

There were also no less than 6 testing-related talks, all very good, given by the usual suspects in the testin…

Slides for 'Creating RESTful Web Services with restish'

For those who are interested, the slides for my PyCon 2010 talk 'Creating RESTful Web Services with restish' are online. Leave comments please if you attended the talk and want to start discussions on some of the topics I mentioned.

Update 02/25/10: the video is up too

Web site monitoring techniques and tools

When monitoring a Web site, you need to look at it both from a 'micro' perspective (i.e. are the individual devices and servers in your infrastructure running smoothly?) and from a 'macro' perspective (i.e. do your customers have a pleasant experience when accessing and using your site?; can they use your site's functionality to conduct their business online?). You can also think about these types of monitoring as 'engineering' monitoring vs. 'business' monitoring. They're both important, but most system administrators obviously focus on the engineering/micro-type monitoring, since it's closer to their skills and interests, and tend to not put much emphasis on the business/macro-type monitoring.

In this post I'll talk about various types of monitoring we're doing for the Evite Web site. I'll focus on what I call 'deep application monitoring', which to me means gathering and displaying metrics about your Web application&#…