ZooKeeper is a centralized service for maintaining configuration information, naming, providing distributed synchronization, and providing group services. All of these kinds of services are used in some form or another by distributed applications. Each time they are implemented there is a lot of work that goes into fixing the bugs and race conditions that are inevitable. Because of the difficulty of implementing these kinds of services, applications initially usually skimp on them, which make them brittle in the presence of change and difficult to manage. Even when done correctly, different implementations of these services lead to management complexity when the applications are deployed.

ZooKeeper aims at distilling the essence of these different services into a very simple interface to a centralized coordination service. The service itself is distributed and highly reliable. Consensus, group management, and presence protocols will be implemented by the service so that the applications do not need to implement them on their own. Application specific uses of these will consist of a mixture of specific components of Zoo Keeper and application specific conventions. ZooKeeper Recipes shows how this simple service can be used to build much more powerful abstractions.

ZooKeeper was originally written by Yahoo! and was open sourced as a Hadoop sub-project.  If you are familiar with Google Chubby, ZK scratches a similar itch.  Basically, you create a quorum of servers that use a Paxos variant to propagate changes between the nodes.  There is plenty of information about ZooKeeper already available, so I won’t go into too any detail — but it helps solve of a number of problems in distributed applications such as naming, configuration, and group membership, barriers, locks, 2PC, and leader election.

I try to take very few liberties when porting code from one language to another in order to get things working first.  Now that I have a good base and tests passing, I am planning on making the client feel a little more “.NET-like” – particularly around watches.  Also, I am far from a socket programming expert, so any performance improvements in this area would be especially welcome – just remember that ZooKeeper communication is guaranteed to be ordered (asynchronous does not help).

The source is available on GitHub (under src/dotnet).  I have already submitted a patch to add C# support for Jute (ZK’s client/server protocol which is a fork of Hadoop’s recordio) and will be contributing the rest of client back to Apache once things get a little more stable.  The client is not being used in production at the moment, so please consider the software quality to be alpha.

The picture below shoes the feature in action, notice the parameters are available now:

If you have any additional features ideas, visit the the user voice forum to add or vote on an issue.  Questions or issue can be directed to the Hibernate Profiler group.

The problem with a quick println is that it is often difficult to quickly differentiate your log lines from that of Grails or Hibernate.  Of course you can turn those log levels off, but always like to see the SQL Hibernate is generating during development.  In order to make my logging stand out, I have started adding this to BootStrap.groovy:

   1: def init = { servletContext ->

   2:     ...

   3:     Object.metaClass.printlnRed = { s ->

   4:         def ansi = new Ansi(Ansi.Attribute.BRIGHT, Ansi.Color.RED, Ansi.Color.BLACK);

   5:         ansi.outln(s);

   6:     }

   7:     ...

   8: }

This will allow me to call “printlnRed ‘Some String’ from any Groovy object in my Grails application.  The Ansi class comes from the LGPL licensed jibs library.  Of course you will want to find out what colors work the best for you.  Don’t forget to remove your println’s – you don’t want to clutter up your code for others.  If it is something worth keeping in the application, use Log4J instead.  You can still get colors by configuring your appenders to use Ansi colors as well – there are numerous examples of how to do this including one in Ant.  FYI, I doubt this works in Windows.

   1: usage: groovy ant2maven.groovy

   2:  -?,--help                 Prints help

   3:  -a,--antfile <a>          The ant build file, defaults to build.xml

   4:  -aid,--artifactId <aid>   The artifactId of the new project, defaults to

   5:                            current directory

   6:  -g,--groovy               Enable groovy support

   7:  -gid,--groupId <gid>      The groupId of the new project

   8:  -n,--name <n>             The name of the new project, defaults to

   9:                            current directory

  10:  -o,--output <o>           Name of the POM file, defaults to pom.xml

  11:  -p,--paths <p>            The ant paths to parse

  12:  -t,--testpaths <t>        The ant paths to parse for the test scope

  13:  -u,--nexusurl <u>         Base URL for Nexus repository to search against

  14:  -v,--version <v>          The version of the new project, defaults to

  15:                            1.0-SNAPSHOT

The script requires at least two parameters, the groupId you want for your generated POM and the name of the Ant path that you want to convert to Maven dependencies.  An Ant path normally looks something like this:

   1: <path id="build.classpath">

   2:   <fileset dir="./lib/">

   3:     <include name="xbean-spring-3.4.jar" />

   4:     <include name="camel-*-2.0.0.jar" />

   5:     <include name="spring-*.jar" />

   6:     <include name="jetty-*.jar" />

   7:     <include name="mysql-connector*.jar" />

   8:     <include name="log4j*.jar" />

   9:   </fileset>

  10: </path>

There are a number of ways that you can define your paths in Ant, and you may not even have an Ant path defined, i.e. you have directly define your path under a <javac /> task.  In those cases, you can just create a path and copy your elements for the script to work.

Once the script finds your dependencies, it searches the public Nexus repository using the REST API to find all of your dependencies. The script also attempts to reduce your POM bloat by not listing repeating transitive dependencies as dependencies in your pom.  If the script cannot find a particular dependency, it will go ahead and list it commented out in your POM so that you can resolve it.  Since the public Nexus repository does not have all Maven repositories that you are probably using (Java.net, JBoss, etc), I would recommend setting up your own Nexus instance to achieve the best results.  You can also upload any of your corporate dependencies to your Nexus repository so that they can be resolved by the script as well.

I hacked this together quickly over a few hours this weekend, so there are lots of room for improvements:

• Better error handling when things go wrong
• Automatically install missing artifacts to your own Nexus repository
• Suggest dependencies by name when the checksum search fails
• Add excludes when two artifacts have conflicting version of transitive dependencies

One thing to note is that when you start the script for the first time, it may appear unresponsive.  That is because Groovy is downloading the necessary Grapes for the script – so be patient on the first run.

I have made the script available at GitHub, so feel free to fork for improvements.

   1: def index = {

   2:     def xml = new groovy.xml.StreamingMarkupBuilder()

   3:     xml.encoding = "UTF-8"

   4:     render xml.bind {

   5:         mkp.xmlDeclaration()

   6:         Transactions {

   7:             Transaction.findAllByDate(new Date(), [max: 10, sort: "date", order: "desc").each {

   8:                 Transaction {

   9:                     ID(it.id)

  10:                     Date(it.date)

  11:                     Total(it.total)

  12:                 } 

  13:             }

  14:         }    

  15:     }.toString()

  16: }

With this, you will get a nice well formed XML document:

   1: <?xml version="1.0"?>

   2: <Transactions>

   3:     <Transaction>

   4:         <ID>193710</TD>

   5:         <Date>2009-10-18 17:09:24</Date>

   6:         <Total>123.12</Total>

   7:     </Transaction>

   8:     ...

   9: </Transactions>

Let’s look at a simple script for monitoring ActiveMQ advisory messages:

   1: import org.apache.camel.*;

   2: import org.apache.camel.impl.*

   3: import org.apache.camel.language.groovy.*

   4: import org.apache.camel.component.activemq.*

   5: import org.apache.activemq.camel.component.*

   6: import org.apache.log4j.*

   7:  

   8: @Grab(group='org.apache.camel', module='camel-groovy', version='2.0-M2')

   9: @Grab(group='org.apache.camel', module='camel-jms', version='2.0-M2')

  10: @Grab(group='org.apache.camel', module='camel-core', version='2.0-M2')

  11: @Grab(group='org.apache.activemq', module='activemq-all', version='5.2.0')

  12: @Grab(group='org.apache.activemq', module='activemq-camel', version='5.2.0')

  13: class Routes extends GroovyRouteBuilder {

  14:   public void configure(){

  15:         def log = { exchange -> if (exchange) println exchange } as Processor;

  16:  

  17:         from("activemq:queue:ActiveMQ.Advisory.>").

  18:           process(log);

  19:     }

  20: }

  21:  

  22: def camelCtx = new DefaultCamelContext()

  23: camelCtx.addComponent("activemq", ActiveMQComponent.activeMQComponent("tcp://127.0.0.1:61616"))

  24: camelCtx.addRoutes(new Routes())

  25: camelCtx.start()

All you have to do is save this to a file and run “groovy Script.groovy” – no uploading JAR files or classpath setup required thanks to Grape.  You can see in the script how simple it is to define a Camel processor using Groovy coercion.  With a few modifications, I also use this script as a base for draining dead letter queues, moving message from one queue to another, etc.

Of course, just printing out advisory messages is not all that useful.  Say you want to get notified by email when there is a slow consumer.  Just add dependencies for the necessary mail JARs and and change your route:

   1: from("activemq:queue:ActiveMQ.Advisory.SlowConsumer.*").

   2:   mail("smtp://monitor-list@mycompany.mailserver")

With all of the different Camel components, there are a lot of things you can do with the information – send an IM, log to a database, or write to a file.

Since many developers use Hiberante with Spring, we have tried to make Spring integration as seamless as possible.  If you are using  Hibernate with Spring, configuring your application to be profiled is as simple adding a JAR and a custom attribute to your Spring configuration.  Here you can see where I have added support for profiling to an Appfuse demo project:

   1: <!-- Hibernate SessionFactory -->

   2: <bean id="sessionFactory" class="org.springframework.orm.hibernate3.annotation.AnnotationSessionFactoryBean">

   3:     <property name="dataSource" ref="dataSource"/>

   4:     <property name="configLocation" value="classpath:hibernate.cfg.xml"/>

   5:     <property name="hibernateProperties">

   6:         <value>

   7:             hibernate.dialect=${hibernate.dialect}

   8:             hibernate.query.substitutions=true 'Y', false 'N'

   9:             hibernate.cache.use_second_level_cache=true

  10:             hibernate.cache.provider_class=org.hibernate.cache.EhCacheProvider

  11:         </value>

  12:         <!-- Turn batching off for better error messages under PostgreSQL -->

  13:         <!-- hibernate.jdbc.batch_size=0 -->

  14:     </property>

  15:     <hprof:profiler />

  16: </bean>

This, of course, is just one of the many different ways that you can configure the profiler.  Here is a screenshot of me profiling an Appfuse application.  The top few sessions are profiling unit tests and the remaining sessions (with the URLs) are profiling the web application:

Currently, the product is in closed beta and we are looking for beta testers.  If you are interested, then please contact either Ayende through his blog and he will get you all setup.  Other places you can provide feedback are the user group and the feedback site.  Although I may be a little biased, I think this application can add a lot of value for anyone who is using NHibernate and Hibernate.

The other interesting thing is that the current goal is to get 80% unit tests passing on all projects (the projects that I work on have 100% all the time, otherwise I would lose my mind).  This is a strategy I disagree with.  Let’s say that over time your project has gotten to a state where you only have 50% of the unit tests passing.  As far as I am concerned, this is equivalent to having 0% of the tests passing.  Why?  Because developers see tests in two colors: red and green.  You will quickly find that the 50% will continue to down because no one is actually noticing that they are breaking tests as they are making code changes.  On a few large projects I have worked on, I have seen where a single failing unit test quickly becomes 20. 

Assuming that because of business goals and quantity of tests to fix, that just fixing all of the tests immediately is not an option.  A much better strategy, in my opinion, would be to ignore the tests that are failing and say that all projects must have 100% unit tests passing.  Then, make your goal to be to work your way down from 50% ignored tests to 0% ignored tests over a reasonable period of time. 

My company has decided to close its facility here in Indianapolis, so I am currently job hunting.  Its turns out that suddenly finding a bunch of free time on your hands is bad for the wallet.  It also does not help that a large number of engineers in our office now have very little to do for the next 60 days but talk about things like computer hardware (in between interviews of course).  My machine at home is close to being three years old, so I have decided that it is fully depreciated.  Inspired by the likes of Scott Hanselmann and James Kovacs I have decided to build my own Ultimate Devloper Rig and write about it.

The System

Case – Antec P182

Pretty original choice, huh?  I really wanted to buy the Cooler Master HAF 932 after reading the review, but then I went to Fry’s.  Here is the size comparison between the P182 and the 932:

I was a little concerned that the 932 might not fit under my desk so I went with the P182.  My other concern was that the the amount of fans in the Cooler Master would  make the noise level of the case unacceptable, but a coworker of mine has one and he says that it is not that bad.  If you are not overclocking your machine, you probably do not even need to hook all of the fans in that case up.

 Processor – Intel Core i7 920

When I first starting looking, I assumed that I would be going with a Core 2 Quad system.  If you go Core i7, you are going to pay a lot more for the motherboard, processor, and RAM, right?  Turns out that you won’t be paying quite the premium you would have paid 3 months ago.  My office is located right across the street from Fry’s and they had an great deal on the Core i7 920 – $229.  NewEgg’s current price for a i7 920 is $289.99 and an entry level quad core processor, the Q8300, is currently running $189.99.  If you look at the benchmarks for the Core i7 920 vs. say the Core 2 Quad Q9650 (currently $339), the 920 performs significantly better in a lot of benchmarks and slightly better in some others.  Also, the i7 is a quad core processor with hyper threading (2 threads per core) which means that the operating system will think that the processor has 8 cores.

 Motherboard – ASUS P6T Deluxe LGA 1366 Intel X58 ATX

The motherboard is definately the place where going with the i7 will hurt you on price.  On average, X58 motherboards are about twice as expensive as their comparable P45 counterparts.  In fact, some of the prices for the Core 2 boards are moving closer to 3 times cheaper.  If you think of your machine as a one time sunk cost, then saving $150-$200 on a motherboard is fairly significant.  However, if you are planning on upgraded your processor and RAM a couple of years down the road, you definately do not want to be stuck with a motherboard that only supports Core 2 processors and DDR2 RAM.

After reading some reviews on the current choices for i7 motherboards, I decided to go with the ASUS board.  The board has features such as dual Gigabit LAN, power on bios, 8 SATA ports, 6 DIMM slots, and 3 PCI express slots.  This board supports both Crossfire and SLI — so you have flexibility in which video card manufacturer you go with.  My choice really came down between this board and the Gigabyte GA-EX58-UD4P.  You could probably save yourself an extra $50-$75 bucks by going with this board or the cheaper ASUS if you don’t want all the bells and whistles of the P6T Deluxe.

 Power Supply – CORSAIR 650TX 650W

After picking my motherboard, I moved on to the power supply.  It is important to make sure that your PSU is certified for both SLI and Crossfire, and this one is.  It has connectors for eight SATA drives.  NewEgg had a really good deal on this particular PSU after instant and mail-in rebates ($79), and I could not find anyone with anything bad to say about it when searching. 

 RAM – G.SKILL 6GB (3xGB) DDR3 1333 Triple Channel Memory

Since there are so many different options for RAM, I decided to use the poll the audience approach for choosing.  I searched for "DDR3 1333 triple channel 6GB" on NewEgg and looked at the results.  There about 10 options, all within $20-$30 of each other.  I decided to go with the G.SKILL because it was inexpensive, had the most reviews, and had a 84% excellent rating.

 Boot Drive – Western Digital VelociRaptor WD3000HLFS 3000 GB 10,000 RPM

This is the item that I thought about the most.  It is probably going to be a few months before there is some good data on reliability and performance for SSDs.  I was actually reading a i7 builder’s guide on AnandTech this morning (which incidentally came out after I made my purchases) and it says that they are working on reviewing the latest cheaper SSD drives.  You can get a 64GB GSKILL SSD for $139 or a 128GB for $234.  Even a 64 GB drive would give you plenty of room as a boot drive.  The Intel X25-M SSDs currently boost much higher read times than the cheaper SSDs and they are significantly more expensive, 80 GB for $339, than the cheaper models.  So, I decided to go with the tried and true VelociRaptor … for now.

 Storage Drive – Samsung Spinpoint F1 HD103UJ 1TB 7200 RPM

Again, I spent some time thinking about what to do for storage and I considered setting up a RAID array for data drives.  However, I did not want to put a huge investment into storage in case we see SSD drives that significantly outperform HDD (and are affordable) in the near future, in which case my VelociRaptor would be fine for holding my data.

I do image backups of all of my drives, so data loss is not a real concern for me.   7200 RPM drives are a dime a dozen, so brand does not really matter that much here.

 Video Card – EVGA 512-P3-N975-AR GeForce 9800GT 512 MB 

This is one area where you will probably want to do differently than what I did.  I am not really into playing games on my computer so I do not really care about the video card that much.  The previous generation video card will do just fine for Visual Studio so I am skimping on this item to save about $100.  In reality, I could probably go with a cheaper card that works just as good, but I couldn’t bring myself to buy the cheapest cards available. 

For anyone who plays games, I would recommend the MSI R4870-T2D1G Radeon HD 4870 1GB per the AnandTech guide.  For about a hundred dollars more than my card, you get a card with a 1GB of RAM that is based on the newer chipset from ATI.  With most 4870 X2 boards costing about double what that card will cost you, that card is definately a better buy.  One video card will do fine for me since it has dual DVI for my two Dell 2407WFP (24" widescreen) monitors.

CPU Cooler – Stock

Really?  No after market cooler you say?  I am not planning on overclocking the system so I don’t see a need for the after market cooler.  I am planning on doing some testing when I get the system setup, so I will let you know what the temperatures are like.  Using this same thought process, I am going to go with the stock case fans and see what happen as well.  Maybe I will be wrong and find out that the temperatures are unacceptable; and I’ll have to get a shiny aftermarket cooler and some new case fans.  But nobody ever really discusses what happens with the stock gear, so I think it will be an interesting experiment.

Totals

About $1300 for the system is not a bad price at all.  You could probably get a very similar system for around $1000 if you go with a Core 2 Quad system.

Benchmarks

Assuming I get all the components in the mail, get the system built, and everything works properly; I will post some benchmarks next week.  I will probably get some stats on how long it takes to build and run the unit tests for Castle, but I would also like to find a big project that uses MSBuild so that I can test some times using its parallel build support (NAnt does not support this).  If there are any suggestions on some other pointless benchmarks, let me know. =)

The new component uses a .NET port of the Yahoo! UI Library Compressor that is available on Codeplex.  The component worked great, but they did not have a strongly signed version of the library release.  I made a request and they quickly turned out a release that same day, which was awesome response time.

You can now have all of your Javascript and CSS combined for you on the fly by using the component:

   1: #blockcomponent(BuildJS with "key=layout")

   2:     $jsBuilder.Add("Content/css/main.css")

   3:     $jsBuilder.Add("Content/css/someOther.css")

   4:     $jsBuilder.Add("Content/js/prototype.js")

   5:     $jsBuilder.Add("Content/js/someOther.js")

   6: #end