Eric Hauser’s Blog

Over the past couple of weeks, I ported the Java client for ZooKeeper to .NET.  What is ZooKeeper?

ZooKeeper is a centralized service for maintaining configuration information, naming, providing distributed synchronization, and providing group services. All of these kinds of services are used in some form or another by distributed applications. Each time they are implemented there is a lot of work that goes into fixing the bugs and race conditions that are inevitable. Because of the difficulty of implementing these kinds of services, applications initially usually skimp on them, which make them brittle in the presence of change and difficult to manage. Even when done correctly, different implementations of these services lead to management complexity when the applications are deployed.

ZooKeeper aims at distilling the essence of these different services into a very simple interface to a centralized coordination service. The service itself is distributed and highly reliable. Consensus, group management, and presence protocols will be implemented by the service so that the applications do not need to implement them on their own. Application specific uses of these will consist of a mixture of specific components of Zoo Keeper and application specific conventions. ZooKeeper Recipes shows how this simple service can be used to build much more powerful abstractions.

ZooKeeper was originally written by Yahoo! and was open sourced as a Hadoop sub-project.  If you are familiar with Google Chubby, ZK scratches a similar itch.  Basically, you create a quorum of servers that use a Paxos variant to propagate changes between the nodes.  There is plenty of information about ZooKeeper already available, so I won’t go into too any detail — but it helps solve of a number of problems in distributed applications such as naming, configuration, and group membership, barriers, locks, 2PC, and leader election.

I try to take very few liberties when porting code from one language to another in order to get things working first.  Now that I have a good base and tests passing, I am planning on making the client feel a little more “.NET-like” – particularly around watches.  Also, I am far from a socket programming expert, so any performance improvements in this area would be especially welcome – just remember that ZooKeeper communication is guaranteed to be ordered (asynchronous does not help).

The source is available on GitHub (under src/dotnet).  I have already submitted a patch to add C# support for Jute (ZK’s client/server protocol which is a fork of Hadoop’s recordio) and will be contributing the rest of client back to Apache once things get a little more stable.  The client is not being used in production at the moment, so please consider the software quality to be alpha.

I have had the privilege of doing some work for Ayende for the past month to add support for Hibernate to his excellent NHibernate Profiler.  What does this mean exactly?  Well, Hibernate is now supported as a first class citizen in the profiler.  We have tried to make sure that there is a 1:1 feature mapping between the NHibernate and Hibernate support.  I am happy to report that all applicable warnings and information that you receive for NHibernate is also displayed for Hibernate.  There are of course differences between NHibernate and Hibernate — such as Hibernate does not support statement level batching options, multiquery, or future queries – which is reflected in the documentation.

Since many developers use Hiberante with Spring, we have tried to make Spring integration as seamless as possible.  If you are using  Hibernate with Spring, configuring your application to be profiled is as simple adding a JAR and a custom attribute to your Spring configuration.  Here you can see where I have added support for profiling to an Appfuse demo project:

   1: <!-- Hibernate SessionFactory -->

   2: <bean id="sessionFactory" class="org.springframework.orm.hibernate3.annotation.AnnotationSessionFactoryBean">

   3:     <property name="dataSource" ref="dataSource"/>

   4:     <property name="configLocation" value="classpath:hibernate.cfg.xml"/>

   5:     <property name="hibernateProperties">

   6:         <value>

   7:             hibernate.dialect=${hibernate.dialect}

   8:             hibernate.query.substitutions=true 'Y', false 'N'

   9:             hibernate.cache.use_second_level_cache=true

  10:             hibernate.cache.provider_class=org.hibernate.cache.EhCacheProvider

  11:         </value>

  12:         <!-- Turn batching off for better error messages under PostgreSQL -->

  13:         <!-- hibernate.jdbc.batch_size=0 -->

  14:     </property>

  15:     <hprof:profiler />

  16: </bean>

This, of course, is just one of the many different ways that you can configure the profiler.  Here is a screenshot of me profiling an Appfuse application.  The top few sessions are profiling unit tests and the remaining sessions (with the URLs) are profiling the web application:

Currently, the product is in closed beta and we are looking for beta testers.  If you are interested, then please contact either Ayende through his blog and he will get you all setup.  Other places you can provide feedback are the user group and the feedback site.  Although I may be a little biased, I think this application can add a lot of value for anyone who is using NHibernate and Hibernate.

Last year, Hammett posted a blog entry on combining Javascript files into a single file using Monorail.  I was looking to enhance the code so that it would minify Javascript and CSS files on the fly, so I took his original code and enhanced it.  Alex Henderson was also looking for the same functionality, so he contributed to the patch as well.  You can download it here:  http://support.castleproject.org/projects/MR/issues/view/MR-ISSUE-457

The new component uses a .NET port of the Yahoo! UI Library Compressor that is available on Codeplex.  The component worked great, but they did not have a strongly signed version of the library release.  I made a request and they quickly turned out a release that same day, which was awesome response time.

You can now have all of your Javascript and CSS combined for you on the fly by using the component:

   1: #blockcomponent(BuildJS with "key=layout")

   2:     $jsBuilder.Add("Content/css/main.css")

   3:     $jsBuilder.Add("Content/css/someOther.css")

   4:     $jsBuilder.Add("Content/js/prototype.js")

   5:     $jsBuilder.Add("Content/js/someOther.js")

   6: #end

Although there are a number of good continuous integration servers out there, TeamCity is one of my favorites.  It has great support for .NET — supporting MSBuild, NAnt, Solution builds, NUnit, MSTest, and FxCop.  The price is right if you have less than 20 users and 20 build configurations – free.  I was also able to a plugin to integrate with our issue tracking systems in a couple of hours timespan (my Spring MVC experience definately helped though).

One of thing that you want to do with a release build is automatically increment your version numbers.  Most approaches to this problem recommend using MSBuild to autogenerate the AssemblyInfo.cs file.  Although that certainly works, I prefer not to have to do that for every project.  Instead, each of our projects references a single CommonAssemblyInfo.cs file.

When TeamCity builds a release build, it is configured to the IncrementVersion task:

   1: <Project xmlns="http://schemas.microsoft.com/developer/msbuild/2003">

   2:     ...

   3:     <Target Name="UnrestrictedExecutionPolicy">

   4:         <Exec Command="powershell set-executionPolicy unrestricted&quot;" />

   5:     </Target>

   6:     <Target Name="IncrementVersion" DependsOnTargets="UnrestrictedExecutionPolicy">

   7:         <Exec Command="PowerShell .\incrementVersion.ps1 $(build_vcs_number_1)" />

   8:     </Target>

   9:     ...

  10: </Project>

The $(build_vcs_number_1) property tells TeamCity to the current revision number from the first source repository attached – in my case SVN — to the project (most projects will only have one repository attached).  Then, we execute a simple Powershell script to replace the file version number:

   1: $buildNumber = $args[0]

   2: (Get-Content ..\src\CommonAssemblyInfo.cs) | Foreach-Object {$_ -replace "(\d\.\d\.\d)\.\d*", "`$1.$buildNumber"} | Set-Content ..\src\CommonAssemblyInfo.cs

If you are more comfortable using a MSBuild task for regex replacement, there is a RegexReplace task that is part of the MSBuild Community Tasks.  For my build, we do not need any other custom tasks, so it was preferable to just use a simple PS script.

There are a lot of times in your application where you will load a lot of entities in a single session (to display a lot of data on a page, build a report, etc).  Since you are not planning on doing a mutable operation on an entity, there is a big saving that you can get from not flushing the session.  To get an idea on everything that happens when you do a flush, check out the AbstractionFlushingEventListener from the NHibernate source.

If you are using explicit SessionScope’s, not flushing the session is trivial:

   1: using (new SessionScope(FlushAction.Never))

   2: {

   3:     repository.DoSomeReadOperation();

   4: }

This will execute the operation without flushing the session at the end.

Another way to control this behavior is by using an interceptor.  For example, if you are using the Castle’s Automated Transaction Facility, you probably already have a way of determining whether or not your methods do mutable operations by their transaction attributes.  You can then use an interceptor to control flushing:

   1: public class UnitOfWorkInterceptor : IInterceptor

   2: {

   3:     private readonly ITransactionManager transactionManager;

   4:

   5:     /// <summary>

   6:     /// Initializes a new instance of the <see cref="UnitOfWorkInterceptor"/> class.

   7:     /// </summary>

   8:     /// <param name="transactionManager">The transaction manager.</param>

   9:     public UnitOfWorkInterceptor(ITransactionManager transactionManager)

  10:     {

  11:         this.transactionManager = transactionManager;

  12:     }

  13:

  14:     #region Implementation of IInterceptor

  15:

  16:     /// <summary>

  17:     /// Intercepts the specified invocation.

  18:     /// </summary>

  19:     /// <param name="invocation">The invocation.</param>

  20:     public void Intercept(IInvocation invocation)

  21:     {

  22:         ITransaction transaction = transactionManager.CurrentTransaction;

  23:         FlushAction flushAction = transaction == null ? FlushAction.Never : FlushAction.Config;

  24:

  25:         ActiveRecordUnitOfWork.Before(flushAction);

  26:         invocation.Proceed();

  27:         ActiveRecordUnitOfWork.After();

  28:     }

  29:

  30:     #endregion

  31: }

Note that in this example, interceptor ordering is important.  If you register the unit of work interceptor before the transaction interceptor in the ATM facility, then transactionManager.CurrentTransaction will always return null.  (The ActiveRecordUnitOfWork class in the above example is just a facade for controlling the session in a thread local as discussed in this post http://erichauser.net/2008/08/06/activerecord-session-scope-and-wcf-redux/).  If you do not want to tie your flushing to transactions, then you can always create your own metadata and read that metadata at runtime in the interceptor.  The ATM facility code is a great example of how to inspect metadata on initialization and use that metadata in your interceptor.

Hopefully, this little trick will some increased performance for your queries that return a lot of results.

Been pretty busy at work, so short post. 

I’ve added a facility in Castle contrib for validating method parameters using Castle Validator.  The goal is to eventually allow have this used for validate parameters in Monorail, but right now I’m using it to do input validation for web service parameters.  I wrote up a brief blurb on how to use it here: http://using.castleproject.org/display/Contrib/Castle.Facilities.MethodValidator.  Should be pretty self explanatory.

If anyone finds any issues, let me know.

I have contributed a couple of enhancements to Castle.Components.Validator that have been committed to the trunk.  Besides using attributes, validations can now be supplied in code using the [ValidateSelf] attribute:

Each of the above validations *could* be done by using an attribute and a custom validator, but expressing validations in code is much simpler for these types of one off validations. You can have as many methods decorated with [ValidateSelf] on an object you want as long as they have the above method signature (void return and one ErrorSummary parameter).  You can also specify the RunWhen and ExecutionOrder just like regular validators.

The second enhancement is the IValidationContributor interface. This allows you contribute to the validation of an object beyond the default validation. The interface is fairly simplistic:

You can extend AbstractValidationContributor so that you can perform initialization for a given type.  The SelfValidationContributor implements the logic for recognizing and executing the self validation feature above.  You can write custom contributors that can be injected into the DefaultValidatorRunner for things like retrieving validations from the container and invoking them on the object.

Enjoy!

I have added a couple of additions to Ayende’s NServiceBus Facility:

• Message handlers and sagas are automatically registered with the container for the defined assemblies
• Message handlers are proxied
• You can register IHandlerFilter instances with the kernel that allow for interception of messages as they are processed
• Using my replay strategy implementation, you can decorate a message handler as [Idempotent(Attemps = 5, Delay = 2000)].  That means that a message will be retried when an exception is thrown processing the message 5 times, delaying 2 seconds between each retry.  This could also be extended so that depending on the type of exception that is being thrown, a default retry strategy is applied.

• The bus is started automatically once all handler dependencies have been satisfied

There were a couple of interesting things that I found when implementing this.  One, NServiceBus retrieives handlers from the kernel using the concrete class instance (looking back Ayende noted this as “yuck”).  That means that we cannot proxy the interface, but instead have to create a class proxy for our message handlers.  That means that when we the message handlers, we have to verify that the message handler methods are virtual and give a nice error if they are not:

Very easy to do with Castle’s fluent interface for registration.  Sagas, on the other hand, have to be registered in the container by each of their interfaces because saga instances are retrieve by calling builder.Build<ISaga<SomeMessage>>().

Out of the box, the way to do message interception with NSB is to have message handlers chained in a specified order.  The IHandlerFilter provides another method for message interception.  If you have a new filter to add, all you have to do is register the IHandlerFilter instance with the container:

The filters follow the chain of responsibility pattern, so they can be short circuited — which is the same as calling bus.DoNotContinueDispatchingCurrentMessageToHandlers().

I have attached the updates to the post: nservicebus-fullduplex-xml-update.  Enjoy.

I got an email earlier this week from a member of our team saying that they had noticed odd behavior in our unit tests.  They said when the ran the tests from Visual Studio, the memory allocation increased by about 500MB.  When they ran the tests again, the memory increased another 500MB, and so forth.  The system component that developer was working on had the most tests out of all of our components, so he was noticing this before others did. 

First of all, it seemed odd that unit tests would cause Visual Studio’s memory to increase since the tests are run in another process.  I knew it would probably be impossible to profile Visual Studio itself, so I profiled out tests from the command line to see what if anything I could figure out.  A couple of tricks to profiling MSTest.exe:

• Launch mstest.exe with the /noisolation option so that the tests are run in process
• When you use the /noisolation option, you have to replace \Program Files\Microsoft Visual Studio 9.0\Common7\IDE\MSTest.exe.config with the App.config for your tests.  You also need to add the this:

    <runtime>
        <assemblyBinding xmlns=”urn:schemas-microsoft-com:asm.v1″>
            <probing privatePath=”PrivateAssemblies;PublicAssemblies”/>
        </assemblyBinding>
    </runtime>

The first thing I noticed after all of the tests ran was that 98% of the memory was being held by System.String.  When I looked to see what objects were referencing System.String, one thing blaringly stood out:

All of the standard out was being held in a single member variable from the whole test run.  Since I was not sure if Visual Studio was doing the same thing, I turned off logging in our application and ran the tests again.  Surprise, that worked.  Visual Studio no longer was increasing in memory usage.

Yes, I realize 800 MB of log output is a lot but that is not really the issue – I’m not sure why VS keeps that output in memory.  Once it has the test results, I believe it writes it out to a file so why keep it around?

Awhile ago, I posted a solution I was using for a project for managing SessionScope with WCF using an ICallContextInitializer. Because of another issue in the codebase (an error in a custom listener), an exception was getting thrown when session.Dispose() was called in the AfterInvoke method.  What was interesting was that it appeared to the client that the WCF call had completed successfully — even though this was not the case.  It turns out that ICallContextInitializer is not the best WCF extension to use to manage the scope. Here’s a quick console application that will show the behavior of throwing an exception in AfterInvoke. Notice how the console window will crash before the ReadKey is hit:  

   1: class Program

   2: {

   3:     static void Main(string[] args)

   4:     {

   5:         using (var host = new ServiceHost(typeof(Operation)))

   6:         {

   7:             var endpointAddress = new EndpointAddress("net.tcp://localhost/optest");

   8:             var endpoint = new ServiceEndpoint(ContractDescription.GetContract(typeof(IOperation)),

   9:                                                new NetTcpBinding(),

  10:                                                endpointAddress);

  11:             endpoint.Behaviors.Add(new ThrowBehavior());

  12:             host.Description.Endpoints.Add(endpoint);

  13:             host.Open();

  14:  

  15:             var op = ChannelFactory<IOperation>.CreateChannel(new NetTcpBinding(), endpointAddress);

  16:             op.Op();

  17:         }

  18:  

  19:         Console.ReadKey();

  20:     }

  21: }

  22:  

  23:
 [ServiceContract]

  24: public interface IOperation

  25: {

  26:     [OperationContract]

  27:     void Op();

  28: }

  29:  

  30: public class Operation : IOperation

  31: {

  32:     public void Op()

  33:     {

  34:     } 

  35: }

  36:  

  37: internal class ThrowBehavior : IEndpointBehavior

  38: {

  39:     public void Validate(ServiceEndpoint endpoint)

  40:     {

  41:     }

  42:  

  43:     public void AddBindingParameters(ServiceEndpoint endpoint, BindingParameterCollection bindingParameters)

  44:     {

  45:     }

  46:  

  47:     public void Apply
DispatchBehavior(ServiceEndpoint endpoint, EndpointDispatcher endpointDispatcher)

  48:     {

  49:         foreach (DispatchOperation operation in endpointDispatcher.DispatchRuntime.Operations)

  50:         {

  51:             operation.CallContextInitializers.Add(new ThrowCallContextInitializer());

  52:         }

  53:     }

  54:  

  55:     public void ApplyClientBehavior(ServiceEndpoint endpoint, ClientRuntime clientRuntime)

  56:     {

  57:     }

  58: }

  59:  

  60: internal class ThrowCallContextInitializer : ICallContextInitializer

  61: {

  62:     public object BeforeInvoke(InstanceContext instanceContext, IClientChannel channel, Message message)

  63:     {

  64:         return null;

  65:     }

  66:  

  67:     public void AfterInvoke(object correlationState)

  68:     {

  69:         throw new InvalidOperationException();

  70:     }

  71: }

My revised code uses a DispatchMessageInterceptor instead:

   1: public class ARSessionScopeBehavior : IEndpointBehavior

   2: {

   3:     

   4:     /// <summary>

   5:     /// Implement to pass data at runtime to bindings to support custom behavior.

   6:     /// </summary>

   7:     /// <param name="endpoint">The endpoint to modify.</param>

   8:     /// <param name="bindingParameters">The objects that binding elements require to support the behavior.</param>

   9:     public void AddBindingParameters(ServiceEndpoint endpoint, BindingParameterCollection bindingParameters)

  10:     {

  11:     }

  12:  

  13:     /// <summary>

  14:     /// Implements a modification or extension of the service across an endpoint.

  15:     /// </summary>

  16:     /// <param name="endpoint">The endpoint that exposes the contract.</param>

  17:     /// <param name="endpointDispatcher">The endpoint dispatcher to be modified or extended.</param>

  18:     public void ApplyDispatchBehavior(ServiceEndpoint endpoint, EndpointDispatcher endpointDispatcher)

  19:     {

  20:         endpointDispatcher.DispatchRuntime.MessageInspectors.Add(new ARDispatcherMessageInspector());

  21:     }

  22:  

  23:     /// <summary>

  24:     /// Implements a modification or extension of the client across an endpoint.

  25:     /// </summary>

  26:     /// <param name="endpoint">The endpoint that is to be customized.</param>

  27:     /// <param name="clientRuntime">The client runtime to be customized.</param>

  28:     public void ApplyClientBehavior(ServiceEndpoint endpoint, ClientRuntime clientRuntime)

  29:     {

  30:     }

  31:  

  32:     /// <summary>

  33:     /// Implement to confirm that the endpoint meets some intended criteria.

  34:     /// </summary>

  35:     /// <param name="endpoint">The endpoint to validate.</param>

  36:     public void Validate(ServiceEndpoint endpoint)

  37:     {

  38:     }

  39:  

  40: }

   1: public class ARDispatchMessageInspector : IDispatchMessageInspector

   2:  {

   3:  

   4:      private static readonly ILogger Logger = LogManager.GetLogger(ARDispatchMessageInspector);

   5:  

   6:      private const string ActiveRecordSessionScopeKey = "wcf.ar.sessionscope";

   7:  

   8:      #region Implementation of IDispatchMessageInspector

   9:  

  10:      /// <summary>

  11:      /// Called after an inbound message has been received but before the message is dispatched to the intended operation.

  12:      /// </summary>

  13:      /// <returns>

  14:      /// The object used to correlate state. This object is passed back in the <see cref="M:System.ServiceModel.Dispatcher.IDispatchMessageInspector.BeforeSendReply(System.ServiceModel.Channels.Message@,System.Object)" /> method.

  15:      /// </returns>

  16:      /// <param name="request">The request message.</param>

  17:      /// <param name="channel">The incoming channel.</param>

  18:      /// <param name="instanceContext">The current service instance.</param>

  19:      public object AfterReceiveRequest(ref Message request, IClientChannel channel, InstanceContext instanceContext)

  20:      {

  21:          var scope = Local.Data[ActiveRecordSessionScopeKey] as SessionScope;

  22:          if (scope == null)

  23:          {

  24:              Logger.Debug("Creating new ActiveRecord SessonScope");

  25:              scope = new SessionScope();

  26:              Local.Data[ActiveRecordSessionScopeKey] = scope;

  27:          }

  28:          return null;

  29:      }

  30:  

  31:      /// <summary>

  32:      /// Called after the operation has returned but before the reply message is sent.

  33:      /// </summary>

  34:      /// <param name="reply">The reply message. This value is null if the operation is one way.</param>

  35:      /// <param name="correlationState">The correlation object returned from the <see cref="M:System.ServiceModel.Dispatcher.IDispatchMessageInspector.AfterReceiveRequest(System.ServiceModel.Channels.Message@,System.ServiceModel.IClientChannel,System.ServiceModel.InstanceContext)" /> method.</param>

  36:      public void BeforeSendReply(ref Message reply, object correlationState)

  37:      {

  38:          var scope = Local.Data[ActiveRecordSessionScopeKey] as SessionScope;

  39:          if (scope != null)

  40:          {

  41:              if (SessionScope.Current == scope)

  42:              {

  43:                  Logger.Debug("Disposing ActiveRecord SessonScope");

  44:                  scope.Dispose();

  45:              }

  46:              else

  47:              {

  48:                  Logger.Warn(

  49:                      "The current Active Record session scope does not equal the correlated session scope from WCF.  This scope will not " +

  50:                      "be disposed, however, this could lead to entities not being properly flushed.");

  51:              }

  52:              Local.Data[ActiveRecordSessionScopeKey] = null;

  53:          }

  54:      }

So far, no problems.  Funny enough, when searching for solutions to this problem, I noticed some other posts where people recommended using the ICallContextInitializer to manage unit of work.  I assume they just have not had any issues when calling Dispose() and have not seen this behavior.