Archive for June, 2007

Tuning your invariants

I’m currently building a pretty complex parallel computer that presently runs across multiple nodes in-process. The idea is to push the design to work across a grid of collaborating nodes, either as separate threads, processes or networked machines. I am currently working on two key areas:

  1. Limiting the number of garbage collections that are necessary.
  2. Achieving high performance in the multi-threaded core engine.

This week has been focused on both axis of performance, but recently I have been pushing my design to improve its scalability. Today I have managed to get the system to run an extreme number of networking downloads with a CPU consumption envelope remaining below 3%, which is pretty impressive. Most of this has come from harnessing IO completion ports as well as being extremely savvy with the use of threads provided by the .Net thread pool. So far I am doing well riding on the back of the .Net thread pool, but I may need to consider dedicated threads at some point – the thread pool scheduler can thrash a little in certain scenarios, but I’m not noticing that yet.

Due to the nature of this system, which I’ll explain more of soon, there can be multiple active downloaders retrieving online data from n sources. Downloaded data comes into the system and is processed via a sophisticated logical pipeline. An earlier naive implementation resulted in a saturation of the thread pool, or a profound number of context switches due to equally primitive locking strategies.

To limit or reduce garbage collections I have also needed to improve the types of data structures I am using, as well as understand better the invariants I need to uphold during runtime. For example, I have a pipeline that needs to have data live and in transit as much as possible, to utilize resources as best as can be achieved, whilst not saturating the CPU or increasing the working set pointlessly. Processors on the pipeline work at different rates, but this means that at various points in time, there could be too much waiting in the pipeline, which stresses the memory footprint.

Part of the solution comes from throttling inputs to the pipeline, and using non-collectable data structures (buffers) to handle the task items waiitng to be processed. The head of my pipeline now uses a ring buffer and a semaphore to help avoid creating thousands of network requests to fill the pipeline. The nature of the downstream elements of the pipeline is to process large volumes of data per item, which naturally results in their execution running slower than the source of the pipeline (hence the possibility of the active objects at the source filling up the buffers and killing the memory footprint.

Through careful steps I have managed to produce a highly performant parallel data pipeline without incurring much resources at all. My original design worked well in every way apart from the two key areas above. I could execute multiple concurrent searches and download and process a lot of data, but CPU utilization was around 80%. By breaking the design down, understanding the finer low-level elements of real-time programming, I have arrived at present invariants that allow my system to download the same data as before at nearly 3x the speed, using no more than 3% CPU. I’m still working on the GC issues, but thats not bad going.

40 Hour Guideline?

For some the notion of a 40 hour week is too difficult accept, deeming it too short to really achieve anything. But the rationale for this working principle is built on sound thinking. Working long hours at fever pitch will not only stress a team out, but will induce mental fatigue to the point that silly mistakes are made, and the team will actually work slower and do less than if they were energised everyday and doing less. Obviously when a release is coming up or other such important deadline, then the amount of work the team does must be as flexible as possible, and working more should be acceptable.

I think there is something important to consider that supports the 40 hour week “guideline”. In a team that is energized and running at full-steam, the amount of things going on at any time could be quite large, with the whole team busy coding, designing or talking. Even when a team member is working on something, they will be actively engaged in ensuring they are aware of what is going on around them while they go about their business. Many people who truly engage with pair programming say they like it, but it can be draining. When we work alone, it is easy to take a “breather”, to sit back a little and reflect. But when a pair is engaged, theres no real breather to talk of – its a continual every rolling train of design ideas, forward thinking, code-reviewing and programming to carve the features into the system. When this process hums along like a spinning top, the energy required by the participants is greater than when working alone (certainly in my experience). This is all worthwhile because the idea is that the features that are being implemented are moving along at a productive steady pace and the quality of that work is second to none.

Even when pair programming is not used, there are other practices that actually support this steady train of relentless progress throughout the day, such as TDD (test driven development).  TDD as a practice and a design process can with experience become a profound tool to develop a system with, but its impact on the way its followers think and work, alone or in pairs, should be considered when reviewing the 40 hour guideline.  Just like a spinning top, TDD is difficult to get right, difficult to get it humming along.  But when it does, the level of delivery and progress can be just as profound as the practice itself, creating the well known “flow” state quite easily, and generating extended periods of high concentration and focused work – times this by at least two if this is being done with pairs.

Sometimes these periods of high productivity are like streaks of lightening across the sky, and the amount of progress in a working week might feel small and unproductive. Ignoring the human element that causes the daily mental, emotional and phyisical states of your workforce to fluctuate, a well running team should be able to accommodate and self-organize to help maintain its steady momentum, and the troughs should be less acute.  And this steady pace is built on the high focus developed from the XP practices and open working environment.  When things are truly rocking, it is not uncommon for me to arrive home absolutely exhausted.  But its a positive exhaustion, because its the effect of a causation that itself is positive.  I can wake up the following morning bright and early and be 100% ready for a day riding the development train, and this is part of the point behind the 40 hour guideline.  Happy people are more productive.  Being more productive further fuels happiness and well-being of the team, because they are riding on a wave of positive steady delivery.

The 40 hour week guideline is in the end just a guideline, and it should be appropriately changed to suite your environment.  But its a worthwhile principle of XP and other agile practices because if things are going well, progress will be steady and moving and theres no need for fever pitch.  If this isnt the case, if productivity is not as high and you’re thinking of increasing the hours to get more work out, it might be that other parts of the process arent working like they should be.  Increasing pressure to work longer hours might seem like the easiest approach, and in some cases it can be.  But there should be no need when the team has developed the steady pace.