All there is is letting go, allowing and embracing change as a natural process of the universe, and aligning with the underlying forces, rather than fighting them in the interminable endeavour of the ego to make things concrete.  Nothing remains the same, not even those grandiose ideas created by our own greatness – they fade away as the rolling waves of time erode their existence into the depths of history.  To remain fixed on our ideas, on our decisions that are the beacons of our greatness in the workplace, is to simply create  coffins out of empty vessels of once interesting ideas.

As 2009 nears its end, I hope the future involves less about the past and more about making the best decisions to allow change to arise from the present moment based on facts as they present themselves Now.  Old ideas about what should work will be replaced with feedback from experience, to help guide decisions on what needs to change, be it process, communication, software development or anything else for that matter.  Its all the same nothingness in the end.

1. I add a few assembly dependencies  to my project.
2. I then think I have remembered a class I am wanting to use and declare it, such as FooBar<double>.
3. Resharper turns FooBar red, which implies the class name isnt known.
4. Problem is FooBar does exist in one of the assemblies that was added, but its not a generic.  Resharper reports: Cannot resolve symbol: FooBar.

To solve this I have to do Ctrl-N and look up the classname either in the browser (since I added assembly dependencies and not project dependencies, which would give me code-access), and then work out its not a generic.  While it doesnt take long to realise my error, I still expect resharper to offer to declare my variable as FooBar without the generic as one of its options that it provides to fix the problem.  As a productivity tool, this little feature would be very helpful.

So consider this.  Process is a dirty word.  So you and your team dont do process.  This is fine, and not uncommon by any means in certain industries.  But theres not such thing as no process.  Every act employed to bring software together, be it feature definition, bug tracking, testing, quality assuring, how these are done and when, is process at some level.  This is very important.

I think it is important to be aware of this, because it opens up the landscape of possibility and choices in evolving the way a team works and improves over time.  Metaphor lost its importance in the later version of XP, but there are people that believe strongly in its use, myself included, see here.  With metaphor our mental vocabulary is enriched, and a deeper understanding of the world can result.  And choice arises from the deeper understanding gained from employing new mental models.  If the number of choices increase, then hopefully the ability to have the answer increases.

The problem I see with the no process approach, is that it immediately shuts down a world of choice that exists that is tried and tested, and seen to lead to more successful projects.  All of this is lost, replaced by ad-hoc bespoking by teams that dont do process.  There will be meetings to ensure no or little process is introduced, yet its all process.  And if its all process, is it not better to choose from those that have a pedigree of success behind them? To spend time manufacturing a new approach to requirements gathering is ok if you are a process expert.  If you’re not a process expert then such time should be spent doing other things where true value can be added.

If process is a dirty word for your team, realize this habitual mental model may be inhibiting the chances of greater success.  There is not such thing as no process.  So choose the process elements you want to re-invent carefully, and for the rest, employ tried and tested principles that have been perfected so that you can get down to the real work of delivering software.

To build or enhance a high performance managed application has a set of constraints, three of which are data structure choice, algorithm choice and the garbage collection envelope. An interesting discussion arises when talking about Array vs List<T>, but this can be quickly addressed by understanding the mechanics of both and where they differ. I use List<T> as the comparator simply because this is the common replacement of choice by most C# programmers I have come across.

Some think data locality is not so important in a managed host environment and you can ignore it, and largely you can, but data locality is just as important as any other axis when building a high performance application. The simple question is, does a poor data locality eat into CPU cycles? Yes. Then it requires just as much respect as choosing a poor algorithm. Data locality is so important there is research into enhancing the garbage collector so that it improves locality of reference as part of its role. So with that said, an array becomes a very good choice because its allocation creates a contiguous block of memory together in one place. To work with an array structure in our programs results in the complete block of memory being brought into the cache pipeline (caveats ignored).

So an array is created and stored contiguously in memory. From this point on you will get the fastest type of performance you can expect, because an index into the data structure is a direct address (as good as can be in a managed world), and this gives you O(1) access time anywhere in the structure. Furthermore, arrays are a fundamental type in IL, meaning their implementation is as raw as you can get it, with minimal levels of indirection.

Consider:

int[] a = new int[100];
a[0] = 0×10;

Disassembly:

At memory location 0x013B1BF4 in my debugger I see the following:

We are basically looking at the heap, and the first red value of 0x791240f0 is the type pointer, followed by the size of the array (64h == 100 denary). On assembly line 0×00000031 0×10 is assigned to the first element, which results in the following changes:

So the resulting JIT compiled code is as direct as you could get it with an array, and the memory is all kept together as shown above, which results in performance and perfect data locality for value types. Note, an array of reference types requires some extra care to ensure a better locality of reference which you don’t get for free as you do with value types.

The issue that arises with arrays is that they are fixed – once created, thats it. To resize and array involves scrapping the current block of memory and requesting another. There may be performance improvements provided by the CLR, but largely this means discarding the original array to be garbage collected and requesting a new block of memory.

Above the array a is created on the heap. Creating a new array will result in this being lost:

The difference in the location of the arrays would occur if in between creating the first array of 100 and the second of 200, more memory was allocated on the heap. For your information, if this were a non-managed situation and you had much more control over memory, you could re-establish the 200 element array at the original address of a, avoiding fragmentation. But in a managed world creating a new array and assigning it to an existing array will not allow the GC to release the memory (the 100 element array), since this will only occur at some non-predetermined time.

For those not interested in the details you can skip this next piece.

The red text marks where the previous instance of a was (a = 100), and the blue text marks where the new array of 200 elements was created, showing how the GC does indeed allocate memory contiguously.

So if the size of an array needs to change, should we use an array, or should we choose a different data structure such as List<T>?

Consider this:

int[] intArray = new int[100];

List<int> intList = new List<int>();

The different between the two is that to get something into box structures is programmatically different, since its a direct memory address with the array, but it requires at least a first call to Add() for the list. With the array we can immediately do something like intArray[99] = 99, but for the list you would need to call Add() x100 before being able to achieve the same. You’d be forgiven for thinking you could do this:

List<int> intList = new List<int>(100);

intList[99] = 99;

The Capacity of a List is simply a statement to the implementation that the initial capacity will be 100, and from there after will double its capacity once exceeded. So if you add 101 items into the list its capacity will immediately become 200, add 201, and the capacity would increase to 400. If you understood a little more about how the List is implemented you might get frustrated with why the above doesnt work, but there is a very good reason why it doesnt.

While I am going to look at the internals of List from this point, let it be said that good engineering is about abstraction and encapsulation, and relying on implementation details is a form of very bad coupling. At the same time it is important to understand how something works, especially in a managed environment that comes with the price of the garbage collector.

So the reason you cant immediately gain access to the 100th element in the list is because the Capacity of a List is not th same as its Count. You cant grumble at that. Internally List is implemented using an Array<T>. When you add the first element, a block of memory will be created the size of Capacity. If you create the list without a size for Capacity, your default array size will be 4 in .Net 2.0.

The interesting stuff in List happens when you call Add:

public void Add(T item)
{
    if (this._size == this._items.Length)
    {
        this.EnsureCapacity(this._size + 1);
    }
    this._items[this._size++] = item;
    this._version++;
}

If the current size has reached Capacity, the first thing to happen is to create a new array inside EnsureCapacity:

private void EnsureCapacity(int min)
{
    if (this._items.Length < min)
    {
        int num = (this._items.Length == 0) ? 4 : (this._items.Length * 2);
        if (num < min)
        {
            num = min;
        }
        this.Capacity = num;
    }
}

This private method deals with working out the next size to expand the array to – notice this._items.Length * 2. The Capacity assignment is where the re-allocation occurs:

public int Capacity
{
    get
    {
        return this._items.Length;
    }
    set
    {
        if (value != this._items.Length)
        {
            if (value < this._size)
            {
                ThrowHelper.ThrowArgumentOutOfRangeException(ExceptionArgument.value, ExceptionResource.ArgumentOutOfRange_SmallCapacity);
            }
            if (value > 0)
            {
                T[] destinationArray = new T[value];
                if (this._size > 0)
                {
                    Array.Copy(this._items, 0, destinationArray, 0, this._size);
                }
                this._items = destinationArray;
            }
            else
            {
                this._items = List<T>._emptyArray;
            }
        }
    }
}

Because the List uses an Array, this means all that I said earlier about arrays is largely applicable here. The issue with a List is that even though the full array is created underneath, you are limited to reaching into the List no further than Count, which can be less than the size of the internal array (as can be seen above). There will be a performance hit with the call to Add because it does some sanity checking on the array – and potentially resizes – as well as the method call itself (a List could be used in a hard realtime scenario as long as the programmer ensured the capacity was such that it obviated a resize).

One last comment to note about the Capacity of a List. Most people know of the impact of not creating a list with a capacity, but I’ll explain it anyway. If you dont create a list with a capacity, after the first Add() your underlying array will have a size of 4. When you exceed 4 it will double to 8, then to 16, … So clearly, if you know you are going to load the list with 1250 items, you should do this : list = new List<int>(1250). What happens if you know your list will become 1250, but you are doing lots of async calls which means it could take some time to fill the list? Still create the list with 1250 – remember, if you decide not to create the list with a Capacity you will incur 10 memory allocations before arriving at a structure that supports your needs. If you are not in a realtime situation, you must still respect the GC, but you can be less aggressive.

From a class role perspective, both Array and List offer the same rich set of interfaces such as IList, ICollection and IEnumerable, allowing both to be interchanged with each other in your code which is useful.

Both List and Array are highly performant data structures, and if you use the List correctly you can achieve comparable performance to an Array. Data locality will be good with both data structures, with the same caveats when creating data structures, associative or otherwise, that cannot be established 100% inplace within the data structure – remember that if the type being stored is a reference type, all you will hold within your data structure is a reference to some place else in the heap.

If you know you will need to handle expansion of an array manually, dont bother reinventing the wheel and just use the List – its as fast you could probably write. The List however falls short as soon as you add another dimension to the underlyng data structure – a List is a one-dimensional concept and will offer little, unless again you opt for creating jagged array style structures. As soon as you move into the world of holding 2 or more dimensions of data, you must resort back to arrays for all the reasons already mentioned. If you need an n-dimensional structure that may need to expand, consider the algorithm for a List – you could use a lot from its implementation in creating an expandable 2D matrix.

6c 00 61 00 74 00 65 00 72 00 7a 00 2e.

1. Limiting the number of garbage collections that are necessary.
2. Achieving high performance in the multi-threaded core engine.

This week has been focused on both axis of performance, but recently I have been pushing my design to improve its scalability. Today I have managed to get the system to run an extreme number of networking downloads with a CPU consumption envelope remaining below 3%, which is pretty impressive. Most of this has come from harnessing IO completion ports as well as being extremely savvy with the use of threads provided by the .Net thread pool. So far I am doing well riding on the back of the .Net thread pool, but I may need to consider dedicated threads at some point – the thread pool scheduler can thrash a little in certain scenarios, but I’m not noticing that yet.

Due to the nature of this system, which I’ll explain more of soon, there can be multiple active downloaders retrieving online data from n sources. Downloaded data comes into the system and is processed via a sophisticated logical pipeline. An earlier naive implementation resulted in a saturation of the thread pool, or a profound number of context switches due to equally primitive locking strategies.

To limit or reduce garbage collections I have also needed to improve the types of data structures I am using, as well as understand better the invariants I need to uphold during runtime. For example, I have a pipeline that needs to have data live and in transit as much as possible, to utilize resources as best as can be achieved, whilst not saturating the CPU or increasing the working set pointlessly. Processors on the pipeline work at different rates, but this means that at various points in time, there could be too much waiting in the pipeline, which stresses the memory footprint.

Part of the solution comes from throttling inputs to the pipeline, and using non-collectable data structures (buffers) to handle the task items waiitng to be processed. The head of my pipeline now uses a ring buffer and a semaphore to help avoid creating thousands of network requests to fill the pipeline. The nature of the downstream elements of the pipeline is to process large volumes of data per item, which naturally results in their execution running slower than the source of the pipeline (hence the possibility of the active objects at the source filling up the buffers and killing the memory footprint.

Through careful steps I have managed to produce a highly performant parallel data pipeline without incurring much resources at all. My original design worked well in every way apart from the two key areas above. I could execute multiple concurrent searches and download and process a lot of data, but CPU utilization was around 80%. By breaking the design down, understanding the finer low-level elements of real-time programming, I have arrived at present invariants that allow my system to download the same data as before at nearly 3x the speed, using no more than 3% CPU. I’m still working on the GC issues, but thats not bad going.

I think there is something important to consider that supports the 40 hour week “guideline”. In a team that is energized and running at full-steam, the amount of things going on at any time could be quite large, with the whole team busy coding, designing or talking. Even when a team member is working on something, they will be actively engaged in ensuring they are aware of what is going on around them while they go about their business. Many people who truly engage with pair programming say they like it, but it can be draining. When we work alone, it is easy to take a “breather”, to sit back a little and reflect. But when a pair is engaged, theres no real breather to talk of – its a continual every rolling train of design ideas, forward thinking, code-reviewing and programming to carve the features into the system. When this process hums along like a spinning top, the energy required by the participants is greater than when working alone (certainly in my experience). This is all worthwhile because the idea is that the features that are being implemented are moving along at a productive steady pace and the quality of that work is second to none.

Even when pair programming is not used, there are other practices that actually support this steady train of relentless progress throughout the day, such as TDD (test driven development).  TDD as a practice and a design process can with experience become a profound tool to develop a system with, but its impact on the way its followers think and work, alone or in pairs, should be considered when reviewing the 40 hour guideline.  Just like a spinning top, TDD is difficult to get right, difficult to get it humming along.  But when it does, the level of delivery and progress can be just as profound as the practice itself, creating the well known “flow” state quite easily, and generating extended periods of high concentration and focused work – times this by at least two if this is being done with pairs.

Sometimes these periods of high productivity are like streaks of lightening across the sky, and the amount of progress in a working week might feel small and unproductive. Ignoring the human element that causes the daily mental, emotional and phyisical states of your workforce to fluctuate, a well running team should be able to accommodate and self-organize to help maintain its steady momentum, and the troughs should be less acute.  And this steady pace is built on the high focus developed from the XP practices and open working environment.  When things are truly rocking, it is not uncommon for me to arrive home absolutely exhausted.  But its a positive exhaustion, because its the effect of a causation that itself is positive.  I can wake up the following morning bright and early and be 100% ready for a day riding the development train, and this is part of the point behind the 40 hour guideline.  Happy people are more productive.  Being more productive further fuels happiness and well-being of the team, because they are riding on a wave of positive steady delivery.

The 40 hour week guideline is in the end just a guideline, and it should be appropriately changed to suite your environment.  But its a worthwhile principle of XP and other agile practices because if things are going well, progress will be steady and moving and theres no need for fever pitch.  If this isnt the case, if productivity is not as high and you’re thinking of increasing the hours to get more work out, it might be that other parts of the process arent working like they should be.  Increasing pressure to work longer hours might seem like the easiest approach, and in some cases it can be.  But there should be no need when the team has developed the steady pace.

I have raised quite a few points from just a small section of code. It is clear a failure to apply abstraction well has subsequently allowed the misplacement of system responsibilities within the AssignmentForm class. The business policy details belong not on a user interface class but on a domain class or something similar. The domain model has the role of representing the business domain, its relationships and its policies. Centralising business logic in the domain model keeps all of the important details in one place, and it should be through abstraction that these details are exposed to the rest of the software in a way that keeps the how hidden behind the what.

Building object oriented software is all about constructing the system from software artefacts that are fashioned to work with the forces of change. As has been mentioned, a good design axiom is to build against abstract concepts rather than concrete representations. This is achieved through the use of abstract types (abstract class or interface), which are used to form the semantic substructure on which the system implementation will be written. Whenever a piece of code is built against an abstract type, it is shielded from whatever may change behind that software construct. New requirements are added to the system by “plugging in” to the foundations of the of the design, the substructure. In our example the AssignmentForm is a client, and Vehicle and Customer are the abstract constructs that the AssignmentForm should depend upon. Actual requirements are built and evolved by plugging them into these existing design elements.

Figure 2

In figure 2 Vehicle and Customer form the semantic bond, each an abstract type of core concepts in the problem domain. Underneath these abstract types there is an ever changing world of evolving requirements being constructed and modified to plug into the abstract foundation. Here the new type CorporateCustomer is introduced as a specialization of the abstract Customer type, and new vehicles are added to the system in exactly the same way. With the client’s dependent only on abstract structures, they are oblivious to these types of modifications made to the system over time. Without this type of complexity management, it would be much riskier to add new functionality to the software.

Substructure Metaphor

I have found the notion of a substructure a useful metaphor to help demonstrate the importance of the abstract types within object oriented software. In construction the substructure makes up the foundations on which the main structure sits. The purpose of these foundations is to withstand the forces of nature, which if ignored can uproot and move the above superstructure with ease. In software construction these forces are the natural evolving requirements. Abstract types are like the footings and foundations that take the load of the rest of the system, and provide a structure on which the requirements can be implemented.

This metaphor should only be used to appreciate the relevance of the internal underpinnings of the design. It is wrong to apply the construction metaphor completely, because here the foundations are built first before any subsequent building can take place. From a software perspective this would require a full design up-front exercise to appreciate what needs to be in the foundations. However agile development principles show us that the construction metaphor falls down when taken literally, since we can build our software safely in an evolutionary manner that allows the substructure to slowly take place as the system grows.

What this all boils down to is the need to code against specifications, not implementations. The designer of the tracking system overlooked the essence of the Vehicle when he modelled it in the software. It seems the essence of the Vehicle is that it has a property named Assignee and a property named AssigneeName, both poorly naming the concepts they represent. The Vehicle has no important role in the system but to collate some data items. A change to the vehicle hiring or returning policy has a greater chance of affecting the AssignmentForm than it does the Vehicle class; this is clearly wrong.

The obscure names chosen for the Vehicle properties fail to capture the true domain, and I am certain I could not talk to the domain experts about Assignee’s and AssigneeName’s without there being confusion. The fix is to focus on mapping the outside world correctly to the inside world of the software, and we do this with the help of the ubiquitous language. Using a ubiquitous language it will be easier to reason about the domain while working with the code, because the domain and the code will the same.

Ubiquitous Language

In Domain Driven Design, Eric Evans presents a technique that bridges the knowledge transfer gap between the developers and the domain experts, the ubiquitous language. This language is built on collaboration with all team members to model the domain the software is being constructed for. The team continually exercises the ubiquitous language, both in conversation as well as in the code. This not only improves the communication and understanding between the domain experts and software team, but its manifestation as the core domain model in the system further reinforces the focus on domain concepts, not technical concepts.

Our next objective then (in part 4) is to introduce just a little bit of abstraction to see how well it affects the design. In the next section I will take the code through a number of refactorings, and my objective will be to improve abstraction, separation of concerns, and better represent the problem domain within the code. By better capturing the essence of the problem I will show how the domain can be modelled in the code to improve the current readability issues.

But age caught up with me – I couldnt run fast enough. However, check me out now – a little older, a few gray hairs, but I still know how to show the ladies a good time : http://www.youtube.com/watch?v=EumQLlJA1rc.

Ouch.

View the Public Entries – some weird, some stupid, some moving.

Consider the following stories:

1. Show ticking stock prices (3 points)
2. Show ticking index prices (1 point)
3. Show ticking hedge prices (1 point)

A possible way to estimate these could be to say that if we do the stock prices story first, it will take 3 points, and the rest should be much quicker, say 1. But theres a problem with this way of estimating – what happens if the customer requests to do the hedge prices story first? To make some sense we would probably have to change the estimates on the other two dependent stories. A second issue is that there is a smell here – if we are building just-enough-infrastructure, adhering to the 80/20 rule when building the code, we shouldnt have a “framework” at the end of the first story. What happens if after doing the framework to allow the other two stories to be implemented much faster, the customer requests to drop the other stories for other more important work?

Another possible way to deal with these dependent stories could be to pull out the technical aspects, create a story for those, and then stabilise our estimates on the actual three stories above:

1. Build ticking engine (3 points) // Technical Story
2. Show ticking stock prices (1 point)
3. Show ticking index prices (1 point)
4. Show ticking hedge prices (1 point)

I wouldnt take this approach any further for the simple fact it sides-steps all of the techniques we are trying to harness to help us deliver just enough software faster. The technical story not only has to be done first (which reduces the customers ability to prioritise based on business value), but after the technical story something could arise that would force the customer to drop the other ticking stories, leaving all of the technical “infrastructure” in place in the system. I have never liked this approach because it completely avoids the process of emergent design that would be used to build the software incrementally around the axis of these three stories.

In the end the principles I like to keep in mind are:

1. Simplicity
2. Individuality
3. Simple Design

Simplicity - keep the estimations and process of dealing with them simple. As soon as it starts to “feel” complicated, thats probably a smell.

Individuality - see work as individual pieces.

Simple Design – possibly one of the hardest principles to learn, but very important to grapple and gain experience with, since it is one of the key enablers to building high-quality software fast. In the original stories above I would see that I would need to build something to allow me to get some ticking in each of the stories. I would be thinking how I could bring something into existence that would be as simple and as lean as possible. This could cause me to lean towards estimating all around the same size, such as 2 or 1.5. Not a true fan of decimal estimations I would also consider just leaving all at 2.