Archive for the ‘ Beta ’ Category

Price Action Analysis

Over the past two years, I have spent many hours (many) making communion with the world of trading and investing.  I build software for a hedge fund, and currently work in HF and Options Market Making.  So I decided to take my financial knowledge to the next level two years ago, and immerse myself in everything related to investing and trading.  I have studied topics such as portfolio theory, CAPM, the efficient frontier, options, forex, futures, commodities, candlesticks, you name it.  I have built my own trading strategy platform and charting package, and I have actually been part time trading, and mostly I remain net positive with my trades.  But I have recently been trying to move deeper into the options trading space, and realized after a rather hit and miss trading record, that it ultimately starts and ends with price action.  Let me explain…

Continue reading

Resharper and Generics (or not)

I am finding quite a few issues with Resharper the more I lean on it.  Despite the issue of having to use Process Explorer to close my VS session when a bug occurs, there are other features that seem to be missing.  One thats been bugging me for some time is the inability for Resharper to help resolve a type if I declare a variable and specify it as a closed-type generic, when in fact the class isnt generic at all.  My scenario goes like this:

  1. I add a few assembly dependencies  to my project.
  2. I then think I have remembered a class I am wanting to use and declare it, such as FooBar<double>.
  3. Resharper turns FooBar red, which implies the class name isnt known.
  4. Problem is FooBar does exist in one of the assemblies that was added, but its not a generic.  Resharper reports: Cannot resolve symbol: FooBar.

To solve this I have to do Ctrl-N and look up the classname either in the browser (since I added assembly dependencies and not project dependencies, which would give me code-access), and then work out its not a generic.  While it doesnt take long to realise my error, I still expect resharper to offer to declare my variable as FooBar without the generic as one of its options that it provides to fix the problem.  As a productivity tool, this little feature would be very helpful.

Theres Method in the Madness

I once read theres no such thing as no communication.  One might not respond to something, but that response itself is communicating something.  Every act, be it to frown, to say something, to close your eyes, or to say nothing, is communicating something at some level.  This is very important.

So consider this.  Process is a dirty word.  So you and your team dont do process.  This is fine, and not uncommon by any means in certain industries.  But theres not such thing as no process.  Every act employed to bring software together, be it feature definition, bug tracking, testing, quality assuring, how these are done and when, is process at some level.  This is very important.

I think it is important to be aware of this, because it opens up the landscape of possibility and choices in evolving the way a team works and improves over time.  Metaphor lost its importance in the later version of XP, but there are people that believe strongly in its use, myself included, see here.  With metaphor our mental vocabulary is enriched, and a deeper understanding of the world can result.  And choice arises from the deeper understanding gained from employing new mental models.  If the number of choices increase, then hopefully the ability to have the answer increases.

The problem I see with the no process approach, is that it immediately shuts down a world of choice that exists that is tried and tested, and seen to lead to more successful projects.  All of this is lost, replaced by ad-hoc bespoking by teams that dont do process.  There will be meetings to ensure no or little process is introduced, yet its all process.  And if its all process, is it not better to choose from those that have a pedigree of success behind them? To spend time manufacturing a new approach to requirements gathering is ok if you are a process expert.  If you’re not a process expert then such time should be spent doing other things where true value can be added.

If process is a dirty word for your team, realize this habitual mental model may be inhibiting the chances of greater success.  There is not such thing as no process.  So choose the process elements you want to re-invent carefully, and for the rest, employ tried and tested principles that have been perfected so that you can get down to the real work of delivering software.

Looking at Arrays and Lists

Most of this ramble relates to .Net 2.0/C#.

To build or enhance a high performance managed application has a set of constraints, three of which are data structure choice, algorithm choice and the garbage collection envelope. An interesting discussion arises when talking about Array vs List<T>, but this can be quickly addressed by understanding the mechanics of both and where they differ. I use List<T> as the comparator simply because this is the common replacement of choice by most C# programmers I have come across.

Continue reading

Tuning your invariants

I’m currently building a pretty complex parallel computer that presently runs across multiple nodes in-process. The idea is to push the design to work across a grid of collaborating nodes, either as separate threads, processes or networked machines. I am currently working on two key areas:

  1. Limiting the number of garbage collections that are necessary.
  2. Achieving high performance in the multi-threaded core engine.

This week has been focused on both axis of performance, but recently I have been pushing my design to improve its scalability. Today I have managed to get the system to run an extreme number of networking downloads with a CPU consumption envelope remaining below 3%, which is pretty impressive. Most of this has come from harnessing IO completion ports as well as being extremely savvy with the use of threads provided by the .Net thread pool. So far I am doing well riding on the back of the .Net thread pool, but I may need to consider dedicated threads at some point – the thread pool scheduler can thrash a little in certain scenarios, but I’m not noticing that yet.

Due to the nature of this system, which I’ll explain more of soon, there can be multiple active downloaders retrieving online data from n sources. Downloaded data comes into the system and is processed via a sophisticated logical pipeline. An earlier naive implementation resulted in a saturation of the thread pool, or a profound number of context switches due to equally primitive locking strategies.

To limit or reduce garbage collections I have also needed to improve the types of data structures I am using, as well as understand better the invariants I need to uphold during runtime. For example, I have a pipeline that needs to have data live and in transit as much as possible, to utilize resources as best as can be achieved, whilst not saturating the CPU or increasing the working set pointlessly. Processors on the pipeline work at different rates, but this means that at various points in time, there could be too much waiting in the pipeline, which stresses the memory footprint.

Part of the solution comes from throttling inputs to the pipeline, and using non-collectable data structures (buffers) to handle the task items waiitng to be processed. The head of my pipeline now uses a ring buffer and a semaphore to help avoid creating thousands of network requests to fill the pipeline. The nature of the downstream elements of the pipeline is to process large volumes of data per item, which naturally results in their execution running slower than the source of the pipeline (hence the possibility of the active objects at the source filling up the buffers and killing the memory footprint.

Through careful steps I have managed to produce a highly performant parallel data pipeline without incurring much resources at all. My original design worked well in every way apart from the two key areas above. I could execute multiple concurrent searches and download and process a lot of data, but CPU utilization was around 80%. By breaking the design down, understanding the finer low-level elements of real-time programming, I have arrived at present invariants that allow my system to download the same data as before at nearly 3x the speed, using no more than 3% CPU. I’m still working on the GC issues, but thats not bad going.

40 Hour Guideline?

For some the notion of a 40 hour week is too difficult accept, deeming it too short to really achieve anything. But the rationale for this working principle is built on sound thinking. Working long hours at fever pitch will not only stress a team out, but will induce mental fatigue to the point that silly mistakes are made, and the team will actually work slower and do less than if they were energised everyday and doing less. Obviously when a release is coming up or other such important deadline, then the amount of work the team does must be as flexible as possible, and working more should be acceptable.

I think there is something important to consider that supports the 40 hour week “guideline”. In a team that is energized and running at full-steam, the amount of things going on at any time could be quite large, with the whole team busy coding, designing or talking. Even when a team member is working on something, they will be actively engaged in ensuring they are aware of what is going on around them while they go about their business. Many people who truly engage with pair programming say they like it, but it can be draining. When we work alone, it is easy to take a “breather”, to sit back a little and reflect. But when a pair is engaged, theres no real breather to talk of – its a continual every rolling train of design ideas, forward thinking, code-reviewing and programming to carve the features into the system. When this process hums along like a spinning top, the energy required by the participants is greater than when working alone (certainly in my experience). This is all worthwhile because the idea is that the features that are being implemented are moving along at a productive steady pace and the quality of that work is second to none.

Even when pair programming is not used, there are other practices that actually support this steady train of relentless progress throughout the day, such as TDD (test driven development).  TDD as a practice and a design process can with experience become a profound tool to develop a system with, but its impact on the way its followers think and work, alone or in pairs, should be considered when reviewing the 40 hour guideline.  Just like a spinning top, TDD is difficult to get right, difficult to get it humming along.  But when it does, the level of delivery and progress can be just as profound as the practice itself, creating the well known “flow” state quite easily, and generating extended periods of high concentration and focused work – times this by at least two if this is being done with pairs.

Sometimes these periods of high productivity are like streaks of lightening across the sky, and the amount of progress in a working week might feel small and unproductive. Ignoring the human element that causes the daily mental, emotional and phyisical states of your workforce to fluctuate, a well running team should be able to accommodate and self-organize to help maintain its steady momentum, and the troughs should be less acute.  And this steady pace is built on the high focus developed from the XP practices and open working environment.  When things are truly rocking, it is not uncommon for me to arrive home absolutely exhausted.  But its a positive exhaustion, because its the effect of a causation that itself is positive.  I can wake up the following morning bright and early and be 100% ready for a day riding the development train, and this is part of the point behind the 40 hour guideline.  Happy people are more productive.  Being more productive further fuels happiness and well-being of the team, because they are riding on a wave of positive steady delivery.

The 40 hour week guideline is in the end just a guideline, and it should be appropriately changed to suite your environment.  But its a worthwhile principle of XP and other agile practices because if things are going well, progress will be steady and moving and theres no need for fever pitch.  If this isnt the case, if productivity is not as high and you’re thinking of increasing the hours to get more work out, it might be that other parts of the process arent working like they should be.  Increasing pressure to work longer hours might seem like the easiest approach, and in some cases it can be.  But there should be no need when the team has developed the steady pace.

Applied Abstraction (part 3)

Quick Review

I have raised quite a few points from just a small section of code. It is clear a failure to apply abstraction well has subsequently allowed the misplacement of system responsibilities within the AssignmentForm class. The business policy details belong not on a user interface class but on a domain class or something similar. The domain model has the role of representing the business domain, its relationships and its policies. Centralising business logic in the domain model keeps all of the important details in one place, and it should be through abstraction that these details are exposed to the rest of the software in a way that keeps the how hidden behind the what.

Continue reading