what i still don't know

Is programming language choice a good thing?

2012-03-08T21:09:00.000Z

James Lewis and I did a ThoughtWorks QTB on programming language choice. You can watch the video here. I'm working on getting the scala code I mention in the talk online, it shows how you can use GPUs from within scala code via OpenCL.

Environment Cost: Are licensing costs damaging software quality?

2011-05-09T23:22:00.000+01:00

Are you unable to test your solution fully except in production? This could well be because your business can't afford to buy licenses for the same software as is in production to be installed on UAT, QA and Development environments. I've seen clients where things like web servers, application servers, messaging middleware and storage solutions just can't be the same as production because it's too darn expensive.
One reason this happens is because these kind of software purchases tend to be costed based only on production needs, most often this occurs when procurement sits off to one side away from the rest of the business. In fact in one organisation I saw a few years ago the development part of the organisation had to negotiate separately to buy the same software since procurement dealt solely with "operational" systems.
Frankly I also think it happens because some software vendors are hoping clients will have to come back and ask for lots more licences once they realise they need to start testing things. I've seen this happen too, although to be fair I also know of vendors who throw in test and dev licences for free. (of course some vendors will gladly sell you expensive test tools to test their expensive complex software, but thats a whole other story....)
The lack of a representative "production like" environment causes numerous problems, in fact most outages I've seen could have been prevented had something more representative of production been available. When I say representative I mean environments where the performance delta's between them are relatively easy to measure and model, that gets much harder when you are running different software or radically different hardware. I'm now of the view that if you can't afford to buy the licences for all of production, UAT, QA and development then you can't afford the software at all.
Once you go down the road of relying on some expensive software (or hardware) to scale things on production you are running increasing risks: an outage becomes more and more likely and you are letting a software or hardware vendor control the cost of scaling your business. I think it's much better to go with commodity hardware, and software where you choose what to pay for and when - this seems to be what works for the best known "internet scale" companies.
I think software licensing and hardware costs can damage software quality as they make scaling the ability to test software too expensive. It's just not enough to scale production, you have to have a cost effective way to scale the means of getting high quality software into producution as well.

Signal to Noise ratio in software testing

2010-12-15T16:06:00.000Z

When using techniques like automated functional testing we want a high signal to noise ratio, we want a failing test to tell us something has really broken and not just that we changed something.
Tracking this ratio of 'useful failures' to 'wasteful failures' gives us a signal to noise ratio for our tests. Ideally we want something like 10:1, say, so only every 10th failure is spurious and just due to a software change and/or fragility in our tests. Unfortunately for many teams this is more like 1:10, so most test failures are due to test fragility and not to something really being broken.
When we get too much noise and not enough signal we tend to start ignoring problems and disabling tests, while this might make the ratio better we do so at the cost of losing some of the signal. You might want to try tracking this ratio for a few weeks, tracking the trend over time can give a way to focus attention on eliminating areas of high noise.
My experience is the sources of noise are often related and are often due to things like timeout issues, hard coding id's or creating unnecessary dependencies on the order things get displayed or happen. Another common source of noise is teams working with very large backlogs of 'low impact' bugs, the issue here is more complex and probably worth a post on it's own - but when prioritizing bugs it is worth considering the impact they have on team productivity and not just the production impact.
Whatever the cause a relatively small effort can sometimes dramatically improve the signal to noise ratio.

Software that requires unnessecary constant internet connectivity

2010-03-10T17:04:00.001Z

I bought a logitech harmony remote a while back and last night needed to reconfigure it, so I fire up the config tool and sit watching connection errors. I finally followed a link to the support web site only to see "routine maintenance" was in progress. I'm left frustrated and wondering why a connection is always required to just reconfigure a couple of buttons on the remote. Why not cache some data locally and have an offline mode?

I had the same issue with my slingbox which I use to stream tv to another room, when I moved house I had no broadband for a week or so - I could not retune the slingbox without internet connectivity as this is required for authentication, again puzzlement and frustration.

Now in the last few days I've been reading about Ubisoft and issues people have had with their "always need to be connected" DRM solution, I'm very glad I've not bought the product involved.

Until the internet achieves utility status (like electricity or water) it seems a huge assumption and big risk to release products that rely on "always on" connections where an offline mode seems entirely possible. I think some packaging that says "Needs an internet connection" needs to be revised to say "Needs a constant internet connection to function at all"

To introduce an unnecessary single point of failure is in my book poor design or poor decision making process, especially where most of the impact of the associated risk ends up with the consumer.

Using Retlang for multi-threaded windows form code

2010-01-27T15:21:00.000Z

I've been a fan of Retlang for a long time and have been meaning to write up some of the ways I've used it. This is the first of 3 blog entries and in this one I'll describe a way of using Retlang to avoid the dreaded InvalidOperationException "Control accessed from a thread other than the thread it was created on".

It avoids lots of nasty boiler plate calls around InvokeRequired(), this isn't new and has been described before but what I want to show how it can also make testing easier and work with existing MVC patterns.

We are going to use Retlang to allow controller code and view code to communicate over message channels in a thread safe way.

Lets imagine a very simple implementation of MVC to illustrate the approach.

Here's the test, we are just adding numbers together and then displaying the answer in the view.

[TestFixture]

public class TestSimpleController

{

private Model model;

private SimpleView view;

private MockRepository repository;

[SetUp]

public void SetUp()

{

repository = new MockRepository();

model = repository.StrictMock<Model>();

view = repository.StrictMock<SimpleView>();

}

[Test]

public void testShouldDoSimpleAdd()

{

model.PushNumber(10);

model.PushNumber(5);

model.SumStack();

LastCall.Return(15);

view.DisplayCurrentTotal(15);

repository.ReplayAll();

var simpleController = new SimpleController(model,view);

simpleController.SendNumber(10);

simpleController.SendNumber(5);

simpleController.Sum();

repository.VerifyAll();

}

I'm using RhinoMock for mocking the view and the model. The controller implementation looks like

public class SimpleController

{

private readonly Model model;

private readonly SimpleView view;

public SimpleController(Model model, SimpleView view)

{

this.model = model;

this.view = view;

}

public void SendNumber(int i)

{

model.PushNumber(i);

}

public void Sum()

{

var result = model.SumStack();

view.DisplayCurrentTotal(result);

}

Now suppose we have to extend the implementation to deal with a long running task, we don't want to block the view thread as that is bad for user experience so we create a new thread to call the model from. Here is the controller code for that:

public void DoPrediction()

{

var thread = new Thread(InvokeModelWork);

thread.Start();

}

private void InvokeModelWork()

{

var result = model.LongRunningCalculation();

view.DisplayCurrentTotal(result);

}

It's worth noting here that a test for this code written in the same way as for our SimpleAdd will quite likely start failing at this point as asserts will be made on the calling thread before the "worker" thread has called the model and the view. Here is that naive and incorrect version of the test

[Test]

public void testShouldInvokeLongRunningCalc()

{

model.PushNumber(33);

model.PushNumber(44);

model.PushNumber(11);

model.LongRunningCalculation();

LastCall.Return(88);

view.DisplayCurrentTotal(88);

repository.ReplayAll();

var simpleController = new SimpleController(model, view);

simpleController.SendNumber(33);

simpleController.SendNumber(44);

simpleController.SendNumber(11);

simpleController.DoPrediction();

repository.VerifyAll();

}

This sort of test is a common problem in many code bases where the threading is not added until after complaints from the users about performance problems - this can create a lot of problems with the testing and is a very common source of bugs in MVC code. It is easy to tie yourself in knots using the mock framework to signal threads that a method was called or, even worse, using Thread.Sleep() to pause inside of the test. My experience has been that both of these are a source of "mysterious" test/build failures and are very hard to maintain.

Anyway even if we make our test pass we'll see the following when we try to run the code for real in a Windows Forms implementation of our view interface:

{"Cross-thread operation not valid: Control 'textBoxResult' accessed from a thread other than the thread it was created on."}

For completeness here is the very simple user control that implements the view.

public partial class SimpleControl : UserControl, SimpleView

{

private readonly SimpleController controller;

public SimpleControl()

{

controller = new SimpleController(new DoesMath(),this);

InitializeComponent();

}

public void DisplayCurrentTotal(int i)

{

textBoxResult.Text = i.ToString();

}

private void buttonSum_Click(object sender, EventArgs e)

{

controller.Sum();

}

private void buttonPredict_Click(object sender, EventArgs e)

{

controller.DoPrediction();

}

private void buttonSubmit_Click(object sender, EventArgs e)

{

var input = int.Parse(textBoxInput.Text);

controller.SendNumber(input);

}

So how can Retlang help?

Retlang lets us create channels which different threads can use to communicate. We can create one of these channels from the dispatch thread of a windows form class.

Here is the new version of the constructor for the controller:

private Channel<int> viewChannel;

public SimpleController(Model model, SimpleView view)

{

this.model = model;

viewChannel = new Channel<int>();

viewChannel.Subscribe(view.Fibre, view.DisplayCurrentTotal);

}

In this simple case we have one channel over which we will send int's and we also have have just one subscriber which is the view.DisplayCurrentTotal() method. We also ask the view to provide use with the Fiber to use, in this case this allows the view to provide a fiber that it knows will be safe to execute the subscriber method(s) on.

In more advanced cases you might want to have many channels or multi-plex different message types over the same channel and use the Dispatcher pattern to route those to the right subscribers on the view. I've done just that in a real example and used reflection to create mappings from each view method to the the correct message type being received from the channel.

On with our simple example; so what do Sum() and DoPrediction() look like now?

public void Sum()

{

var result = model.SumStack();

viewChannel.Publish(result);

}

public void DoPrediction()

{

var thread = new Thread(InvokeModelWork);

thread.Start();

}

private void InvokeModelWork()

{

var result = model.LongRunningCalculation();

viewChannel.Publish(result);

}

So instead of calling methods on the view we publish messages instead, in this case just an int.

What about the test? We'll need a few changes to SetUp as well, Retlang gives us a nice stubbed implementation of fiber we can use for testing.

[SetUp]

public void SetUp()

{

fiber = new StubFiber();

repository = new MockRepository();

model = repository.StrictMock<Model>();

view = repository.StrictMock<SimpleView>();

SetupResult.For(view.Fibre).Return(fiber);

fiber.Start();

}

[Test]

public void testShouldInvokeLongRunningCalc()

{

var finished = new ManualResetEvent(false);

model.PushNumber(33);

model.PushNumber(44);

model.PushNumber(11);

model.LongRunningCalculation();

LastCall.Return(88);

view.DisplayCurrentTotal(88);

// note the type of m below must match the parameter type of the above method

LastCall.Callback((int m) => finished.Set());

repository.ReplayAll();

var simpleController = new SimpleController(model, view);

simpleController.SendNumber(33);

simpleController.SendNumber(44);

simpleController.SendNumber(11);

simpleController.DoPrediction();

Assert.IsTrue(finished.WaitOne(1000),"Timed out");

repository.VerifyAll();

}

We use a manual reset event to allow us to wait for a particular event to occur, in this case the call to view.DisplayCurrentTotal() is followed by the use of a callback to allow this to happen. We then wait for the event (or a timeout) before calling VerifyAll(). So we can't avoid the fact that things happen asynchronously, but by using a message channel we make things explicitly so and importantly we have consistency across all the controller/view interactions. If we treat our test code as just code we can take advantage of this consistency to extract methods etc and this helps keep the code readable.

Here are the required changes for the view:

private readonly FormFiber formFiber;

public SimpleControl()

{

formFiber = new FormFiber(this, new BatchAndSingleExecutor());

controller = new SimpleController(new DoesMath(),this);

InitializeComponent();

formFiber.Start();

}

public IDisposingExecutor Fibre

{

get { return formFiber; }

}

We no longer need to worry about surrounding things with DispatchRequired() calls, our implementation of the view interface stays nice and clean. We also avoid having to create a view Proxy, this is another solution to the DispatchRequired() that requires a lot of repetitive boiler plate code.

I've tried to keep things simple for this example, some things to think about for a real world implementation are

That Methods on the view interface will be of the form void Method(Message msg)
Any methods on the view that do need to return something must be called from the same thread as created the form (but to be thread safe this must be done anyway).
Whether to multiplex multiple message types over the same channel and then use a Dispatcher to call the correct method on the view or to have a channel per message type
Whether to create different message channels for different styles of message flow and quality of service

I wonder if this is more Model Channel Controller as opposed to Model View Controller?

In the next entry I'll describe using a similar idea to allow high performance low latency updates of a view direct from a domain model by sidestepping the controller but without seriously compromising encapsulation. In the final entry of the 3 I'll describe how to use Retlang channels to efficiently share work across multiple worker threads and hence CPU core. The Retlang developers have done a great job and created a simple to use but highly useful library.

Ban The Debugger

2009-11-24T15:27:00.001Z

How much logging do you really need in your application? I was visiting a client recently to help diagnose some problems with one of their applications. I asked for their support people to send me the log file, seemed like a good place to start. I got a file of about 2.5K, that seemed kinda small to me. I opened it up and found just one exception and it's stack trace so I got back to the support people to check. Yep, that was it, logging cranked up to max and the only output was one stack trace. Not even the date and time of the problem. Wow! A technique I find very useful prior to go live is to Ban the Debugger. Developers get very used to just firing up the debugger when fixing issues or diagnosing problems. This is fine but it means that no one looks at the log files from the point of view of someone who only has those available to find out what is going on. For our colleagues in support and operations it is only the log files that they can use to find and fix issues. So during development prior to go live I stop the developers from using the debugger, instead I ask them to spend at least some time trying to fix the issue based solely on the log files and whatever else our friends in support will have available once we have gone live. This usually leads to a big upswing in the amount of logging and it's logging we know helps to fix issues. Of course sometimes you do need the debugger, but hopefully after we've used the logging to narrow down the problem area and to understand what the users were trying to do at the time. So back to the client above - it turned out the technical lead had never worked in support and that the support team had not really been represented during development. We owe our colleagues better than this, perhaps Banning the Debugger for a time during development might help.

Technology Lightning Talks in Chicago

2009-11-04T09:34:00.002Z

Some of my colleagues from the ThoughtsWorks Technology Advisory Board members will be delivering Lightning Talks next week in Chicago, I'm not speaking myself but will be attending. If you are in Chicago and would like to come along follow this link for registration information. I think it's a pretty great list of speakers and am looking forwards to hearing them myself.

Databases and Separation of Concerns

2009-11-02T16:57:00.002Z

A continuing source of pain on projects is Object Relational Mappings and databases. A contributor to this pain is the mixture of two concerns, this mixture seems to occur on nearly every project that uses a database. I think spending a moment to think about these two different kinds of usage is worthwhile. So what are these two concerns? A. Persist State Needed for Recovery This is just working state saved so that when the application restarts we can continue processing. For example perhaps the current state of a customer order or work in progress on a very long running calculation. B. Data saved for Reporting and Querying This is data saved so it can be queried later on, perhaps to allow end of month reports to be generated or tracking of user trends. It is not needed to recover the working state of an application. Many teams try to overload (A) to achieve (B), the sorts of problems this can cause are i) Data Volume - the volumes of data needed for (A) tend to be smaller, it's current working data as opposed to historical data. This can show itself as performance issues for the application as queries become slow over time. ii) The object design needed for (A) and (B) is not necessarily the same. This often shows itself as fields or object relationships being created with names like "history" or "recordOf", so an object design created for (A) becomes overloaded with things needed for (B). This again causes performance issues as the number of objects and data getting pulled into memory by the ORM can increase. It also means a simple state update can touch a lot of tables as we try to achieve (B) at the same time, chances are the indexes needed for historical queries can start to impact the update speed for these. iii) Confusing Code As with any other area where we fail to achieve separation of concerns the code can become confusing, for example state change operations become implicitly overload to create and persist data needed for historical reasons. iv) Archive of historical data becomes problematic, so you can't cleanly identify what data in the DB can be safely moved out to a historical database or deleted without impacting the functionality of the application itself. It wont always help but separation of these two concerns can provide clarity and address some kinds of performance issues. I certainly think it's worth calling out these concerns as different kinds of requirement even if you end up using the same implementation for both.

See you at JAOO?

2009-09-30T19:53:00.001+01:00

I'm very excited to be attending JAOO this year, I'll be hanging out at the ThoughtWorks stand so why not head over and say hello.

Great OS X app for those who change location a lot

2009-09-30T19:50:00.003+01:00

Finding this extremely useful http://www.symonds.id.au/marcopolo/ Lets you define rules to work out where you currently located based on all sorts of criteria and then define actions for that location: enable screen saver password, default printer, mount a drive, etc.

Incremental Internet Product Launch - responding to feedback over predicting needs

2009-09-02T13:09:00.002+01:00

I've spent a lot of time working with companies looking to launch offerings on the internet, I've been asked to get involved at various stages of the life cycle including starting over after the previous launch failed. There seem to be two main approaches to product development for the internet and interestingly these approaches often correlate with whether the company is a "traditional" company or not. By traditional I mean a company who previously sold and marketed products via non-internet means, the internet is something they've had to react to. So by implication the other kind of company is one that exists solely because of the internet, an "internet company" if you will, they are built around the web and have none of the more traditional retail sales channels. Traditional Companies The more traditional companies often seem to have a very fixed approach to product development, perhaps as a legacy of selling through offline means such as the high street? They go through a series of (often costly) exercises to try and fully describe a product offering, so to anticipate needs and to fully design a web product. There are usually a series of gates to pass through to get approval for budget, not least because the whole product has been designed and needs to be built so there is a lot of money involved. Product launches are costly infrequent events for this kind of company, usually with a lot riding on them. As the product offering has been designed upfront traditional methodologies are often used in delivery, no need for a more Agile approach when we already know what the users want. As such a ability to react quickly to change is rarely valued in this kind of product development with the real users often not involved at all. There is a high degree of risk around adoption with this sort of approach; will people actually want to use it? One technique often used to try to mitigate this risk is a big expensive marketing campaign and lots of publicity around the launch, although this in turn increases risks around performance problems and scaling at launch time. It also tends to push companies towards big upfront expenditure on fixed infrastructure sized for the peak demand. I'm sure we can all think of examples of products launched in a blaze of publicity only then to very publicly fail due to performance issues. Finally even if things go well and you do manage a good launch you've got a much bigger problem: non-traditional companies seem able to react extremely quickly and update or create competing offerings, often in weeks and not the months or years it took the traditional company. Internet Companies Internet companies seem to do things quite differently, perhaps because they've never had to sell products that had to be 100% fully formed at launch via traditional retails channels. They do something few traditional companies seem willing to do - they create and launch a product with an absolute bare minimum of functionality, just enough to hit the core proposition and nothing more. They then put that out into the wild and see what happens; often with minimum publicity but always with good feedback mechanisms in place. If no one picks up on the offering it can be just as quietly be canned, while money has been lost it was a small amount compared to a fully fledged product launch. In fact as this is a relatively inexpensive approach there is a chance to try out a whole series of ideas and do so quickly and often. If people do start to use it then the feedback mechanisms come into play and you can respond to that feedback to start incrementally improving the product. There is a very direct relationship with the users and their feedback here. You'll probably see a gradual ramp up in users as things pick up and can start to scale up numbers of machines etc. If you've used "cloud" like technologies from day one this is probably pretty easy to do. By necessity your delivery mechanisms have to be fast and quality has to be a fundamental part of the process: no time for a big 1 month long test period at the end, users need rapid response to their feedback for them to stay engaged with the product. Conclusion If you are a traditional company looking to go head to head with an internet company you may need to change how you approach internet product development. You'll be going up against companies that constantly and rapidly respond to user feedback. The key seems to be understanding the bare minimum you can launch with and then not being afraid of having a more direct relationship with end users. If someone is telling you that the bare minimum product definition "has to be" very large you need to be very sure some other company can't see a way to launch something simpler (faster) and just incrementally respond to their end user feedback.

Test Code Is Just Code

2009-04-22T17:58:00.003+01:00

Test Code is Just Code Some Anti-Patterns to avoid and some techniques for making sure Test Code doesn't slow you down Maintaining Test Code can become Costly One of the key objections to Test Driven Development is that the tests will become a barrier to change; that the maintenance and reshaping of the tests needed as new business requirements come into play will start to become prohibitively expensive. In fact there is evidence that this indeed happens, over time many Agile teams spend more and more time fixing tests that have been broken by new requirements as opposed to working on those new requirements. I've seen too many teams in exactly this situation and it usually due to difficult to understand test code that is hard and costly to maintain. We already know how to keep code in good shape This problem has, however, been solved and many high performing teams are able to introduce new functionality or make rapid changes to existing code without needing to start deleting tests or abandon the XP principle of TDD (Test Driven Development). The key is that they behave towards test code as they do any other code, Test Code is Just Code. All the things we have learnt about keeping production code in shape through high discipline, techniques like refactoring, and the use of patterns all apply to test code. And yes, sometimes this even means writing tests to make sure our test code behaves as expected. Common Traps I want to say more on why people seem to treat test code differently later but first I want to describe some of the common traps that I've seen people fall into and some ways of avoiding them. Here are five I see again and again, they can cause a lot of pain and I've met developers who have been put off TDD by falling into them. Cut & Paste The first one is cut & paste, for some reason when it comes to unit tests people suddenly start cutting and pasting all over the place. Suddenly you find a file with 20 tests each of which repeats exactly the same few lines of code. I don't think I need to describe why this is bad and I expect we've all seen the outcome: at some point later those tests all start breaking at the same time, if we are unlucky a few tweaks have happened to the cut & pasted code in each so we spend a lot of effort figuring out how to make each one pass again. There is no rule that says we can't have methods in test fixtures so ExtractMethod still applies, and using SetUp sensibly often helps. The same rational for avoiding cut & paste and the same solutions we know from production code apply to test code. Poor Encapsulation The classic example of a failure of encapsulation in test code is when you start to see database specific code creeping into every test, perhaps you have "Naked SQL" in your tests? I'm going to focus on the database here but the same ideas apply to other areas as well, for instance security, auditing and messaging. The pain this trap causes often shows itself at the worse time when you are optimising the database or trying sort out a referential integrity issues, suddenly the tests become a millstone around your neck and start breaking on a very frequent basis — because of Cut & Paste this can often be a very large number of tests. I've seen teams avoid addressing things like foreign keys in the database because it has become too difficult to get the test code to create and delete things in an appropriate order. The solution is to introduce proper encapsulation around the test code that looks after the database. The builder and fluent interface patterns are very useful here, if we can hide the implementation and separate concerns we can also often open to door to more advanced testing techniques. Of course you'll want to make sure that builder code is properly tested, that it really puts things into the database in the order you expect and deletes them properly as well, but that should be work you only need to do once. Bloated SetUp The next example is Bloated Setup, hopefully the name is self evident, you end up with a SetUp method in your test fixture that becomes huge. As with any bloated method it becomes code that is hard to understand and error prone. Perhaps changing the order of calls in SetUp fixes one test but causes 5 others to fail and the reasons are just not obvious? If you've seen this then you are probably suffering from bloated SetUp. This pain point has two main causes, the first is poor coding practice such as described above, and the second is the structure of the production code itself. It is often dependencies within the production code that force us to do too much work in SetUp, we have to create all the things each class under test requires and in turn their dependencies as well. The pain grows as SetUp code then gets Cut & Pasted into other test fixtures. As opposed to the first two examples the solution here lies not just into our approach to the test code, we also need to look at the way the production code is structured. The first step is to make sure we use a well understood pattern such as DI (Dependency Injection) to express and understand the dependencies in the production code. The second step is to make use of stubbing or mocking techniques in our test code so that we can isolate the class under test and exercise it without needing to instantiate the whole tree of dependencies. A lot has been written on mocking and DI already, some of the best articles are here & here. Mocking has its own traps not least of which is only having the real wire up of all the components done for the first time in production (more later) but used well it is an invaluable technique for keeping test code easy to maintain and understand. Too hard to write the Test The previous pain point described a situation where the solution lay in looking at both our test code and the production code, this one is about testing pain that comes entirely from the production code. Finding a test hard to write can tell us a lot about the production code and a pain point people often refer to is "it's just too hard to write a test for that". My view on this is that code that is hard to test is poorly written code. To write a new unit test, and hence make a change to make it pass, the code we are testing needs to be easy to read and understand. If we can't work out which class to write the test against that means we have poor separation of concerns or a lack of coherence in our production code. If at first you can't write the new test then refactor the production code until you can, it's the production code that is at fault and not the TDD technique. Code Integration Test Pain The last example I want to talk about is integration testing. As this is such an overloaded term I first need to describe by what I mean by code level integration testing. In essence it is making sure that when we wire up all the components in our solution they behave as expected together. If we've used patterns like DI, a DI framework such as Spring and a test technique such as Mocking then we need to make sure we are fully testing the wired up solution as well, unfortunately this can get forgotten. Often we end up with a special test wire-up class or a Bloated SetUp doing the wire-up job, either way there is a bug waiting to happen if the production wire-up code never gets called until actual deployment. In essence SetUp for our code integration tests ought to be calling the production wire-up code. I often hear that this is impossible because the production wire-up code is hard wired to a specific environment. If that is the case the team is probably heading for trouble, too many teams don't spend enough time thinking about how to handle configuration for multiple environments such as Dev, QA and Production. If you can solve that problem first then exercising the full production wire-up code should not be an issue, just point it at the appropriate environmental configuration. Why do people treat test code differently? Test Code is only throw away code if the production code is throw away. I think people often think of test code as throw away, that once in production it becomes superfluous and so there is little use devoting time to keeping it in good shape. When people say this to me I like to ask if they plan on ever changing the production code or doing another release, the answer is always yes. Test Code is an intrinsic part of the solution, like the launch tower for a rocket it provides an essential stepping stone in the solution and one that it's worth expending time and effort to keep in good shape. There are very few engineering disciplines where regular monitoring and testing do not form part of ongoing maintenance activities, in fact I can't think of a single one. We are lucky with software because if we look after our tests they are a very low cost solution to making sure things can be evolved with minimal risk. We ought to treat testing as an intrinsic part of software engineering instead of a one off 'gate keeper' activity or a problem for the QA team to solve . Test Code is as much a part of a modern software solutions as the production code itself.

Words of wisdom from Ron Jeffries

2009-02-25T12:25:00.002Z

Via Patrick Logan Think you are doing Scrum and/or XP? Then read this. My take away is that yet again people want to latch on to a silver bullet instead of working out what is really wrong with their organisation.

Acceptance Testing Silverlight with white

2009-02-25T09:25:00.004Z

I've used White to acceptance test WPF applications a few times so was pleased to discover it just works with Silverlight. It seems that IE passes the usability api calls into the contained Silverlight app so to an outside caller the app just looks like extra controls within IE. There were a few things I had to make sure were in place though 1. You need to make sure you start IE in a way White can 'find', so if you just pass the URL as the executable name IE (or your default browser) will start but White will not then be able to find that application and window. Obviously you'll need to alter the values to match your install and set up. const string internetExplorer = @"C:\Program Files\Internet Explorer\IEXPLORE.EXE"; const string url = "http://localhost:1138/"; const string windowTitle = "silverlightApp - Microsoft Internet Explorer"; var ie = new ProcessStartInfo(internetExplorer, url); var application = Core.Application.Launch(ie); var window = application.GetWindow(windowTitle, Core.Factory.InitializeOption.NoCache); 2. Make sure you set the automation name property. <TextBox Name="textBox1" Text="some text" AutomationProperties.Name="textbox"/> 3. Use the same Name when you try to find the control. var textBox = window.Get("textbox"); That is it. You'll need to make sure the names you use don't clash with the names of the IE controls, I'd suggest using GUI Spy for that but I've not been able to find a working link to download that tool for about 6 months now. I'm using IE version 6.0 and Silverlight 2. I've not tried this with any other browsers.

The Tyranny of corporate licences

2009-02-17T09:35:00.003Z

I often hear enterprise architects say "we can use product X as it is free", as I know X is actually quite pricey I ask "wow, XYZ is giving you X for free?" To which they reply "No, we have a corporate license and that comes out of someone elses budget" So this is a special kind of free which I'd not come across before, one that actually means "free, as in we don't care about the cost to the company as a whole". This sort of individualism is revealed as doubly corrosive when you talk to the developers who find product X is way more complex than they need and actually increases development costs by requiring specialist skills and damaging developer productivity. This disconnect between people who actually have to use software and those who make the purchasing decisions is not at all unusual. The fact there is a corporate licenses leads to some nasty behaviors as well, it is argued that the software is "free" when someone raises cost as an objection and it is argued as "highly expensive" when someone questions how much value it really adds (i.e. an erroneous equation of expensive and high ROI). The other things that gets thrown into the mix is standards - I just ask people go and look if the standards are being driven and defined by companies that make big complex enterprise software products. They can then draw their conclusion on how much value those particular standards have and whether they truly help avoid lock in. All of these things allow an easy defense for those purchasers who failed to exercise due diligence in their decision and create a a situation that is almost impossible for the people who have to use those products day-to-day to challenge. I suggest projects that use a license pay a proportion of its costs and any loss (or gain) the developers see is carefully tracked. Before you comment you may want to check that your friendly enterprise software vendor hasn't placed you under a gagging order, no really, I'm not joking.

problems with ./configure and make on osx

2009-02-03T15:14:00.003Z

I've been trying out various AMQP implementations, most still require a build from source on OSX. So the usual ./configure make I'm sure others have seen this error before /bin/sh: line 0: cd: someDirNameHere: No such file or directory

More of a reminder for me than anything else, the solution is usually

export CONFIG_SHELL=/bin/bash
./configure
make

Five kinds of technical debt

2009-01-05T09:43:00.001Z

Martin has described a concept called Technical Debt and this seems to have entered the vocabulary of our projects. It is a useful term but I worry it is being abused somewhat. It seems that the term is being used to described a multitude of evils. I've found it very useful to try and categorise the different kinds of debt. In fact I'd go further: I think it is dangerous not to understand the type of Technical Debt we are dealing with and, even more importantly, what the root cause of the Debt was (or continues to be). The other thing I've seen with technical debt is that can be transparent or opaque. Transparent debt is 'on the books' so to speak, we incur it knowingly and the team and the client are part of the decision. Opaque debt is off the books, either developers don't realise they are incurring debt or the team decides, for whatever reason, to hide the debt from the client. Anyway here are five different kinds of "Technical Debt", I'm sure we won't all agree with this list and it certainly isn't meant to be exhaustive but I do think there is merit in better categorising and understanding these issues: Ignore Skip doing something we know is really needed. So take a quicker path ignoring, say, the need to build something horizontally scalable. Of course there are sensible ways to do this, such as splitting a story into two parts and playing another part later. Unfortunately this doesn't always happen, instead developers sometimes take a unilateral decision to ignore a requirement and hence incur debt that is not visible to the team as a whole (at least not until later). I've heard people use 'yagni' to justify this behaviour. Get Surprised Your client comes up with some new requirement that means a change in direction we cannot easily accommodate with existing code. This often shows itself as a 'code base of two halves': pre and post change of direction. We may live with two ways of doing things in the code base and this might never become debt at all, or at least not until some cross cutting requirement forces the issue. Disregard some data A classic example of this is build time, or more specifically the time it takes to run all the unit tests. Projects teams often see this time starting to creep up but it gets ignored until such time as it starts to really impact productivity. They are slowly incurring more debt and as is the nature of debt the longer you wait to repay it the tougher it gets. Repeat a mistake We incur Technical Debt, often without even knowing it, when we repeat previous mistakes. The one I see the most is Hibernate hiding from the developers that there is a database is involved, to the extent they forget about the cost of a database call. I think this one could also be called 'ignore advice'. Either way we storing up trouble. Guess This usually happens in relationship to a non-functional requirement, such as performance, and again it is largely hidden incurrence of debt. Basically we just guess that something will work as expected and don't bother to check , we just assume it will be ok. It is easy to avoid guessing either by using a spike or just a 'back of the envelope' calculation. And finally... Phantom Debt This is 'technical debt' as invoked by a developer who has decided they don't like part of the code base and hence want to rewrite it. Technical Debt certainly sounds better than 'egotistical refactoring session' ;-) (Many Thanks to all my colleagues for the internal discussion and feedback I got for the first version of these ideas, no doubt we will continue to argue about this)

Don't delete unit tests even if you have acceptance test coverage

2008-08-08T10:56:00.005+01:00

I think it is well understood that the sooner you find a defect the less it will cost. A defect costs the most once into production whether that be a car that needs to be recalled or a software bug that stops your companies orders being processed. If you use TDD and CI then you get to find some defects at the development stage where it is clearly a lot cheaper to fix them than finding those same bugs in QA, UAT or worse still in production. What is really odd then is that people delete unit tests and while there can be legitimate reasons such as code no longer being used this is not why most tests get deleted. Some bogus reasons for deleting tests are a) "They keep failing when I change things" - Well that's because you just broke some existing functionality! And guess what: it's a lot cheaper to fix that defect now at development time than once it gets into production. Of course maybe someone else has to deal with production bugs so you could just be selfish and ignore the cost to them and your company and go ahead and delete that failing test. b) "They are very hard to maintain" - This is more insidious and is a symptom of something that actually happens quite a lot: developers forget that tests are code too. So for some reason once developers are working on unit testing all those things we know are bad practice are allowed. Unsurprisingly the end result is poorly factored code that is hard to maintain. So should you delete those tests? No! You should do the same as you would when you find hard to maintain code anywhere else in the codebase (that is slowing you down), after all test code is still just code. c) "I have coverage of that code from acceptance tests" - So what if you have coverage of that code from a higher level acceptance test? Is it ok to delete the unit test then? I think the answer must still be no and the reason is to do with the cost of finding and fixing a defect. A while ago I was working on a big complex distributed application with great acceptance level tests but relatively poor coverage around the messaging layers. These acceptance tests failed frequently and developers would spend a lot of time trying to track down the problem. We added units tests to flag those issues much earlier and in a far clearer way. So while we were adding test coverage to code that was already tested we did so to lower the cost of finding and fixing defects. Deleting test code should only be done if it will not increase the cost of finding a defect.

Patterns for Concurrency over Concurrent Language Features

2008-07-24T13:07:00.005+01:00

The Multi-core Problem I continue to see lots of posts and articles about multicore and rightly so, multicore CPUs present some real challenges for the software industry. I think by now most people get it: by default most software is written in a way that can only make use of one CPU. If I buy a new faster CPU that code will run faster, but if I buy an additional CPU or a multi-core CPU then that code will not run faster. This is because it will use only one of the available CPUs, the rest will sit idle, in fact it might even run slower as the individual cores in a multi-core CPU often run slower than a single core CPU. At the moment this problem is masked to some degree, if you've a 2 CPU machine you probably run enough pieces of software at the same time such that, along with the operating system, both cores are kept reasonably busy. When we have 8, 16 or 32 cores the problem will be a lot more evident. I'm guessing the issue will really surface when large organizations start to upgrade their desktop machines and find out that instead of running faster, as has always been the case before, the software on those machines actually runs no faster or even slower. Nor a good return on their investment. Evolution or Revolution A lot of people are suggesting ways in which we get around this problem, the approaches advocated seem to fit broadly in two categories. Evolutionary - We need to learn how to use existing threading libraries and features properly from within existing programming languages. Revolutionary - We need new to start using different programming languages to write our code (i.e. Erlang) What surprises me is the concentration on programming languages and language level features, I believe both approaches will fail unless we look at the architecture and patterns we use to deliver software. Evolutionary Lets look first at the evolutionary approach. Many people are recommending we learn how to use threads, locks, semaphores et al correctly but this is no use if you just try and bolt multithreading onto patterns you would choose for a non-concurrent design. All to often everything is designed and written ignoring the need for concurrency with the hope the bottlenecks can be tackling by adding a 'little bit' of concurrency later. To give a concrete example consider the observer pattern, all too often this is implemented in a way that means the order that the observers are invoked is fixed. A common approach to 'bolting on' concurrency to a non-concurrent design is to service each observer on a different thread. In fact this often looks likes its working when tested on a single core machine (where the order is deterministic) but soon becomes a source of bugs once it's running on a multi-core machine and the order that the observers are called can change. Another example is taking work from a queue, why not just have multiple threads consuming from the queue? If you've designed for this you'll be fine, but often that does not happen and it turns out that when the order items are taken from the queue is changed the outcome changes i.e the cancellation of an order overtaking the creation of that order. So the cancel is processed first and discarded (the order does not exist yet!) and then the order creation arrives..... Concurrency is also something that is hard to compartmentalise and isolate to one part of software, the impact of concurrency tends to be pervasive. So you need to design for multi-core from the beginning and choose an architecture and patterns that create a suitable abstraction around the concurrency required for your problem domain. You may then find someone has already created or documented approaches and patterns that meet your need and you never actually need to start creating your own threads, locks, etc. This is where choosing from existing concurrency libraries comes in. Revolutionary So what about the revolutionary approach? I think here it's even clearer we need to change the patterns and architecture we use. I really hope people don't think that by just coding in Erlang, say, that software will magically scale to multiple cores. What's different about Erlang is that it has language features that provide really great support for a particular architectural approach. If you ignore those features and just write sequential code and design the solution as you always have you should not be surprised when it doesn't scale to many cores. Patterns So the evolutionary and revolutionary approaches are both valid if we adjust the architectures and the patterns we use to take account of concurrency. If we do that we'll create software that can better scale to multiple cores. If we try and ignore concurrency when making design decisions and instead expect languages or language features to just solve the problem for us we'll be no better off than we are today. There is a reason we don't normally write assembly code directly, we have much higher level abstractions and languages. Using threads directly is not that dissimilar to assembly language, it directly exposes the underlying hardware level implementation. Therefore we should not be surprised that using threads directly is difficult and often unproductive, we need higher level abstractions for concurrency. Some languages come with features that make using certain patterns for concurrency much easier to implement than others, Erlang is the obvious example. Other languages have libraries that provide implementations of those same patterns, for example Retlang. Other people have written code that scales incredibly well across multiple CPUs in C and C++ as well. It's not about the language, it's about having a design that takes account of concurrency and then choosing patterns that provide suitable abstractions around it. Conclusion When it comes to software we have multiple patterns available and we try and choose the best ones for the problem at hand. We need to develop and use similar patterns to help use with concurrency and in fact many exist already. Many of the patterns described by http://www.enterpriseintegrationpatterns.com/ play equally well at the software level as they do the message level, many patterns in EDA (i.e. Event Collaboration Patterns) also lend themselves to concurrent solutions. There are plenty of other examples. Its here at the level of patterns we should be focusing our efforts around solving the multi-core problem and not at the language level.

Tools: Enablement or Control

2008-07-22T13:16:00.005+01:00

I've used a lot of tools during my software development career and it seems they fall primarily into two categories - tools that seek to control and tools that seek to enable. Enabling Tools

Concentrate on addressing one key issue rather than the whole development life cycle
Don't dictate a particular work flow or way of working
Are often written by people who use them
Require a degree of trust between end users (so someone can ignore the agreed approach if they want)
Tend to be quick and easy to use
Are open and easy to integrate with other tools
Are adaptable to many situations and ways of working

Leading to: The ability to adapt to changing situations and constantly refine process leading to incremental improvements in productivity over time

Controlling Tools

Come with a built in vision on how a whole series of activities should be done
Dictate work flow and the ordering of activities
Assume no trust between end users (you must follow the tool's defined approach)
Tend to be complex and hard to use
Only work in one situation, you change to work their way or you spend all your time fighting the tool
Are often closed, proprietary and hard to integrate with other tools

Leading to: A lack of flexibility and consequently much frustration when trying to respond to changing circumstances, ultimately this constrains the productivity of the team

Another thing I've noticed over the years is no two projects are alike and no one approach fits every situation. If you don't have trust and good working relationships among your team members then the tools are the least of your worries. So I prefer enabling tools over controlling tools - I'd much rather trust people and be flexible than constrain both people and the process.

Marketing Driven Development

2008-07-21T19:14:00.004+01:00

Marketing Driven Development is a process where the features and priorities for a new release of software are driven primarily by a marketing department. So a feature is valued more for the ability to create a marketing campaign around it over any usefulness or value that it may have for an end user. Marketing Driven Development is primarily a reaction to the fear of commoditization. It's consequences often include

Well known bugs (or to use the correct term: sub-optimal feature) remain for many versions as fixing them is never a priority
Usability does not improve as this is rarely seen as valuable by marketing and anyway it was sold as easy to use the first time around
It tends to be expensive to buy, maintain and customize i.e. marketing driven development does seem to allow the avoidance of commoditization.

.net: A fluent Facade around System.CodeDom

2008-07-17T19:24:00.002+01:00

I've been doing quite a lot of work with System.CodeDom recently (more on exactly what later) and have frankly found the api quite hard to use. So I've found myself growing a fluent interface facade around the api, for example the following code creates a class with one method that always returns the integer four.

  [TestFixture]
  public class TestsForWriteUp
  {
      [Test]
      public void testShouldCreateClassWithOneMethod()
      {
          ClassBuilder builder = new ClassBuilder(new ResolveTypeFromAssemblies());
          builder.AddMethod("theMethod", typeof (int)).Number(4).Return();

          Compiler compiler = new Compiler("CodeGenTests.dll");
          ReturnsAnInt dynamicallyBuildCode = compiler.compile<ReturnsAnInt>(builder);
          Assert.AreEqual(4,dynamicallyBuildCode.theMethod());
      }
  }

  public interface ReturnsAnInt
  {
      int theMethod();
  }

So I get hold of a classbuilder, add a method to it, then add the number four and finally return. The reason things look a bit back to front is that the facade is using a stack under the covers, so what I'm really saying is

Push a statement (a constant integer of value four) on to the stack and then
Pop whatever is at the top of stack and return it.

Finally I take the builder and dynamically compile the contents of the builder into an assembly, in this case in a way that gives me back an instance of a particular type so I can immediately start working with. (I can also save the output to a dll if needed) The new ResolveTypeFromAssemblies() injected into the ClassBuilder allows it to resolve any types that are used, in this case it just checks the loaded assemblies. The "CodeGenTests.dll" passed into the compiler lets it know the generated code depends upon that assembly (i.e. for the interface ReturnsAnInt). Actually that first example is not very interesting, how about a slightly more complex example:

       [Test]
       public void testShouldCreateAddMethod()
       {
           MethodBuilder methodBuilder = builder.AddMethod("theMethod", typeof (int));
           methodBuilder.DeclareParameter(typeof (int), "a").DeclareParameter(typeof (int), "b");
           methodBuilder.VarUsage("a").VarUsage("b").BinaryOperator(CodeBinaryOperatorType.Add).Return();

           TakesTwoArgs takesTwoArgs = compiler.compile<TakesTwoArgs>(builder);
           Assert.AreEqual(3, takesTwoArgs.theMethod(1, 2));
           Assert.AreEqual(9, takesTwoArgs.theMethod(4, 5));
       }

       public interface TakesTwoArgs
       {
           int theMethod(int a, int b);
       }

Still pretty simple, declare a method, add two parameters, sum them and then return the result. Again note the back-to-front invocation style. This is considerably less code than you'd need to accomplish the same thing using the System.CodeDom classes directly. The next example shows the use of a field, a constructor with an argument and then how to create an instance using that constructor. Hopefully much more terse than CodeDom but still easy to understand.

        [Test]
        public void testShouldCreateConstructorAndField()
        {
            string fieldName = "seed";
            builder.DeclareVariable(typeof (int), fieldName);
            builder.AddConstructor().DeclareParameter(typeof (string), "input")
                .FieldUsage(fieldName).VarUsage("input").UnaryOperator(typeof(int),"Parse").VarAssignment();
            builder.AddMethod("theMethod", typeof (int)).FieldUsage(fieldName).Return();

            Parameter argument = new Parameter(typeof(string),"42");
            ReturnsAnInt returnsAnInt = compiler.compile<ReturnsAnInt>(builder,argument);
            Assert.AreEqual(42, returnsAnInt.theMethod());
        }

Again things look back to front, if you read the line ending in .VarAssignment() from right to left you'll probably find it clearer to see what is happening. The reason for the use of the stack is that it greatly simplifies the underlying code, in effect I'm using Reverse Polish Notation Finally here is the c# version of the code that is compiled:

namespace GeneratedCode {
    using System;
    using CodeGen;
    
    
    public class DynamicallyCompiledCode : CodeGen.CompiledBase, CodeGenTests.TestsForWriteUp.ReturnsAnInt, CodeGen.ICompiled {
        
        public int seed;
        
        public DynamicallyCompiledCode(string input) {
            this.seed = int.Parse(input);
        }
        
        public virtual int theMethod() {
            return this.seed;
        }
    }
}

I'd be interested in comments, is this something people would find useful? It's grown out of a formula compiler, a dynamic proxy generator and a compiler compiler so is reasonably complete. I'll be posting more about the compiler compiler and how it is using this facade around CodeCom to bootstrap itself soon. (btw I never meant to write the compiler compiler, it just grew out of some work I've been doing around DSL's)

Cutting the Cost of Change over Just Cutting Cost

2008-04-23T11:46:00.002+01:00

Many businesses talk about cutting cost, especially at the moment, and IT is often an easy target. The problem with focusing on just cutting cost is that you can increase your future cost of change. The most obvious example is when you let people go who are experts in your legacy systems, you certainly cut cost as you are not paying their salary any longer - but at the same time you've just constrained your ability to deliver changes to those legacy systems. Now if a business has been focused on cutting cost for a while chances are that any initiatives to pull functionality out of a legacy systems (where change might be costly) into newer systems (where, hopefully, changes are cheaper) will not have happened. So you've been cutting opex, which funds the ability to change existing systems, and often also cutting capex which funds the creation of new systems or the movement of existing functionality into systems that require less opex. So the end result of cost cutting is that the cost of change will increase over time, you are removing capability to change systems while at the same time restricting the ability to replace those systems. Eventually you reach a point where cost of change is high enough to mean you can't make the business case for even small changes. At that point you might decide to invest in replacing the legacy systems that have become so expensive to change - but chances are that you no longer understand what those systems do and that it's beyond the capacity of the business to fund replacing them. Even if you can afford to replace them what do you do in the mean time, it might takes months or years to build the replacement. So what is the alternative? One might be to focus on cutting the cost of change as opposed to just cutting cost. So incrementally invest to move functionality off legacy systems into new systems or to keep software in a reasonable shape in the first place. Consider viewing systems as things that constantly evolve as opposed to things you grow to a certain point and then throw over the fence into support. Think about architectural approaches that reduce coupling and dependencies between systems. These are just suggestions and there are a lot of other things you could do, chances are the people on the ground will have some good idea of where to start. If you can cut the cost of change that means you can get more done for less money, which is cutting your cost. Focus on the cost of change may well be the thing to measure and to drive down over time, as opposed to just cost per se.

Great post on "Bad Behaviour" in projects

2008-04-08T11:45:00.001+01:00

These all ring very true.

Will cost per transaction kill your business?

2008-03-19T10:17:00.005Z

Imagine a business, Major Widgets Plc, that makes money by selling things in bulk, they sell a batch of 20,000 widgets on to a high street retailer who then sells the individual units. Each time Major Widgets sells a batch it costs them $1.5 to process that sale, as the overall cost of a batch is $20,000 that is less than 1 tenth of a percent cost overhead, pretty good. As they only sell 1000 batches per month they also process all the orders at the end of month in a legacy batch processing system. They are feeling pretty pleased with themselves. Now imagine a new business, Overlooked Upstart Inc., who also sell widgets. They do something rather different though, they sell widgets directly to the consumer as individual units. They currently sell only around 10% of the total number of widgets per month of Major Widgets, so instead of 1000 transactions per month this means they have to do 2 million. It actually costs them $0.001 to process each transaction and as they charge only $1 per widget that is actually more cost overhead in percentage terms than Major Widgets. Overlooked Upstart also process each transaction as and when it happens; as their business is spread reasonably over the month that is about 66000 per day or around 47 per minute. They run this system on a couple of commodity blade servers and know they can scale up to a few hundred orders per minute without too much cost. They too are feeling pretty pleased with themselves because..... ...in this hypothetical universe it turns out consumers really like the convenience of buying widgets online directly without the high street retailer taking a cut. Major Widgets sees a downturn, the retailers only want 15,000 widgets per month, this is a problem and if it continues they'll be in serious trouble in a few years time. So Major Widgets decides to launch online sales themselves and set the goal of having 10% of total widget sales online. The first challenge is that all the orders are currently keyed in by hand at the rate of about 30 per day, there is no electronic way to capture the orders at the required rate of 66000 a day without hiring a few new people(!). The second challenge is that the legacy billing system can only run on the last Monday of the month within a 4 hour time window - which comes out at around 140 transactions per second (around 130 per second faster than their current system can do). The harsh truth is that they cannot currently compete selling individual widgets due to their high cost per transaction and aged infrastructure. Ok, not the end of the world, they just need to spend money on a few new systems. But it turns out it's not that simple, they'd been seeing IT as a inconvenient cost for some time and over the past few years have been cutting both operational and capital spending as well outsourcing the support as part of a multiyear contract. To make things worse all the people who knew how all the systems were tied together have left, the person who wrote the cobol transaction processing system retired and all the web developers have been poached by some company called Overlooked Upstart Inc. Major Widgets come to realize that they are best off being taken over by someone who has infrastructure with low cost per transaction and high scalability........ Of course this is a simplified made up example and I'm lucky enough to work with clients who see IT as an integral part of what they do, they are well aware of their legacy systems and what online offerings can require in terms of scaling and cost. However I am still amazed to hear of some companies who talk as if the internet is going to go away and that IT is just a cost to be controlled. What are they going to do when someone new moves into their market? This isn't really a new phenomena either, market de-regulation in equity trading led to a massive up turn in volumes that put some small brokerages out of business. I'm sure there are other examples as well.