July 16, 2014

What Level Should We Automate Most Of Our Tests At?

So this blog post has been a long time in the making. Well, a long time in the procrastinating, at any rate.

I have several clients who have hit what I call the "front-end automated test wall". This is when teams place greatest emphasis on automating acceptance tests, preferring to verify the logic of their applications at the system level - often exercised through the user interface using tools like Selenium - and rely less (or not at all, in some cases) on unit tests that exercise the code at a more fine-grained level.

What tends to happen when we do this is that we end up with large test suites that require much set-up - authentication, database stuff, stopping and starting servers to reset user sessions and application state, and all the fun stuff that comes with system testing - and run very slowly.

So cumbersome can these test suites become that they slow development down, sometimes to a crawl. If it takes half an hour to regression test your software, that's going to make the going tough for Clean Coders.

The other problem with these high-level tests is that, when they fail, it can take a while to pin down what went wrong and where it went wrong. As a general rule of thumb, it's better to have tests that only have one reason to fail, so when something breaks it's alreay pretty well pinpointed. Teams who've hit the wall tend to spend a lot time debugging.

And then there's the modularity/reuse issue: when the test for a component is captured at a much higher level, it can be tricky to take that chunk and turn it into a reusable chunk. Maybe the risk calculation component of you web application could also be a risk calculation component of a desktop app, or a smartwatch app. Who knows? But when its contracts are defined through layers of other stuff like web pages and wotnot, it can be difficult to spin it out into a product in its own right.

For all these reasons, I follow the rule of thumb: Test closest to the responsibility.

One: it's faster. Every layer of unnecessary wotsisname the tests have to go through to get an answer adds execution time and other overheads.

Two: it's easier to debug. Searching for lost car keys gets mighty complicated when your car is parked three blocks away. If it's right outside the front door, and you keep the keys in a bowl in the hallway, you should find them more easily.

Three: it's better for componentising your software. You may call them "microservices" these days, but the general principles is the same. We build our applications by wiring together discrete components that each have a distinct responsibility. The tests that check if a component fulfils its reponsibility need to travel with that components, if at all possible. If only because it can get horrendously difficult to figure out what's being tested where when we scatter rules willy nilly. The risk calculation test wants to talk to the Risk Calculator component. Don't make it play Chinese Whsipers through several layers of enterprise architecture.

Sometimes, when I suggest this, developers will argue that unit tests are not acceptance tests, because unit tests are not written from the user's perspective. I believe - and find from experience - that this is founded on an artificial distinction.

In practice, an automated acceptance test is just another program written by a programmer, just like a unit test. The programmer interprets the user's requirements in both cases. One gives us the illusion of it being the customer's test, if we want it to be. But it's all smoke and mirrors and given-when-then flim-flam in reality.

The pattern, known of old, of sucking test data provided by the users into parameterised automated tests is essentially what our acceptance test automation tools do. Take Fitnesse, for example. Customer enters their Risk Calculation inputs and expected outputs into a table on a Wiki. We write a test fixture that inserts data form the table into program code that we write to test our risk calculation logic.

We could ask the users to jot those numbers down onto a napkin, and hardcode them into our test fixture. Is it still the same test? It it still an automated acceptance test? I believe it is, to all intents and purposes.

And it's not the job of the user interface or our MVC implementation or our backend database to do the risk calculation. There's a distinct component - maybe even one class - that has that responsibility. The rest of the architecture's job is to get the inputs to that component, and marshall the results back to the user. If the Risk Calculator gets the calculation wrong, the UI will just display the wrong answer. Which is correct behaviour for the UI. It should display whatever output the Risk Calculator gives it, and display it correctly. But whether or not it's the correct output is not the UI's problem.

So I would test the risk calculation where the risk is calculated, and use the customer's data from the acceptance test to do it. And I would test that the UI displays whatever result it's given correctly, as a separate test for the UI. That's what we mean by "separation of concerns"; works for testing, too. And let's not also forget that UI-level tests are not the same thing as system or end-to-end tests. I can quite merrily unit test that a web template is rendered correctly using test data injected into it, or that an HTML button is disabled running inside a fake web browser. UI logic is UI logic.

And I know some people cry "foul" and say "but that's not acceptance testing", and "automated acceptance tests written at the UI level tend to be nearer to the user and therefore more likely to accurately reflect their requirements."

I say "not so fast".

First of all, you cannot automate user acceptance testing. The clue is in the name. The purpose of user acceptance testing is to give the user confidence that we delivered what they asked for. Since our automated tests are interpretations of those requirements - eevery bit as much as the implementations they're testing - then, if it were my money, I wouldn't settle for "well, the acceptance tests passed". I'd want to see those tests being executed with my own eyes. Indeed, I'd wanted to execute them myself, with my own hands.

So we don't automate acceptance tests to get user acceptance. We automate acceptance tests so we can cheaply and effectively re-test the software in case a change we've made has broken something that was previously working. They're automated regression tests.

The worry that the sum total of our unit tests might deviate from what the users really expected is mitigated by having them manually execute the acceptance tests themselves. If the software passes all of their acceptance tests AND passes all of the unit tests, and that's backed up by high unit test assurance - i.e., it is very unlikely that the software could be broken from the user's perspsctive without any unit tests failing - then I'm okay with that.

So I still have user acceptance test scripts - "executable specifications" - but I rely much more on unit tests for ongoing regression testing, because they're faster, cheaper and more useful in pinpointing failures.

I still happily rely on tools like Fitnesses to capture users' test data and specific examples, but the fixtures I write underneath very rarely operate at a system level.

And I still write end-to-end tests to check that the whole thing is wired together correctly and to flush out configuration and other issues. But they don't check logic. They just the engine runs when you turn the key in the ignition.

But typically I end up with a peppering of these heavyweight end-to-end tests, a feathering of tests that are specifically about display and user interaction logic, and the rest of the automated testing iceberg is under the water in the form of fast-running unit tests, many of which use example data and ask questions gleaned from the acceptance tests. Because that is how I do design. I design objects directly to do the work to pass the acceptance tests. It's not by sheer happenstance that they pass.

And if you simply cannot let go of the notion that you must start by writing an automated acceptance test and drive downwards from there, might I suggest that as new objects emerge in your design, you refactor the test assertions downwards also and push them into new tests that sit close to those new objects, so that eventually you end up with tests that only have one reason to fail?

Refactorings are supposed to be behaviour-preserving, so - if you're a disciplined refactorer - you should end up with a cluster of unit tests that are logically directly equivalent to the original high-level acceptance test.

There. I've said it.

July 9, 2014

What Problem Does This Solve?

It seems that, every year, the process of getting started with a new application development becomes more and more complicated and requires ever steeper learning curves.

The root of this appears to be the heterogenity of our development tools, which grows exponentially as more and more developers - fuelled by caffeine-rich energy drinks and filled with the kind of hubris that only a programmer seems to be capable of - flex their muscles by doing what are effectively nothing more than "cover versions" of technologies that already exist and are usually completely adequate at solving the problem they set out to solve.

Take, for example, Yet Another Test Automation Tool (YATAT). The need for frameworks that remove the donkey work of wiring together automated tests suites and running all the tests is self-evident. Doing it the old-fashioned way, in the days before xUnit, often involved introducing abstractions that look very xUnit-ish and having to remember to write the code to execute each new test.

Tools like JUnit - which apply convention over that kind of manual configuration - make adding and running new tests a doddle. Handy user-friendly test-runner GUIs are the icing on the cake. Job done now.

For a bit of extra customer-centric mustard, add on the ability to suck test data for parameterised tests out of natural language descriptions of tests written by our customers. We cracked that one many moons ago, when heap big Turbo C++ compilers roamed the earth and programmer kill many buffalo etc. Ah yes, the old "merge the example data with the parameterised test" routine...

Given that the problem's solved, and many times over, what need, asks I, to solve it again, and again? And then solve it again again?

The answer is simple: because we can. Kent Beck learns new programming languages by knocking up a quick xUnit implementation in it. Pretty much any programmer beyond a certain rudimentary ability can do it. And they do. xUnit implementations are the Stairway To Heaven of programming solutions.

Likewise, MVC frameworks. They demonstrate a rudimentary command of a programming language and associated UI frameworks. Just as many rock guitar players have at some point a few weeks into learning the instrument mastered "The Boys Are Back In Town", many developers with an ounce of technical ability have gone "Look, Ma! I done made a MVC!" ("That's nice, dear. Now run outside and rig up an IoC container with your nice friends.")

But most cover versions of Stairway To Heaven (and The Boys Are Back In Town) are not as good as the originals. And even if they were, what value do they add?

Unless you're embuing your xUnit implementation with something genuinely new, and genuinely useful, surely it's little more than masturbation to do another one?

Now, don't get me wrong: masturbation has a serious evolutionary purpose, no doubt. It's practice for the real thing, it keeps the equipment in good working order, and it's also enjoyable in its own right. But what it's not any good for is making babies. (Unless it's immediately proceeded by some kind of turkey baster-type arrangement.)

It's actually quite satisifying to put together something like an xUnit implementation, or an MVC framework, or a Version Control System, or a new object oriented programming language that's suspiciously like C++.

The problems start when some other developers say "Oh, look, a new shiny thing. Let's ditch the old one and start using this one that does exactly the same thing and no better, so we shall."

Now, anyone looking to work with that team has got to drop X and start learning X', so they can achieve exactly what they were achieving before. ("But... it's got monads...")

And thusly we find ourselves climbing a perpetually steepening learning curve, but one that doesn't take us any higher. I shudder to think just how much time we're spending learning "new" technologies just to stand still.

And, yes, I know that we need an xUnit implementation for x=Java and x=C# and x=Object Pascal and so on, but aren't these in themselves self-fulfilling prophesies? A proliferation of sort-of-similar programming languages giving rise to the need for a proliferation of Yet Another 3rd Generation Programming Language xUnit ports?

Genuinely new and genuinely useful technologies come by relatively rarely. And while there are no doubts tweaks and improvements that could be made to make them friendlier, faster, and quite possibly more purple, for the most part the pay-off is at the start when developers find we can do things we were never able to do before.

And so I respectfully request that, before you inflict Yet Another Thing That's Like The Old Thing Only Exactly The Same (YATTLTOTOETS - pronounceed "yattle-toe-totes"), you stop and ask yourself "What problem does this solve? How do this make things better?" and pause for a while to consider if the learning curve you're about to subject us to is going to be worth the extra effort. Maybe it's really not worth the effort, and the time you spend making it and the cumulative time we all spend learning it would be better spent doing something like - just off the top of my head - talking to our customers. (Given that lack of customer involvement is the primary cause of software development failure. Unless you've invented a tool that can improve that. and, before anybody says anything, I refer you back to the "sucking customer's test data into parameterised tests" bit earlier. Been there. Done that. Got a new idea?)

Brought to you by Yet Another Blog Management System Written In PHP That's Not Quite As Good As The Others

June 23, 2014

What's My Problem With Node.js?

So you may have guessed by now, if you follow me on The Twitters, that I'm not the biggest fan of Node.js.

Putting aside that it's got ".js" on the end, and is therefore already committing various cardinal sins in my book - the chief one being that it's written in JavaScript, the programming language equivalent of a Victorian detective who falls through a mysterious space-time warp into 1970's New York and has to hastily adapt to hotpants, television and disco in order to continue solving crimes - my main problem with Node.js is that it makes it easier to do something that most development teams probably shouldn't ought to be. Namely, distributed concurrent programming.

If programming is hard to get right, then distributed concurrent programming is - relatively speaking - impossible to get right. You will almost certainly get it wrong. And the more you do of it, the more wronger what it do be.

The secret to getting concurrency right is to do as little of it as you can get away with. Well-designed applications that achieve this tend to have small, isolated and very heavily tested islands of concurrency. Often they have signs on the shore warning travellers to "turn back, dangerous waters!", "beware of rabid dogs!", "danger: radiation!" and "Look out! Skeletor!" You know; stuff that tends to send right-minded folk who value their lives running in the opposite direction.

Node.js is a great big friendly sign that says "Come on in. Hot soup. Free Wi-Fi.", and it's left to salvage specialists like me to retrieve the broken wrecks.

So, yes, Node.js does make it easier to do distributed concurrency, in much the same way that a hammer makes it easier to drive nails into your head. And both are liable to leave you with a hell of a headache in the morning.

June 15, 2014

A Hippocratic Oath for Software Developers - What Would Yours Be?

Good folk who take the whole notion of a profession of software development seriously are fond of comparing us to medical doctors.

For sure, a professional developer needs to keep abrest of a very wide and growing body of knowledge on tools, techniques, principles and practices, just like a good doctor must.

And, for sure, a professional developer - a true professional - would take responsibility for the consequences of their decisions and the quality of their work, just like doctors must - especially in countries where patients have access to good lawyers who charge on a "no win, no fee" basis.

To my mind, though, what would truly set us apart as a profession would be a strong sense of ethics.

Take, for example, the whole question of user privacy: we, as a profession, seem to suffer from extreme cognitive dissonance on this issue. Understandable, when you consider that we're simultaneously users and creators of systems that might collect user data.

As users, we would wish to choose what information about us is collected, stored and shared. we would want control, and want to know every detail of what's known about us and who gets to see that data. Those of us who've had our fingers burned, or have seen those close to us get burned, by a lax attitude to user privacy, tend to err on the side of caution. We want to share as little personal information as possible, we want as few eyes looking at that data as possible, and we want to know that those eyes are attached to the brains of trustworthy people who have our best interests at heart.

What we've learned in recent years is that none of this is true. We share far more data about ourselves than we realise, and that data seems to be attracting the gaze of far too many people who've shown themselves to be untrustworthy. So, quite rightly, we rail against it all and make a big fuss about it. As users.

But, wearing our other hats, as developers, when we're asked to write code that collects and shares personal data, we don't seem to give it a second thought. "Duh, okay." seems to be the default answer.

I've done it. You've done. We've all done it.

And we did it because, in our line of work, we pay scant attention to ethics and the public good most of the time. At best, we're amoral: too wrapped up in the technical details of how some goal can be achieved for our employers to step back and ask whether it should be achieved at all. Just because we can do it, that doesn't mean that we should.

When was the last time your team had a passionate debate about the ethics of what you were doing? I've watched teams go back and forth for hours over, say, whether or not they should use .NET data binding in their UI, while blithely skimming over ethical issues, barely giving them a second thought.

And just because the guy or gal writing the cheques told us to do it, that doesn't mean that we must. Sure; one way to interpret "professional" is to think it's just someone who does something for money. But some of choose to interpret it as someone who conforms to the standards of their profession. The only problem being that, in software development, we don't have any.

So, if we could take such a thing as a Hippocratic Oath for software development, what would it be?

I suspect, given recent revelations, privacy might figure quite largely in it now. As might user safety in those instances where the software we write might cause harm - be it physical or psychological. You could argue that applications like Twitter and Facebook, for example, have the potential to cause psychological harm; an accidental leak of personal information might ruin someone's life. And then we're back to privacy again.

But what other ethical issues might such an oath need to cover? Would it have anything to say about - just off the top of my head - arbitrarily changing or even withdrawing an API that hundreds of small businesses were relying on? Would it have anything to say about having conflicting financial interests in a development project? Should someone who profits from sales of licenses for Tool X be allowed to influence the decision of whether to buy Tool X? And so on.

What would your oath be?

June 13, 2014

Real Long-Term Apprenticeships In The UK - We Found One!!! Are There More Like This?

So, I'm inspired again.

Someone sent me a link to this web page detailing software apprenticeships in the South West of England and in Wales for Renishaw Plc.

I think this might be the first genuine, long-term apprenticeship I've seen offered by a real, credible employer in the UK. You may know of other examples, but more about that in a minute.

Years of personal research into this had led me to conclude that the apprenticeships on offer - what little their were - simple were not credible. They tended to be short. They tended to be technology or vendor-specific. They tended to be, well, a bit "Mickey Mouse", preferring to skip the tough stuff and the disciplines of writing software under commercial constraints in favour of "cool stuff" that may play well in digital agencies but is a poor foundation for a resurgence in UK tech.

There are three things I'd look for in an apprenticeship:

1. Is it long enough and in-depth enough for such a complex discipline?

2. Does it lead to a degree or equivalent internationally-recognised qualification? (Let's face it, even the best devs - without a degree - can hit a glass ceiling.)

3. Does the employer pay for it - or at the very least, doesn't charge for it.

The Renishaw software apprenticeship appears to tick all three boxes.

I'm not aware of any other schemes that currently do. Perhaps you are?

I'd like to hear about it if you do, so I can compile an online resource for potential apprentices interested in learning to be a good software developer.

June 12, 2014

Next Public Test-driven Development Course, London July 19th

Just a short hoorah to mention I'll be running another public TDD workshop on Saturday July 19th, aimed at and priced for self-starters who self-fund their own training. (Though you're more than welcome to come if the boss is paying, too.)

Still an inflation-beating £99 for a packed day of craftsy goodness. Places are limited. Hope you can join us.

June 8, 2014

Reliability & Sustaining Value Are Entirely Compatible Goals

This is a short blog post about having your cake and eating it.

The Agile Software Development movement has quite rightly shifted the focus in what we do from delivering to meet deadlines to delivering sustainable value.

A key component in sustaining the delivery of value through software is how much it costs to change our code.

The Software Craftsmanship schtick identifies primary factors in the cost of changing software; namely:

1. How easy is it to understand the code?

2. How complicated is the code?

3. How much duplication is there in the code?

4. How interdependent are all the things in the code?

5. How soon can we find out if the change we made broke the code?

By taking more care over these factors, we find that it's possible to write software in a way that not only delivers value today, but doesn't impede us from delivering more value tomorrow. In the learning process that is software development, this can be critical to our success.

And it's a double win. Because, as it turns out, when we take more care over readability, simplicity, removing duplication, managing dependencies and automating tests, we also make our software more reliable in the first instance.

Let us count the ways:

1. Code that's easier to understand is less likely to suffer from bugs caused by misunderstandings.

2. Code that is simpler tends to have less ways to go wrong - fewer points of failure - to achieve the same goals

3. Duplicated code can include duplicated bugs. Anyone who's ever "reused" code from sites like The Code Project will know what I mean.

4. Just as changes can propagate through dependencies, so can failures. If a critical function is wrong, and that function is called in many places and in many scenarios, then we have a potential problem. It's possible for a single bug in a single line of code to bring down the entire system. We call them "show-stoppers". It's for this reason I dreamed up the Dependable Dependencies Principle for software design.

5. High levels of automated test assurance - notice I didn't say "coverage" - tends to catch more programming errors, and sooner. This makes it harder for bugs to slip unnoticed into the software, which can also have economic benefits.

So there's your cake. Now eat it.

May 30, 2014

From Where We're Standing, "Perfect" And "Good Enough" Are The Same Destination

This morning, I heard another tale of woe from a developer working a company that had hired a "quality manager" who goes around warning developers not to overdo software quality.

Demotivational posters have gone up with slogans like "Perfection is the enemy of Good Enough".

I make this point often, but I guess I'll just have to keep making it: for more than 99% of teams, there is no immediate danger of making their software too good.

Indeed, so far are the majority of teams from even nearing perfection, that - from their vantage point - Perfect and Good Enough are effectively the same destination.

Imagine the quality of your software is Miami, Florida. And let's imagine that Perfection for your software is Manhattan, New York.

"Good Enough" would probably be somewhere around Brooklyn, NY. That is to say, if you're in Miami, Florida, the answers to the questions "How do I get to Manhattan?" and "How do I get to Brooklyn?" are essentially the same.

Fear not perfect software. Just because you aim for it, it doesn't meet it will ever happen. But, for the vast bulk of development teams, falling just short of it could well put them where they need to be.

So, here's my demotivational poster about quality:

Aim for perfection, because Good Enough is on the way

May 27, 2014

Don't Say "Junior", Say "Trainee"

It's been a very rough coouple of weeks, so I'm just starting to climb back on the horse. But there was something I really felt needed commenting on, so here's a short post about the experience balance of development teams.

A typical model followed by employers in the UK is to hire one or two strong and very experienced developers and get them to lead a team mostly made up of less experienced and less skilled developers.

The problem with this pyramid model is that it just doesn't work. Writing software is complicated. It tales many years to become competent enough to make a decent fist of it at all. Per se, a competent software developer is not a "junior" software developer. They've been doing it, and learning about it, for quite some time.

We've probably all worked on teams where we've seen a couple of good devs scrambling on a daily basis to undo the damage caused by less skilled and/or less disciplined developers. The phrase I hear a lot in this context is anti-work. Far from adding value by throwing more and more warm-but-inexperienced bodies at a problem, it just makes it worse. Two strong developers working together on a problem would normally fare better than the same two strong developers trying to herd a team of half a dozen inexperienced cats. The hedring will take up most of their time - been their, done that - and the rest of their time will be taken up clearing up after them because they're not house-trained yet. (Okay, analogy over.)

It's understandable, in a world where most employers believe that 9 women can have 1 baby in 1 month, that hiring cheaper "junior" developers sounds like good value. But the reality is that it's actually very poor value indeed.

Better, in practice, to hire a smaller number of stronger developers who will get more valuable work done iinstead of wasting their talents on cat-herding and cleaning up the mess.

Ah, but how, then, do "junior" developers learn and become stronger developers?

This is where we turn the pyramid upside-down; smaller teams of stronger developers, mentoring one or two apprentices. So, there's less cat-herding and less litter-cleaning, but still there are the learning opportunities for our developers-in-waiting.

Financially, this model makes the most sense. It takes care of the now, by freeing up more time for good developers to do good development, but still makes room for the future by creating real-world learning opportunities for the good developers of the future.

On the practical side, this shifts the emphasis of what "junior" developers do from the day-to-day grunt work to a focus on learning. We seek good learning opportunities, and - because the "senior" developers are taking care of business themselves - we can prioritise and reward learning over delivery to begin with, so our trainees can focus on doing it right.

May 14, 2014

5 Years Of Codemanship

This month sees the fifth birthday of my training and coaching company Codemanship.

I created the company in 2009 to take advantage of the growing demand I was getting to run workshops and coach teams in what we're currently calling software craftsmanship.

I wanted to put into practice ideas that had been swimming around in my silly old head for several years, especially relating to how good development teams happen.

I firmly believe, based on my own experiences and on the available evidence, that good software development is a product of good software development cultures, and that culture happens bottom-up over a long period of time.

The boss cannot demand good development culture. Management edicts just don't work in this space. Nor can we expect it to happen overnight. You can't send your teams on a 2-3 day course and expect them to come back transformed.

Through Codemanship, I focus on developers and their habits and instincts, helping those who want to improve to gradually shift the patterns that are their default behaviour.

Our BBC TV Platforms case study illustrates how this can be done and what effect it might have, but there are many ways to skin a rabbit, and many routes to becoming better crafters of software. Over the last 5 years, I've worked with many different organisations and thousands of individual developers, and helped them to discover their own path, just as I once discovered mine.

Outside of the day-to-day work, Codemanship has also been busy in the community, and I've been blessed to be involved with some great events and initiatives, like producing a West End comedy show to celebrate the code-breakers of Bletchley Park, organising the original international Software Craftsmanship conference, and curating and contributing to the Music By Programmers album to raise funds for a programming club at The National Museum of Computing.

I'm most proud, though, of one extra-curricular activity; my apprenticeship with Will Price, a CS undergraduate who's working with me to help plug the gaps in his education on the practical side of "just how do we write software?" We're reaching the end of our first academic year - though there'll be much for us to do during the summer break - and I'm very pleased with how it's turned out so far. I'll be traveling the length of breadth of the land talking about it, and hopefully encouraging a few of you to try something similar.

Finally, just to say a huge thank you to all our customers, without whom none of this would be possible. I'm very lucky to have the chance to work with you all.

So, that's 5 years of Codemanship. So far, it's been the most fun, the most challenging and the most rewarding experience of my career, and I've got my fingers crossed that there'll be many more years of this to come.