October 16, 2014

Dear Aunty Jason, I'm An Architect Who Wants To be Relevant Again...

"Dear Aunty Jason,

I am a software architect. You know, like in the 90's.

For years, my life was great. I was the CTO's champion in the boardroom, fighting great battles against the firebreathing dragons of Ad Hoc Design and the evil wizards of Commercial Off-The-Shelf Software. The money was great, and all the ordinary people on the development teams would bow down to me and call me 'Sir'.

Then, about a decade ago, everything changed.

A man called Sir Kent of Beck wrote a book telling the ordinary people that dragons and wizards don't actually exist, and that the songs I'd been singing of my bravery in battle had no basis in reality. 'You', Sir Kent told them, 'are the ones who fight the real battles.'

And so it was that the ordinary people starting coming up with their own songs of bravery, and doing their own software designs. If a damsel in distress needed saving, they would just save her, and not even wait for me - their brave knight - to at the very least sit astride my white horse and look handsome while they did it.

I felt dejected and rejected; they didn't need me any more. Ever since then, I have wandered the land, sobbing quietly to myself in my increasingly rusty armour, looking for a kingdom that needs a brave knight who can sings songs about fighting dragons and who looks good on a horse.

Do you know of such a place?

Your sincereley,

Sir Rational of Rose."

I can sympathise with your story, Sir Rational. I, too, was once a celebrated knight of the UML Realm about whom many songs were sung of battles to the death with the Devils of Enterprise Resource Planning. And, I, too, found myself marginalised and ignored when the ordinary folk started praying at the altar of Agile.

And, I'm sorry to say, there are lots of kingdoms these days where little thought is given to design or to architecture. You can usually spot them from a distance by their higgledy-piggledy, ramshackle rooftops, and the fact that they dug the moat inside the city walls.

But, take heart; there are still some kingdoms where design matters and where architecture is a still a "thing". Be warned, though, that many of them, while they may look impressive from the outside, are in fact uninhabited. Such cities can often be distinguished by gleaming spires and high white walls, beautiful piazzas and glistening fountains, and the fact that nobody wants to live there because they built it on the wrong place.

You don't want to go to either of those kinds of kingdom.

Though very rare, there are a handful of kingdoms where design matters and it matters that people want to live in that design. And in these places, there is a role for you. You can be relevant once again. Once again, people will sing songs of your bravery. But you won't be fighting dragons or wizards in the boardroom. You'll be a different kind of knight.

You'll be fighting real battles, embedded in the ranks of real soldiers. Indeed, you'll be a soldier yourself. And you'll be singing songs about them.




October 15, 2014

Software Craftsmanship 2015. Yes, You Heard Right.

After much discussion and beer and brainstorming and beer and then more beer, I've decided that there will be a Software Craftsmanship 2015.

It's going to be a little different to the previous 4 SC conference, in some key ways:

Firstly, not only will there be no presentations (as usual), there will be no sessions at all. How is this possible? Well, the fun in SC20xx was always in the doing, so SC2015 will be 100% doing. Instead of proposing sessions, I'll be asking people to devise problems, games, challenges, projects - fun stuff that folk can pick up and pair with other folk on for as much of the day as they want. We'll provide the space, the Wi-Fi and the sausages (yes, there will be sausages). You provide the laptop and your brains and enthusiasm.

Each mini-project will have a point to it. This won't be just some random hackathon. They'll be devised to exercise some of your code crafting muscles, hoppefully in entertaining and interesting ways. There'll be a handout that explains it, links to useful resources, and the person/people who devised the challenge will be nearby and easily identifiable, ready to answer questions and give feedback. They will,, of course have completed the challenge themselves beforehand. SC20xx has never been home to people who only talk the talk.

So, with a good wind behind us, you will learn something. But in your own time and at your own pace. How you structure your day will be entirely up to you. We'll open the doors, say "howdy" to y'all and then point you to the rooms/spaces where different kinds of challenges can be found, and off you go. There'll be scheduled coffee breaks and lunch and wotnot, naturally, but your day is your own. Want to gra a challenge and then go sit outside with a chum and do it on the lawn? Dandy.

You can spend an hour on each challenge you like the look of, all all day doing the same challenge, or do a 1-hour challenge several times, with different people, wearing different disguises. It's entirely up to you. The estimated time needed will be clearly sign-posted, so you don't pick an 8-hour project thinking it's going to take half an hour (or vice-versa.)

In the evening, as is the tradition for SC20xx, there'll be diversions, and sausages. Or sausages and diversions, depending on your priotities. Or diverting sausages, if that's your sort of thing.

Of course, there will be discussions. As well as the usual technical stuff we like to natter about, I'm going to provide "Conversation Cards" - yes, exactly like in Monty Python's The Meaning Of Life - on the theme of Developer Culture, Ethics & Diversity. If you want to find a space and have a debate, feel free. If you want to write a blog post based on one of the cards, fill yoour boots. If you just want to chat about it over coffee, all good. if you don't want to talk about it at all, no pressure. Nothing's compulsory.

Blog posts, GitHub wotsits, photos, videos and all manner of outputs from the day will be aggregated, collated, assigned to pastel-coloured categories and then fed to the huddled masses who are starving for a bit of quality.

It's likely to happen in the Spring, when the weather's warming up (I'd put my money on May). We'll be fieelding proposals for challenges within the next few weeks, and all materials will need to be finalised a month before the event. I shan't call it a "conference", because that's the one thing it's not going to be.

So, there you have it: Software Craftsmanship 2015 (#sc2015), May-ish 2015, very probably at Bletchley Park, very probably all profits from ticket sales will fund programming clubs or something along those lines.

Any questions?



October 8, 2014

Programming Well (Part II)

This is a short series of blog posts aimed at teachers and students who are interested in progressing on to more challenging programming projects. Here we explore the disciplines programmers need to apply to make it easier to add new features and change existing features in our programs, which ultimately makes it easier for us to incorporate our learning and make our programs better.

------------------------------------------------------------------------------------------------------------

In my previous post, we looked at why it can be very helpful to be able to test (and re-test) our programs very frequently, and how automating those tests can enable this.

We also looked at why it's very important to write code that other programmers can easily understand.

In this post, we'll talk about how we break down our programs into smaller units, and how we can organise those units to make changing our programs easier.

3. Compose programs out of small, simple pieces that do one job each

Most modern programming languages give us mechanisms for breaking down our code into smaller, simpler and more manageable "chunks". This enables us to tackle more complex programming problems without having to consider the whole problem at once, and it also enables us to make chunks of our code reusable, so we don't have to keep repeating ourselves, and ultimately we write less code.

This principle is applicable at multiple levels of code organisation:
* Blocks of code can be broken down into functions that do a specific job
* Groups of related functions - functions that are used together to do a job - can be split out into their own modules
* Groups of related modules - modules that are used together for a purpose - can be packaged together into a unit of release (e.g., an executable program file, or a code library)

If our goal is to make change easier, breaking the code down into cohesive units like this - units of code that are used together, and are likely to change together - helps to localise the impact of changes.

We balance the need for cohesion in functions, modules and packages with our need to not repeat ourselves, reusing code whenever we get the chance to minimise how much code we have to write.

And it's important that each unit of code has only one reason to change. If I write a function that does two distinct jobs, I lose the ability to change how one of those jobs is done without affecting the other. I also lose the ability to reuse the code that does each of those jobs independently of the other.

Let's take a look at a coupe of specific examples, one at the function level, another at the module level:



In this example, the Java function has two distinct jobs: it reads the scores from a file, and then it calculates the total score.

This presents us with several potential barriers to change.

Firstly, this is harder to test. If I write an automated test for this function, it could fail for a whole bunch of reasons (wrong file path, data in file not formatted correctly, total calculated incorrectly, and so on.) So if anything breaks, I'm probably going to end up in the debugger trying to figure out what actually went wrong.

It also prevents me from changing how either of these jobs is to be done without potentially affecting the other. It's all in the same function, and functions either work as a whole or they don't. Break one part, you break all of it. Ideally, functions should only have one reason to fail.

It's also a barrier to reuse. Maybe when I want to calculate a total, I don't always want scores to be read from a file. Maybe when I want scores read from a file, it's not always because I want to calculate their total. I might want to reuse either of these chunks of code independently of the other.

ON DUPLICATION OF CODE: Reuse is how programmers remove duplication from their code. Instead of, say, copying and pasting a block of code when we want new code to do something similar, we can create a function that accepts parameters and reuse that instead, and we can group functions together into reusable modules, and modules into reusable packages, and so on. The benefit of doing this is that wehn we need to make a change to duplicated logic, we only need to make that change in one place. If we copied and pasted, we'd have to change it everywhere we pasted it - effectively multiplying the effort of making that change.

When I refactor this code so that each function has a distinct responsibility - only one reason to fail - we can see how the code becomes not only easier to test, but easier to change independently and to reuse independently, which makes accomodating change even easier going forward.



In this refactored version of the code, these two distinct jobs of reading the scores from a file and adding up the total are delegated by the original function to two new functions, each with a distinct responsibility.

The function totalScore() is now just composed of calls to these other functions that do the actual work. totalScore doesn't know how the work is done, and reading scores and totalling scores know nothing about each other. As we'll see, composing our program in such a way that all the different bits - all playing their particular role - know as little about the details of how each other does their work becomes very important in making change easier.

Also notice that the two comments in the original function have disappeared? I took the opportunity to incorporate the useful information in the comments into the new functions I created. The original function now pretty much just tells the story that the comments told us. This is an example of how breaking our program down can also help us to make it easier to read and understand.

Which brings me to our second example, and...

4. Hide the details of how work is done, and make it so we can swap those details easily

In our refactored example above, the details of how scores are read from a file, and how those scores are totalled, are hidden from the totalScores() function. This gives us a degree of flexibility, allowing us to compose our program in different ways (e.g., we can total scores that weren't read from a file, or read scores for some other purpose.)

This flexibility - the ability to compose programs in differet ways while making only small changes to the code - can become a very powerful tool for making change easier. The more ways there are to compose the working units of our program, the more things we can make our program do with less effort.

In object oriented programming, using languages like Java and Python, we can go a step further in achieving greater flexibility.

In our example, even though we now have reading scores and totalling scores neatly separated, there is still rigidity in our code. What if I want to read scores from different sources? Let's say, for example, that our program is intended to be used in different settings - sometimes a desktop application that stores the scores in a file on the hard drive, sometimes a web application that stores the scores in a database like MySQL or Microsoft Access. How could we organise our code so that we can actually choose our strategy for storing scores without having to change a lot of the code?



Refactoring the code further, I can move the function that reads scores into its own class (an object oriented module from which we create instances - "objects" - that do the work), and the function that totals the scores into another class. Each class has a distinct job - a single responsibility.

Then - and this is the key to this trick - I set up an interface for each of those classes, and make it so that our RobotScoreboard only knows about the interfaces. I do this by passing in instances of the reader and the totaller in the constructor - a special function that we call when an object is created to properly initialise it before can use it - referencing only the interfaces when I do. So RobotScoreboard has no knowledge of exatly what kind of object will be doing the work. All it knows is what functions it will be able to call on these objects, which is all the interface defines.

If I wanted to use my RobotScoreboard with scores stored in a MySQL database, I write a new class that implements the same ScoresReader interface, and pass an instance of that into the constructor instead.

This gives me the ability to substitute different implementations of ScoresReader and Totaller without having to make any changes to RobotScoreboard, and to compose the program in many different ways.

This is something that's very difficult to do in programming languages that don't have a built-in mechanism for making it possible to subsitute one implementation of a function or a module with another without changing the code that uses those them. Which is why we very highly recommend that on more challenging programming projects, you use languages that support what we call polymorphism (that is, the ability for functions or modules to look the same from the outside but work differently on the inside).

DESIGNING FOR TESTABILITY: Another thing worth noting in this example is that, by introducing this extra level of flexibility, it now potentially becomes easier to write unit tests for our RobotScoreboard. Recall from the previous blog post how important it can be that we test our program as frequently as possible. For this to be possible, our tests need to run quickly. Reading data from files, or from databases, takes considerable computing resources, and tests that read files or depend on databases tend to run slowly. In our refactored version of the code, it now becomes possible for me to write a fake implementation of the ScoresReader - one that just builds a list of scores in program code, rather than reading it from a file - purely for the purposes of our tests. Making our programs easier and quicker to test automatically is another great benefit of organising our code in this sort of way.

Inexperienced programmers often have a problem with organising their programs into small, reusable, subsititible chunks like this. They see lots of new little functions and new modules and say "It's more complicated". But if you look closely, at the individual "chunks", you see that they are each simpler. when we lump all our code together into gib functions that do lots of things, and big modules with many, many functions, we lose a great deal of flexibility.

The key is to keep the individual parts as simple as we can, and to compose our programs out of them in such a way that we can more easily swap parts with new or different parts with less impact on the rest of the code.


In the next blog post, we'll look at how we manage our code and how we can more effectively work with other programmers on the same programs.





October 7, 2014

Programming Well Part I - A Basic Guide For Students & Teachers

This is the first part of a series of blog posts aimed at new programmers who have gotten the hang of the basics and want to progress on to more challenging projects, as well as teachers and mentors who may be helping people to progress as programmers. I'll add more installements as time allows.

-----------------------------------------------------------------------------------------------------------------

If you're a teacher working with the new computing curriculum, then you probably have a lot on your plate.

But there are some things I would like you to think about on the subject of programming that, as far as I'm aware, aren't covered in the curriculum. I believe they should be, because they're actually very important things every programmer should know.

Once a programmer has learned enough to write simple programs that successfully compile and run, and kind-of, sort-of work, they are ready to move on tackling more complex problems and more ambitious projects.

This is where it all can get, well, complicated. Programming is like Jenga. Programs are easy to start, but can be very difficult to keep going.

As the code gets more complicated, we run into difficulties: in particular, it gets harder and harder to make any additions or changes to the code without breaking it. You may be surprised to learn that changing code without breaking it is the predominant focus of most programmers.

And if there's one thing we know about software, it's that it is going to change. That's why we call it "software". We write a program that we think solves our problem. Then we try to use it, and inevitably we learn how we could do it better. Then we go back and change the code to incorporate what we learned. It's one of the main reasons why we often wait for "Version 2.0" of a program before we consider adopting it. When change is difficult, software doesn't improve.

And you might also be surprised how quickly you can run into difficulties. I've run workshops where code attendees wrote in the morning was holding them back in the afternoon.

For these reasons, professional programmers understand only too well the importance of programming well, even on what might seem like trivial projects.

What is "programming well"?

Let's break it down into a handful of simple disciplines (simple to understand, not so simple to master, as it turns out):

1. Test Continuously

By far the most empowering tool a programmer can have is the ability to find out if a change they just made broke their program pretty much straight away. If we break it and don't find out until later, it can take much more time to debug. For starters, we're no longer working on that part of the code, so it's not fresh in our minds. We've got to go back, read the code and understand it again - a bit like loading it into main memory - before we can fix it. And if we've made a bunch of changes since then, then it can be much harder to pinpoint what went wrong.

For this reason, we find it's good idea to test the program as frequently as we can - ideally we'd want to re-test it after making a single change to the code.

Doing this by hand over and over again can be very laborious, though. So we automate those tests. Basically, we write programs that test our programs. This practice is very highly recommended. And it can be done in pretty much any programming language - Visual Basic, Python, C#, Java, FORTRAN, C++, you name it. If we can call functions we've written in our program, we can write code that drives our program - using those functions - and asks questions about the outcomes.

There are special frameworks that can help us to do this, and they're available for most programmig languages (e.g., PyUnit for Python is a "unit testing" tool that lets us automatically test discrete chunks of our software).

Automated tests, at the end of the day, are just programs. If the kids can write programs, they can write automated tests for those program. Simples.

Here's an example of an automated unit test for a Java program:



Tests have three components:

* A set-up - that initialises the test data or objects we intend to use for the particular test case we're interested in. In this example, we set up a DartsPlayer object with an initial score of 501, then take 3 throws, each scoring treble 20.

* An action - this is the code that we want to test. In this case, we're testing that throwing darts correctly changes the player's current score

* An assertion - we ask a question about what the output or outcome of the action should be in that particular set-up

Try it. I suspect you may find it makes a huge difference to how easy the going is, especially on more ambitious projects.

2. Write code other programmers can understand

Here's a fun fact for you: what do programmers spend most of their time doing? Reading code. It's estimated we spend between 50%-80% of our time just reading programs and trying to understand them. Understanding programs is the bulk of the work in programming.

And I'm not just talking about other people's code either. I've spent many a joyful hour looking at code I wrote - months ago, weeks ago, yesterday, before lunch - and wondering what the hell I was thinking.

Most of the programming we do is on programs that already exist. We're adding new features, changing existing features, fixing bugs, making it run faster or use less memory or disk space... There are all sorts of reasons why we spend most of our time wrestling with existing code. Add other programmers (hell is "other programmers"), and that effect magnifies to epic proportions. Working on projects with our friends, or collaborating with programmers out there on Teh Internets, makes readable code a very high priority. We're not just communicating with the computer, we're communicating with each other.

The best kind of code clearly tells the story of what the code does.

Look at this very simple example:



At first glance, it may not be entirely obvious what this code does. Programs give us an opportunity to make them more self-explanatory by choosing more meaningful names for the things in our code - functions variables, constants, modules and so on. If I refactor this code to make it more readable ("refactoring" is the art of improving our code to, for example, make it more readable, without changing what it does), I might end up with something like:



Some people - silly people - will try to convince you that the way to make programs easier to understand is through the use of comments. In prcatice, we find comments are often less than helpful. Firstly, they clutter up the code, which can make it even harder to read. And, like all documentation, comments can get out of step with the code being described. Comments can be an indicator that the code is difficult to read, and we've learned that it's better to try and tackle that issue directly rather than rely on comments or other kinds of documentation in all but the most extreme cases.

In the next blog post, we'll look at how we break our programs down, and a couple of simple principles for organising our programs that can make them easier to change.






October 4, 2014

How I Deal With "Bugs" On Payment-On-Results Projects

Those of us who do bits and bobs of development work for clients, as I occasionally get asked to do, know how important it can be to get absolute clarity on what it is we've agreed I'm going to deliver.

I'm a big fan of getting paid on results, rather than just being paid for my time and expenses.

The upside is that my time remains my time; I don't respond well to clients attempting to manage it for me, especially as my weeks can get complicated in that respect with multiple concerns for me to address for multiple clients.

The potential downside is that I don't deliver what the client thought we'd agreed I'd deliver. And so extra effort and care needs to be put into how we agree these things.

Driving design with executable acceptance tests can help enormously in this respect. Books like Gojko Adzic's Specification By Example explain expertly how acceptance tests can lead us all to a clear shared understanding of what the software must do.

On fixed price - or payment on results - development deals, bugs can be a contentious issue. My preferred way of billing is to exclude the cost of fixing bugs. They are, after all, my failure.

But sneaky clients, given enough leeway, can try to sneak in new features and change requests as bug reports, hoping to get them done for free. I'm distinctly uncomfortable with the industry's traditional solution to this problem, which is to make the customer pay for their mistakes.

Executable acceptance tests can provide much clarity on what is a bug and what isn't.

If any of the agreed tests fail, then it's definitely a bug. Therefore, I don't deliver the software unless all the acceptance tests pass.

Let's say, for example, we agreed that a feature to transfer credits from one user's account to another's should debit the payer and credit the payee. I deliver the software that passes the tests we agreed for that. But none of those tests defined what should happen if the payer and the payee are the same user account. The customer might try to report it as a bug if the software allows it, but when acceptance criteria are precisely defined using tests, I classify that as a change request. The customer is asking me to make the software do something we hadn't agreed it should. It is undefined behaviour.

There is one kind of undefined behaviour that I would classify as a bug, and fix at my own cost, though. When development starts, we'll agree a set of behaviours that should never be allowed - for example, when would anyone ever want their software to throw an unhandled exception?

So if the implementation leaves it such that an undefined behaviour triggers one of these gotchas - e.g., a null reference exception - I fully accept that it is a programming error, and that my code should either not allow that scenario to arise, or meaningfully handle it. I usually refer to these as universal acceptance criteria, and they are requirements about things that should never happen, regardless of the problem domain.

Universal acceptance criteria are almost exclusively about low-level programming errors - trying to invoke methods on null objects, trying to reference an array index that's too high, and so on - and can often be eliminated with a bit of extra care. There are, for example, static analysis tools that can find these things. Inspections help enormously with these kinds of errors.

This approach neatly compartmentalises work into 3 distinct categories:

1. Things we explicitly agreed the software will do (requirements)
2. Things we didn't explicitly agree the software will do (more requirements)
3. Things we explicitly agreed the software must never do (bugs)

Done correctly, only the third category is "bugs", and they should be rare occurrences, leaving us more time to focus on what the customer needs.




September 29, 2014

Code Inspections: A Powerful Testing Tool

Quick post about code inspections.

Now, when I say "code inspections" to most developers, it conjures up images of teams sitting in airless meeting rooms staring at someone's code and raising issues about style, about formatting, about naming conventions, about architectural patterns and so on.

But this is not what I really mean by "code inspections". For me, first and foremost, inspections are a king of testing. In fact, statistically speaking, inspections are still the most effective testing technique we know of.

Armed with a clear understanding of what is required of a program (and in this context, "program" means a unit of executable code), we inspect that code and ask if it does what we think it should - "at this point, is what we think should be true actually true?" - and look for possibilities of when it could go wrong and ask "could that ever happen?"



For example, there's a bug in this code. Can you see it?

Instead of writing a shitload of unit tests, hoping the bug will reveal itself, we can do what is often referred to as a bit of whitebox testing. That is, we test the code in full knowledge of what the code is internally. The economics of inspections become clear when we realise the full advantage of not going into testing blind, like we do with blackbox testing. We can see where the problems might be.

There are numerous ways we can approach code inspections, but I find a combination of two specific techniques most effective:

1. Guided Inspection - specifically, inspections guided by test cases. We choose an interesting test case, perhaps by looking at the code and seeing where interesting boundaries might occur. Then we execute the code in our heads, or on paper, showing how program state is affected with each executable step and asking "what should be true now?" at key points in execution. Think of this as pre-emptive debugging.

2. Program Decomposition - break the code down into the smallest executable units, and reason about the contracts that apply to each unit. (Yes, everything in code that does something has pre- and post-conditions.)

In practice, these two techniques work hand in hand. Reasoning about the contracts that apply to the smallest units of executable code can reveal interesting test cases we didn't think of. It also helps us to "fast-forward" to an interesting part of a wider test scenario without having to work through the whole process up to that point.

For example, let's focus on the program statement:

sequence[i] = sequence[i - 1] + sequence[i - 2]

What happens if the result of evaluating sequence[i - 1] + sequence[i - 2] is a number too large to fit in a Java int? The maximum value of an int in Java is 2,147,483,647. This code might only work when the sum of the previous two numbers is less than or equal to that max value.

Fibonacci numbers get pretty big pretty fast. In fact, the sum of the 48th and 49th numbers in the sequence is greater than 2,147,483,647.

And there's our bug. This code will only work up to the 49th Fibonacci number. To make it correct, we have choices; we could add a guard condition that blocks any sequenceLength greater than 49. (Or, for fans of Design By Contract, simply make it a pre-condition that sequenceLength < 50.)

Or we could change the data type of the sequence array to something that will hold much larger numbers (e.g., a long integer - although that, too, has its limits, so we may still need to guard against sequence lengths that go too high eventually.)

Now, of course, unit tests might have revealed this bug. But without exception, when teams tackle the Fibonacci sequence TDD exercise in training, if they spot it all, they spot it by reading the code. And, in this example, it would be extraordinary luck to choose exactly the right input - sequenceLength = 50 - to tell us where the boundary lies. So we'd probably still have to debug the code after seeing the test fail.

Reading the code, you may well spy other potential bugs.

And, yes, I know. This is a very trivial example. but that only reinforces my point: it's a very trivial example with bugs in it!

Try it today on some of your code. Are there any bugs lurking in there that nobody's found yet? I'll bet you a shiny penny there are.



September 28, 2014

Software Development Isn't Just Programming

Alerted to a piece on wired.com this morning about a new website, exercism.io, that promises programmers the "deep practice" they need to become good enough to do it for a living.

You won't be at all surprised to learn that I disagree, or why.

Software development, as distinct from programming, requires a wide and complex set of disciplines. The part where we write the code is, in reality, just a small part of what we do.

In particular, how can such sites exercise that most important of developer muscles, for collaborating?

The Codemanship Team Dojo, which I occasionally enjoy putting client teams through and have run with some large groups at conferences and workshops, provides a very clear illustration of why a team full of strong programmers is not necessarily a strong team.

Problems that seem, in isolation, trivial for a programmer to solve can quickly become very complicated indeed the minute we add another programmer (Hell is other programmers.)

Folk have got to understand each other. Folk have got to agree on stuff. Folk have got to coordinate their work. And folk have got to achieve things with limited time and resources.

This, it turns out, is the hard stuff in software development.

I know some technically very strong developers who consistently seem to fall flat on their faces on real teams working on real problems for real customers.

Don't get me wrong; I think the technical skills are very important. But I consider them a foundation for a software developer, and not the be-all-and-end-all.

For my money, a well-rounded learning experience for an aspiring software developer would include collaboration within something approximating a real team, would include something approximating a human customer (or customers, just to make it more interesting), would require the resulting software to be deployed in front of something approximating real users, and would encompass a wider set of technical disciplines that we find are necessary to deliver working software.

And their contention that professional programmers need "hundreds of hours" of practice is missing a trailing zero.






September 25, 2014

Functional Programming Is Great. But It Ain't Magic.

An increasing annoyance in my day-to-day job as a coach and trainer is what I call FPF, or "Functional Programming Fanaticism". Typically, it emanates from people who've recently discovered FP in the last few years, and have yet to realise that - like all programming innovations since the 1940's - it doesn't actually solve all the problems for us.

Putting aside the widely-held perception that functional programs can be considerably less easy to understand, even for experienced FP-ers, (and this is no small consideration when you realise that trying to understand program code is where programmers spend at least half our time), there is the question of side effects.

More specifically, people keep telling me that functional programs don't have any. This is patently not true: a program with no side effects is a program which does nothing of any use to us. Somewhere, somehow, data's got to change. Try writing a word processor that doesn't have side effects.

FP helps us write more reliable code - in particular, more reliable concurrent code - by limiting and localising side effects. But only if you do it right.

It's entirely possible to write functional programs that are riddled with concurrency errors, and, indeed, that's what many teams are doing as we speak.

How can this be so, though, if functions are said to be "clean" - side-effect free? Well, that bank account balance that gets passed from one function to next may indeed be a copy (of a copy of a copy) of the original balance, but from the external user's perspective, whatever the current balance is, that is the balance (and it has changed.)

The moment we persist that change (e.g., by writing it to the database, or through transactional memory, or however we're handling shared data), the deed is done. Ipso facto: side effect.

Languages like Haskell, Clojure and that other one that sounds like "Camel" don't do our concurrent thinking for us. If joint account holder A checks their balance before trying to use the debit card, but joint account holder B uses their debit card before A does, then - you may be surprised to learn - these languages have no built-in feature for reconciling joint account transaction paradoxes like this. You have to THINK ABOUT HOW YOUR SOFTWARE SHOULD HANDLE CONCURRENT SCENARIOS from the user's perspective.

In non-FP work, we seek to make concurrent systems more reliable and more, well, concurrent, by strictly limiting and localising concurrent access to shared data. FP just embeds this concept within the languages themselves, making that easier and more reliable to do.

Just as workflow frameworks don't decide what should happen in your workflows, functional programs don't decide how your application should handle side-effects. The best they can do is give you the tools to realise the decisions you make.

What I'm seeing, though, (and this was case when we were all prostrating before the Great Workflow Ju Ju In The Sky a decade or so ago), is that teams mistakenly lean on the technology, believing through some kind of magic that it will handle these scenarios for them. But, like all computer programs, they will only do exactly what we tell them to.

It's not magic, folks. It's just code.


September 17, 2014

The 4 C's of Continuous Delivery

Continuous Delivery has become a fashionable idea in software development, and it's not hard to see why.

When the software we write is always in a fit state to be released or deployed, we give our customers a level of control that is very attractive.

The decision when to deploy becomes entirely a business decision; they can do it as often as they like. They can deploy as soon as a new feature or a change to an existing feature is ready, instead of having to wait weeks or even months for a Big Bang release. They can deploy one change at a time, seeing what effect that one change has and easily rolling it back if it's not successful without losing 1,001 other changes in the same release.

Small, frequent releases can have a profound effect on a business' ability to learn what works and what doesn't from real end users using the software in the real world. It's for this reason that many, including myself, see Continuous Delivery as a primary goal of software development teams - something we should all be striving for.

Regrettably, though, many software organisations don't appreciate the implications of Continuous Delivery on the technical discipline teams need to apply. It's not simply a matter of decreeing from above "from now on, we shall deliver continuously". I've watched many attempts to make an overnight transition fall flat on their faces. Continuous Delivery is something teams need to work up to, over months and years, and keep working at even after they've achieved it. You can always be better at Continuous Delivery, and for the majority of teams, it would pay dividends to improve their technical discipline.

So let's enumerate these disciplines; what are the 4 C's of Continuous Delivery?

1. Continuous Testing

Before we can release our software, we need confidence that it works. If our aim is to make the software available for release at a moment's notice, then we need to be continuously reassuring ourselves - through testing - that it still works after we've made even a small change. The secret sauce here is being able to test and re-test the software to a sufficiently high level of assurance quickly and cheaply, and for that we know of only one technical practice that seems to work: automate our tests. It's for this reason that a practice like Test-driven Development, which leaves behind a suite of fast-running automated tests (if you're doing TDD well) is a cornerstone of the advice I give for transitioning to Continuous Delivery.

2. Continuous Integration

As well as helping us to flag up problems in integrating our changes into a wider system, CI is also fundamental to Continuous Delivery. If it's not in source control, it's going to be difficult to include it in a release. CI is the metabolism of software development teams, and a foundation for Continuous Delivery. Again, automation is our friend here. Teams that have to manually trigger compilation of code, or do manual testing of the built software, will not be able to integrate very often. (Or, more likely, they will integrate, but the code in their VCS will likely as not be broken at any point in time.)

3. Continuous Inspection

With the best will in the world, if our code is hard to change, changing it will be hard. Code tends to deteriorate over time; it gets more complicated, it fills up with duplication, it becomes like spaghetti, and it gets harder and harder to understand. We need to be constantly vigilant to the kind of code smells that impede our progress. Pair Programming can help in this respect, but we find it insufficient to achieve the quality of code that's often needed. We need help in guarding against code smells and the ravages of entropy. Here, too, automation can help. More advanced teams use tools that analyse the code and detect and report code smells. This may be done as part of a build, or the pre-check-in process. The most rigorous teams will fail a build when a code smell is detected. Experience teaches us that when we let code quality problems through the gate, they tend to never get addressed. Implicit in ContInsp is Continuous Refactoring. Refactoring is a skill that many - let's be honest, most - developers are still lacking in, sadly.

Continuous Inspection doesn't only apply to the code; smart teams are very frequently showing the software to customers and getting feedback, for example. You may think that the software's ready to be released, because it passes some automated tests. But if the customer hasn't actually seen it yet, there's a significant risk that we end up releasing something that we've fundamentally misunderstood. Only the customer can tell us when we're really "done". This is a kind of inspection. Essentially, any quality of the software that we care about needs to be continuously inspected on.

4. Continuous Improvement

No matter how good we are at the first 3 C's, there's almost always value in being better. Developers will ask me "How will we know if we're over-doing TDD, or refactoring?", for example. The answer's simple: hell will have frozen over. I've never seen code that was too good, never seen tests that gave too much assurance. In theory, of course, there is a danger of investing more time and effort into these things than the pay-offs warrant, but I've never seen it in all my years as a professional developer. Sure, I've seen developers do these things badly. And I've seen teams waste a lot of time because of that. But that's not the same thing as over-doing it. In those cases, Continuous Improvement - continually working on getting better - helped.

DevOps in particular is one area where teams tend to be weak. Automating builds, setting up CI servers, configuring machines and dealing with issues like networking and security is low down on the average programmer's list of must-have skills. We even have a derogatory term for it: "shaving yaks". And yet, DevOps is pretty fundamental to Continuous Delivery. The smart teams work on getting better at that stuff. Some get so good at it they can offer it to other businesses as a service. This, folks, is essentially what cloud hosting is - outsourced DevOps.

Sadly, software organisations who make room for improvement are in a small minority. Many will argue "We don't have the time to work on improving". I would argue that's why they don't have the time.







September 16, 2014

Why We Iterate

So, in case you were wondering, here's my rigorous and highly scientific process for buying guitars...

It starts with a general idea of what I think I need. For example, for a couple of years now I've been thinking I need an 8-string electric guitar, to get those low notes for the metalz.

I then shop around. I read the magazines. I listen to records and find out what guitars those players used. I visit the manufacturers websites and read the specifications of the models that might fit. I scout the discussion forums for honest, uncensored feedback from real users. And gradually I build up a precise picture of exactly what I think I need, down to the wood, the pickups, the hardware, the finish etc.

And then I go to the guitar shop and buy a different guitar.

Why? Because I played it, and it was good.

Life's full of expectations: what would it be like to play one of Steve Vai's signature guitars? What would it be like to be a famous movie star? What would it be like to be married to Uma Thurman?

In the end, though, there's only one sure-fire way to know what it would be like. It's the most important test of all. Sure, an experience may tick all of the boxes on paper, but reality is messy and complicated, and very few experiences can be completely summed up by ticks in boxes.

And so it goes with software. We may work with the customer to build a detailed and precise requirements specification, setting out explicitly what boxes the software will need to tick for them. But there's no substitute for trying the software for themselves. From that experience, they will learn more than weeks or months or years of designing boxes to tick.

We're on a hiding to nothing sitting in rooms trying to force our customers to tell us what they really want. And the more precise and detailed the spec, the more suspicious I am of it. bottom line is they just don't know. But if you ask them, they will tell you. Something. Anything.

Now let me tell you how guitar custom shops - the good ones - operate.

They have a conversation with you about what guitar you want them to create for you. And then they build a prototype of what you asked for. And then - and this is where most of the design magic happens - they get you to play it, and they watch and they listen and they take notes, and they learn a little about what kind of guitar you really want.

Then they iterate the design, and get you to try that. And then rinse and repeat until your money runs out.

With every iteration, the guitar's design gets a little bit less wrong for you, until it's almost right - as right as they can get it with the time and money available.

Custom guitars can deviate quite significantly from what the customer initially asked for. But that is not a bad thing, because the goal here is to make them a guitar they really need; one that really suits them and their playing style.

In fact, I can think of all sorts of areas of life where what I originally asked for is just a jumping-off point for finding out what I really needed.

This is why I believe that testing - and then iterating - is most importantly a requirements discipline. It needs to be as much, if not more, about figuring out what the customer really needs as it is about finding out if we delivered what they asked for.

The alternative is that we force our customers to live with their first answers, refusing to allow them - and us - to learn what really works for them.

And anyone who tries to tell you that it's possible to get it right - or even almost right - first time, is a ninny. And you can tell them I said that.