November 25, 2014

Continuous Inspection II - Planning & Executing CInsp

In this second blog post about Continuous Inspection (CInsp, for short), I want to look at how we might manage the CInsp process to get the most value from it.

While some develoment teams are now using CInsp tools to analyse their code to get early warnings about code quality problems when they're easier and cheaper to fix, it's fair to say that this area of the develoment discipline has to date evaded the principles that we apply to other kinds of requirements.

Typically, as a kind of work, CInsp is ad hoc, unplanned, untracked and most teams who do it have only a very vague idea of what kind of cost it has and what kind of benefits they're reaping from it.

CInsp is rarely prioritised, leaving the field wide open to waste a lot of time and effort on activities that add little or no value.

Non-functional requirements obey the same laws as functional ones, which is why we need to attack them using the same principles and techniques.

In this post, I want to examine how we plan and execute CInsp on projects starting from scratch. (In a future post, I'll talk about applying CInsp to existing code bases with a build-up of code quality issues.)

Continuous Inspection Requirements.

There are an infinite number of properties we could look for in our code, but some have value in finding and most don't. Rather than waste our time arbitrarily searching our code for "stuff", it's important we have a clear idea of what it is we're looking for and why.

Extreme Programming, for example, has a perfectly usable mechanism for describing the things we want to inspect for, and the benefits of catching those kinds of code quality problems early.

A Code Quality Story is a non-functional user story that briefly summarises a code quality "bug" we wish to avoid and the pay-off we might expect if we can avoid introducing it into our code.



Note first of all that I've chosen here to use a blue index card. This might be in a system where we write functional user stories on green cards, report bugs on red cards, and record other outcomes - "miscellaneous tasks", like setting up the build and implementing code quality gates - on blue cards.

Why do this? Well, I've found it very useful to know roughly how much of a team's time is split between delivering working features, fixing bugs (ideally, zero time), and "shaving yaks" when the yaks being shaved are sufficiently large and not part of the work of delivering specific features.

The importance of the effort split becomes apparent as time goes by and the software evolves. A healthy project is one where the proportion of effort devoted to delivering working features remains relatively constant. What typically happens on teams who set out at an unsustainable pace is that they begin development with their time devoted mostly to the green cards, and after a few months most of their time is spent tackling red cards and making a lot less progress on new features. This is a good indicator of the rising cost of change we're seeking to avoid, so we can sustain the pace of development and deliver value for longer. This information will help us better judge how well-spent the time devoted to things like CInsp is.

So we have a placeholder for our code quality requirement in the form of a blue index card. What next?

Planning Continuous Inspection

This is where I, and a lot of teams, have gone wrong in the past. What we should never, ever do is allow the customer to choose when and whether we tackle non-functional requirements. And in "customer" I include proxy customers like business analysts and project managers. The overwhelmingly common experience of development teams is that purely technical issues, like code quality, get sidelined by non-technical stakeholders.

We must not give them the chance to drop our Feature Envy story in favour of a story about, say, sorting columns in an HTML table if we strongly believe, as professionals, that avoiding Feature Envy is important. If, as the evidence suggests, care taken over code quality helps to maintain productivity and deliver greater value over time, then we risk presenting customers with a confusing false dichotomy between work that enhances quality and work that directly delivers working features.

The analogy I use is to pretend we're running a restaurant using the planning practices of Extreme Programming.

Every job that needs doing gets written on a card, and placed into a backlog of outstanding work. There will be user stories like "Take table 3's order" and "Serve french fries and beer to table 7" and "Get the bill for table 12". These are stories about work that will make the restaurant money.

There will also be stories like "Wash the dishes in the sink" and "Clean out pizza oven" and "Repaint sign over door". These are about tasks that cost money, but don't directly bring in revenue by themselves.

If we allowed our restaurant's shareholders - who themselves have never worked in a restaurant, but they have a stake in it as a business - to prioritise what stories get done at the expense of others in a world where backlogs always outweigh the available time and resources, then there's a very real danger that the kitchen will rarely get cleaned, the sign above the door will fade until nobody can see it, and we'll run out of clean plates halfway through service.

The temptation for teams who are driven solely by the priorities of non-technical stakeholders is that non-functional issues like code quality will only get tackled when a crisis emerges that blocks progress on functional requirements. i.e., we don't wash up until we run out of plates, or we don't clean the kitchen until the inspector shuts us down, or we don't repaint the sign until the customers have stopped coming in.

One thing we've learned about writing software is that it's cheaper and easier to tackle problems proactively and catch them earlier. Sadly, too many teams are left lurching from one urgent crisis to the next, never getting the chance to get ahead of the issues.

For this reason, I strongly advise against involving non-technical stakeholders in planning CInsp. (As well as other technical work.)

Now put yourself in the diner's shoes: you pick up the menu, and every dish lists all of the tasks restaurant staff have to do in order to deliver it. Let's say we charge £11 for fish and chips, with a clean grill, mopping the floors, cashing up that evening, doing the accounts, getting up early to take delivery of fresh fish, and so on.

Two questions:

1. If we hadn't told them, would the diner even care?

2. If we make it the diner's business, are we inviting them to negotiate the price of the fish and chips down by itemising what goes in to running the restaurant? ("I'll have the fish & chips, but I'm not paying for your trainee chef's college course" etc)

The world is full of work that needs doing, but nobody thinks they should pay for. In order for the world to keep turning, for fish & chips to appear on our dining tables, this work has to get done one way or another, and it has to be paid for.

The way a restaurant squares this circle is to build it into the cost of the meal and to not present diners with a choice. Their choice is simple: don't like the price, don't order the dish.

Likewise in software development, there's a universe of tasks that need doing that do not directly end with a working feature being delivered to the customer's table. We must build this work into the price ("feature X will take 3 days to deliver") and avoid presenting the customer with bewildering choices that, in reality, aren't choices at all.

So planning Continuous Inspection is something that happens within the team among technical stakeholders who understand the issues and will be doing the work. This is good advice for any non-functional requirements, be they about build automation, internal training or hiring developers. This is just "stuff that has to happen" so we can deliver working software reliably, economically and sustainably.

The key thing, to avoid teams disappearing up their own backsides with the technical stuff, is to make sure we're all absolutely clear about why we're doing it. Why are we automating the build? Why are we writing a tool that generates code? Why are we sending half the team to the Software Craftsmanship conference? (Some companies send entire teams.) And the answer should always be something of value to the customer, even if that value might not be realised for months or years.

In practice, we have planning meetings - especially in the early stages of a project - that are for technical stakeholders only. Lock the doors. Close the blinds. Don't tell the boss. (I have literally experienced running around offices looking for rooms where the developers can have these discussions in private, chased by the project manager who insists on sitting in. "Don't mind me. I won't interfere." Two seconds later...)

Such meetings give teams a chance to explicitly discuss code quality and to thrash out what they mean by "good code" and "bad code" and establish a shared set of priorities over code quality. It's far better to have these meetings - and all the inevitable disagreements - at the start, when we can take steps to prevent issues, than to have them later when we can only ask "what went wrong?"

Executing Continuous Inspection

On new software, the effort in Continuous Inspection tends to be front-loaded, and with good reason.

As I've mentioned a few times already, it tends to be far cheaper to tackle code quality "bugs" early - the earlier the better. This means that adding new code quality requirements later in development tends to catch problems when they're much more expensive to fix, so it makes sense to set the quality bar as high as we can at the start.

There's good news and there's bad news. First, the bad news: on a new project, from a standing start, it's going to take considerable effort to get automated code inspections in place. It will vary greatly, depending on the technology stack, availability of tools, experience levels in the team, and so on. But it's not going to take an afternoon. So you may be faced with having to hide a big chunk of effort from non-technical stakeholders if you attempt to start development (from their perspective, when they're actively involved) at the same time as putting CInsp in place. (Same goes for builds, CI, and a raft of other stuff that we need to get up and running early on.)

Another very strong recommendation from me: have at least one iteration before you involve the customer. Get the development engine running smoothly before you wind down the window and shout "Where to, guv'nor?" They may be less than impressed to discover that you just need to build the engine before you can set off. Delighting customers is as much about expectations as it is about actual delivery.

Going back to the restaurant analogy, consider why restaurants distinguish between "service" and "preparation". Service may start at 6pm, but the chefs have probably been there since 9am getting things ready for that. If they didn't, then those first orders might take hours to reach the table. Too many development teams attempt the equivalent of starting service as the ingredients are being delivered to the kitchen. We need to do prep, too, before we can start taking orders.

Now, for the good news: the kinds of code quality requirements we might have on one, say, JEE project are likely to be similar on another JEE project. CInsp practitioners tend to find that they can get a lot of reuse out of code quality gates they've already developed for previous projects. So, over months and years, the overall cost of getting CInsp up and running tends to decrease quite significantly. If your technology stack remains fairly stable over the years, you may well find that getting things up and running can eventually become an almost push-button process. It takes a lot of investment to get there, though.

Code Quality stories work the same way as user stories in their execution. We plan what stories we're going to tackle in the current timebox in the same way. We tackle them in pairs, if possible. We treat them purely as placeholders to have a conversation with the person asking for each story. And, most importantly, we agree...

Continuous Inspection Acceptance Tests

Going back to our Feature Envy code quality story, what does the developer who write that story mean by "Feature Envy"?

Here's the definition from Martin Fowler's Refactoring book:

"A classic [code] smell is a method that seems more interested in a class other than the one it is in. The most common focus of the envy is the data."

It's all a bit handwavy, as is usually the case with software design wisdom. A human being using their intelligence, experience and judgement might be able to read this, look at some code and point to things that seem to them to fit the description.

Programming a computer to do it, on the other hand...

This is where we can inhabit our customer's world for a little while. When we ask our customer to precisely decribe a business rule, we're putting them on the spot every bit as much as a computable definition of Feature Envy might put me and you on the spot. In cold, hard, computable terms: we don't quite know what we mean.

When the business problem we're solving is about, say, mortgages or video rentals or friend requests, we ask the customer for examples that illustrate the rule. Using examples, we can establish a shared vocabulary - a language for expressing the rule - explore the boundaries, and pin down a precise computable understanding of it (if there is one.)

We shouldn't be at all surprised that this technique also works very well for rules about our code. Ask the owner of a code quality story to track down some classic examples of code that breaks the rule, as well as code that doesn't (even if it looks at first glance like it might).

This is where the real skill in CInsp comes into play. To win at Continuous Inspection, development teams need to be skilled as reasoning about code. This is not a bad skill for a developer to have generally. It helps us communicate better, it helps us visualise better, it makes us better at design, at refactoring, at writing tools that work with code. Code is our domain model - the business objects of programming.

Using our code reasoning skills, applied to examples that will form the basis of acceptance tests, we can drive out the design of the simplest tool possible that will sound the alarm when the "bad" examples are considered, while silently allowing the "good" examples to pass through the quality gate.

As with functional user stories, we're not done until we have a working automated quality gate that satisfies our acceptance tests and can be applied to new code straight away.

In the next blog post, we'll be rolling up our sleeves with an example Continuous Inspection quality gate, implementing it using a variety of tools to demonstrate that there's often more than one way to skin the code quality cat.








November 22, 2014

Continuous Inspection I - Why Do We Need It?

This is the first of a series of posts about Continuous Inspection. My goals here is to give you something to think about, rather than to present a complete hands-on guide. The range (and maturity) of tools and techniques we can apply to Continuous Inspection (I'll call it CInsp from now on to save a few keystrokes) is such that I could write 1,000 blog posts and still not cover it all. So here I'll just focus on general CInsp principles and illustrate with cherrypicked examples.

In this first post, I want to summarise what I mean by "Continuous Inspection" and argue that there's a real need for it on most software development teams.

Contininuous Inspection is the practice of - and stop me if I'm getting too technical here - continuously inspecting your code to detect non-functional issues in the software.

CInsp is just another kind of Continuous Testing, which is a cornerstone of Continuous Delivery. To have our software always in a shippable state, we must take steps to assure ourselves that the software is always working.

If we follow the thinking behind continuous testing (and re-testing) of our software to check that it still works, the benefit is that we never stray more than a few minutes from having something we could ship if the business wanted us to.

To date, the only practical way we've found to achieve Continuous Testing is to automate those tests as much as possible, so they can be run quickly and economically. If it takes you 2 weeks to re-test your software, then after each change you make to the code, you are at least 2 weeks away from knowing if the software still works. Manual testing makes Continuous Delivery impractical.

In recent years, automated testing - and especially automated unit testing - has grown in popularity, and the effects can be seen in teams delivering more reliably and more sustainably as a result.

But only to a point.

What I've observed across hundreds of teams over the last decade or more is that, even with high levels of automated testing, the pace of delivery still slows to unacceptable levels.

In order to sustain the pace of change, the code itself needs to remain open to change. Being able to quickly regression test our software is a boon in this respect, no doubt. But it doesn't address the whole picture.

There are other things that can hamper change in our code. If the code's complicated, for example, it will be more likely to break when we change it. If there's duplication in our code - if we've been a bit trigger-happy with Copy+Paste - then that can multiply the cost of making a change. If we've not paid attention to the dependencies in our code, small changes can cause big ripples through the code and amplify the cost.

As we make progress in delivering functionality we tend also to make a mess inside the software, and that mess can get in our way and impede future progress. To maintain the pace of innovation over months and years and get the most out of our investment over the lifetime of a software product, we need to keep our code clean.

Experienced developers view design issues that impede progress in their code as bugs, and they can be every bit as serious as bugs in the functionality of the software.

And, just like functional bugs, these code quality bugs (often referred to as "code smells", because they're indicatice of your code "rotting" as it grows) have a tendency to get harder and more expensive to fix the longer we leave them.

Duplication has a tendency to grow, as does complexity. We build more dependencies on top of our dependencies. Switch statements get longer. Long parameter lists get longer. Big classes get bigger. And so on.

Here's what I've discovered form examining hundreds of code bases over the years: code smells that get committed into the code are very likely to remain for the lifetime of the software.

There seems to be a line that once we've crossed it, our mistakes are likely to live forever (and impede us forever). From observation, I've found that this line is moving on.

In the Test-driven Development cycle, for example, I've seen that when developers move on to the next failing test, any code smells they leave behind will likely not get addressed later. In programming, "later" is a distant and alien land where all our little TO-DO's never get done. "Later" might as well be "Narnia".

Even more so, when developers commit their code to a shared repository, at that point code smells "petrify", and remain forever trapped in the amber of all the other code that surrounds them. 90% of code smells introduced in committed code never get fixed.

This is partly because most teams have no processes for identifying and addressing code quality problems. But even the ones who do tend to find that their approach, while better than nothing, is not up to the task of keeping the code as clean as it needs to be to maintain the pace of change the customer needs.

Why? Well, let's look at the kinds of techniques teams these days use:

1. Code Reviews

There's a joke that goes something like this: "Ask a developer what's wrong with a line of code, and she'll give you a list. Ask her what's wrong with 500 lines of code, and she'll tell you it's fine."

Code reviews have a tendency to store up large amounts of code - potentially containing large numbers of issues - for consideration. The problem here is seeing the wood for the trees. A lot of issues get overlooked in the confusion.

But even if code reviews identified all of the code quality issues, the economics of fixing those issues is working against us. Fixing bugs - functional or non-functional - tends to get exponentially more expensive the longer we leave them in the code, and for precisely the same reasons (longer feedback cycles).

In practice, while rigorous code reviews would be a step forward for many teams who don't do them at all, they are still very much shutting the stable door after the horse has bolted.

2. Pair Programming

In theory, pair programming is a continuous code review where the "navigator" is being especially vigilent to code quality issues and points them out as soon as they spot them. In some cases, this is pretty much how it works. But, sad to say, in the majority of pairs, code quality issues are not high on anyone's agenda.

This is for two good reasons: firstly, most developers are not all that aware of code smells. They don't figure high in our list of priorities. Code quality isn't sexy, and doesn't get you hired at IronicBeards.com.

Secondly, with the best will in the world, people have limitations. When Codemanship does pairing to assess a developer's skill level in certain practices, the level of focus required on what the other person's doing is really quite intense. You don't take your eye off the screen in case you miss something. But there are dozens of code smells we need to be vigilant for, and even with all my experience and know-how, I can't catch them all. My mind will have to skip between lots of competing concerns, and when my remaining brain cells are tied up trying to remember how to do something with Swing, I'm likely to take my eye off the code quality ball. It's also very difficult to maintain that level of focus hour after hour, day-in and day-out. It hurts my brain.

Pair programming, as an approach to guarding against code smells, is good when it's done well. But it's not that good that we can be assured code written in this way will be maintainable enough.

3. Design Authorities

By far the least effective route to ensuring code quality is to make it someone else's job.

Hiring architects or "technical design authorities" suffers from all the shortcomings of code reviews and pair programming, and then adds a big bunch of new shortcomings.

Putting aside the fact that almost every architect or TDA I've ever met has been mostly focused on "the big picture", and that I've seen 1,000-line switch statements waved through the quality gate by people obsessing over whether classes implement certain interfaces they've prescribed, turning design authorities into design quality testers never seems to end well. Who wants to spend their day scouring other people's code for examples of Feature Envy?

I'll say no more, except to summarise by observing that the code I've seen produced by teams with dedicated design authorities counts amongs the worst for code quality.

4. Coding Standards

In theory, a team's coding standards are a codification of what we all agree we mean by "good code".

Typically, these are written down in documents that nobody ever reads, and suffer from the same practical drawbacks as architecture documents and company mission statements. They're aspirational affirmations at best. But, in practice, everybody just ignores them.

Even on those more disciplined teams that try to adhere to coding standards, they still have major drawbacks, all relating back to things we've already discussed.

Firstly, coding standards are a list of "stuff" we need to be thinking about along with all the other "stuff" we have to think about. So they tend to take a lower priority and often get overlooked.

Secondly, as someone who's studied a lot of coding standards documents (and what joy they bring!), they have a tendency to be both arbitrary and by no means universally agreed upon. Often they've been written by some kind of design or development authority, usually with little or no input from the team they're being imposed on. It's rare for issues that affect maintainability to be addressed in a coding standards document. Programmers are a funny bunch: we care deeply about some weird stuff while Elephants In The Room creep in without being questioned and sit on us. Naming conventions, therefore, have little relation to how easy the code will be to read and understand. And it's rare to see duplication, dependencies, complexity and so on even being hinted at. As long as all your instances have names beginning with obj and all your private member variables beging with "m_", the gods of code goodness will be appeased.

And then there's the question of how and when we enforce coding standards. And we're back to the hard physics of software development - time, money and cost. Knowing what we should be looking for is only the tip of the code quality iceberg.

What's needed is the ability to do code reviews so freqently, and do them in a way that's so effective, that we never stray more than a few minutes from clean code. For this, we need code reviewers who miss very little, who are constantly looking at the code, and who never get tired or distracted.

For that, thankfully, we have computers.

Program code is like any other domain model; we can write programs to reason about the design of other programs, expressed in terms of the structure of code itself.

Code quality rules are just like any other computable business rules. If the rule is that a block of code in one class should not make copious references to features in another class ("Feature Envy"), it's possible to write an automated test that reads code and looks at those references to determine if that block of code is in the right place.

Let's illustrate with a technology example. Imagine we're working in Java in, say, Eclipse. We could write code for a plug-in that, whenever we make a change to the code document we're working on, reads the code's Abstract Syntax Tree (basically, a code DOM) and does a calculation for the ratio of internal and external dependencies in that Java method we just changed. If the ratio is too low, it could flag it up as a warning while we're writing the code.

The computational power of computers is such today that this sort of continuous background code reviewing is practically possible, and there have already been some early attempts to create just such plug-ins.

In the article I wrote a few years ago for Visual Studio Journal, Ever-decreasing Cycles, I speculate about the impact such short code quality feedback loops might have on the economics of development.

It's my belief that, just as continuous automated unit testing has had a profound effect on the "bottom line" of software development for many teams and businesses, so too would Continuous Inspection.

In the next blog post, I'll talk about the CInsp process and look at practical ways of managing CInsp requirements, test automation and how we action the code quality problems it can throw up.




November 19, 2014

In 2015, I Are Be Mostly Talking About... Continuous Inspection

Just a quick FYI, for event organisers: after focusing this year on software apprenticeships, in 2015 I'll be focusing on Continuous Inspection.

A critically overlooked aspect of Continuous Delivery is the need to maintain the internal quality of our software to enable us to sustain the pace of innovation. Experience teaches us that Continuous Delivery is not sustainable without Clean Code.

Traditional and Agile approaches to maintaining code quality, like code reviews and Pair Programming, have shown themselves to fall short of the level of rigour teams need to apply. While we place great emphasis on automated testing to ensure functional quality, we fall back on ad hoc and highly subjective approaches for non-functional quality, with predictable results.

Just as with functional bugs, code quality "bugs" are best caught early, and for this we find we need some kind of Continuous Testing approach to raise the alarm as soon after code smells are introduced as possible.

Continuous Inspection is the missing discipline in Continuous Delivery. It is essentially continuous non-functional testing of our code to ensure that we will be able to change it later.

In my conference tutorials, participants will learn how to implement Continuous Inspection using readily available off-the-shelf tools like Checkstyle, Simian, Emma, Java/NDepend and Sonar, as well as rigging up our own bespoke code quality tests using more advanced techniques with reflection and parser generators like ANTLR.

They will also learn about key Continuous Inspection practices that can be used to more effectively manage the process and deliver more valuable results, like Non-functional Stories, Clean Code Check-ins, Build Inspections and Rising Tides (a practice that can be applied to incrementally improving the maintainability of legacy code.)

If you think your audience might find this interesting, drop me a line. I think this is an important and undervalued practice, and want to reach as many developers as possible in 2015.




November 6, 2014

Personal Craftsmanship Coaching Sessions To Raise Money To Feed People In Need This Winter

As part of an Indiegogo campaign to raise £1,000 for hot meals for people in need this winter, I'm offering 10x 2-hour personal coaching sessions for developers in TDD, refactoring, SOLID and other aspects of software craftsmanship.

These would be online pairing sessions, and we can work in any language you like on any problem you like, just as long as the emphasis is on craftsmanship.

The money raised from one coaching session would pay for 100 hot meals for people struggling with poverty this winter.

So if you fancy a bit of one-on-one coaching to help you with any questions or obstacles you're wrestling with in your quest to become a better code crafter, and would like to help people in real need at the same time, please take a look.






October 24, 2014

Refactoring Is Dead (Hard)

Jason Yip tweets jokingly that he's waiting for someone to write "Refactoring Is Dead".

Well, here it is.

The "TDD is Dead" meme started with a blog post with that clickbaity title, arguing that folk are doing TDD wrong. So it seems in that same spirit to argue that "refactoring is dead" for a similar reason.

But that would be innacurate. It would be more accurate to say that refactoring was never really alive in the first place.

As a practice, refactoring has never really taken off. It's one of those things that many claim to have done on the CVs, but few actually ever have, safe in the knowledge that the person interviewing them is probably lying about their refactoring experience, too.

Pair with 100 Agile developers, and if you're lucky you may find a handful who even understand what refactoring really is. having paired with thousands of developers over the years, I know that perhaps 1% of us can actually do it for real.

And the reason for this is simple. Refactoring, as a skill and a discipline, takes a lot of time and study and practice to get competent at. It's like swimming - it exercises all of the muscles developers use. There has to be unit tests. You have to have an eye for code quality problems ("code smells"). You have to understand the mechanics of each refactoring, which requires the ability to reason about code in a way that's more rigorous than most of us are used to. And, once you get past the basics to the good and most useful stuff, it requires the ability to think several refactorings ahead - like a chess player - so that you can restructure and manipulate complex code into the desired forms. If TDD is riding a bicycle, then refactoring is flying a helicopter.

In summary, refactoring is hard. Dead hard.


UPDATE: Yes, I know that TDD involves refactoring. But I also know that a majority of developers who say they do TDD do little or no refactoring as part of it. And the ones who do are doing very simple refactorings. Legacy code is a whole different ball of wax.





October 23, 2014

Software Craftsmanship 2015 - Registration Page Preview

So, Sc2015 preparations are gathering pace. we should be opening soon for registration, and here's a preview of the blurb that will be on the registration page, to wet your appetite:



Software Craftsmanship 2015 is the sixth international software craftsmanship conference, organised by Codemanship.

http://www.softwarecraftsmanship.co.uk

All of the profits from SC2015 will be put towards programming and maths education projects at Bletchley Park and The National Museum of Computing.

People who came to previous SC20xx events told us that what they enjoyed best was exploring and getting hands-on with programming challanges, talking to and pair programming with lots of differet people.

We want to maximise that experience at SC2015.

So, at SC2015, not only will talks still be banned, but we're doing away with the whole idea of sessions and schedules altogether.

Want to spend all day trying different programming challenges?
Want to spend all day tackling one programming challenge 8 times with 8 different pairing partners?
Want to do fun programming stuff in the morning, and then spend the afternoon shooting the breeze with interesting people who care about the things you do?
Want to take your laptop dow to the lake and do live blogging all day?

At SC2015, you can!

Our goal is to make SC2015 as loose as possible. Your day is your own.

What we'll provide is a unique venue with an amazing history, a decent Wi-Fi connection, power sockets for your laptops, refreshments, and 100 of the most passionate and open-minded software developers for you to bounce ideas off, learn from and socialise with.

We'll be asking people to submit their ideas for programming challenges and mini-projects which will hopefully fire the creative juices on the day, and you'll be able to pick these up and try any that take your fancy.

These could be things like "Build your own code metrics tool", or "Write a program that plays another program at poker and start a poker tournament at SC2015" or "Write a program that reverses time and visit the dinosaurs", or whatever flight of fancy our contributors dare to take. As long as it has an element of craftsmanship to it, and exercises the code craft muscles, we're interested in hearing about it. use the organiser's email address at the bottom to get in touch if you have an idea(s). A proper grown-up RFP will go out soon.

Their creators will be around on the day to answer questions and give tips and advice based on their own experiences tackling the project. Some might take about an hour. Some might take you all day. Estimated completion times will be clearly signposted to help you plan your day.

If none of them take your fancy, do your own thing. Or take a wander around beautiful and historic Bletchley Park, including the amazing National Museum of Computing, and see if the muse grabs you.

This year, we'll also be promoting the theme of Developer Culture, Ethics & Diversity in the informal discussions going on throughout the day. We'll be asking a group of volunteers to prepare "Conversation Cards" - yes, just like in Monty Python & The Meaning of Life - with interesting statistics, thought-provoking facts and (hopefully) conversation-inspiring questions to inspire discussion, brainstorming and debate. And, yes, if you just want to spend the entire day discussing issues with your peers, at SC2015 you can.

Doors will open at 9am, where you can register and pick up your information pack, grab a tea of a coffee, and then relax for a bit, or jump straight in. It's entirely up to you.

If you decide to stay with us for the evening, there will be sausages! Oh yes! (And veggie sausages, too.) And booze, soft drinks and entertaining diversions. None of which are compulsory, of course. If you want to sit outside and gaze at the stars instead, at SC2015 you can.

Of course, it can't be complete anarchy. We do have a very simple schedule:

09:00 Doors open for registration (with Tea & Coffee)
09:30 Welcome by organiser Jason Gorman
09:35 Your Time
11:00 Tea & Coffee served
12:30 Lunch
15:00 Tea & Coffee served
16:30 Bar opens
18:00 Mansion closed briefly for evening changeover
18:30 Dinner served
20:45 Bar closes
21:00 Mansion closes

Won't Someone Please Think Of The Software?!

One hope I harbour for putting proper computing - especially programming - back into the British school curriculum is that, having had a go at making computers do stuff by typing text into them - today's kids will grow up to be the kind of responsible, enlightened adults who know from experience that it takes time and costs money to write software (and many times more to change software that's already been written.)

Because I've noticed that, right now, with a generation of adults most of whom have no practical experience of programming, that thought barely registers.

Exhibit A is the commentator insisting that scrapping Greenwich Meantime in the UK (our clocks go back an hour in the winter) would be an easy win for the government as it would be "a very popular move that would cost nothing". I beg to differ. Time and time zones and daylight savings and GMT and all that paraphernalia takes up a lot of source code; code that would need to be changed. Changing code is very expensive.

This is quite typical of how computer muggles think - or, more accurately, don't think - when it comes to matters involving software. More often than not, it just never crosses their minds.

Nowhere is this more visible than in politics; parties gleefully dream up policies like the now-very-very-late-and-getting-later Universal Credit benefits system or a national database of this-that-and-the-other or an online system for geo-tagging naughty donkeys, or whatever the hair-brained policy is, without giving any thought to how much the IT would cost, or if it's even feasible. (TIME SAVING TIP: If it involves IT and government, it's usually not feasible. "Hello, World" could cost billions and still not work after a decade, if the right - i.e., wrong - suppliers got involved. And they always do get involved.)

But this is the Information Age, and that ought to be a clue that anything any government, business, charity or other organisation wants to do on a large scale is going to involve processing large amounts of information, and that mean's it's going to involve computers and very probably new software or changes to existing software.

Even after all this hoo-hah about the Year Of Code and computing in schools and TechCityUK and all that palaver, politicians still haven't seen the light. The fact that political parties don't have a dedicated spokesperson for government computing says it all. They don't think about this stuff until they absolutely have to, by which time the bill has passed, the policy is set in stone and necks are on the chopping block. And once everyone's committed, it seems no amount of money is too much to flush down the toilet of bad IT strategies. Essentially, we have to wait until that government gets voted out and a new broom can come in and acknowledge her predecessors mistakes and scrap the programme, by which time hundreds of millions, or even billions, of public funds have been poured down the drain.

My appeal to them: won't someone please think of the software?!


October 16, 2014

Dear Aunty Jason, I'm An Architect Who Wants To be Relevant Again...

"Dear Aunty Jason,

I am a software architect. You know, like in the 90's.

For years, my life was great. I was the CTO's champion in the boardroom, fighting great battles against the firebreathing dragons of Ad Hoc Design and the evil wizards of Commercial Off-The-Shelf Software. The money was great, and all the ordinary people on the development teams would bow down to me and call me 'Sir'.

Then, about a decade ago, everything changed.

A man called Sir Kent of Beck wrote a book telling the ordinary people that dragons and wizards don't actually exist, and that the songs I'd been singing of my bravery in battle had no basis in reality. 'You', Sir Kent told them, 'are the ones who fight the real battles.'

And so it was that the ordinary people starting coming up with their own songs of bravery, and doing their own software designs. If a damsel in distress needed saving, they would just save her, and not even wait for me - their brave knight - to at the very least sit astride my white horse and look handsome while they did it.

I felt dejected and rejected; they didn't need me any more. Ever since then, I have wandered the land, sobbing quietly to myself in my increasingly rusty armour, looking for a kingdom that needs a brave knight who can sings songs about fighting dragons and who looks good on a horse.

Do you know of such a place?

Your sincereley,

Sir Rational of Rose."

I can sympathise with your story, Sir Rational. I, too, was once a celebrated knight of the UML Realm about whom many songs were sung of battles to the death with the Devils of Enterprise Resource Planning. And, I, too, found myself marginalised and ignored when the ordinary folk started praying at the altar of Agile.

And, I'm sorry to say, there are lots of kingdoms these days where little thought is given to design or to architecture. You can usually spot them from a distance by their higgledy-piggledy, ramshackle rooftops, and the fact that they dug the moat inside the city walls.

But, take heart; there are still some kingdoms where design matters and where architecture is a still a "thing". Be warned, though, that many of them, while they may look impressive from the outside, are in fact uninhabited. Such cities can often be distinguished by gleaming spires and high white walls, beautiful piazzas and glistening fountains, and the fact that nobody wants to live there because they built it on the wrong place.

You don't want to go to either of those kinds of kingdom.

Though very rare, there are a handful of kingdoms where design matters and it matters that people want to live in that design. And in these places, there is a role for you. You can be relevant once again. Once again, people will sing songs of your bravery. But you won't be fighting dragons or wizards in the boardroom. You'll be a different kind of knight.

You'll be fighting real battles, embedded in the ranks of real soldiers. Indeed, you'll be a soldier yourself. And you'll be singing songs about them.




October 15, 2014

Software Craftsmanship 2015. Yes, You Heard Right.

After much discussion and beer and brainstorming and beer and then more beer, I've decided that there will be a Software Craftsmanship 2015.

It's going to be a little different to the previous 4 SC conference, in some key ways:

Firstly, not only will there be no presentations (as usual), there will be no sessions at all. How is this possible? Well, the fun in SC20xx was always in the doing, so SC2015 will be 100% doing. Instead of proposing sessions, I'll be asking people to devise problems, games, challenges, projects - fun stuff that folk can pick up and pair with other folk on for as much of the day as they want. We'll provide the space, the Wi-Fi and the sausages (yes, there will be sausages). You provide the laptop and your brains and enthusiasm.

Each mini-project will have a point to it. This won't be just some random hackathon. They'll be devised to exercise some of your code crafting muscles, hoppefully in entertaining and interesting ways. There'll be a handout that explains it, links to useful resources, and the person/people who devised the challenge will be nearby and easily identifiable, ready to answer questions and give feedback. They will,, of course have completed the challenge themselves beforehand. SC20xx has never been home to people who only talk the talk.

So, with a good wind behind us, you will learn something. But in your own time and at your own pace. How you structure your day will be entirely up to you. We'll open the doors, say "howdy" to y'all and then point you to the rooms/spaces where different kinds of challenges can be found, and off you go. There'll be scheduled coffee breaks and lunch and wotnot, naturally, but your day is your own. Want to gra a challenge and then go sit outside with a chum and do it on the lawn? Dandy.

You can spend an hour on each challenge you like the look of, all all day doing the same challenge, or do a 1-hour challenge several times, with different people, wearing different disguises. It's entirely up to you. The estimated time needed will be clearly sign-posted, so you don't pick an 8-hour project thinking it's going to take half an hour (or vice-versa.)

In the evening, as is the tradition for SC20xx, there'll be diversions, and sausages. Or sausages and diversions, depending on your priotities. Or diverting sausages, if that's your sort of thing.

Of course, there will be discussions. As well as the usual technical stuff we like to natter about, I'm going to provide "Conversation Cards" - yes, exactly like in Monty Python's The Meaning Of Life - on the theme of Developer Culture, Ethics & Diversity. If you want to find a space and have a debate, feel free. If you want to write a blog post based on one of the cards, fill yoour boots. If you just want to chat about it over coffee, all good. if you don't want to talk about it at all, no pressure. Nothing's compulsory.

Blog posts, GitHub wotsits, photos, videos and all manner of outputs from the day will be aggregated, collated, assigned to pastel-coloured categories and then fed to the huddled masses who are starving for a bit of quality.

It's likely to happen in the Spring, when the weather's warming up (I'd put my money on May). We'll be fieelding proposals for challenges within the next few weeks, and all materials will need to be finalised a month before the event. I shan't call it a "conference", because that's the one thing it's not going to be.

So, there you have it: Software Craftsmanship 2015 (#sc2015), May-ish 2015, very probably at Bletchley Park, very probably all profits from ticket sales will fund programming clubs or something along those lines.

Any questions?



October 8, 2014

Programming Well (Part II)

This is a short series of blog posts aimed at teachers and students who are interested in progressing on to more challenging programming projects. Here we explore the disciplines programmers need to apply to make it easier to add new features and change existing features in our programs, which ultimately makes it easier for us to incorporate our learning and make our programs better.

------------------------------------------------------------------------------------------------------------

In my previous post, we looked at why it can be very helpful to be able to test (and re-test) our programs very frequently, and how automating those tests can enable this.

We also looked at why it's very important to write code that other programmers can easily understand.

In this post, we'll talk about how we break down our programs into smaller units, and how we can organise those units to make changing our programs easier.

3. Compose programs out of small, simple pieces that do one job each

Most modern programming languages give us mechanisms for breaking down our code into smaller, simpler and more manageable "chunks". This enables us to tackle more complex programming problems without having to consider the whole problem at once, and it also enables us to make chunks of our code reusable, so we don't have to keep repeating ourselves, and ultimately we write less code.

This principle is applicable at multiple levels of code organisation:
* Blocks of code can be broken down into functions that do a specific job
* Groups of related functions - functions that are used together to do a job - can be split out into their own modules
* Groups of related modules - modules that are used together for a purpose - can be packaged together into a unit of release (e.g., an executable program file, or a code library)

If our goal is to make change easier, breaking the code down into cohesive units like this - units of code that are used together, and are likely to change together - helps to localise the impact of changes.

We balance the need for cohesion in functions, modules and packages with our need to not repeat ourselves, reusing code whenever we get the chance to minimise how much code we have to write.

And it's important that each unit of code has only one reason to change. If I write a function that does two distinct jobs, I lose the ability to change how one of those jobs is done without affecting the other. I also lose the ability to reuse the code that does each of those jobs independently of the other.

Let's take a look at a coupe of specific examples, one at the function level, another at the module level:



In this example, the Java function has two distinct jobs: it reads the scores from a file, and then it calculates the total score.

This presents us with several potential barriers to change.

Firstly, this is harder to test. If I write an automated test for this function, it could fail for a whole bunch of reasons (wrong file path, data in file not formatted correctly, total calculated incorrectly, and so on.) So if anything breaks, I'm probably going to end up in the debugger trying to figure out what actually went wrong.

It also prevents me from changing how either of these jobs is to be done without potentially affecting the other. It's all in the same function, and functions either work as a whole or they don't. Break one part, you break all of it. Ideally, functions should only have one reason to fail.

It's also a barrier to reuse. Maybe when I want to calculate a total, I don't always want scores to be read from a file. Maybe when I want scores read from a file, it's not always because I want to calculate their total. I might want to reuse either of these chunks of code independently of the other.

ON DUPLICATION OF CODE: Reuse is how programmers remove duplication from their code. Instead of, say, copying and pasting a block of code when we want new code to do something similar, we can create a function that accepts parameters and reuse that instead, and we can group functions together into reusable modules, and modules into reusable packages, and so on. The benefit of doing this is that wehn we need to make a change to duplicated logic, we only need to make that change in one place. If we copied and pasted, we'd have to change it everywhere we pasted it - effectively multiplying the effort of making that change.

When I refactor this code so that each function has a distinct responsibility - only one reason to fail - we can see how the code becomes not only easier to test, but easier to change independently and to reuse independently, which makes accomodating change even easier going forward.



In this refactored version of the code, these two distinct jobs of reading the scores from a file and adding up the total are delegated by the original function to two new functions, each with a distinct responsibility.

The function totalScore() is now just composed of calls to these other functions that do the actual work. totalScore doesn't know how the work is done, and reading scores and totalling scores know nothing about each other. As we'll see, composing our program in such a way that all the different bits - all playing their particular role - know as little about the details of how each other does their work becomes very important in making change easier.

Also notice that the two comments in the original function have disappeared? I took the opportunity to incorporate the useful information in the comments into the new functions I created. The original function now pretty much just tells the story that the comments told us. This is an example of how breaking our program down can also help us to make it easier to read and understand.

Which brings me to our second example, and...

4. Hide the details of how work is done, and make it so we can swap those details easily

In our refactored example above, the details of how scores are read from a file, and how those scores are totalled, are hidden from the totalScores() function. This gives us a degree of flexibility, allowing us to compose our program in different ways (e.g., we can total scores that weren't read from a file, or read scores for some other purpose.)

This flexibility - the ability to compose programs in differet ways while making only small changes to the code - can become a very powerful tool for making change easier. The more ways there are to compose the working units of our program, the more things we can make our program do with less effort.

In object oriented programming, using languages like Java and Python, we can go a step further in achieving greater flexibility.

In our example, even though we now have reading scores and totalling scores neatly separated, there is still rigidity in our code. What if I want to read scores from different sources? Let's say, for example, that our program is intended to be used in different settings - sometimes a desktop application that stores the scores in a file on the hard drive, sometimes a web application that stores the scores in a database like MySQL or Microsoft Access. How could we organise our code so that we can actually choose our strategy for storing scores without having to change a lot of the code?



Refactoring the code further, I can move the function that reads scores into its own class (an object oriented module from which we create instances - "objects" - that do the work), and the function that totals the scores into another class. Each class has a distinct job - a single responsibility.

Then - and this is the key to this trick - I set up an interface for each of those classes, and make it so that our RobotScoreboard only knows about the interfaces. I do this by passing in instances of the reader and the totaller in the constructor - a special function that we call when an object is created to properly initialise it before can use it - referencing only the interfaces when I do. So RobotScoreboard has no knowledge of exatly what kind of object will be doing the work. All it knows is what functions it will be able to call on these objects, which is all the interface defines.

If I wanted to use my RobotScoreboard with scores stored in a MySQL database, I write a new class that implements the same ScoresReader interface, and pass an instance of that into the constructor instead.

This gives me the ability to substitute different implementations of ScoresReader and Totaller without having to make any changes to RobotScoreboard, and to compose the program in many different ways.

This is something that's very difficult to do in programming languages that don't have a built-in mechanism for making it possible to subsitute one implementation of a function or a module with another without changing the code that uses those them. Which is why we very highly recommend that on more challenging programming projects, you use languages that support what we call polymorphism (that is, the ability for functions or modules to look the same from the outside but work differently on the inside).

DESIGNING FOR TESTABILITY: Another thing worth noting in this example is that, by introducing this extra level of flexibility, it now potentially becomes easier to write unit tests for our RobotScoreboard. Recall from the previous blog post how important it can be that we test our program as frequently as possible. For this to be possible, our tests need to run quickly. Reading data from files, or from databases, takes considerable computing resources, and tests that read files or depend on databases tend to run slowly. In our refactored version of the code, it now becomes possible for me to write a fake implementation of the ScoresReader - one that just builds a list of scores in program code, rather than reading it from a file - purely for the purposes of our tests. Making our programs easier and quicker to test automatically is another great benefit of organising our code in this sort of way.

Inexperienced programmers often have a problem with organising their programs into small, reusable, subsititible chunks like this. They see lots of new little functions and new modules and say "It's more complicated". But if you look closely, at the individual "chunks", you see that they are each simpler. when we lump all our code together into gib functions that do lots of things, and big modules with many, many functions, we lose a great deal of flexibility.

The key is to keep the individual parts as simple as we can, and to compose our programs out of them in such a way that we can more easily swap parts with new or different parts with less impact on the rest of the code.


In the next blog post, we'll look at how we manage our code and how we can more effectively work with other programmers on the same programs.