Thoughts on sustainable product development

I’ve been working across several client products over the last couple of years and have noticed a couple of common problems which contribute negatively to the longevity of the projects.

Knowledge Loss

When projects move between teams or people working on the product move on (or even the project moving between companies, which can be common in an agency setting) some knowledge is lost of

• How the code works
• The features of the system
• Why particular decisions were made
• What to do it something goes wrong

I’ve been thinking recently about how we can minimise the impact of these issues to make projects more sustainable. Some ideas I’ve got so far

• Pair programming
• Executable specifications
• Architectural decision records
• Incident reports
• Lightweight documentation (such as the C4 Model)
• Consistent use of ubiquitous language
• Automate everything (it has a secondary effect of being documentation)

Brittle code that’s hard to change

When developers aren’t confident in their knowledge of the system they’re working on I find they can be more likely to try and make smaller changes and refactor less. Over time this leads to

• duplication
• code that’s hard to understand
• code that’s hard to change
• low test coverage

Once a code base has started down this path I see there a few things needed to correct this

• Increase test coverage on areas of code that change frequently
• Refactoring to improve code
• Gradual refactoring to improve the architecture

I’ll probably write some more on this over the coming months as I work on more client products.

Some queries for visualising IIS logs in Kibana

We pump all our IIS logs into an an ELK stack, partly as a centralised log store and partly so we can easily analyse them.

A few useful queries

Get all errors

scstatus: ([500 TO 599] OR [400 TO 403] OR [405 TO 499])

Requests that took 200ms or more

timetaken: [200 TO *]

Requests that took 200ms or more that aren’t POST requests.

timetaken: [300 TO *] AND NOT method: POST

While these queries are interesting on their own, using aggregations in the visualisation section can give us further insite. For example, here is the number of slow requests per minute.

Using and becoming familiar with these aggregations can quickly help you build useful dashboards based around what’s valuable and what you’re working on at the moment. Right now, I’m doing some profiling so response time matters, but know when slow requests happen doesn’t really help much.

If I change to use the significant terms aggregation I can find which pages are most common in my search.

The blacked out parts are the URLs of the slowest pages. Now I can start profiling those pages specifically and see if I can track down any bottle necks. As you can see the pages on the left of the chart are where I need to focus my attention.

Hopefully this shows you how easy it is to do some basic analysis to point you in the right direction.

Naming boolean methods

I’ve been thinking a bit over the last few days about something I’ve seen quite a lot in a code base I’m working on: method names that exactly describe the contents of a method returning a boolean.

For example,

bool IsUkCultureAndIsNotPublished(Page page)
{
    return page.Culture == "en-GB" && !page.IsPublished;
}

Forgetting about all the othe rthings wrong with this code it introduces some duplication between the name and what the method does. Next let’s look at the place it’s used.

var pages = GetPages();
pages.Where(IsUkCultureAndIsNotPublished).ToList().ForEach(Publish);

What if we need to add another condition? Do we add it inside our method and change the name?

bool IsUkCultureAndIsNotPublishedAndIsValid(Page page)
{
    return page.Culture == "en-GB" && !page.IsPublished && page.IsValid;
}

That seems like at some point someone won’t remember to change the name. Maybe we should add it outside the method and wrap that in another method?

bool IsUkCultureAndIsNotPublishedAndIsValid(Page page)
{
    return IsUkCultureAndIsNotPublished(page) && page.IsValid;
}
...
var pages = GetPages();
pages.Where(IsUkCultureAndIsNotPublishedAndIsValid).ToList().ForEach(Publish);

That seems a bit confusing too and now we’ve introduced more duplication.

The name of our IsUkCultureAndIsNotPublished method doesn’t tell us the intent. It breaks encapsulation because it’s telling us the internals of the method. I’d suggest that a better name might be something like IsReadyToPublish. Let’s have a look at our calling code now.

var pages = GetPages();
pages.Where(IsReadyToPublish).ToList().ForEach(Publish);

Now we can see why we are publishing the pages. Instead of having some pages that are Uk culture and not published, we have pages that are ready to publish.

Describing the intent of the method allows us to have more readable code that you’re more likely to understand when you come to read your code in 6 months time and reduces the number of places you need to update if your requirements change.

Quickly checking golden master files based on stdout

A couple of people were interested in quick ways of checking program outputs against golden masters last night at XP Manchester. Here’s a couple of one liners I use. This assumes your program outputs to stdout and you’ve already created a file with the output.

Firstly using powershell if I’m on windows without bash

(compare-object (get-content goldenmaster.txt) (yourcommand) | Measure-Object).Count

Or alternatively using bash

diff goldenmaster.txt <(youcommand) >/dev/null; echo $?

Technical Interviews

I’ve been thinking a bit about technical interviews recently. I’ve interviewed dozens of people over the last few years, but I’m still not really satisfied with how any of them have gone.

I still rarely come out of an interview thinking “I know this person has the technical skills I’m looking for”.

I tend to favour working through katas or refactoring problems to see peoples design and TDD skills while talking to them about times they’ve done similar things or probing for more detail if I think there’s more there.

But even after sitting and writing code with them I find it hard to know whether someone didn’t perform well because of nerves, pressure or they just don’t know what they’re doing. Questioning can help, but doing that can feel like it’s adding even more pressure.

I don’t know what the answer is here, but I’m sure there’s a better way than what I’m doing now.

DotNet Core on Ubuntu 16.04

I’ve been playing around with dotnet core on linux on an old laptop. I just went to install it on my new laptop running ubuntu 16.04 and found that it won’t be supported until the RTM release. In the meantime here is a work around where I basically installed the old version of libicu (which I think is something to do with unicode).

wget http://security.ubuntu.com/ubuntu/pool/main/i/icu/libicu52_52.1-3ubuntu0.4_amd64.deb

sudo dpkg -i libicu52_52.1-3ubuntu0.4_amd64.deb

sudo sh -c 'echo "deb [arch=amd64] https://apt-mo.trafficmanager.net/repos/dotnet/ trusty main" > /etc/apt/sources.list.d/dotnetdev.list'

sudo apt-key adv --keyserver apt-mo.trafficmanager.net --recv-keys 417A0893

sudo apt-get update

sudo apt-get install dotnet-dev-1.0.0-preview1-002702

Strangling Old Applications

I’ve been a bit quiet recently, but here is a piece I wrote for our company blog.

Strangling Old Applications

Some links on dealing with rework in a kanban system

I’ve been thinking a bit over the last couple of days about how to deal with rework. Below are a few links that have helped guide my thought process.

https://www.targetprocess.com/blog/2014/06/how-we-handle-bug-fixes-and-rework/

https://leanandkanban.wordpress.com/2009/04/08/bugs-and-rework/

http://leansoftwareengineering.com/2007/11/25/accounting-for-bugs-and-rework/

Avoiding online snooping

This post is a bit different from my usual software related posts and probably has a different target audience, but here goes…

There has been more and more talk over the last couple of years about how much we are watched online. This can be by governments, criminals, advertising agencies or even social media companies like facebook or twitter.

In this quick post I’m not going to try and convince you that you need these things (but you probably do), but I’m going to share a few easy ways you can make it harder for people to track you online.

VPN

A VPN connection (virtual private network) can help make you more anonymous online. It has the following advantages

• your web traffic is encrypted so can’t easily be read by anyone monitoring you. This means that if a government agency (in the UK at least) want to see what you’ve been browsing they will need a court order first, which helps you not be involved in mass surveillance operations
• your IP address is hidden making it harder to be tracked and people will find it harder to find your physical location
• depending on where your VPN is hosted you can “pretend” you are in that country. So, for example, you can appear to be in america and watch things that are only available on the american Netflix.

The main downside (other than having to set it up) is that it can slow down browsing slightly. I’ve not really had much trouble with this, but I know that for some people this is pretty important.

I use Private Internet Access. They have lots of servers all over the world and are very cheap if you pay annually (about £1.50 a month). It’s easy to set up and they even have iPhone and android apps.

Adblockers

Adverts on web pages can be used to track your activity as you move between websites, download malware onto your computer and are generally pretty annoying. I’m not going to get into the ethics of this (a lot of websites use adverts as their main stream of income), but I block ads with a simple browser plugin called adblock. This one works great on Chrome, but there are lots of alternatives around if your browser isn’t supported or it doesn’t work well for you.

Anti-Tracking browser plugins

Websites often use tracking software to track your activity online. Some of this is advertising networks, some of it is malicious, some of it is for website analytics, but for me it’s all unwanted. I use two plugins, which between them cover most bases: Ghostery and Disconnect.

Social Media

Every page that has a like this on facebook button or share on twitter button is probably helping to let facebook and twitter know your browsing habbits. Staying signed out of twitter and facebook unless you are using them can help stop this.

BONUS: Two factor authentication

This isn’t directly related to privacy, but it will help you stay safe online. Two factor authentication means that when you sign into a website they will text you a code to verify that it is actually you, not just someone who knows your username and password. This is a pretty fail safe way of making sure no one can access your accounts, but does rely on  you always having your phone with you.

Hopefully these simple and easy to implement tools will help you stay safe and unwatched online. If you have any questions then feel free to message me or ask on twitter.

Thoughts on purescript after a couple of weeks of use

Most of the work I do at the moment involves dynamic languges (mostly ruby and javascript) and I often feel like I have to work slightly differently without a type system to support me, especially in javascript.

A few weeks ago, I saw Becky Conning give a talk at the Manchester Lambda Lounge on purescript, which peaked my interest in finding an alternative to javascript for work in the browser.

Purescript is a haskell like strongly typed language, which compiles to javascript. It supports javascript interop via it’s FFI module and compiles to much more readable javascript than other alternatives, such as clojurescript.

I’ve spent a couple of weeks using it and mostly I like it. Here are a few highlights for me.

• I find being able to drive the structure of your program by defining types makes it a lot easier to think about how changes will affect other parts of your program.
• The FFI is wonderfully simple to use. It allows you to define type signatures for external code, call multi argument functions as if they were purescript functions using Fn2 and even curry them. It’s a little bit fiddly the first time you use it, but I found it fairly easy once I got my head round it.
• The Eff monad, is similar to haskell’s IO monad and is used for defining types of side effects. Unlike haskell you need to be more specific about the type of effects. For example, a function which outputs a log message to the console would have the following constraint for Eff:

forall e. Eff (console :: CONSOLE | e) Unit

This probably looks a bit different to what you would writ in haskell, which would probably be something like:

Unit -> IO()

because in purescript you have to be explicit with forall, whereas in haskell it is implied (I think).

• QuickCheck. I’ve used ports of quickcheck in other languages (ruby, .Net etc.), but they’ve never been as elegant as haskell or scala, but the purescript port is excellent.
• Easy to install and good tooling. Purescript is available in npm, as is the preferred build tool, pulp. I found getting started with pulp, to be very simple and easy. Just run the following and you have deployable code ready for your browse:

pulp init
pulp browserify

When I get chance, I’ll try and put up some examples of what I’ve been working on, but in the meantime head over to purescript’s github page and to find out more.