When I started following @OctopusDeploy on twitter in preparation for adopting it for Redacted Financial Services Inc., I saw that they were sending representatives to the Kansas City Developer’s Conference (@kc_dc, #kcdc16). I looked up the conference and saw that @jeffreypalermo was going to be there as well. Excited, I reached out to @darrencauthon to see if he would be there too. When I got the confirmation, I’d made up my mind to go.

Living in the Northwest, I’d never heard of this conference. I’m super glad I came. There 1600 people at the conference. The topics’ focus are a little .NET centric but range toward the philosophical as well as the technical. In addition to .NET, there are presentations on R, Ruby, Javascript, leadership, etc.. The keynote encouraged attendees to learn more deeply about something they’re already doing and also learn something completely new.

This is exactly what I was wishing for in a conference. I’ve enjoyed the opportunity to talk with the gentlemen at @OctopusDeploy. I was impressed by the quality of Jeffrey Palermo’s talk about Continuous Delivery. I greatly enjoyed hearing about new technologies such as Semantic UI. I was fascinated by the differences between Angular, Ember, Knockout, and ReactJS. I’m now very interesting in trying out React.

The highlight was meeting Darren Cauthon in person after 5 years of being twitter friends. Darren & I are kindred spirits in that we both believe in TDD and strive for quality, well-crafted code. We both have backgrounds in .NET and experience with Ruby (though his experience with Ruby is far superior to mine).

I’ll come to this conference again–perhaps as a speaker next time.

I’m in the midst of setting Redacted Financial Services Inc. up to use Octopus Deploy. An issue we’ve had with automated deployments in the past is that the tooling reports that everything is okay, but later we find out the software is misconfigured for the environment it’s installed in. To resolve this problem we’ve started including a service health api into our deployed web services & sites. At the end of any deployment we can issue a GET to the api and have the site tell us whether or not it is at least configured in such a way that it can access its dependent databases and services. Anything other than a 200 and we fail the deployment. We are also handing this api call off to Nagios for monitoring.

The code sample below demonstrates the implementation of this api. We deliver as a code-based add-in using our internal nuget feed. It includes health checks for Sql Server, Couchbase, Rabbit MQ, and standard REST services. This code is longish and we’ll probably open source it after we’ve had a chance to bake it. If you try it out let me know how it works for you.

Technical Debt is a metaphor to describe software rot. The idea is that each time software engineers take shortcuts in the code they incur “debt” in the code base. Each time future engineers must read and/or modify the indebted code, they pay “interest” on the debt in the form of longer project times and increased risk of defects due to unreadable code.

Some argue, and I’m one of them, that sometimes it is necessary to incur technical debt for the sake of speed. In a personal example I had a feature that was required to be implemented inside a week. My team estimated the work at 2 weeks. This was unacceptable due to an externally imposed deadline. The team offered a hacky solution to the problem that resulted in a quick turnaround. We offered the solution on the condition that we would immediately be given the 2 weeks to correct the design flaw. Agreement was reached and we released the feature in 1 week. We finished the feature in 3 weeks.

When you take on technical debt, you don’t reduce the cost of the feature–you increase it. You take on the work of the hacky solution and the work of reworking the hacky solution.

The reason we were able to reach this agreement is that our product owner and our team all understood that bad code is more expensive. We deliberately wrote bad code for the sake of instant gratification and then we immediately paid the full price for good, tested code.

We’ll fix it later

My colleague @jrolstad likes to say

Technical Debt is the lie we tell ourselves that we’ll come back and fix it later.

This is a phenomenon widely observed by many software engineers. We complain that the software is rotting and are promised an opportunity to “fix it later,” but “later” never seems to come.

What is always coming but never arrives? Tomorrow

— Children’s joke

Your code is your design

There’s an antipathy toward useless documentation in the Agile community, especially documentation about what code does. “The best documentation of the code is the code.”

When it comes to the design of their software, many developers make the mistake of thinking of the system in terms of their aspirations for the code base. The design of the system is always its current state. If you take a policy of accumulating technical debt in your code base then your actual design is a mess–not the gleaming structure of rationality you imagine it will be when you fix the technical debt… later.

What can we do?

If your organization has a legitimate need to take on technical debt, we can insist that the work to repair the debt be placed on the work calendar immediately. Most of the time the “business value” of getting a feature delivered fast is an attempt to pretend that a feature doesn’t cost as much as it does.

As a software engineering professional, we should not pretend that features do not cost what they do. We should not lie to the business, nor help them lie to themselves.

Estimation

These ideas have an implication with respect to estimation. If you owned a home and wanted to add a room on the second floor, how would you react if your contractor said “Well, these beams are rotted. We can probably build the room without replacing the beams, but your house may collapse in ten years. What do you want to do?” If you are anything like me, you would be appalled that the contractor even offered the option. The fact that the beams are rotted requires that they be replaced. This is not optional. The problem is not “solved” by building a room on rotted beams that will collapse in 10 years. It is not solved even if we know we are going to sell the house in 5 years (I’m looking at you startups).

The engineer should not offer to build the room on rotted beams. We should not offer or suggest alternatives to the business that result in the accumulation of technical debt. We should be honest with ourselves and with our employers about the real cost of the work they have requested.

But what if there’s a legitimate reason?

Is it always wrong to accrue technical debt? I don’t think so. As I stated earlier, I believe it can be acceptable to take some short-term shortcuts to get a solution out fast. However, this should be anomalous and the resultant mess should be cleaned up immediately following the achievement of the business goal. It needs to be understood by all parties that the shortcut costs more, not less.

How do I sell this to the business?

This is a complex topic and I don’t have all the answers. I have some answers and some promising leads.

First, stop presenting hacky alternatives in your estimate. Look at the code and honestly assess what it will take to alter it correctly. What is it going to take to add appropriate tests where they are missing? To clean the code so it will tolerate the change? To make the change? To repair an architectural deficiency? Estimate and present your estimate confidently and do not offer hacky options. If the business wants to negotiate, ask which features they would like to drop from the project. Do not offer to take engineering shortcuts.

Another tactic you can take is to measure estimates vs. technical debt in the code. You’ll have to start collecting some data for this approach and it will take some time. You’ll need:

• Project estimates
• How long the projects actually took
• Cyclomatic complexity for the affected code.

You’ll want to relate the accuracy of estimates to complexity. You should see that estimation accuracy decreases as complexity increases. For extra points, you can look at actual project estimates for similar features in code bases with different complexities. You should see that that more complex code bases are harder to maintain–even from an estimation perspective–than their simpler counterparts. You can use this information to put teeth into the claim that unclean code costs the company money.

I’d love to hear any other ideas you might have in the comments!

Sometimes we fall into the trap of thinking that since no one is complaining about our work then everyone must be happy with it. This is a dangerous mode of thought because our customers may not in fact be happy with our work, and because it inhibits us from improving ourselves. Getting accurate customer feedback about our performance is critical if we are to continually improve.

A Personal Example

At Redacted Financial I have some resource needs that are slow to be filled. There’s a process in place to make sure you get the right resources, but budgetary constraints mean that resource needs must be demonstrated before they are doled out. This makes sense to a degree but:

Budgets are there to keep you from being irresponsible. They should not keep you from being smart. 
--Chris McKenzie

Yes, I’m quoting myself–but I really like that formulation :). The process to get more resources is just painful enough that my team members are averse to going through it. Our team has similar needs across the board so iterating through the approval process for each member of the team strikes me as wasteful. I’d rather identify a baseline for all of my team members and start each person with those resources. When I proposed this idea I was told:

We have resources allocated in wildly different ways and nobody is complaining. Clearly there is no standard baseline for your team.
--The Gatekeeper

I thought this attitude was interesting. In the mind of this engineer, “nobody is complaining” is equivalent to “everybody has what they need.” Sometimes as engineers we operate on the assumption that everyone complains if something is bothering them.

This is false. Marketers have known this for years. That’s why they spend so much time and effort to get you to fill out customer satisfaction surveys. Sometimes the most valuable information you can mine for is “where am I failing?” Marketers know that for every person who is vocal about there complaints, there are hundreds they will never hear from.

How can you find out if your customers are happy?

Let me ask a different question first: “Who are your customers?” There are different people who are impacted by your work. They are all stakeholders, but not every stakeholder is your customer. If you can’t look at your stakeholders and clearly identify your primary customer, consider using a Responsibility Assignment Matrix.

I found it useful in a recent project to use the RASCI matrix which is defined as follows:

• Responsible – This is you in our example
• Accountable – This is your customer. This is who you are accountable to. This isn’t your boss (necessarily). This is the person who will use the end-product of your work. It’s to enable them in their goals that you are doing yours.
• Support – These are stakeholders who will support you in your efforts, but who do not directly consume the end-product of your work.
• Consulted – These are stakeholders who should be consulted about your work. They may need to approve some aspect of what you’re doing, or they may have important insight about how you should go about your job.
• Informed – These are stakeholders who should be informed about your work and/or your progress.

These matrices are often used to facilitate the repair of organizational disfunction, but they are also useful simply to clarify the roles & responsibilities of all of the stakeholders in your personal work ecosystem.

Once you’ve identified your primary customer, the simplest way to find out if they are happy with your work is… ask them. Face-to-face conversations are nice where possible. If you are lucky enough to be able to have a face-to-face conversation with your customers, try to adopt an attitude that is open to criticism. Don’t interrupt what they’re saying with explanations or excuses–even if you disagree with what they’re saying or if you think they’re wrong about something. There will be time for responding later. For now, your task is to listen and gather as much information about their assumptions and agendas.

Some customers will be confrontation-averse so a face-to-face conversation may not yield honest results. As mentioned before, surveys might be useful. An anonymous comment box (or anonymized email account) could work. It’s on you to figure out how to mine the information.

What do I do with customer feedback once I get it?

It may be hard to get the information. It may also be hard to hear it. You should take some time to reflect on the feedback before you respond, especially if it’s negative. Try to distance yourself from any initial emotional reaction so that you can consider more than just what the feedback says about you and your work. What is the underlying agenda your customer is trying to achieve? Are you helping or hindering that agenda? Are they lacking any key pieces of information? Are there other easy solutions to their problems?

When your customer gives you feedback, the worst thing you can do is not respond to it. If you fail to respond to feedback–positive or negative–you send the message that you do not value it. If you don’t value the feedback, it shows you don’t value your customer. You should respond to feedback even if you’re not sure you can do anything to address their underlyng complaint.

In our recent town hall, it was expressed that working on project teams instead of product teams isn’t ideal from the perspective of collective code-ownership. The response was “The Business doesn’t want to work that way.” We are not resourced to have product teams for the 50 or so applications we manage. Business priorities shift and sometimes require all of our resources to concentrate on a few applications at a time. This wasn’t happy news for our development teams, but it is understandable. My point is even if you can’t do anything to address the critical feedback due to issues beyond your control, you can respond to the feedback by explaining the other constraints to your customer.

The response to the town hall was overwhelmingly positive. Sometimes, it is enough for your customer to hear a good reason why it’s hard to accomodate their needs to reduce their frustration.

This week at Redacted Financial we had a “town hall” style meeting that included the entire software engineering organization. The meeting was run lean coffee style. No work-related topic was off limits. Our director made himself available for 2 hours to answer any questions about why our organization is run the way it is.

We discussed:

• Why we have a separate project management group.
• The rationale behind our work-from-home policy.
• Why we tend toward project teams rather than product teams.
• How we can address technical debt & product direction using project teams.
• Why our teams are sized the way they are.
• Maintenance and dissemenation of current best practices, standards, and guidelines for different kinds of softare.

This meeting is similar to a retrospective at the end of a project or sprint. It is different in that there are things that we can’t change because they are constraints imposed on us by the business, but understanding where those lines are and what we can change was valuable to everyone. One of the things that we are changing is our work-from-home policy. Those who advocated for a more liberal policy are involved in a small team working with the Director to write up a new policy.

Our Director offered to have a version of this meeting annually. The team immediately objected “We should have this meeting quarterly!” So we are. We’re going to allocate 1hr per quarter for this style of meeting.

The value-add as I see it is that the team can be involved in changing how their organization works within the larger enterprise. Where there are barriers, the team can be informed about what they are and why they’re there. For those who feel strongly that a given barrier shouldn’t be there, they can direct their attention to problem-solving how to remove the barrier while still accomplishing the business objectives that caused the barrier to be erected in the first place. A side-effect of empowering team members to make change is that it reduces the amount of work management has to do to identify opportunities for change and push the through.

The town hall meeting was a very positive experience for our team. I wonder if anyone else has done anything similar and what their results have been.