Deep SDLC Analysis

…a process for

planning, creating, testing, and deploying

an information system

http://en.wikipedia.org/wiki/Systems_development_life_cycle

Cycles Decomposed

Requirements Code Testing

Deployments Defects

Software

Piecing it together

Get some requirements

Write some code

Commit code

Commit tests

Build and test

Manual testing/verification

Deploy

We start “analyzing”

• Create a ‘backlog’

• Start estimating

• Work (sprint) planning

• Work assignments

• Estimated delivery date

• Burn down charts

Don’t regularly “analyze”

• Code

• Automated tests

• Commit messages

• We could though

Some “analysis” is done

• Build fail/pass

• Areas with high failed tests

• Defect churn

Never “analyze”

• Deployments

Analysis of analysis

• Inconsistent across SDLC areas

• Shallow

• Silo’d

• Very immature

• Missing software & data

Inconsistency & Immaturity

• Data is silo’d by team function

• Don’t believe that areas are generating meaningful data

• Don’t know what to look at

• Don’t know how to look at the data

Develo

per

Test

er

Develo

per

BA

Silos

Work

Ite

ms

Sourc

e C

ontr

ol

Conti

nuous

Inte

gra

tion

Test

Resu

lts

Deplo

ym

ents

Meaningful DataExample: Source Control

Each check-in• Date-time

• Committer

• Comment

Metadata• Defect/Work Item #

• Files changed/added

• Contents changed/added

• Hash/commit #

• CR-LF, LF, <sigh>

What to look at?Example: Work Items & Defects

• Defect age

• Open-to-close duration

• Frequency by application area

How to look at the data?

• Don’t get too granular (i.e. individual defects)

• Look for trends• Change in code coverage

• Change in defect open time

• Change in estimate vs actual spread

Getting More

• Stop silo-ing• By application

• By job description

• Tie the current pieces together

• Add the missing pieces

• Your biggest integration project yet

Add value by adding nothing

• Start using existing analysis scripts• git_stats

• gitstats

• rewind

rewind

• Git history analysis

• github.com/gilesbowkett/rewind

rewind

rewind

Tying systems together

• Work Items & Defects

• Source Control

Defects & Source Control

• Commit with defect/work item # in message

Defects & Source Control

Defects & Source Control

Query this:

- All defects that have been closed since <enter date>

- All commit hashes from returned defects

- Get all commit log entries for those hashes

- Get all modified/added files from those commits

== Modified files for this release

Add a bit more

• Work Items & Defects

• Source Control

• Continuous Integration

Some Continuous Integration

CI information

• Get last pinned buildlocalhost/httpAuth/app/rest/builds/pinned:true,project:Umbraco,sinceBuild:127100

• Get changes included in the buildlocalhost/httpAuth/app/rest/changes?locator=build:(id:122960)

• Get files in the changeslocalhost/httpAuth/app/rest/changes/id:209381

So……?

We know the files changed between pinned builds

We also know highly volatile files (remember rewind?)

Can we tell if we have high risk files in this release?

Well……sorta……

Predicting code risk

• Multiple ways to try to accomplish this

• Active Analysis• You have unit tests right?

• You measure code coverage right?

• Static Analysis• Soooo many options

rewind to rewind

• Shows past performance

• Strong indicator of the chance for change

• Weak indicator of the risk of the changes

Automated Testing

• Some tests are good

• Some tests are not so good

• Tests against a class is a sign that you tried to care

• Not how much you cared

• Their existence is a mediocre indicator of risk

Piecing it together

Get some requirements

Write some code

Commit code

Commit tests

Build and test

Manual testing/verification

Deploy

Linking active analysis

• From CI we know the files that have changed

• CI will give us the names of all tests that were runlocalhost/httpAuth/app/rest/testOccurrences?locator=build:(id:122960)

• We can compare the lists to find files that changed that don’t have pattern matched test names

Customer.cs � CustomerTests.cs

InvoiceServices.cs � InvoiceServicesTests.cs

etc

More active analysis

• Determining confidence in matched files/classes

• Code coverage• How many lines of code were executed by tests?

Linking more active analysis

• From CI we know the files that have changed

• Code coverage will give us the %ages for each file/class

• We can compare the lists to find files that have low levels of coverage. This is the riskiest code under test.

Better predictions

• Test existence and code coverage are, at best, mediocre indicators

• What if we have 100% coverage on 100% of the files?

• Deep static analysis

Piecing it together

Get some requirements

Write some code

Commit code

Commit tests

Build and test

Manual testing/verification

Deploy

Which metrics to use

• Afferent/Efferent Coupling• How many types do I rely on?

• How many types rely on me?

• Cyclomatic Complexity• How many different logic paths are there that can be executed?

• Instability (Efferent Coupling/Total Coupling)• How resilient is the object to change?

Linking more static analysis

• From CI we know the files that have changed

• Can determine threshold crossing on different metrics

• Files found in both lists were changed, but probably weren’t easy to change

So you’ve identified some files…

…that are at risk for having introduced defects, so how do you reduce/limit those yet to be found defects?

• Manual testing

It’ll catch some of the problems

Different eyes see different things

• Will they test for nuanced technical constraints?

Developer enters a bar.

Orders a Βeer.

Orders a βeer.

Orders a Вeer.

Orders a Beer or Drink=Drink.

Orders a Beer; Update Drinks Set Price=0.00;

• Peer code reviews

Piecing it together

Get some requirements

Write some code

Commit code

Commit tests

Build and test

Manual testing/verification

Deploy

Triggering code reviews

1. CI compiles code

2. CI runs automated tests

3. CI runs code coverage

4. CI runs analysis scripts (rewind)

5. CI runs static analysis (nDepend)

6. Successful build triggers 2nd build project1. CI parses results from #3, #4, #5

2. Broken rules � call REST endpoint to create code review task

If it moves, automate it.

If it doesn’t move, kick it and then automate it.

Deltas

•T in code coverage %

•T in LoC : commits

•T in coupling

•T in any metric you’ve decided to care about

• Probably not commit-to-commit

• Probably a release-to-release metric

Deltas

• Systems need to be able to look at past data

• Many systems are limited in this fashion

• You’re probably going to have to write your own tool

• Make delta comparisons a build step like code coverage

So we’ve done some cool stuff

• We can identify code that is at risk for change

• We can identify code that has changed

• We can identify how much risk there is in a release

• We can pre-empt some of that risk earlier in the cycle

But we’ve only focused on the code

…and PMs, BAs and QAs *hate* it

when we focus solely on the code

Nothing matters more than delivering

How long would it take for your organization to deploy a change that involves just one single line of code? Do you do this

on a repeatable, reliable basis?

Mary Poppendieck

How long does it take you to deliver feature/defect X?

Measuring delivery/cycle time

Create defect/work item

Close defect/work item

Deliver the build to production

Getting that info

• Deployment system ties to a package

• Package ties to a CI Build #

• CI Build can be used to determine included commits

• Each commit is linked to a defect tracking entry Time created

Time deployed

Cycle Time

Performance

• Do you perf test?

• How often?

• Do you track changes in perf?

Performance

• Part of a CI project • its own project since it will take time

• its own build agent to ensure isolation

• Make the output an artifact these can support deltas

• Changed files can focus analysis of perf output

• Quantifiable performance values• first run gives a baseline

• Supporting deliverables when deploying• Look! It didn’t get slower when we added that feature!

How do you decide what to fix?

• Everything is always “High” or “Critical”

• Its always the finder’s baby/priority issue

• More users == more conflicting “High” issues

You fix it and find out “oh, we only use it once a year”

or “We use a different system, but thanks”

Helping to prioritize

• Don’t take their word for it…do they *really* use it?

• System instrumentation• Log by feature relate them back to other systems

• Log each use able to count the number of uses

• Log when used determine frequency of use

Adding instrumentation should be easy. Use AOP. Be thorough.

Prioritizing

Feature with defect but low use count might not be as critical to fix.

Feature used once a month might not have to be fixed and delivered tomorrow.

So where does that leave you?

• Have some of these systems in place

• Generating a lot of data that you’re never using

• Probably have some gaps

So where does that leave you?Get some requirements

Write some code

Commit code

Build and test

Manual testing/verification

Easy winsGet some requirements

Write some code

Commit code

Build and test

Manual testing/verification

Code Reviews

Static Analysis

More workGet some requirements

Write some code

Commit code

Commit tests

Build and test

Manual testing/verification

Deploy

Code Reviews

Code Coverage

Static Analysis

Still more workGet some requirements

Write some code

Commit code

Commit tests

Build and test

Manual testing/verification

Deploy

Code Reviews

Code Coverage

Static Analysis

Performance Tests

The most workGet some requirements

Write some code

Commit code

Commit tests

Build and test

Manual testing/verification

Deploy

Code Reviews

Code Coverage

Static Analysis

Performance Tests

The gain

• Better planning

• More accurate estimates

• Less defects getting to testers

• Better understanding of the health of our software

gracias

Donald Belcham@dbelcham

donald.belcham@igloocoder.com