Cucumbers Have Layers - RubyConf 2015

A Love Story

CUCUMBERS HAVE LAYERS

Sam Livingston-Gray RubyConf 2015

!

7

Welcome!

I'd like to get a quick [ADVANCE] show of hands.

AUDIENCE POLLHow many people in this audience...

Have used Cucumber (on any project)? Have used Cucumber more than once? Would use Cucumber again?

❓

8

[REVEAL EACH] - How many of you have used Cucumber on any project, big or small?- How many have used Cucumber on more than one project?- And regardless of how many times you've used it before, how many would use it again?

This talk is directly aimed at those of you who may have used Cucumber in the past, and decided not to use it again. I hope to at least offer you a different perspective, and convince you to at least give Cucumber another look.

9

For those of you who haven't used Cucumber, you should go get The Cucumber Book before you start. It'll give you a much better introduction than I possibly could, even if this were a full 45 minutes of "Cucumber 101."

But, just so you're not completely lost...

CUCUMBER 101❓

10

In Cucumber, you describe features of your software in a language called [ADVANCE] Gherkin.

CUCUMBER 101: GHERKIN

❓

11

Gherkin is a DSL for writing acceptance tests.

Now, because this is a Ruby conference, and we have a tendency to say "DSL" when we mean "API", I have to clarify: when I say Gherkin is a DSL, I mean that it is an actual domain-specific language, with its own grammar and semantics. Gherkin is not Turing complete, but it can be used to tell a Turing-equivalent machine what to do.

As I was saying, Gherkin is a DSL for describing software. Each separate Gherkin file is called a feature, and here's a feature that I pulled straight from the Cucumber website.

Feature: Addition

In order to avoid silly mistakesAs a math idiot

I want to be told the sum of two numbers

Scenario: Add two numbers

Given I have entered 50 into the calculatorAnd I have entered 70 into the calculator

When I press add

Then the result should be 120 on the screen12

A feature has one or more scenarios, and a scenario has one or more steps: those are the given/when/then that you see toward the bottom left of the screen.

Aside from a few keywords, which I've highlighted here in green, everything else is written in whatever natural language works for you. Gherkin's grammar is really simple: everything from a keyword to the end of the line is basically treated as a single token by the Gherkin parser.

This is quite useful just as documentation, but Cucumber also lets you use these feature files to automate tests, which is why the people behind Cucumber talk about "executable specifications".

To go from human-readable documents to running tests, you have to write a bunch of [ADVANCE] step definitions.

Given /I have entered (.*) into the calculator/ do |n|

@calculator = Calculator.new

calculator.push(n)

end

13

A step definiton is a regular expression, plus a block of code. This is how you translate from human-friendly blobs of text to something that Ruby can execute.

When Cucumber wants to run a step, it tests it against every one of the regular expressions you've given it. When it finds a match, it executes the block that follows that regex. There's also a mechanism for taking capturing groups from the regex and passing them as arguments to the block, which is how you can get interesting data into your tests.

Gherkin Features Scenarios Steps

Cucumber Parses Gherkin Steps → Step definitions Runs each Scenario step by step

CUCUMBER 101: CUCUMBER❓

14

In my mind, Gherkin and Cucumber are almost two separate things.

[REVEAL EACH] - Gherkin gives you a human-friendly way to describe software,- and Cucumber processes your Gherkin files and uses them as a script for automating tests.

Cucumber can be... kind of unwieldly. I basically put up with Cucumber because I really like Gherkin.

GHERKIN IS:AWESOME

#

15

[PAUSE BRIEFLY] Gherkin is a domain-specific language where the domain is "talking to other humans about software." It's very free-form, so it lets you talk about your application's domain using whatever natural language makes sense to you and your team.

Gherkin has just enough structure that it can be used to drive a lot of machinery for automating tests. But...

GHERKIN IS NOT:CODE

!

16

Gherkin is NOT a programming language! This is a critically important point that can be easy to overlook when you're starting out with Cucumber. Programming languages are great, but they require us to get all the details right up front, and that process tends to shift our focus onto how to do a thing.

Gherkin, on the other hand, exists to help us think about what thing to do, why we're doing it, and who we're doing it for.

CUCUMBER IS NOT: TDD

!

17

I think it's also important to realize that Cucumber is not a tool for doing test-driven development. Cucumber and TDD complement each other nicely, but it's been my experience that Cucumber works on a very different rhythm than TDD.

Cucumber Scenario

Unit TestFail

Pass Refactor

Fail

Pass

Refactor

18

I think of Cucumber as a set of guide rails for TDD, and my workflow for using it goes something like this...

[REVEAL EACH] - I start with a Cucumber scenario. I run the scenario...- and watch it fail. I look at the error message to find out why it failed, and use that information to go...- write a unit test.- I watch the unit test fail...- make it pass...- and refactor. At this point, I have a choice. If I know what the next unit test you need to write is,- I do that, and go back around the red/green/refactor cycle again, usually several times. But if I'm stuck...- I go back and run the Cucumber test again. It's probably still failing, but it's failing for a different reason. So I go back into the TDD cycle again. At some point, I run the Cucumber test and...- it passes. I do a little dance,- refactor,- move to the next scenario.

Cucumber Scenario

Unit TestFail

Pass Refactor

Fail

Pass

Refactor

19

When I'm working this way, I spend most of my time in that tight inner loop doing TDD: red/green/refactor. The fact that the tests and the code under test are both written in the same language makes it easy to go around the inner TDD loop very quickly—sometimes a test every minute or so. This is where I'm focusing on how the thing works, and it's good, satisfying, detail-oriented work.

Cucumber Scenario

Unit TestFail

Pass Refactor

Fail

Pass

Refactor

20

But when I start to lose sight of the forest for the trees, I jump back up to Cucumber. The shift from Ruby back to Gherkin helps remind me to get out of that hyperfocused how mode, and come back to thinking about what, why, and who... and that helps me remember what the next thing to do is.

"For me, Cucumber worksmore like a mind hack than a testing tool.

It reminds and invites me to think about the experience of using software separately from

the details of its implementation." -Tom Stuart

http://codon.com/notes-on-counting-tree-nodes

MIND HACK!

21

Tom Stuart wrote something about Cucumber that really resonated with me. He described it as [REVEAL] "more like a mind hack than a testing tool", because it helps him think about the big picture rather than the details.

http://codon.com/notes-on-counting-tree-nodes

"I wish more people knew it works bestas a thinking and collaboration tool,not just an automated checking tool"

-@mattwynne

A THINKING TOOL$

22

As I was putting this talk together, I asked Matt Wynne, co-author of The Cucumber Book, if there was anything he wanted people to know about Cucumber. He tweeted back that he wished more people knew Cucumber as [REVEAL] a thinking and collaboration tool, not just something for test automation.

Both of these quotes lead back to something I said a few slides ago.

I asserted that Gherkin is not code, and that Cucumber is not for TDD. But negative definitions aren't very useful. Or, to put that another way, a positive definition is much more useful than a negative one. What are these things for? Well, I already talked about how I use Cucumber as guide rails around an inner TDD loop, but let's talk about Gherkin some more.

Describing software

At the level ofUSER INTENT

(and maybe automatingtests with Cucumber)

GHERKIN IS FOR:!

23

By the way, everything I'm saying in this talk is my own opinion, based on my own experience. I don't speak for the Cucumber team here.

I think that Gherkin is for: [REVEAL EACH]- describing software- at the level of user intent.

And you might choose to use Cucumber to turn your Gherkin artifacts [REVEAL] into automated tests.

Let's unpack that one piece at a time.

Acceptance CriteriaDESCRIBING SOFTWARE

If this doesn't work,our users will abandon us.

!

24

By "describing software", I mean that Gherkin lets you capture [REVEAL] "acceptance criteria".

And by "acceptance criteria", I mean [REVEAL]"the system has to do this stuff or you don't get paid."

When I add a new widgetUSER INTENT

NOT: When I visit "/widgets" And I click the "New Widget" link And I fill in "Widget Name" with "My Widget" And I click the "Create Widget" button

!

25

By "user intent", I mean that your Gherkin paints its picture [REVEAL] in broad strokes, without getting bogged down in a lot of details. Details are what TDD is all about.

[REVEAL] I've worked with scenarios that look like this, and what I've found is that every time I tweak my user interface, twenty of my Cucumber scenarios will explode, and I have to spend an hour or two editing them, which... is not the best use of my time.

(maybe)AUTOMATING TESTS

!

26

Just because Cucumber is pitched as a tool for writing automated tests, you're not obligated to use it that way. Personally, I think Cucumber's greatest value is as a tool for facilitating conversations between developers and the people who pay us. I've written Gherkin files, thrown them away, and felt like my time was very well spent.

!

27

Describing software

At the level ofUSER INTENT

(and maybe automatingtests with Cucumber)

So this is what I now think Cucumber should be used for. But it took me a long time to figure this out, and I made a lot of mistakes along the way. Some of those mistakes were rather painful. In the hope that you can learn from my mistakes, I'm going to share them with you.

Inconceivable!CLASSIC BLUNDERS

%

28

Yes, this is the part of the talk where you all get to laugh at me. [REVEAL]

This is by no means an exhaustive list of my Cucumber fuckups; these are just some of the more interesting, entertaining, or educational ones.

When I visit "/test/js-tests" Then I see 99 passing QUnit tests

WTF%

29

I'm going to show you a Cucumber scenario that I helped write in a real live codebase.

Before I show it to you, let me reiterate: it is okay to laugh at me.[REVEAL AND... WAIT FOR IT]

This visited a route that was only defined in the test environment. That route rendered a static view that, in turn, required a JavaScript file that contained all of the unit tests for our front-end helper functions.

That number 99 actually started out somewhere around 40 or 50. But we kept adding more JavaScript tests, and every time we did that, we had to update this Cucumber scenario with the new number.

How did this happen? Well... we had a fairly large Rails project with an extensive Cucumber test suite, and we needed to run some unit tests in the browser, and we thought, "well, here's this thing that's already set up to drive Selenium..."

...and apparently there weren't any grownups around to stop us.

Given the Rails application Then CRUD works for the Widgets controller

ACCEPTANCE ASTRONAUT%

30

Here's something else I did in an actual project. [REVEAL]

What this did was- visit the new page for the widgets controller,- fill out the form with randomly generated data,- submit the form,- check that the same data it submitted was displayed on the show page,- click the edit link,- change each value on the edit form, click "Save", make sure that the changes were visible... and so on.

To make this work, we wrote something that could automatically mutate values, and to make *that* work, we added CSS classes to indicate that a given field contained numbers, or names, or addresses... but that's just good semantic markup, right? At least, that's what we told ourselves.

Seriously, this was a lot of fun, but while we were gold-plating our Cucumber suite, we were avoiding writing actual features that our actual customer actually cared about, and pretty soon they actually fired us.

Next up...

CUKING REGRESSION TESTS%

31

When you get a bug report, it's a good idea to add an automated test to reproduce the bug, and prevent any regressions. It is not a good idea to put these tests in Cucumber.

Gherkin is a great way for you to tell the story of your application. Ideally, you can print out all of your feature files and hand them to a new developer or manager. They should be able to read through those in an hour or two and come away with a pretty good high level idea of what your software does. Cluttering that up with a bunch of regression tests turns your Hemingway into Charles Dickens.

Doing this is also a good way to commit the next blunder, which is [ADVANCE] having too many scenarios.

*(to run on every commit)TOO MANY SCENARIOS*

%

32

Opinions vary on how long is a reasonable time to wait for a test suite. Personally, I'm willing to wait about five minutes, up to three or four times a day. Any more than that and my tests might as well not be running.

If you do find yourself in this situation, you might consider tagging a critical subset of your tests to run before every commit, then let your CI server run everything else after you push.

Another way to commit the "too many scenarios" blunder is [ADVANCE] to automate every feature you write.

@FYI / @TBDAUTOMATING EVERY FEATURE

%

33

It's perfectly okay to use Gherkin to facilitate a conversation with someone—possibly yourself—and then throw away the feature file once you've learned what you needed to learn from it!

If do you feel the need to hang on to it for posterity, go ahead and check it in to your features directory, but tag it as [REVEAL] "@FYI" or "@TBD", and change your configuration so that scenarios with that tag never run.

ABUSING STEP DEFINITIONSToo many step definitions Huge step definitions Too many huge step definitions Step definitions that call other step d e҉ finiti ons Just about any l

o

gi

c

whatsoever in a

s

t ep

d ę

f

i

n

i

t

҉

i

ô

n

&

Given /some rubbish/ do

# ...

end

34

And, finally, step definitions. There are *so* many ways that step definitions can make you hate life.If you get the Cucumber book—and you should get the Cucumber book—it will tell you to define a bunch of helper functions and invoke them from your step definitions, and that does help. But... I started using Cucumber years before the Cucumber book was published. Which means I've made mistakes like:

[REVEAL EACH. DO NOT AD-LIB NUMBERS; THEY'RE COMING SOON]

</RANT>%

35

After making all of those mistakes and more, I found myself feeling very conflicted about Cucumber. I really loved the expressiveness of Gherkin, and I wanted to believe in this idea that programmers and managers could sit down in a room and write acceptance tests together in universal harmony.

But I struggled to reconcile that with the project I'd been working on, which had hundreds of scenarios backed by 750 step definitions that contained almost 5,000 lines of code, and the whole test suite took about 90 minutes to run.

Eventually, I found myself asking an interesting question...

❓

THE QUESTIONHow would you write scenarios

if you didn't know what the UI was going to be?

36

[REVEAL AND READ] [PAUSE]

If you can tell from reading your Cucumber features whether you're using a web application or a desktop application or a CLI... you're probably letting too much detail leak into your features.

Here's an example...

❓

When I visit "/widgets" vs.

When I view the list of available widgets

THE QUESTION

37

[PAUSE 5 SECONDS. THE SILENCE WILL NOT KILL YOU.]

Here's another example...

❓

When I POST <some JSON> to "/widgets/42" vs.

When I save my changes to the current widget

THE QUESTION

38

[PAUSE 5 SECONDS. BREATHE.]

❓

THE QUESTIONHow would you write scenarios

if you didn't know what the UI was going to be?

39

This question floated around in my head for a while as I worked on other things, until...

THE PROJECTSalesperson Commissions Multiple compensation schemes Schemes change every quarter or two Described in quasi-legalese

!

40

Once upon a time, I was brought in to work on one part of a rather large monolithic Rails app that calculated [REVEAL] salesperson commissions. Now, you might hear "salesperson commissions" and think "okay, so you add up how much each person sold, multiply the total by some percentage, and cut a check, right?"

But that would be *far* too simple. There were usually [REVEAL] half a dozen compensation schemes in effect at any one time. These schemes changed [REVEAL] a couple of times a year, sometimes quite dramatically. And they were worked out by the sales department, who would put together something like fifteen pages of [REVEAL] dense, confusing quasi-legalese to describe each plan, which we then had to read through and somehow translate into working code. So one of my goals as I worked on this project was to be able to describe every aspect of these schemes using Gherkin.

This is a work of fiction.DISCLAIMER

Any resemblance to actualsales compensation schemes is...

unfortunate.

!

41

I'm going to talk about a simplified, fictionalized version of how just one of those compensation schemes worked. It's not super important for you to catch all of the details here. They're not really relevant to the talk; they're just here to give you some concrete examples of how I wrote and organized my features for this project.

TARGETSSales Target: $100,000 / month Target Bonus: $100

!

42

We'll start with the concept of a "sales target" and a "target bonus". This is, basically, the company saying "if you sell $100k of widgets in a month [REVEAL] (that's the sales target), we'll pay you $100 over your base salary [REVEAL] (that's the target bonus)."

By the way, these numbers aren't realistic; I just picked them to make it easy to convert between percentages and dollars in my head.

Anyway, there's a scaling factor here: if you miss your target, you get paid less. If you exceed your sales target, you get paid more. So far, so good—but there's a catch.

PAY CURVE!

% to Sales Target % of Target Bonus paid

0% 0%

50% 25%

100% 100%

150% 175%

43

The catch is this little thing called a "pay curve". The pay curve is a simple function: you put in what percentage of the sales target you hit, and you get back the percentage of your target bonus that you'll get paid. To describe the pay curve, the sales department actually gave us a spreadsheet with example rows for every possible input value from zero to 250.

Fortunately, since it was already in a spreadsheet, it was easy to build a chart so we could see the curve.

PAY CURVE!

Perc

ent o

f Bon

us

0%

75%

150%

225%

300%

Percent to Target0% 25% 50% 75% 100% 125% 150% 175% 200% 225% 250% 275%

44

Looking at the chart shows that this is a "piecewise linear function", and that helped me wrap my head around what was happening. So [ADVANCE] I went back to the spreadsheet...

PAY CURVE!

% to Sales Target % of Target Bonus paid

0% 0%

50% 25%

100% 100%

150% 175%

45

and from there, it was quite straightforward to convert that spreadsheet into a Cucumber table, and use it to drive a scenario outline.

Feature: Pay Curve for Sales Associates

Scenario Outline: "Sales Associate" Pay Curve Given the "Sales Associate" pay curve When the rep hits <Percent to Target> of their sales targets Then their monthly payout is <Percent of Bonus> of target monthly bonus

Examples: | Percent to Target | Percent of Bonus| | 0 | 0.0 | | 1 | 0.5 | | 2 | 1.0 | | 49 | 24.5 | | 50 | 25.0 |

46

In case you're not familiar with it, a scenario outline is basically a [REVEAL] template for a scenario, followed by [REVEAL] a table. The template gets executed once for each row in the table, with [REVEAL] values from the appropriate column filled in wherever you put a placeholder value.

Now, because I knew that this was a piecewise linear function, I was able to get away with not turning all 251 rows of the spreadsheet into a giant Cucumber table. I put in a few examples around each of the inflection points just to make sure I got the boundaries right, and then moved on to the compensation scheme...

Feature: Compensation Scheme for Sales Associates

Background: * I am a sales rep using the "Sales Associate" scheme * I have a monthly sales target of $100,000 * I have a monthly target bonus of $100

Scenario Outline: Using the right pay curve When I make <Sales> in sales Then my monthly sales bonus is <Bonus> Examples: | Sales | Bonus | | $ 0 | $ 0 | | $ 50,000 | $ 25 | | $ 100,000 | $ 100 |

47

At first glance, this looks remarkably similar to the scenarios for the pay curves. When I first wrote this, it felt like I was repeating myself, but this actually introduces quite a few new concepts:

[REVEAL EACH] - compensation schemes,- sales target and target bonus in dollars instead of percentages,- actual sales in dollars, and- bonus amount in dollars.

With those concepts in place, I could then introduce the next feature of this scheme, which is the "safety net".

Feature: Commissions Plan for Sales Associates

Scenario Outline: Safety Net Given my safety net IS deployed When I make <Sales> in sales Then my monthly sales bonus is <Bonus> And my safety net bonus is <Safety Net>

Examples: | Sales | Bonus | Safety Net | | $ 0 | $ 0 | $ 100 | | $ 50,000 | $ 25 | $ 75 | | $ 100,000 | $ 100 | $ 0 | | $ 175,000 | $ 225 | $ 0 |

48

This is the last one, I promise.

The safety net is a feature to help out new hires as they're getting up to speed for the first few months. This is basically a guarantee that you'll always get paid at least the amount of your target bonus. If you don't hit your sales target, we'll kick in the difference. If you do better than your sales target, we'll pay you more than your target bonus -- but you'll never make less, at least until we take your safety net away.

Anyway, I think that's enough to give you a sense for how I organized and wrote the Gherkin features for this project. I do want to talk about an underutilized element of Gherkin's grammar, though.


FREE-FORM TEXT GOES HERE!

Scenario: Dolor sit amet

49

I omitted this on earlier slides so I could make the text bigger, but Gherkin gives you some space at the top of the file where you can write whatever you want.

Some of the examples and tutorials I've seen show that space being used for [ADVANCE] "As a / I want / so that"...

Feature: Lorem Ipsum

As a ______ I want ______ So that ______


50

...but in practice, I find that people tend to [ADVANCE] fill in that template without really thinking about it...

Feature: Lorem Ipsum

As a manager filling in this form I want to make up some plausible BS So that the developers do what I tell them to, damn it


51

[PAUSE]

So sometimes I just skip this part. For this project, though...


This scheme is based on gross sales, and uses the "Sales Associate" pay curve.

This scheme features a "safety net". If the appropriate flag is set, sales reps on this scheme are guaranteed at least 100% of their target bonus. This tends to be used in a new hire's first few months so they can earn a reasonable living while they get up to speed.

Background: * I am a sales rep using the "Sales Associate" scheme *[...]

52

...I used that space to provide some context about why this feature exists, or what makes this feature interesting in comparison to other features that may be similar.

[KEEP GOING]

Feature: Commissions Plan for Sales Associates

This plan is based on gross sales, and uses the "Sales Associate" pay curve.

This plan features a "safety net". If the appropriate flag is set, sales reps on this plan are guaranteed at least 100% of their target bonus. This tends to be used in a new hire's first few months so they can earn a reasonable living while they get up to speed.

Background: * I am a sales rep using the "Sales Associate" plan *[...]

53

For this scheme, I was worried that the examples of how the safety net worked in specific cases might not fully explain what that aspect of the scheme was for. So I took a few lines to explain it, using the simplest language I could.

After I handed this project off, one of the bits of feedback I got was that this documentation, in particular, was extremely helpful in making sense of these frankly ridiculous compensation schemes.

Again, this is the sort of thing you may overlook if you're trying to treat Gherkin as a programming language. If you're thinking of Gherkin as code, then this section of the file just feels like a big block comment. But if you're thinking about Gherkin as a medium for communicating with other people about your project, this freeform text area can be really useful, because it gives you a place to talk about things without having to fit it into a step-by-step recipe.


Scenario Outline: Safety Net Given my safety net IS deployed When I make <Sales> in sales Then my monthly sales bonus is <Bonus> And my safety net bonus is <Safety Net>

Examples: | Sales | Bonus | Safety Net | | $ 0 | $ 0 | $ 100 | | $ 50,000 | $ 25 | $ 75 | | $ 100,000 | $ 100 | $ 0 | | $ 175,000 | $ 225 | $ 0 |

54

The last thing I want you to notice about these features is that absolutely every word of this is expressed in terms of the domain, not the interface. If you sat down and read through all of these features, you'd learn a lot about how this organization thinks it can motivate its salespeople, but you won't have any idea what kind of app they're using to do it.

Now let's talk briefly about [ADVANCE] the architecture I settled on for the app.

ARCHITECTURE'

Controllers & Views

AR & Service Objects

Core logic in POROs

55

I did choose to use Rails, but I wanted to try a more disciplined approach than I usually see in Rails apps. So I organized the code into three main layers.

[REVEAL EACH]

- The UI is standard Rails controllers and views.

- The UI layer talks to a mix of ActiveRecord objects and some service objects.

- And that layer, in turn, talks to a set of plain old Ruby objects, or POROs, that model the rules for the compensation schemes themselves.

Anyway, this is a little weird for Rails, but it's nothing earth-shattering.

ARCHITECTURE

Architecture: The Lost Years-Robert Martin, Ruby Midwest 2011

'

56

If you want to explore those ideas a little more, Bob Martin gave a talk in 2011 called "Architecture: The Lost Years". Personally, I found this talk really hard to watch because I felt a lot of his rhetorical techniques detracted from the important things he had to say.

And while I was working on this sales commission project, I also ran across a really good presentation that Jim Weirich gave to Cincinatti.rb called "Decoupling from Rails."

ARCHITECTURE

Decoupling From Rails-Jim Weirich, Cincinatti.rb, Oct 2013

'

57

This is a talk with some good SOLID ideas in it, and it's well worth watching just for the main topic. But right at the very end of the video, after he finished the talk and the Q&A, Jim said something really interesting.

58

[CLICK TO PLAY]

When I heard that, I just about fell out of my chair, because that was basically what I was doing in this project!

Unfortunately for me, I never met Jim. So I didn't get a chance to talk to him about this "interesting experiment."

But I do get to share it today with all of you, so that's cool.

-------------------- TRANSCRIPT OF AUDIO --------------------

JIM WEIRICH: Let me tell you a goal that I have. [...] You can do integration testing coming in at this level and test all the way down to the database and back. That runs pretty doggone fast. The only thing you're not testing is the controllers, the webby stuff, the views and things.

I would like to demonstrate that if you do your Cucumber tests right, you could run your Cucumber tests at this level, or throw a switch and run it at this level. So if you want fast integration tests, run them here; if you want complete "include the web" integration tests, you can run them at this level. I think that would be an interesting experiment.

TESTING LAYERS!

AR & Service Objects

Core logic in POROs

59

Controllers & Views

@model

@ui

@core

My initial idea was basically what Jim had described. Using the tagging feature of Cucumber, I wanted to be able to mark a scenario as being [REVEAL] "@ui", [REVEAL] "@model", or both.

The scenarios tagged with "@ui" would use Capybara to interact with the full Rails stack in the way that most people think about using Cucumber with Rails.

But the scenarios tagged with "@model" would run directly against the ActiveRecord layer, so they could be faster. The ActiveRecord layer would exercise the POROs indirectly, and I thought that would be good enough.

However, I discovered very quickly that the PORO layer was complicated enough on its own that I didn't want to have to think about the relational data model at the same time, so almost immediately, I added [REVEAL] a "@core" tag as well.


Scenario Outline: "Sales Associate" Pay Curve

Given the "Sales Associate" pay curve

When the rep hits <Percent to Target> of their sales targets Then their monthly payout is <Percent of Bonus> of their target monthly bonus

Examples:

| Percent to Target | Percent of Bonus | | 0 | 0.0 |

| 1 | 0.5 | | 2 | 1.0 |

60

Here's how that played out in one of the feature files I showed earlier.

Once I had written a scenario, I would start by tagging it with the name of the layer I wanted to run it at, plus a [ADVANCE] "WIP" suffix to indicate that it was work in progress.


@core_WIP


Given the "Sales Associate" pay curve When the rep hits <Percent to Target> of their sales targets

Then their monthly payout is <Percent of Bonus> of their target monthly bonus

Examples: | Percent to Target | Percent of Bonus |

| 0 | 0.0 | | 1 | 0.5 |

| 2 | 1.0 |

61

I'd run the scenario and watch it fail, and from there, I'd drop down to RSpec and do the usual small, fast TDD cycles until the scenario passed. Once the scenario was passing, I'd [ADVANCE] remove the "WIP" suffix.


@core





| 0 | 0.0 | | 1 | 0.5 |

| 2 | 1.0 |

62

If I wanted to reuse the scenario at the next layer up, I would then add another tag for that layer, again with the [ADVANCE] "WIP" suffix.


@core @model_WIP





| 0 | 0.0 | | 1 | 0.5 |

| 2 | 1.0 |

63

And again, I'd write a bunch of RSpec tests until I got the same scenario passing at the new higher layer, at which point I would [ADVANCE] remove the "WIP" suffix from that layer.


@core @model





| 0 | 0.0 | | 1 | 0.5 |

| 2 | 1.0 |

64

So that's how I approached this from the Gherkin side of things.

But it took me a while to figure out a good way to implement this...

Messing with Cucumber load pathTHROWING A SWITCH

!

65

I wanted Cucumber to run all of the @core scenarios with one set of step definitions in place, then run all of the @model scenarios with a completely different set of step definitions, and then run all of the @ui scenarios with a *third* set of step definitions.

Manipulating the load path was painful, but it basically worked. The problem was that, any time I changed a step, I had to edit three different regular expressions in three different files and make sure they all matched. Which... [ANIMATION]

Swapping in different step driversTHROWING A SWITCH

!

Given /(.*?) in sales/ do |amount| step_driver.record_sales(amount) end

66

So instead, I wound up consolidating down to one set of step definitions that invoked something I called a step driver. [REVEAL, PAUSE]

This has the wonderful effect of making step definitions small and simple.

With this change, I was then able to define three different step drivers to interface with each layer of my application. I put all three of them on the load path, and used an environment variable to decide which step driver to instantiate.

Further details are beyond the scope of this talk; feel free to ask me about them later if you're curious.

OBSERVATIONS(

67

I just have a few observations to make before I wrap up.

First, a piece of advice from my inner five-year-old:

THE STEP DEFINITIONS ARE LAVA

68

(

Given /(.*?) in sales/ do |amount| step_driver.record_sales(amount) end

The step definitions are lava.

Because of the way Cucumber works, the step definitions have to be there, but they should be a very thin adapter between Gherkin steps and a custom driver for automating your application. [REVEAL] Ideally, a step definition should be one annoyingly obvious line of code.

Step definitions exist in a PHP-style flat namespace with one global scope for sharing variables between them. Moving all of the interesting logic out into a step driver lets me use Ruby's full toolset to organize and refactor code, and lets me keep the step definitions so simple that I never have to think about them.

This worked so well for me that I would write a step driver for a new project even if I wasn't using a multi-layered approach.

BREAK IT DOWN(

69

These commissions plans involved something like fifteen pages of legalese, which I then had to translate into code. It was just too complicated for me to hold all of it in my head at once.

Breaking the problem down across architectural layers allowed me to focus on just the core logic, then figure out how to adapt it to ActiveRecord, and then fill in the UI once everything else was already working. And honestly, looking back at it, this was probably the only way I could have completed this app in any reasonable amount of time.

SUSTAINABILITY"I can see why you did it that way" "It was really nice to have such clear documentation"

(

70

The developers who took over the project from me weren't as enthusiastic about this Cucumber setup as I was... but they did say things like [REVEAL EACH]

- "Yeah, I can see why you did it that way" and- "it was really nice to have such clear documentation about what these business terms meant."

So I'll call that a win.

(

PERFORMANCE

71

I do have to talk about performance.

At Cascadia RubyConf in 2011, Ryan Davis gave a talk called "Size Doesn't Matter," in which he talked about the speed and relative size of various testing frameworks in Ruby. Cucumber shows up in nine different slides from Ryan's slide deck, and they basically all look like this:

72Cascadia Ruby Conf 2011 @ Seattle, Cascadia

Ryan Davis, Seattle.rbSize Doesn’t Matter

Lines of Code(including dependencies)

bacon

minitest

shoulda

test-unit2

rspec2

cucumber 17430

7484

6381

4236

1166

380

+ 62,985 lines of C!

Monday, August 1, 11

In every single metric that Ryan chose to present, Cucumber came in dead last. When I saw this talk, I laughed and winced, because at the time, I was dealing with that 90-minute test suite. So when I started this project, I was fully expecting to pay a huge performance penalty.

PERFORMANCELayer # Scenarios Time

Core 64 0.7s

Model 118 7.2s

UI 11 3.0s

Total wall clock time (WIP + done) ~40s

(

73

With that in mind, here are the numbers from this project.

There were 64 scenarios tagged with "@core", and they ran in [REVEAL] under a second.

At the model layer, 118 scenarios ran in [REVEAL] under eight seconds.

At the UI layer, I used Capybara to drive the application. I didn't care about Javascript, so I was able to use the Rack::Test driver, but even with that advantage, these were by far the slowest: [REVEAL] 11 scenarios in three seconds.

Of course, these are the times reported by Cucumber, so they don't include the time it takes to load all of Rails. I had a Rake task set up to run the WIP versions of all three layers, then run the non-WIP versions, for a total of six separate test runs, so I loaded Rails six times. The total wall clock time for that Rake task came in at [REVEAL] about 40 seconds.

So, yeah. Cucumber is no MiniTest. It's never gonna run thousands of tests per second. But I was... pretty happy with these numbers.

TAKEAWAYSDOMAIN > INTERFACE

THE STEP DEFINITIONS ARE LAVA

!

74

In conclusion... if you do decide to try Cucumber, I suggest that you keep these two things in mind:

[REVEAL EACH]

- Your features should describe your domain, not your interface, and- The step definitions are lava.

John Wilger Dana Scheider

Matt Wynne(we're hiring!)

(in Hawaii!)

Noel Rappin Matt Van Horn Jim Weirich

@geeksamTHANKS!

!

75

I know some of you are ready to get up and move around a bit, so I'm not going to keep you. If you have questions or feedback, or if you want another sticker for your collection, come talk to me after I pack up my stuff.

Thank you.

Technology

Cucumbers Have Layers - RubyConf 2015