Test Driven Development with Meteor, Cucumber & Chimp

Meteor is changing and old packages are being phased out. Among the changes: Velocity is now deprecated. Meteor 1.3 will introduce new unit testing options. But right now, we have excellent end-to-end testing with Chimp.

The team behind Velocity is very seriously developing Chimp and I’ve had a good experience so far using it for a critical, real world project. But there were some tricks. In this tutorial I will explain some basic setup and ‘Behavior Driven Design’ that we can use to crank out some features.

This tutorial walks through a simple application, using Chimp to run our tests.

See the finished tutorial on github.

Cucumber

First, we will use Cucumber. Chimp can also run Mocha, and that probably works great, but my team decided to use Cucumber, to help with client relations and product specifications.

In Cucumber, you will create 2 types of files. 1st, the feature, a file literally ending with the extension .feature. This file includes plain english descriptions of features and specific scenarios, like when a user visits pages, fills out a form, and clicks a button. A single line in that file might be ‘When I create a widget’. You can learn about the specific syntax, called Gherkin. In the 2nd file, called a ‘step definition’, we write JavaScript code that actually drives the test browser to locate that particular button and click it.

Features and steps. English and JavaScript. That’s our test suite.

Unit versus End-to-End Testing

Meteor has been strongly oriented around Packages, a proprietary means of packaging meteor-specific functionality. Developers can create unit tests within their packages, using TinyTest. Until we get Meteor 1.3, other means of unit testing are not straightforward, because Meteor relies so heavily on its own unique file loading patterns and global namespace tricks.

So, it’s simple to write unit tests for meteor packages, but it’s not simple to write unit tests around other code in your actual app.

End-to-end testing is another option. An end-to-end test mimics a real person. Imagine your trusty QA sidekick. You give him a list of instructions, like ‘visit the main page, create a widget, and check that you see both a success message and your new widget on the page”. The human being will carefully set aside his coffee mug, move his Reddit tab into the background, don his headphones and studiously execute your instructions. Later, he will tap you on the shoulder and explain what actually happened.

Chimp, along with Cucumber and your custom features and step definitions, also does all that, only without coffee, Reddit, or fingers. Spin up your app, then spin up chimp, and chimp will execute your Cucumber steps in an actual browser, and display in green and red which steps passed and failed.

Meteor 1.3 will introduce simple unit testing. But, until then, end-to-end testing with Cucumber offers another take on the same idea— write red tests first, and then write the code that makes those tests green. The difference is, while you might write many separate unit tests to exercise a series of functions, you might write a single end-to-end feature, which requires a bunch of different units of code.

There’s a name for that, called ‘behavior driven design’, because the ‘features’ describe how the app should behave. But you can still think of it as ‘test driven’, because you are writing tests first, before you write your application code.

The most important difference between end-to-end testing and unit testing, is this: with end-to-end tests, you strive to test only that which your user can see on the screen. Instead of testing that the ‘doubleNumber’ method, when given 2 will return 4, you will test a feature, ‘User can provide a number and see it doubled’. You can test front end too, because you can just as easily test whether certain HTML elements exist on the page, with certain class names or Ids.

This Tutorial

Let’s create a practice app, using Cucumber and writing tests first. This app will create widgets with names. We will setup chimp, write Cucumber features, and make our tests pass.

First step: Install Chimp

We’ll assume we have already installed Meteor and worked with it. Also: Node and NPM.

If you need instructions, look here:

Install Meteor

Install Node

Next, install chimp:

$ npm install -g chimp

You could also take a moment to review the Chimp docs.

We have the tools we need. Now we can create our app.

Create app


Our app will be called ‘Widget Maker’, and it’s purpose will be to make widgets. Create it:

$ meteor create widget-maker

And then change into the new directory:

$ cd widget-maker

Before we write any code, let’s install React. That’s a bonus in this tutorial. There’s some debate these days about using Blaze versus React. If you are in the Blaze camp, try to keep an open mind, we will not be writing that much front-end code.

$ meteor add react

Testing with Cucumber

We will practice ‘test driven design’. We will write a test, check that something happens, see that test fail, and then write the code so that test passes. Think this: tests first. Then write code to make the test pass.

This works beautifully with Cucumber, because we can describe literally how our app should behave, and test whether it behaves that way. If it does not, we know something is off, and we can write the appropriate code.

It’s also nice, because our tests will operate very similar to our preferred user: an actual human. We will describe the feature we want, and then we will give steps that instruct Chimp exactly what to do, from hitting a route in our app to filling out a form to clicking a button. And lastly, we will tell Chimp what it should see on the screen, and Chimp will check and let us know what it saw.

This is similar to handing a sheet of instructions to a human QA tester, who clicks and pecks and notes what’s acceptable and what’s not. (Yet another term: Acceptance Testing).

With that background, let’s write our first feature.

Our First Feature

Our tests will live in a new directory: /tests/features.

Chimp will look inside this directory and automatically detect our tests.

We will first create our feature file, which literally ends with the extension .feature. And because it is a specific filetype, we get to use a specific syntax: Gherkin.

Let’s write the whole feature and then review what it all means:

Feature: Create a Widget

 As a visitor to the site,
 so that I can manage widgets,
 I want to create and delete widgets on my home page.

 Background:
    Given I am on the site

 Scenario: Visitor creates a widget
    When I name a widget "alpha"
    And submit the form
    Then I should see a list of widget names containing “alpha”

# /tests/features/create-widget.feature

Let’s quickly review this file.

Feature: A human friendly name for this feature, something a client might ask for. Just under ‘Feature’ is a short user story. User stories describe who the user is in this case, what he wants to do and why he wants to do it.

Background: This indicates specific steps that Chimp will take before every scenario. You can create fixtures, navigate to certain routes, or setup the app so it resembles what the user will actually see as he starts to engage this feature. In this feature, we want to make sure the browser first navigates to our site, otherwise it might get stuck testing against a blank URL.

Scenario: These are the steps pertinent to the feature at hand. In this feature, we will create a new widget with a given name and marvel as our brand new widget proudly appears on our home page.

Given, When, Then and And: These words hold special meaning. ‘Given’ indicates a step that should setup some background, whatever we want to occur before we start behaving like a real user. ‘When’ indicates stuff we do, as in ‘When I create a widget’. ‘Then’ indicates the result we want to see, as in ‘Then I should see a list of widgets’.

The word ‘And’ will repeat whichever precedes it. So we could say,

Given I bought a cheeseburger  
And I grabbed some ketchup  

We used ‘And’ instead of 'Given'. It's equivalent to:

Given I bought a cheeseburger  
Given I grabbed some ketchup  

This also works for ’When’ and ‘Then’:

When I create 5 widgets  
And I delete one widget  
Then I should see that one widget disappear  
And I should see that “4” widgets remain

is the same as

When I create 5 widgets  
When I delete one widget  
Then I should see theat onewidget disappear  
Then I should see that “4” widgets remain  

Each line is a ’step’ and each corresponds to a ‘step’ in a separate JavaScript file, which we will soon create.

Cucumber gives us this 2-phase pattern:

  • First phase: the human friendly, or ‘Business Readable’. That’s the ‘.feature’ file.

  • Second phase, actual code that translates the background and scenarios into computer friendly code. JavaScript, in this case.

Here’s that file, create-widget_steps.js:

module.exports = function () {  
 'use strict';

 // steps go here
};

// /tests/features/create-widget_steps.js

Note: Include the ‘use strict’ line so that we can use ES2015 features.

Now, a pause. We don’t need to create all those steps by ourselves. We can run chimp, and chimp will note those missing steps, and chimp will provide copy & paste templates.

Run Chimp

Chimp will execute our tests against a running instance of our app. But before we run our app, using the meteor command, let’s examine the Chimp command.

The Chimp docs give us the command to test a meteor app:

$ chimp --ddp=http://localhost:3000 --watch —path=tests

This command includes 3 different arguments.

ddp tells chimp how to connect via DDP to our Meteor app, which hooks into the Meteor magic we love so much. Very importantly, note the port. Later, we will change that.

watch tells Chimp to only run scenarios tagged @watch, and re-run whenever we save a test file or restart our app.

path tells Chimp where to look for our Features directory.

Run the command, but beware. Error ahead:

Chimp did not connect to our app.

Chimp is running and it is using a regular browser like a normal person, trying to connect to our Meteor app. But, we did not start our Meteor app!

In a separate terminal tab, run the meteor app:

$ meteor

And then, restart chimp, same as before:

$ chimp --ddp=http://localhost:3000 --watch --path=tests

This time, Chimp runs! But, with very little output, because we never did tag our scenario with @watch:

0 scenarios, 0 steps

0 scenarios, 0 steps

Back in /tests/features/create-widget.feature, add the tag:

 @watch
 Scenario:
    When I name a widget "alpha"
    …
// /tests/features/create-widget.feature

Save that. Now, because Chimp was running, it detected the file change, re-ran, and told us something special. We have no steps, but we get some copy & paste steps to start:

Steps for us to copy & paste

Steps for us to copy & paste

If you didn’t create this file before, do it now: /tests/features/create-widget_steps.js. Note, this is a normal JavaScript file, but we have appended _steps to its name.

After including the steps, the file looks like this:

module.exports = function() {  
 'use strict';

 this.Given(/^I am on the site$/, function () {
      // Write the automation code here
      pending();
 });

 this.When(/^I name a widget "([^"]*)"$/, function (arg1) {
   ‘// Write the automation code here
   pending();
 });

 this.When(/^submit the form$/, function () {
       // Write the automation code here
       pending();
 });

 this.Then(/^I should see a list of widget names containing "([^"]*)"$/, function (arg1) {
      // Write the automation code here
   ’pending();
 });
};
// /tests/features/create-widget_steps.js

Save that, and look again out our test results. This time, we see, Chimp tells us that our first step is present but pending.

1 step pending, 3 steps skipped

1 step pending, 3 steps skipped

Let’s review a ‘step’.

Each step establishes a Given, When or Then step. Each is a function, taking 2 arguments. 1st, a string or regex that matches the Feature text. Using regex, anything in the Feature wrapped by double quotes, becomes a variable we can use in the step. The 2nd argument to the step function is a callback containing the actual logic for that step. If the Feature had anything in double quotes, the text inside the quotes is passed as an argument to this callback.

The logic within the step requires the most work. That’s where we instruct the browser which URL to visit, which form field to fill, which buttons to click. And, that’s where we state our actual tests. Imagine telling a person, ‘Visit the home page, provide a name for a widget, submit the form, and check that we see the name when you are done.’ That includes all our steps for the feature, and here they are in code, one at a time.

First:

this.Given(/^I am on the site$/, function () {  
 browser.url(‘http://localhost:3000');
});

// /tests/features/create-widget_steps.js

Here we encounter our first most important helper object, browser. That object gives us access to Webdriver, a library that calls itself ‘Selenium 2.0 bindings for NodeJS’. This utility lets us drive an actual browser, and in this step, we tell the browser to go to our app’s root URL.

Next:

this.When(/^I name a widget "([^"]*)"$/, function (arg1) {  
 browser.setValue('input[name="widgetName"]', arg1);
});

// /tests/features/create-widget_steps.js

Here, we see an argument. In the Feature, we wrote When I enter "alpha" in the name field . The value “alpha” will be parsed by the regex, put into a regex group, and passed as an argument to the callback. So, in that callback, the value of arg1 will be the string “alpha”.

This way, we can tell the browser what to enter into the text field that it finds on the page. Using arguments like this is very important. You can keep your Feature very precise, and pass along all the information you need, and let your JavaScript steps accept incoming arguments.

Next:

this.When(/^submit the form$/, function () {  
 browser.click('input[type="submit"]');
});

We use another Webdriver method, click, passing along the selector specifying that which we want to click, the form’s submit button.

Lastly, we will check that we see the correct result:

this.Then(/^I should see a list of widget names containing "([^"]*)"$/, function (arg1) {  
   let _el = '.widget-name';
   browser.waitForVisible(_el);
   expect(browser.getText(_el)).toEqual(arg1);
});

A few things here. First, I like to specify the element we want as a variable, _el. Here, we say that our desired element has the class .widget-name. Second, we tell the browser to wait until it sees that variable. If you do not tell the browser to wait, then it will chug along at light speed, not waiting for the server to accomplish even this tiny task, and thus the new widget will not be created in time. But, here, we do wait, and on the last line, that widget should have been created, and we should see that widget’s name.

Looking just at the last line, we see 2 important bits. One, we use the Webdriver API yet again, this time to extract text from our chosen element. Two, we use a Jasmine matcher to assert that this extracted text will equal whatever argument we got, in this case, our widget name “alpha”.

Here is the entire file, all at once:

module.exports = function() {  
  'use strict';

  this.Given(/^I am on the site$/, function () {
    browser.url('http://localhost:3000');
  });

  this.When(/^I name a widget "([^"]*)"$/, function (arg1) {
    browser.setValue('input[name="widgetName"]', arg1);
  });

  this.When(/^submit the form$/, function () {
    browser.click('input[type="submit"]');
  });

  this.Then(/^I should see a list of widget names containing "([^"]*)"$/, function (arg1) {
    let _el = '.widget-name';
    browser.waitForVisible(_el);
    expect(browser.getText(_el)).toEqual(arg1);
  });
};

Save that, and see what happens in chimp? Our steps fail.

First failing steps.

First failing steps.

This is beautiful. Our tests us our steps are failing and our code is wrong. We know what we need to fix first: There should be a text field with the name “widgetName”, but Webdriver found no such element on the page.

Passing the first step

We need a form, with a text input where we can enter the name “alpha”. As a bonus, we will put our HTML into a React component.

Let’s follow the first steps of Meteor’s own React tutorial.

First, edit the main widget-maker.html:

<head>  
  <title>widget-maker</title>
</head>

<body>  
  <div id="render-target"></div>
</body>  
<!-- widget-maker.html -->  

And next, we need to use JSX. Rename widget-maker.js to be a jsx file: widget-maker.jsx. And inside it, put:

if (Meteor.isClient) {  
  Meteor.startup(function () {
    ReactDOM.render(<App />, document.getElementById("render-target"));
  });
}

if (Meteor.isServer) {  
}
// widget-maker.jsx

As the client starts, React will find our #render-target div and render inside it our component. Create a new file, app.jsx and put in our first few lines of client side code:

App = React.createClass({  
  render() {
    return (
      <div>
        <h1>Widget Maker</h1>
        <form>
          <input type="text" name="widgetName" placeholder="Name your new widget..."/>
        </form>
      </div>
    )
  }
})
// app.jsx

Now, with chimp running, we start to see some green!

2 passing steps

2 passing steps

Chimp by default boots up a Chrome browser, which Webdriver drives through our steps. Looking at that browser, we see that yes, our form displayed properly and Webdriver entered “alpha” into the text field, but the tests failed because we don’t have a submit button yet.

In the browser, no submit button

In the browser, no submit button.

Let’s put that submit button into the component and get the next step to pass.

Back in the app.jsx component:

 <form>
   <input type="text" name="widgetName" placeholder="Name your new widget..."/>
   <input type="submit" value="Create Widget" />
   </form>

<!-- app.jsx -->  

Now, all steps pass except the last:

All steps pass except the last

All steps pass except the last.

We have met the crux of this challenge: writing code that actually handles our business logic. Handling the form submission, creating the widget, and showing the new widget.

If we were unit testing, we might test each little chunk of code. We could test each and every little function or unit of code in our app, to see that each one does it’s job.

But with this type of end-to-end testing, we have one major assertion— ‘the new widget should appear’, which is like a giant unit, made of many little units. Instead of testing each little unit, we test the outcome of running all the units together.

Because our tests re-run each time we save a file, we can pause and reflect with every failing run: why does the test fail? And then we can write code that makes the test pass.

Here’s some code that will bring us a little further to green. This will update the component’s state with the name we enter, and on submit, will attempt to insert a new widget with that name:

App = React.createClass({  
  handleChange(e) {
    this.setState({widgetName: e.target.value});
  },

  handleSubmit(e) {
    e.preventDefault();
    Widgets.insert({name: this.state.widgetName});
  },

  render() {
    return (
      <div>
        <h1>Widget Maker</h1>
        <form onSubmit={this.handleSubmit}>
          <input type="text" name="widgetName" placeholder="Name your new widget..." onChange={this.handleChange}/>
          <input type="submit" value="Create Widget" />
        </form>
      </div>
    )
  }
});

// app.jsx

Sometimes with Chimp and Meteor, the error we want to see shows up in the browser, not the terminal. Notice in the test browser, we have no Widgets collection!

Widgets is not defined

Widgets is not defined.

We can proceed to create that Widgets collection, publish it and, last, subscribe to it in the component so that new widgets can automatically appear.

First, the widgets collection. At the top of widget-maker.jsx:

Widgets = new Meteor.Collection('widgets');

if (Meteor.isClient) {  
  ...
// widget-maker.jsx

Second, the publication, just underneath, in the Meteor.isServer block:

if (Meteor.isServer) {  
  Meteor.publish('widgets', function() {
    return Widgets.find();
  });
}
// widget-maker.jsx

Third, subscribe in component and show the widgets, this time in app.jsx:

App = React.createClass({  
  mixins: [ReactMeteorData],
  getMeteorData() {
    let subscription = Meteor.subscribe('widgets');

    return {
      widgets: Widgets.find().fetch()
    };
  },
...
  // app.jsx

Note: We still have ‘autopublish’, which automatically publishes all our data. That’s a convenience we don’t need. Remove it:

$ meteor remove autopublish

After all that code, we still haven’t finished. Our test fails:

One last thing to do

One last thing to do.

Do you know what we forgot?

We need to put in the actual HTML in the component that displays widgets.

 renderWidgets() {
    if (this.data.widgets) {
      return this.data.widgets.map(function(widget) {
        return (<p className="widget-name">{widget.name}</p>);
      });
    }
  },

  render() {
    return (
      <div>
        <h1>Widget Maker</h1>
        { this.renderWidgets() }
      ...

//app.jsx

The function renderWidgets() returns an array of paragraph elements, and we call that function to display those paragraphs in the main render function.

Now, our code is in place. We can create a widget and see that widget appear, courtesy of Meteor and React.

One last thing. If you are like me, your tests ran several times, and actually created several widgets, and you end up seeing this:

We are not clearing our DB before each run.

We are not clearing our DB before each run.

Every time our tests run, we insert a new widget. But, in a testing environment, we want to start fresh with each run. So, we will add a before action in our test, so that before each test, we remove all the widgets.

This trick requires some thought. Chimp works very well with Meteor and allows us to execute server code using a special package: xolvio:backdoor

Add it now:

$ meteor add xolvio:backdoor

Like the name suggests, this gives us a ‘backdoor’ that chimp can use to enter, run server code, and get a response. In this case, we want to run code on the server that removes all widgets, like so:

 this.Before(function() {
    server.execute(function() {
      Widgets.remove({});
    });
    browser.pause(100);
  });
 // create-widget_steps.js

Note, we tell the browser to pause a moment so the server can finish before we move on to the rest of the tests.

Side note: In that snippet, we passed a callback into the server.execute function. You may want to also pass along variables. Say you want to remove a widget with a particular name:

    server.execute(function(foo) {
      Widgets.remove({name: foo});
    }, foo);

Running server code is powerful. You can also execute code in the client, using a similar syntax:

    browser.execute(function() {
      Widgets.remove({});
    });

See the chimp docs for more.

But for now, see our test browser? Just a single widget, named “alpha”.

One widget named 'alpha'

One widget named ‘alpha’

Side note: Our input never reset, it still contains the text we entered. Homework! In the component, clear the input after submitting the form.

And our tests? All green!

All tests pass

All tests pass

Helper File

When the tests run, all your steps are loaded into a single module, regardless which file you put them in. You could have a single step, say ‘create a new widget named “bob”’, in any file, and you can use that step across any feature.

Meaning, we can create new file called ‘helpers.js’, and
include code that we specifically want to access from every
feature. Let’s refactor.

From create-widget_steps.js, we will cut these lines:

 this.Before(function() {
    server.execute(function() {
      Widgets.remove({});
    });
    browser.pause(100);
  });

  this.Given(/^I am on the site$/, function () {
    browser.url('http://localhost:3000');
  });

 // /tests/features/create-widget_steps.js

And paste them into a new file, /tests/features/helpers.js, so the whole file looks like:

module.exports = function() {  
  'use strict';

  this.Before(function() {
    server.execute(function() {
      Widgets.remove({});
    });
    browser.pause(100);
  });

  this.Given(/^I am on the site$/, function () {
    browser.url('http://localhost:3000');
  });
};

// /tests/features/helpers.js

After this refactor, the tests still pass. Chimp is gathering up all our steps and running them correctly.

This helper file can include any sort of steps that might be shared across many features. In this case, every feature we run will first clear out old widgets because we included that in our ‘before’ function.

Advanced Topics

Currently, we are running our app in the same environment for both testing and development, and we are using the same database for each environment.

A more advanced setup uses multiple databases, but because of how meteor boots up, this becomes tricky.

Xolvio has created a wonderful resource here (https://github.com/xolvio/automated-testing-best-practices), and it includes an advanced NPM setup. This setup creates a mirror app, runs your tests against a different db than the development db, and even gets you ready for continuous integration.

The entire guide is worth reading, to better understand test driven development.

Other advancements we could make in our Widget app might be:

  • using fixtures
  • creating a specific ‘test’ environment that uses different app settings
  • stubbing out email (see this tip)
  • setting up Continuous Integration

Conclusion

In this tutorial, we created a simple test suite that specified a particular requirement. Those tests failed, but we then added the code that made them pass. Our app can create widgets! We can proceed building the app, feature by feature, starting with plain english specifications, letting our red tests guide us into green. And then we can refactor confidently, because our tests will inform us when something breaks.

Meteor is in a state of flux. As of this writing, The Meteor Development Group is readying Meteor 1.3, and they promise better unit testing. Meteor is quite wonderful and I hope they focus on unit testing, so that they can continue earning web developer’s respect, and so developing Meteor apps can be the best possible experience for you and me.

But still, Chimp gives us a solid, fast means to develop Cucumber specs and run them against our app, in a Test Driven manner. And it will likely be the recommended approach to app-wide approach, even in Meteor 1.3.