Automated Testing

All code should be covered by unit and end-to-end (E2E) tests (as appropriate).

Unit Testing

Unit testing tests our code at the low level, i.e. the “units” of code – and their exposed methods and properties. It is a method of “white box” (or “clear box”) testing, because you have knowledge of the internal functionality of the system.

Unit tests are written for the Jasmine framework (2.x) and are run with Karma in the PhantomJS engine (or alternatively, actual browsers).

E2E Testing

End-to-end testing, which could also be called integration or acceptance testing, tests our code at the high level – the UI components, workflows, and product features. It is a method of “black box” testing, because we shouldn’t need to know exactly how it works – we only want to check the end result.

E2E tests see the application the same way an end user sees it in the browser. In fact, browser automation is used to run the tests.

E2E tests are written for Jasmine with the Protractor framework and are run with Selenium.

• Troubleshooting E2E Tests
• Selenium Builder is a Firefox add-on that can help you record, modify, and export JS code for tests.

Best Practices

Currently we don’t dictate any particular methodologies, such as TDD. Our only goal is to have good test coverage on all our projects. To that end, any code change must have corresponding tests added and/or updated.

Readability

In general, tests should be human-readable. If you take the descriptions from the test blocks, it should read like a story:

  Application Module
    ✓ should include ":: Banno" in the <title>

    after loading a route
      ✓ should not display the preloader icon
      ✓ should begin with the route name in the <title>
      ✓ should still include ":: Banno" in the <title>

  Select Institution modal

    when the user belongs to no institutions
      ✓ should not show the loading message
      ✓ should not show any error messages
      ✓ should show the "no institutions found" message
      ✓ should not allow the user to continue

    when institutions cannot be retrieved
      ✓ should have an empty list of institutions
      ✓ should show the loading message
      ✓ should show an error message
      ✓ should not allow the user to continue

    when there are <= 10 institutions
      ✓ should not display the search box

This means that the Jasmine describe() blocks should describe the preconditions of the test – try to have it begin with a preposition such as “when” or “after”. Jasmine it() blocks should describe what is expected to happen – try to have it begin with “should”.

// The top-level describe() can describe the thing (component, workflow, feature, etc) that is being tested.
describe('Select Institution modal', function() {

  // Nested describe() blocks should describe the state that is being tested.
  describe('when the user belongs to no institutions', function() {

    beforeEach(function() {
      // Set up the state of the app to be tested against in a beforeEach/beforeAll.
    });

    afterEach(function() {
      // Any cleanup can be placed in an afterEach/afterAll.
    });

    // it() blocks should describe what the expect() statements are doing in a human-readable description.
    it('should not show the loading message', function() {
      expect(inst.loading.isDisplayed()).toBe(false);
    });

    it('should not show any error messages', function() {
      expect(inst.errors.count()).toBe(0);
    });

    it('should show the "no institutions found" message', function() {
      expect(inst.noneFound.isDisplayed()).toBe(true);
    });

    it('should not allow the user to continue', function() {
      expect(inst.submitBtn.isEnabled()).toBe(false);
    });

  });

  // A describe() could also name the thing being tested, if it is a subset of the top-level thing.
  describe('filtering', function() {
    // ...
  });

});

Tasks

For consistency, we usually define the npm tasks as:

• test:unit – runs the unit tests
• test:e2e – runs the E2E tests
• test – runs both test:unit and test:e2e, usually in sequence (rather than in parallel) so their output is separated

Because E2E tests are significantly slower than unit testing, they are usually excluded from the “watch” task and CI.

Karma

Because we use Jasmine and PhantomJS for unit testing, you’ll need to install those modules alongside Karma:

$ npm install --save-dev karma karma-jasmine karma-phantomjs-launcher jasmine-core phantomjs-prebuilt es5-shim

Your Karma configuration (usually in a separate karma.conf.js file) should resemble the following:

module.exports = function(config) {
  'use strict';
  config.set({
    basePath: '../', // base path for your files
    files: [
      // Include any files your tests will need. This might include libraries
      //   (bower or npm modules), the project code, and the test files themselves.
      // This is heavily dependent on the project; the stuff below is only an example.
      // See the Karma docs for syntax.
      'npm_modules/angular/angular.js',
      'npm_modules/angular-cache/dist/angular-cache.js',
      'node_modules/es5-shim/es5-shim.js',
      'node_modules/angular-mocks/angular-mocks.js',
      'dist/angular-briefcache.js',
      'test/unit/*Spec.js'
    ],
    frameworks: ['jasmine'], // we use Jasmine
    browsers: ['PhantomJS'], // we use PhantomJS
    reporters: ['dots'] // "dots" keeps the output small, but you can use "progress" if you want more info
  });
};

Protractor

Protractor uses WebDriver to run its tests.

$ npm install --save-dev jasmine-spec-reporter protractor

The protractor.conf.js file should contain our standard configuration, shown below. All of Protractor’s options can be found in its reference config.

exports.config = {
  baseUrl: 'https://localhost:8080/',
  // Usually we only test in Chrome, but other browsers can be used:
  //   https://github.com/angular/protractor/blob/master/docs/browser-setup.md
  multiCapabilities: [{
    browserName: 'chrome'
  }],
  onPrepare: function() {
    'use strict';

    // Use the Jasmine spec reporter.
    var SpecReporter = require('jasmine-spec-reporter');
    jasmine.getEnv().addReporter(new SpecReporter({ displayStacktrace: 'all' }));

    // Sets the browser window to approximately desktop dimensions.
    browser.driver.manage().window().setSize(1600, 800);
  },
  jasmineNodeOpts: {
    // Hide the original reporter.
    print: function() {}
  }
};

Protractor is specfically designed for Angular. If you want to load non-Angular pages, you need to tell Protractor to not look for Angular:

browser.ignoreSynchronization = true;
browser.get('http://localhost:8080/non-angular-page');

Jasmine

Jasmine is a BDD framework. As such, tests should be written to follow a BDD-style narrative; here’s an example from an E2E test suite:

describe('Login form', function() {

  var loginModal = new LoginModal();

  beforeEach(function() {
    loginModal.load();
  });

  it('should be displayed', function() {
    expect(loginModal.form.isDisplayed()).toBe(true);
  });

  it('should display the form elements', function() {
    expect(loginModal.emailField.isDisplayed()).toBe(true);
    expect(loginModal.passwordField.isDisplayed()).toBe(true);
    expect(loginModal.forgotPassword.isDisplayed()).toBe(true);
    expect(loginModal.submitBtn.isDisplayed()).toBe(true);
  });

  it('should begin with blank fields', function() {
    expect(loginModal.emailField.getText()).toBe('');
    expect(loginModal.passwordField.getText()).toBe('');
  });

  it('should begin disabled', function() {
    expect(loginModal.submitBtn.isEnabled()).toBe(false);
  });

  it('should not display any errors', function() {
    expect(loginModal.getError()).toBe('');
  });

  describe('when partially filled in', function() {

    beforeEach(function() {
      loginModal.emailField.sendKeys('test@example.com');
    });

    it('should still be disabled', function() {
      expect(loginModal.submitBtn.isEnabled()).toBe(false);
    });

  });

  describe('when successfully submitted', function() {

    beforeEach(function() {
      loginModal.emailField.sendKeys('test@example.com');
      loginModal.passwordField.sendKeys('foobar');
      loginModal.submitBtn.click();
    });

    it('should close', function() {
      expect(loginModal.form.isPresent()).toBe(false);
    });

  });

});

Check out Jasmine’s documentation – it provides good examples with annotations.

Execution Order

Be aware that function blocks (beforeEach, it, etc) are executed in a specific order by Jasmine at test runtime. Anything outside a function block will be executed first!

For example, this won’t work:

// Warning: this won't work!
describe('example suite', function() {
  var moment;

  beforeEach(inject(function(_moment_) {
    moment = _moment_;
  });

  describe('example using MomentJS', function() {
    var date = moment().subtract(30, 'days'); // problem!

    it('should test MomentJS', function() {
      expect(date).toEqual(jasmine.any(Object));
    });
  });
});

In the example above, date is not defined inside a block. Therefore, it is executed before the beforeEach where moment is defined – creating an error because it is undefined at this point.

To fix this, the date definition should be inside a beforeEach:

describe('example suite', function() {
  var moment;

  beforeEach(inject(function(_moment_) {
    moment = _moment_;
  });

  describe('example using MomentJS', function() {
    var date;

    beforeEach(function() {
      date = moment().subtract(30, 'days');
    });

    it('should test MomentJS', function() {
      expect(date).toEqual(jasmine.any(Object));
    });
  });
});

In general, don’t put code outside before/beforeEach/it/etc blocks except for variable declarations (e.g. var foo;). Code should only go outside these blocks if you are sure that it is not dependent on any code inside a function block, and only needs to run once.

Extra Matchers

Out of the box, Jasmine offers a good set of matchers. If you need functionality that isn’t included, first check out Jasmine-Matchers. It can easily be added to your setup:

1. npm install --save-dev karma-jasmine-matchers
2. Add it to the frameworks in your Karma config: frameworks: ['jasmine', 'jasmine-matchers']

You can also write your own matchers, as shown in the Jasmine docs.

Unit Testing

Unit tests should cover the individual components, services, etc.

As you create your tests, remember you are only testing your project. Your dependencies should already be tested by those who created them.

E2E Testing

Mocking in Node (new way)

First read Writing Mocks for Services to understand how mocks are written and used.

The APS E2E tests are a good guideline.

Tips for writing E2E tests against mocks:

• The server at the beginning of a test will have the state of the previous test. Usually you want to reset the state in a beforeEach() block before your tests run.
  • Call the /_reset endpoint to reset to the original data.
  • Other browser objects (such as session cookies) may need to be reset.
  • For example, the APS reset() helper calls the /_reset, DELETE /login, and DELETE /users/:id/selected-institutions endpoints to make sure that the user is not logged in and has no institution selected.
• Any HTTPS request made in a NodeJS module (such as Protractor) needs the list of SSL certificates in order to trust the connection to the server. See the APS fetch() helper as an example.
• Generally mocks don’t check for CSRF tokens, so there is no need to include an XSRF-TOKEN in requests to the server. Of course, it doesn’t hurt, and you will need to include it if you run E2E tests against live APIs.
• Because the server state is shared across clients, you may not want to develop or test the site in a browser while E2E tests are running.

Checking the Browser Logs

Protractor provides a way to access the browser console logs.

Here’s an example of checking the console for any logs after every test:

afterEach(function() {
  browser.manage().logs().get('browser').then(function(browserLog) {
    expect(browserLog.length).toEqual(0);
    if (browserLog.length) { console.log(require('util').inspect(browserLog)); }
  });
});

Astrolabe

For any non-trivial suite of E2E tests, Astrolabe is an optional but recommended addition to Protractor.

Astrolabe objects should be separated from the tests themselves, e.g. in a components folder. See the APS project for an example on using Astrolabe.