As a team with a strong taste for automated testing we focused from the beginning on how to test our Angular app. In this post we want to describe our different types of tests and how they form a Test Pyramid.
Level 4: Selenium Tests.
Level 3: E2E (Scenario) Tests.
Level 2: Directive Tests.
Level 1: Unit Tests.
For each kind of test, we will explain the following aspects:
- Purpose: What’s the idea behind this type of tests. When to use and what is being tested (the scope of the test). Principles how to write the test.
- Implementation: How does it look like in source code. Examples, technical issues.
- Our experience: Reflection about our specific experience. Some heuristic data.
Level 1: Unit Tests
Purpose:
The foundation of our test pyramid are Unit Tests. The basic idea is the same as for a Unit Test in other languages or environments: The test granularity is very fine and a single test only covers a very small amount of logic. We test only one line of code (LOC) up to around a dozen, seldom more.
Every Unit Test (when implemented with TDD, which we assume here) serves three distinct purposes:
- It helps to clarify the goal of the implementation, because the Unit Test is written before the implementation.
- With every execution of the test it is checked, if the implementation still works as expected.
- It’s a documentation for the idea behind the implementation.
As general guideline for a good test we use the F.I.R.S.T principles:
– Fast – Isolated – Repeatable – Self-validating – Timely
Implementation:
We are using Karma to run the tests and Jasmine as our general testing framework.The Angular Mocks function loads a single Angular module and with inject we can get the instance of the service to test.
To achieve the goal of isolation, we mock every dependency, which could let the test fail if the dependency itsself is broken.
For example if we want to test a service ‚cars‘ which depends on ‚carRepository‘ the basic test setup looks like this:
describe('The service cars', function () {
var cars;
var carRepositoryMock;
beforeEach(function () {
carRepositoryMock = {
saveCar: function () {
// mocking behaviour here
}
};
// Loading the module containing the service
module('carModule');
// After loading we mock the carRepository
module(function ($provide) {
$provide.value('carRepository', carRepositoryMock);
});
// get the cars instance we want to test
inject(function (_carService_) {
carService = _carService_;
});
});
it('should do something ...', function () {
…
});
});
Since Angular 1.2 there is a better syntax to mock dependencies: For details see Issue 3746.
If we are testing a controller we use the $controller service to instantiate the controller we want to test:
inject(function ($controller, $rootScope) {
$scope = $rootScope.$new();
myController = $controller('ControllerToTest', { $scope: $scope });
});
Creating a filter for tests can be achieved with the $filter service:
inject(function ($filter) {
myFilter = $filter('FilterToTest');
});
To verify the interaction with the dependencies we use Jasmine spies.
If we need to test timing related aspects, we use the timer methods of sinon.js.
Our experience:
It’s not a real statistic and we don’t think it should be overemphasized, but we have a 1:3-1:4 ratio ratio of productive loc vs test loc. We consider it a smell if the ratio goes up. Most of the times this means the service or controller is doing to much (violating the SRP) and we should start refactoring.
Level 2: Directive Tests
Purpose:
Directive Tests are a way to test a small part of an app through interaction with the DOM.
Besides some technical things, the Angular Unit Tests are a lot like other Unit Tests (e.g. in Java). Directive Tests on the other side are an Angular speciality.
The main idea is to let Angular process a html snippet, which contains most of the time only a custom element, defined by a Directive. (Therefore the name „Directive Test“.) The result is a DOM Element.
This is a very powerful way to test: It can be focused on one specific part of an app without loading the whole. But this part is tested on a high level, because it is tested through the DOM.
What to test exactly is of course highly depended on the Directive, but we came to the conclusion to test everything what is not pure CSS or HTML (e.g. the background colour of a button)
In other words: We test everything which is handled by Angular: Displaying content (e.g.{{car.name}}), visibility (ng-show), dynamic classes (ng-class) etc.
This means we also test, that every click handler (ng-click) and every other user interaction works as expected.
Because this is done on the DOM level we don’t need any other test to verify that e.g. a click on a button works. And the tests are very stable and fast, because only the parts really needed are loaded.
One thing to keep in mind is the relation between the overall design (or architecture) of an Angular app and the possibility to use Directive Tests. The principle is the same like for Unit Tests: If the code is not designed for, its very hard to write tests. For Unit Tests the solution is straight forward: Just try to use TDD as much as possible: Because the test is written before the implementation there is no problem with bad designed code.
On the other hand to be able to use the full power of Directive Tests, it means that an app should be a composite of loose coupled, feature oriented directives. For example an directive should not depend on some other directive or service which has nothing to do with the feature.
Implementation:
Our Directive Tests are always executed along with our Unit Tests: Karma is the runner and Jasmine the test framework.
A directive is tested through a html snippet, which is compiled and linked to a $scope.
As an example, if we have a directive ‚car‘, the basic test setup looks like this:
describe('The car directive',function(){
var element;
var $compile;
var $scope;
beforeEach(function(){
// the module, which contains the directive
module('carModule');
// fill the template cache with the car html
// This is done via the Karma ng-html2js-preprocessor
module("app/cars/cars-template.html");
// Sometimes a mock is needed, just like in a Unit-Test
// For example we don't want to talk to a real server
module(function($provide){
...
});
// The html snippet, we want to test
element = angular.element('<car some-attribute="test-data"></car>');
inject(function(_$compile_,$rootScope){
$compile = _$compile_;
$scope = $rootScope.$new();
});
// compile, link and trigger a watch cycle
$compile(element)($scope);
$scope.$digest();
});
// Now test the element
it('test element',function(){
...
});
});
This test is much more a integration test than a Unit Test: We test the correct directive configuration, the controller, the linker and perhaps some services. But we don’t want to test everything: If the directive is making a http request, this is mocked.
Because we are testing against the DOM, we can simulate user interactions:
it('click on details should show details of the engine',function(){
element.find(".details").click();
expect(element.find(".engine-details").length).toEqual(1);
});
Our experience:
In the daily work flow we don’t separate Directive from Unit Test. They are both executed by Karma all the time in the background while programming. They are completely natural like Unit Tests and the difference in the concepts don’t bother.
Level 3: E2E Tests
Purpose:
E2E Tests or Scenario Tests are black box application tests. This means the tests don’t know anything about the implementation. They test a feature from the user point of view.
Basically every Acceptance Test is a good candidate for a E2E Test.
Implementation:
The E2E Tests are run by Karma and the syntax is Jasmine like (but it is not Jasmine, it is only the same syntax). The basic idea is to load the Angular app in an iframe and send Javascript events to simulate a user interaction (e.g. click on button).
Here is a example, which tests a button sorts a list of cars:
describe('E2E Test', function () {
beforeEach(function () {
browser().navigateTo('/url/to/test/index.html');
});
it("should sort cars", function () {
element('.cars-sort').click();
var r = repeater('.car');
expect(r.row(0)).toEqual(['Car1']);
expect(r.row(1)).toEqual(['Car2']);
expect(r.row(2)).toEqual(['Car3']);
});
});
We decided to mock our real backend for the tests with a Node.js server. This server emulates our real backend, which is written in Java and integrated in a much larger application.
The Node server also serves the static content for the E2E Tests (index.html, angular.js, css files etc.)
E2E Tests are not very well documented and the setup with Karma can be a little bit tricky. For a full working example see here.
The best documentation for the available expressions is the actual source code.
The Angular team is currently working on a replacement for E2E Tests: Protractor. It has the same purpose, but is based on Webdriver.
Our experience:
We currently have around 20 E2E Tests vs. ~700 Unit/Directive Tests. Our E2E Tests are testing all use cases including error handling. We developed an extension to temporary modify the Node server for specific tests: node-config-ngscenario-dsl. Our E2E tests need less than a minute. A release and test run (implemented with Grunt) less than 2 minutes. We think fast feedback cycles are very important and less than 2 minutes are acceptable for us.
Level 4: Selenium Tests
Purpose:
Selenium Tests serve the same purpose as E2E Tests: They are black box application tests. They only differ in the technical implementation. While E2E Tests are designed for Angular apps, Selenium Tests can be used to test any web page.
Implementation:
Selenium Tests control the browser via Webdriver. Thats the difference to E2E Tests: While E2E Tests are executed on the JavaScript level, Webdriver tries to simulate a user on a more generic level.
Our experience:
We have currently only one Selenium Test, which is only a Deployment Smoke Test: Its goal is to verify, that our app is packaged and deployed as expected. To be more specific we only test on the existence on one div element, which implies that Angular is started. The test is written in Java.
Conclusion
The pyramid described here is not a general solution, which works for every app. Its just what served very good our needs. For example our Smoke Production Test implemented with Selenium could be replaced with something written with CasperJS. But as a general rule we think its worth always to consider on which level of the pyramid the test should be. We favour the lower levels, because a Unit Test is faster and more stable as a E2E Test. For example we only implement a E2E Test if its really needed, because it tests a aspect, which can’t be tested with a Unit or Directive Test.
We would appreciate any kind of feedback. Of course especially other testing approaches or ideas.