Some time ago I’ve created a suite of (FitNesse) acceptance tests for the app we are working on. Some of them were of the form of: Given that a customer exists with customer name "John"
Given that a search results page is displayed for search criteria customer name "John"
Then at least one one customer should be displayed
Our QA (correctly) noted that this test is far from perfect because it doesn’t check that the found customer is the one with name “John”. On the other hand, there could be more than customer with this name. This means that we should not only verify that all found customers have the name “John”, but also that we have found all customers with such name. So we should first look into the database to find out what customers should be in the list and then check that we display only them. Correct? No!
The problem with this approach is that the test starts to mirror the application. It queries the database for customers with the name “John” – but isn’t this what the app does? If we made some mistakes in the app when implementing the query in the app, then who can guarantee that we won’t make the same mistakes when implementing that same query in the test?
So we limited ourselves with this: Given that a customer exists with customer id "mock-customer-1" and name "John" and surname "Smith"
Given that a search results page is displayed for search criteria id "mock-customer-1"
Then at least one one customer should be displayed with name "John" and surname "Smith"
And no customers should be displayed that don't have name "John" and surname "Smith"
Yes, this test does not verify that we found all the customers we searched for, but the only test that verifies application completely would be the application itself. Tests should be a stabilizing harness, not a copy, otherwise there’s no point in writing them.
Tests shouldn’t mirror the application
Some time ago I’ve created a suite of (FitNesse) acceptance tests for the app we are working on. Some of them were of the form of:
Given that a customer exists with customer name "John"Given that a search results page is displayed for search criteria customer name "John"
Then at least one one customer should be displayed
Our QA (correctly) noted that this test is far from perfect because it doesn’t check that the found customer is the one with name “John”. On the other hand, there could be more than customer with this name. This means that we should not only verify that all found customers have the name “John”, but also that we have found all customers with such name. So we should first look into the database to find out what customers should be in the list and then check that we display only them. Correct? No!
The problem with this approach is that the test starts to mirror the application. It queries the database for customers with the name “John” – but isn’t this what the app does? If we made some mistakes in the app when implementing the query in the app, then who can guarantee that we won’t make the same mistakes when implementing that same query in the test?
So we limited ourselves with this:
Given that a customer exists with customer id "mock-customer-1" and name "John" and surname "Smith"Given that a search results page is displayed for search criteria id "mock-customer-1"
Then at least one one customer should be displayed with name "John" and surname "Smith"
And no customers should be displayed that don't have name "John" and surname "Smith"
Yes, this test does not verify that we found all the customers we searched for, but the only test that verifies application completely would be the application itself. Tests should be a stabilizing harness, not a copy, otherwise there’s no point in writing them.