Behind the scenes at SimplyRETS, we focus a lot of our effort on testing to make sure our API, website, and other web services are always functioning properly. Our infrastructure relies heavily on continuously testing at each step of the development process. One of the tools we rely on the most is Assertible, a tool for testing and monitoring APIs and websites. We use Assertible to monitor and continuously test our API, website, blog, and any other web services in our infrastructure.
The primary components of our continuous testing pipeline are:
- Automated post-deploy testing for every push
- Scheduled monitoring
- Add tests for all new bugs or regressions
Automated post-deploy testing for every push
We use GitHub to host our code and push all new features and bug fixes to Pull Requests. Pull requests are very a convenient way to manage proposed changes to our code. They also offer a ton of hidden features like GitHub Deployments and GitHub status checks.
One reason we love Assertible is that it integrates directly with GitHub and utilizes these hidden features to improve the experience of building web services.
Every time we push code to a pull request we deploy the changes to a staging environment. Immediately after the code is deployed, Assertible runs API integration along with functional website tests against the staging environments. When the tests are finished running, we can view the statuses directly on our pull requests. Additionally, the Assertible Deployments API creates links directly to our deployed web services so we can jump to them for review and sharing:
Because of this strategy, we don’t have to worry about manually testing old features which allows us to focus on the features we are currently building without breaking something. While we also use unit tests extensively, we rely heavily on Assertible for integration testing and functional validation of our API release candidates.
Scheduled monitoring
Scheduled monitoring is another critical component of our continuous testing strategy. We run our tests on scheduled intervals to monitor performance and availability to ensure everything always works and is not causing bugs for users. The same test-suites are reused in the post-deploy step mentioned above.
Scheduled API monitoring and website monitoring has several advantages over post-deploy testing alone:
- Ability to validate and test data that constantly changes
- Identify regressions that take time to manifest (e.g. background jobs that change the database)
- Continuously monitor for downtime across a wide array of endpoints
- Continuously monitor for functional defects that aren’t picked up by more simple monitoring services like Pingdom.
We believe that scheduled monitoring is a critical piece of the puzzle and helps us track down bugs before our users hit them!
Add tests for all new bugs or regressions
As part of our overall continuous testing strategy, we rigorously add new tests every time a new bug or regression is uncovered. Because we end up adding tests often, we follow some simple guidelines to ensure the tests aren’t flaky and constantly blowing up our alerting system:
-
Create the smallest test possible
This generally means confining the test to validate a single feature, regression, or bug.
-
Test a single specific interface
This is straight forward in our infrastructure. We limit our tests to a single interface and try not to make tests that rely on several services, when possible.
-
Avoid allocating blobs of data
This goes along with creating the smallest test possible. We’ve found that tests which rely on allocating too much data become flaky. While this is necessary sometimes, it’s important to only allocate the data that’s absolutely necessary to test a single thing.
-
Refactor or remove old flaky tests
Testing is hard. Often, tests can become flaky or raise false-positives often. Because of this, we refactor and/or remove tests which are causing constant problems. When possible, we try to add tests at the unit testing level.
Some of these approaches are described in more detail on the Assertible Blog in the 4 techniques to reduce API testing errors and improve your QA automation post.
— Christopher @ SimplyRETS
Drop us a line if you have questions or you’re interested in learning more about SimplyRETS