At Chase, the key to our software development life cycle (SDLC) is delivering quality software quickly and predictably to production so we can deliver the best experiences to our customers. Validating every mobile code change before it’s integrated using automated unit and functional UI tests is rare in the industry and immensely difficult to accomplish but something we believe is essential.
To speed up our SDLC, we’re focused on solving this problem using automated continuous integration continuous delivery (CI/CD) through a multi-step automation flow that allows code to be written, fully tested, deployed and if needed, rolled back quickly. We call this TrueCD and it empowers engineers, reduces toil and improves software quality.
The Not-So-Simple Process of Mobile Testing
All channels are highly sensitive to quality issues, and mobile can be even more daunting, so getting code right at the core level is critical. Chase has many teams that work on the various features within our mobile banking applications, and these teams use numerous code repositories that receive incoming changes at all hours of the day and night. One bad code merge can bring the development and testing of hundreds of engineers to a standstill.
Although unit tests and diligent code reviews find many issues before they’re introduced, catching all the problems without compiling and running on a mobile test device is nearly impossible.
Validating incoming changes with unit testing is imperative, but even when they all pass something as crucial as being able to authenticate in the app may stop working. Merging these incorrect pull requests can prevent many teams from moving forward. To further complicate the matter, additional code may get merged on top of the problem, making it even more complicated and time consuming to find the offending change.
We worked to introduce functional UI tests into the continuous integration pipeline that validates applications work properly before merging all changes to the code upstream.
Breaking Down our TrueCD Automation Project
Mobile UI tests have long been perceived as slow, flaky and challenging. Done poorly, they introduce more problems than benefits, often leading teams to abandon the cause altogether. Long-running tests can get in the way of progress when teams are waiting for them to complete. If the tests aren’t reliable, teams become skeptical of the results and they’re ignored. Tests that are complicated to run and force developers to switch from their current activities quickly become a burden that they avoid at all costs.
It’s unfeasible for just one team to test every aspect of an application the size of ours. Each team must have a solution that empowers them to provide high value at low cost. That’s where TrueCD comes in.
Chase’s TrueCD (True Continuous Delivery), commonly known in the industry as continuous integration/continuous delivery (CI/CD), is a set of automated practices for testing across applications with the goal of building and releasing code changes in less than one hour. TrueCD helps overcome teams’ challenges such as:
- Cognitive overload for software developers — they need to know infrastructure, ADA compliance, security, cloud and much more.
- Manual errors and issues that can interrupt customers’ access to their money.
- Time wasted coordinating and chasing after manual development cycles.
Using Chase’s TrueCD testing approach, teams now have a defense mechanism to keep their work from getting blocked.
Implementing Automation for Mobile
Taking our TrueCD approach from multiple, non-mobile channels, we implemented automation as a solution for engineers to be able to get coding right the first time in mobile. While we work in multiple channels when coding, we “follow our customers” in mobile. This solution provides teams an easy way to automate Gherkin domain specific language specifications in a way that runs in their existing software build pipelines, and was founded by applying Chase’s TrueCD principles with a focus on addressing the following needs:
- Fast — Tests must run quickly to not get in the way. Tests should run in seconds, not minutes. Long running tests won’t be run often, if at all.
- Repeatable — Tests that don’t consistently return the same results are not trusted nor useful, and flaky tests are seen as a waste of time. Tests should not have to be run repeatedly to get them to pass.
- Simple — Context switching kills productivity. If testing is a part of the team’s daily routine, it can’t pull them out of what they’re doing. Writing new code should not require setting up and running the tests shouldn’t be a complicated affair that introduces friction.
To ensure tests run fast, we use industry-leading mobile testing frameworks that build tests into a separate application that runs natively within the same device as the app being tested. Tests are synchronized with the application being tested so they know when to wait and resume automatically. This eliminates the need for costly waits, retry loops and other tactics that make tests slower.
Synchronizing the tests with the app being tested not only makes them faster but it also makes them more reliable. Mobile applications are inherently asynchronous. Using a testing framework that handles this for you makes your tests easier to automate and less error prone. Reducing testing boundaries only to our project, without integration with external dependencies, prevents instability from outside. All web services are mocked to ensure the same response is received and is more easily controlled by the test itself.
Finally, to ensure tests are simple to create, run and maintain they must be accessible within the Integrated Development Environment (IDE), making it simpler to understand what failed and where without having to juggle additional tools to edit tests, run tests and even modify the application being tested. If a test is having issues, using a powerful IDE provides the means to inspect view hierarchies or profile the application to get a deeper look at what’s happening while the test runs.
Putting the Tests in Action
Automation doesn’t require writing additional tests beyond what’s in the story. User stories that are refined collaboratively with representation from Quality Assurance (QA), Development and Product teams provide the best and most important tests for the product. Declaratively written test scenarios define what the feature is, something that is immensely more important to convey than how we test it. Not only are they simple to read and understand by all, they’re also far less fragile. Acceptance criteria are written using the Gherkin domain specific language and each scenario contains steps that avoid imperative actions such as “clicks,” “scrolls” and other phrases that would couple them too tightly to frequently changing user interfaces. Teams automate the steps with powerful software development languages they’re already familiar with. When a developer refactors UI code in their IDE, the IDE is also aware of the test code and automatically refactors it too.
The most valuable tests are tagged to run as part of the continuous integration pipeline on all pull requests to our repositories. In just a few minutes we get a report on the pull request indicating what passed and any failures. To easily understand why a test has failed, app logs, screenshots and even result files can be used within the IDE to pinpoint the location of the issue.
Ongoing Benefits of TrueCD
UI testing is rare and difficult to do, but we are committed to providing our customers with fast, reliable products and services that require constant innovation and refinement. Adopting TrueCD in mobile has brought more developers to the table who were avoiding the challenges of functional testing. When developers are a part of the functional automation process, they produce code that’s easier to test. Test automation engineers spend less time maintaining flaky tests so they can move on to more productive activities. With fast, repeatable and simple tests we see more team members automating more than ever. In the end, all this translates to getting more quality features to our customers, faster.
For Informational/Educational Purposes Only: The opinions expressed in this article may differ from other employees and departments of JPMorgan Chase & Co. Opinions and strategies described may not be appropriate for everyone and are not intended as specific advice/recommendation for any individual. You should carefully consider your needs and objectives before making any decisions and consult the appropriate professional(s). Outlooks and past performance are not guarantees of future results.
Any mentions of third-party trademarks, brand names, products and services are for referential purposes only and any mention thereof is not meant to imply any sponsorship, endorsement, or affiliation.