What is QA?
Quality assurance (QA) is a way of reducing bugs in products and avoiding problems when delivering solutions or services to customers. With frequent updates comes frequent testing, and we’re no stranger to the concept. Unlike other solutions who update their apps quarterly or sometimes bi-annually, Blackthorn releases monthly updates to our Events and Payments AppExchange solutions. As our apps grow, so does the need for efficiencies. Testing our new features for bugs or errors is an ongoing process, and automation greatly increases efficiency.
Why automated testing? Manual testing is time-consuming, and regression based (existing functionality) bugs can be missed while manually testing. This benefits by avoiding hours of testing small errors that a system can easily catch.
These tests usually don’t take long to make and target actual use cases! We target how end-users use the system, and how our own team would use the apps we provide.
How do you know what to create? We target any issues that can cause an inconvenience for our customers.
- Customer Success Team receives a bug report – internally or externally.
- The bug is documented to be fixed by developers
- Quality Assurance then creates automated tests addressing the bug
- For new features we create automated tests before any bugs are found
- This ensures on any future updates, we won’t have the same issue.
How does it work in production?
- Our development team releases an update internally
- We test the update with a fresh install in a new Salesforce developer org that is generated via Salesforce DX, scripts, and scratch orgs.
- We test the update with an upgrade install in an existing Salesforce developer org
- If there’s a bug, we fix it. If there isn’t, we have the green light for release!
The automated system our team chose to use is Uilicious. We explored other options before deciding what route to go. The first was Selenium with a Salesforce web driver, and the second was Provar. Selenium is an older tool used for testing, so we weren’t certain on capabilities moving forward. On top of that, Selenium isn’t user-friendly for anyone to pick it up and run with it. Provar was codeless and seemed very easy for anyone to pick up and use, we strongly considered this option. However, due to the lack of communication and the price tag (a steep $23,000 starting for one user), we decided to pass.
Uilicious is derived from Selenium and customized heavily by their development team to be simple and easy to use. It is a happy medium between code and codeless automation and has a free version. We opted for the Studio Basic, and our package is $90 per month. The customer support is very helpful and quite responsive for our time zone, being based in Singapore while we are in the United States. The documentation they provide is excellent and covers plenty of scenarios many people would want to test. Some undocumented features are being worked on, and if you ask for them, they will gladly provide you with the resources that are developed so far.
You have all this automated, now what’s next?
Glad you asked! In our ecosystem, we push updates every month. Given the size of our team, we update automation as frequently as possible, but sometimes releases get pushed back. This automation is built in terms of scalability. Manually testing one or two features isn’t too much of a feat; however, testing entire packages after a release, hotfix, etc., is time consuming to test manually. Automation can be intimidating when diving in, but investing the initial time pays off tenfold in the long run.
For those that are interested in the technical aspects – keep reading!
Blackthorn manages automated testing in two different ways.
Uilicious allows for end-user testing. If a flag is thrown for an error, we can investigate further and fix the issue. This allows us to test many scenarios that our end users would do. This can include any…
- Clicking
- Scrolling
- Keyboard commands (arrow keys, enter, tab, etc.)
- And many more scenarios that users would navigate our platform.
Uilicious isn’t all sunshine and rainbows, though. Every good piece of software always has the desire for more. The below list of items we have struggled with as a Salesforce app company testing in Salesforce:
- Salesforce is a tricky beast to tame; we use iframes built into our Visualforce and Lightning components. Uilicious usually does well with handling these, but when testing embedded VisualForce iframes “ it gets a little trickier to automate.
- Timeouts occasionally, their servers will have timeout issues after running tests that are quite lengthy.
- While Uilicious tests user journeys, it is impossible to replicate exactly how an end-user navigates the page.
- The largest desire was for bug fixes. Just like us, Uilicious is a fairly new company. Their system has bugs; however, they are very quick at fixing most of them.
Below is an image of the start to creating a new test; they give a simple setup guide here, with plenty more documentation listed here.
Part of the way through, it starts looking something like this:
By the end of testing, the lines start to add up quickly. This is the total from one of our tests that end up being ~1200 lines long. This test is currently a work in progress, testing everything from start to about 80% of full testing.This script is the start of simply going to the Salesforce sign up page, creating an account, and doing our fresh install testing. A brand-new Salesforce developer account is made each time we release a package to run through testing. This allows our team to work on other tasks, while testing the package at the same time.
Uilicious took some time to implement for our use case but was well worth it. We also use it for checking if any of our features are down, this includes DocumentLink and downloading the documents from the web page, PayLink, and our Events app. If the system detects a page that can’t be rendered, download links are working, etc. it flags the test, takes screenshots of what happened, and automatically emails our team to let us know there are issues with one of our services.
We test both our Payments and Events app in Uilicious “ below are a couple of video examples.
The first, Payments. In this example, we test our custom Visualforce Component, Virtual Terminal. This was exceptionally important for us to constantly be testing, as it’s a very powerful feature that many of our customers use. For example, a test would look like this.
- Selects a new charge
- Adds a payment method (including name, credit card number, zip code, etc.)
- Adds the amount to charge
- Checks out
- Checks the Transaction Record to see that it is completed and the correct amount was charged
Check our Payments Virtual Terminal being tested here.
Secondly, Events. For an example, this test:
- Opens the event associated with this certain test
- Registers as an attendee
- Applies a discount code
- Checks out the cart.
At the end of the test, the system checks the ERS and Sales Document to ensure the customer was recorded and charged appropriately.
If you’d like to check out a video of this being tested, please click here.
The second form of automation we use is a tool called Cypress. On top of Cypress, we use a behavior-driven development framework called Cucumber.
Why did we choose Cypress?
- Powerful browser testing platform
- Large community support
- Ease of setup and use
a. No need to stand up and configure dedicated servers or use third-party services
b. Aligns well with our JS-centric development team - Many convenient features out of the box
a. Extensible with many available add-ons
Why did we use Cucumber?
- Allows us to define our end-to-end tests behaviorally, as mentioned earlier.
- Natural language makes tests understandable to non-technical persons
- Test definitions translate easily between automated and manual process
a. Anyone (including non-developers) can verify a test by simply following the steps
b. Anyone can define a testing scenario without needing to be technical - Re-usability of test steps provides uniformity in defining and understanding what is being performed
a. Example 1: “I see something”.
b. Example 2: “I click on something”. - Imposes certain limitations that prevent us from “cheating” a test
- Naturally forces us to very explicit and clear about what is getting done
If you’d like to check out a video of Cypress in action, please click here.
Sounds a lot like Uilicious, huh? Here’s the difference – Cypress is a very short path for the development team to raise confidence on every release by simply ensuring that our app works consistently. On the other hand, Uilicious allows us to have a fully integrated end to end testing (between Salesforce, and the UI we build for our web application), which we cannot do with our cypress tests. Uilicious tests also act as independent validation to work on the platform side and provide a filler for tests we cannot perform with Cypress such as testing on Internet Explorer and other browsers not supported by Cypress.