E2E Puppeteer: Flaky Test Failures and Retries

Problem

In project WooCommerce Payments, besides unit and integration tests for PHP and JavaScript, we have E2E (end-to-end) tests for some critical flows through GitHub Actions. The problem we face is that there were so many flaky tests without any clear reason.

Research

A little bit about WooCommerce Payments architecture. Besides running code as a WordPress plugin, it interacts with a few servers through HTTP requests as REST API. For running E2E tests, we do not mock these servers, and they use the real servers.

Initially, we noticed that many tests failed. Looking at them closely, there was no specific pattern. They generally happen randomly. After monitoring them more, I noticed the reason was that many HTTP request responses had error statuses, such as status 503 (Service Unavailable) or 429 (Too Many Requests). They were usually resolved by sending these HTTP requests again, or in the context of E2E tests, running tests once more.

With that in mind, our unified approach for this issue is to re-run failed tests.

Approach and Code!

To re-run failed tests, that can be many things:

❌ Trigger the whole E2E tests (GitHub Action) again. This is definitely inefficient as the build and the site setup happens again, and it’s just a waste of time and computing resources.
❌ Use the built-in jest.retryTimes() but it does not serve our specific test nature well because Jest will retry failed tests in sequence right away before it goes to the next test. The same issues, such as 429 Too Many Requests, likely happen again.
✅ Re-try only specific spec/test file in the new E2E run. This fits perfectly with our organization of tests: many tests rely on previous tests (that’s understandable, e.g. adding a card and then removing it) in the same spec/test file, and many flaky results have only one failed test file.

This PR includes all relevant changes as well as all details. In general, I just extracted the test results of all E2E tests for the first time, filtered out all failed spec files, and re-run these files only. This approach has a lot of advantages beside resolving the main flaky issue:

Minimal time usage – No need to re-run the whole process, including setting up E2E test environment.
Minimal change – No E2E setup is altered, and it’s simple to understand.

Lessons Learned

Flakiness is almost unavoidable, especially, in our setup, we use the real server for testing clients. However, we can make the flakiness rate reasonable by understanding what E2E tools offer and the real cause of failures.

PHP and JavaScript: Produce the Same Result for Now and Future

Problem

In WooCommerce Payments, we have a long-standing issue with displaying fee details in order notes. Before that, my team has already had logics in JavaScript to extract data from JSON (fetching via REST API request), and dynamically display fee details in a React component. The issue I am working on is to build static HTML content (order note) from the same output but this code must be used in PHP.

Approaches

The first option seems reasonable and avoids duplication. It’s updating the current JavaScript code to generate HTML, which can be used for both order notes and React components. However, we quickly noticed that this can be complicated due to:

Plenty of changes must happen so that the current generation logic can be switched from React component to HTML.
React consuming and displaying HTML content is not safe. Function name dangerouslySetInnerHTML declares that safety concern explicitly.
For PHP to get this HTML content, it needs to send a request to the site itself for triggering the JavaScript code. It requires even more thoughts to how to approach that correctly and safely.

And then, the second option is to write new PHP code reflecting the same JavaScript code. In other words, PHP and JavaScript codes are duplicated for this purpose.

Way to Final Solution

When decided to write PHP code, a new goal is risen: how to ensure that PHP and JavaScript can really produce the same result. Then, when a developer makes changes in PHP or JavaScript, they will need to do the same for the other language.

I approach that requirement by running unit testings for both PHP and JavaScript with the same fixtures and test results, which are actually encoded in JSON files.

With the current code, I refactor the JavaScript code first so that each line of fee details are handled individually. Then, based on that, I add JSON test files (based on the current JavaScript code result), add up new PHP code and its tests. All of these have been done in the second PR. The different fixtures actually aid me a lot in writing and tweaking PHP code to get the final result, which is pretty stable and almost have no problem.

Updates

August 2023, we found a bug, fixed it, and added a new fixture file. But the fix is still simple and effective for the long future.
September 2023, there are some changes in JavaScript logic, and developers can pick up that quickly to fix the PHP part too.

Currency Conversion – My React Side Project

I often travel abroad and currency exchange is somewhat I need to deal with in every trip. There are a lot of great apps out there but they also usually come with ads. After having finished a few courses and guides about React, I wanted to get started with a real project whose features and scopes are defined by myself. That’s my main motivation for this project!

Have a quick look at demo or code on Github!

Continue reading →

MERN Course on Udemy

In half of this year, I have spent time on freeCodeCamp to learn thoroughly all JavaScript courses from the basic one to the most complicated one. I had a basic understanding of all of them. However, when it comes to how to organize and connect all pieces into a real project, I was still not sure.

From the recommendation, I finished this course on Udemy two weeks ago. This is a great course to gather all knowledge fragments about MongoDB, ExpressJS, React/Redux, and NodeJS.

Continue reading →

JavaScript: setTimeout with Promise

const waitXSeconds = (x) => {
  return new Promise((resolve, reject) => {
    setTimeout(()=>{
      resolve('Done!')
    }, x * 1000 )
  })
}

// Usage
async () => {
  // Wait for 5 seconds
  await waitXSeconds(5);      
  // Do something else 
  // ...
}

Logical Operators: PHP vs JavaScript

I’ve come across this condition from this PR (scroll down to the bottom) and wondered how it really works.

if ( 'object' === typeof window && window.app && window.app.isDebug ) {
	// something here
	// redacted to focus on the if condition
}

Continue reading →