Engineering Year in Review 2021

Vercel

Last year we split out our site and our app into separate packages. This year we handed over the reins to the website to marketing and operations which now runs in Webflow. The few site pages that are dynamic or rely on app data that engineering still own were running on Next.js being hosted with FAB and previewed with Linc. With the acquisition of Linc by Cloudflare the projects haven’t kept pace with platform changes and we were forced to move our deployments elsewhere. We ended up going with Vercel (we went live at time of writing!) it’s working well, but the deployment scenarios need a lot of work and it seems confused to whether its a deployment or build tool (preview greys the lines a little I guess)

SWR

The stale-while-revalidate pattern and specifically SWR Hooks by Vercel was our darling of 2021. It transforms how you think of remote data (cache not state) and has really improved the quality of our front-end code by reducing the amount of state management required. React Hooks can’t be conditionally called, so the pattern of passing in condition ? url : null to conditional data fetches takes a little while to get used to, but conditional fetches are not the norm for us.

The infinite pagination support is also awesome

ESM

As node marches towards ESM we are continuing to make sure we are future facing, this year we have upgraded all our library packages. The addition of Typescript project references also makes our build times much better and improves developer productivity. There are still rough edges with Typescript refusing to believe that project references are a build pipeline and add support for hooks or copying other files during the build... but baby steps.

Lazy Loading Packages

After splitting app and site into separate packages we went about separating app into micro front-ends which are loaded on-demand using Reacts Lazy loading. This improves the load times for users as well as better modularity of code for engineers.

Automation and Testing

DB migrations

Our DB migrations were one of the last manual process in our CI/CD.  Engineers would merge a feature or promote to production and forget to run the database updates. This was mainly due to our build server (CircleCI) not having access to our database. This year we made a lambda inside our environment that runs our migrations on demand and is triggered by the deploy build. We haven’t forgotten to run them since 😃

The last thing to tackle (maybe in 2022?) is to automate the addition of new service schemas and permissions.

E2E Testing

We made great progress on e2e testing of our new services. With a tight focus on wiring, integration points and auth; our tests are lighter, easier to write and give us the security knowing that things are wired up correctly. Our orchestration services which pull data from a lot of sources still prove to be difficult to test and we have to balance the effort vs reward on these.

2022 Focus

Consolidating

It’s been a whirlwind of getting out an MVP and iterating that in a more general direction. We still have a bit of work to amalgamate all the different paths through the product as well as our code base. We’ll be balancing iterating on our new product with fully integrating the old products.

Getting back to Kanban

2021 the product team fell into a more waterfall based approach to delivering large pieces of an essentially new product. Our epics/projects were large and spanned months from scoping through design to build. This increased the cost of change once engineering was heavily involved, hindered cross functional collaboration as functional teams were working on very disparate areas at the same time, and made estimating and managing timelines difficult.

In 2022, we aim to drive the push back to a more iterative, small feature approach where we can work closer with design. The challenge will be how do we go from broader product vision to scoping smaller steps towards that vision.

Continued Front End code base improvement

Whilst we made great inroads in 2021 on the front end code base there are still a few areas of focus for 2022.

• Move to latest next.js and server side components

• Migrate away from styled components. Rather than loading style files from our components we will transition towards a modern style-loading paradigm such as CSS Modules or CSS in JS. This will enable us to upgrade to the latest next.js and prevent style collisions.

• Improve our package naming. We currently have library packages, micro front-end packages and application shells. Our naming convention is a little confusing as to what each package is.

Enzyme

Through 2021 we noticed a bit of a schism forming or formed between React and Enzyme with adapters being slow to be released. There is a competing testing library from React and we’ll have to keep an eye on the situation. It makes us a little nervous as we have a large body of enzyme tests, which would be costly to replace if enzyme started blocking us from React upgrades.

‍