This blog post focusses on the critical technology decisions (specifically on tools) that I believe are missed by techies on non-trivial software projects. At least I have missed them in the past because they seemed like “over-engineering”, and the bricks of YAGNI would be hurled around.
I insist and recommend all fellow techies to consider these strategies upfront in your projects, even if their necessity isn’t immediately apparent. I’ve dubbed this approach The ‘Reverse-YAGNI’.
Having worked on several software projects throughout my career, temporally ranging from a few weeks to a few years, I have realised that sometimes, the decisions that I wrote off as YAGNI, eventually became imperative to the success of the project. These may be obvious to many, but turned to be lessons learnt from (sometimes bitter) experiences to me. This is especially applicable to greenfield software products, but not limited to them.
For e.g., say you start a non-trivial software development project. It’s not Google scale, but not a To-Do list either. Say it’s the cliched simple digital library management system. I would generally start with a YAGNI attitude with a simple modular monolith and Boring technology. But humans are capricious, and a simple book catalog can turn into the library of Babel.
It’s much safer to make provisions in the project for at least the following ones, to ensure you are protected from the unpredictable future needs from the product. These are tech-focussed and simple enough to not need very high upfront investment, but help in longer term.
A Procedural Workflow (Orchestration) Framework
The lesson (and recommendation) here is that every non trivial application eventually needs a step by step procedural workflow that has human and non human inputs. It is very obvious in apps like bank customer onboarding where typical steps include customer self sign up, human approval of documents, system check of credit etc. But even in a simple commerce app, I have seen workflow needs arise and many a times much later in the project. For example, order orchestration can get incredibly complex because ad-hoc choreography with no central visibility is not useful enough. Integrating a Workflow (either custom or a SaaS tool) later in a project is difficult and impacts timelines and budget.
You likely don’t need the Camundas of the world, but a simple framework like Temporal, AWS Step Functions or Uber’s Candence are good enough for most cases. Of course, if you felt the need for Camunda initially, then there was never a confusion about the tool choice in the first place.
Auth Framework — AuthN and AuthZ
It’s worth investing in a robust Authentication (AuthN) and Authorization (AuthZ) framework from the start, even if your current system only requires a basic user login with a single role. While I agree that something like Zanzibar or OPA would be an overkill, a simple, configuration-driven framework can be invaluable.
In 99.9% of cases, you should avoid creating your own authentication system from scratch. Many web development ecosystems, such as Rails and Node.js, offer lightweight yet feature-rich AuthN/AuthZ frameworks. These abstract the underlying systems, making it easier to scale or migrate to enterprise-level solutions like Auth0 or Okta as your needs evolve. There are other SaaS ones like Firebase Auth, AWS Cogito etc., but that’s a matter of familiarity and taste.
The main point here is “Have a decent AuthZ/AuthN sysem in place even if you feel YAGNI-sh”
A Scheduler / Batch job Tool
We live in a real time streaming world and many devs (including me) feel antiquated an un-cool while implementing batch jobs, when I can use Kafka streams and Apache Flink in my 10 requests per second app (/s). Although, in most cases, these are not required. A simple scheduler (either batch based or using a delayed queue) is enough for most asynchronous use cases.
And there are some good use cases. Assume that your distributed system employs the Retry Strategy for your sychronous Order Creation APIs to handle failures; what if all the synchronous retries are exhausted? You would want a closure that Order creation, isn’t it? It’s bad to leave a customer in lurch. A scheduled job running silently and separately can do some more retries later with exponential backoffs and then send an email to the customer notifying of success or failure. (Everyone deserves a closure)
You can use Spring Batch (Java), BullMQ (Node JS) or SideKiq (rails) that provide useful features without the steep learning curve or an overwhelming feature set. Have one of these tools integrated in your system upfront even if your colleagues call you an anti-YAGNI.
Seed / Master data management framework
This may seem obvious, but most applications would need some sort of seed or master data that is easily cacheable and provides features to create and edit such data. In banking systems, it is generally bank branches data, or in commerce it’s Serviceable zip codes list or distance between them.
And most users will eventually want to use Excels and CSVs to manage such data. (Believe me, businesses still run primarily on spreadsheets)
A simple PostGrest with the right caching configurations or a Redis instance can do wonders. But more importantly, have a framework that can read dynamic CSVs and create tables automatically from them, while also supporting UPSERTS. I have felt this sorely missed in many cases.
Mock/Stub API Service
I haven’t worked on a project in my career that didn’t use APIs, either managed by teams within the company, or external SaaS based. The timelines and priorities of the different API never ever align, putting your plans at risk. And god forbid you have to rely on an API from the government! (many banking apps need this). Many systems don’t have a staging API or a dev playground.
You would ideally want your system to be decoupled from the upstream and downstream API dependencies for development. While end to end integration testing cannot happen without all the APIs being available, the development need not stop. To achieve this operational decoupling, a mock/stub based API service that can provide a configurable (an preferably UI driven) API provisioning with mock requests, responses and based on conditions. For e.g. I should be able to upload my OpenAPI format and the tool should generate API stubs for all cases. I should be able to get reponses for HTTP 2XX, 4XX and 5XX, so these systems can be developed independently. Contract testing can ensure any deviations are caught early in the development cycle.
I have used MounteBank, Postman, APIGEE in the past and they all work well, but the efforts to get it right are high. I am still looking for a SaaS solution that solves this problem for me (and better if it can use LLMs), so that even non-tech team members can leverage it.
A simple Config / Feature Flag service
Not much to say here. A simple (config file based or something like LaunchDarkly) can be used to manage features in the system. Many features may need to be rolled out behind an activation flag. And a simple forethought on a platform wide feature is useful.
The key is balance. There will always be people who will blame you of being short-sighted on one hand, or or doing too much over-engineering on the other.
There are many other tools I feel are important like an Auditing frameworks that log all important business events in a data warehouse, or a system logs indexing framework that can parse logs in real time to provide predicable analytics of systems, or a service mesh in Kubernetes system, or a Data Product/ML tool for easy data aggregation and model execution etc. but they did not seem too obvious to be included.
Add any others if you feel. Thanks for reading!
