In fact, many of the products we use every day began as a single application connected to a single database. That is usually the right decision.

Yet, if you spend enough time building software, there is a good chance you will eventually hear a familiar statement:

"We need to move to microservices."

The statement often arrives after the system has grown beyond its original expectations.

The engineering team has grown.

Features are being delivered faster.

Deployments are becoming riskier.

Different parts of the system are competing for resources.

What once felt simple now feels fragile.

At this point, many teams begin looking at microservices as the solution.

Unfortunately, this is where many of them make a costly mistake.

They focus on splitting the application into smaller services but fail to prepare for the realities that come with distributed systems.

• Network failures.

• Service dependencies.

• Message duplication.

• Observability challenges.

• Complex debugging.

• Cascading failures.

I have seen engineers successfully break apart a monolith only to discover that operating ten services is significantly harder than operating one.

This is why I believe the conversation around microservices is often incomplete.

The challenge is not moving from a monolith to microservices.

The challenge is building APIs that continue to work when services fail, traffic spikes, dependencies become unavailable, and production incidents occur.

"The challenge is not moving to microservices. The challenge is building APIs that survive production."

In this article, I will walk through the practical journey from monolith to microservices. More importantly, I will show how to design APIs that are resilient, observable, and capable of surviving production.

The Day Your Monolith Starts Fighting Back

I often tell engineers that there is nothing wrong with a monolith.

In fact, I would rather inherit a well-structured monolith than a poorly designed microservices architecture.

Many teams rush into microservices far too early.

They hear success stories from companies like Netflix, Amazon, and Uber and assume microservices are the natural next step for every application.

They are not.

A monolith is usually the fastest way to deliver value during the early stages of a product.

Everything lives in one codebase.

Everything is deployed together.

Debugging is straightforward.

Transactions are simple.

Development is faster.

A typical monolithic architecture looks something like this:

Everything appears manageable.

Until it isn't.

Imagine an e-commerce platform.

At the beginning, the application supports a few thousand users.

The backend handles user management, orders, payments, inventory, notifications, and reporting.

The system performs well.

Deployments are easy.

Everyone is happy.

Then the business grows.

The marketing team launches successful campaigns.

Traffic increases.

New engineers join the team.

Additional features are introduced.

What was once a clean application slowly becoming more difficult to manage.

Now a change in the notification module requires deploying the entire application.

A bug in reporting can affect order processing.

A spike in product searches forces the entire application to scale, even though only one module is experiencing heavy traffic.

Deployment windows become stressful.

Engineers become increasingly cautious about making changes.

At some point, the monolith starts fighting back.

Not because monoliths are bad.

But because the needs of the business have evolved beyond the architecture's original assumptions.

This is usually the point where teams begin exploring microservices.

The problem is that many engineers think microservices exist primarily to scale servers.

In my experience, that is only part of the story.

The real reason microservices exist is much more interesting.

Microservices help organizations scale teams.

"Microservices exist to scale teams far more than they exist to scale servers."

And understanding that distinction changes how you design systems.

In the next section, I'll explain why microservices exist, when they make sense, and why moving to them too early can create more problems than they solve.

Why Microservices Exist (And Why Most Teams Misunderstand Them)

One of the biggest misconceptions I see is the belief that microservices exist primarily to help applications handle more traffic.

Traffic is certainly part of the conversation, but it is rarely the root problem.

I have seen monolithic applications comfortably handle millions of requests.

I have also seen microservices architectures struggle under workloads that a well-designed monolith could have managed without difficulty.

The question is not:

Can my application handle more traffic?

The question is:

Can my engineering team continue to deliver changes safely and efficiently as the product grows?

That is where microservices begin to make sense.

As systems grow, the bottleneck often shifts from infrastructure to people.

A team of three engineers can comfortably work within a monolith.

A team of thirty engineers working on the same codebase is a very different story.

The challenges become less about CPUs and memory and more about coordination.

Teams begin stepping on each other's toes.

Deployments become frequent.

Merge conflicts increase.

Release cycles slow down.

Changes that should take hours begin taking days.

At this point, the architecture is no longer serving the organization effectively.

This is where microservices can help.

Not because they magically improve performance.

But because they allow teams to move independently.

Scaling Servers vs Scaling Teams

Let's consider an e-commerce platform.

Initially, the system consists of a single backend application responsible for:

• User Management

• Product Catalog

• Inventory

• Payments

• Notifications

A single engineering team owns everything.

This works well.

But as the business grows, things become more complicated.

Different teams emerge.

One team focuses on ride operations.

Another team focuses on payments.

Another team focuses on customer communication.

Now imagine the payments team wants to release a critical fix.

Unfortunately, they must wait because another team is preparing a large deployment involving ride matching logic.

The two changes are completely unrelated.

Yet they are tightly coupled because everything lives inside the same application.

This is one of the first signs that the architecture may be limiting organizational growth.

Microservices address this problem by allowing responsibilities to be separated into independently deployable units.

Instead of one large application, we begin moving toward something like this:

Now each service can evolve independently.

The Payment Service can be updated without redeploying the Order Service.

The Notification Service can scale independently during peak communication periods.

The Notification Service can adopt a different technology stack if there is a legitimate reason to do so.

The architecture begins to mirror the structure of the business itself.

A Microservice Is Not Just a Smaller Application

This is another mistake I frequently see.

Many teams take a monolith and simply split it into smaller pieces.

Unfortunately, this often creates what I call a distributed monolith.

The services are physically separated, but they remain tightly coupled.

Every service depends on every other service.

Changes ripple through the system.

Deployments remain coordinated.

Failures spread quickly.

The team has inherited the complexity of microservices without gaining their benefits.

A good microservice is not defined by its size.

It is defined by its responsibility.

Each service should own a specific business capability.

Examples include:

• User Service

• Payment Service

• Inventory Service

• Notification Service

• Order Service

Notice that each of these represents a business function.

Now consider these examples:

• Utility Service

• Common Service

• Shared Service

• Database Service

These names usually indicate unclear boundaries.

When service boundaries are unclear, ownership becomes unclear.

And when ownership becomes unclear, complexity increases.

A useful question I often ask is:

"What business capability disappears if this service is removed?"

If the answer is obvious, the boundary is probably healthy.

If the answer is vague, the service may need to be reconsidered.

The Database Conversation Nobody Likes

Many teams successfully split their application into multiple services.

Then they connect every service to the same database.

At that point, they have not really solved the coupling problem.

They have simply moved it.

A shared database creates hidden dependencies.

One service can unintentionally break another.

Schema changes become risky.

Teams lose autonomy.

The database becomes the new monolith.

A fundamental principle of microservices is data ownership.

Each service should own its data.

For example:

Order Service

• Orders

• Order Statuses

Payment Service

• Transactions

• Refunds

Notification Service

• Message History

• Delivery Status

This separation allows teams to evolve their services independently.

It also introduces new challenges around consistency, which we will discuss later.

Every architectural decision is a trade-off.

Microservices are no exception.

Before You Reach for Microservices

Whenever someone asks me whether they should adopt microservices, I usually ask a different question:

What problem are you trying to solve?

If the answer is:

• We want to look modern.

• Netflix uses microservices.

• Everyone else is doing it.

Then I strongly recommend staying with the monolith.

If the answer is:

• Teams are blocked by each other.

• Deployments are becoming risky.

• Different domains need to scale independently.

• Organizational growth is creating bottlenecks.

Then microservices may be worth considering.

The goal is never to collect services.

The goal is to create a system that allows the business and engineering teams to move faster without sacrificing reliability.

That sounds simple.

In practice, this is where the real challenge begins.

Because the moment services become independent, they need a way to communicate.

And communication is where many microservices architectures either succeed or fail.

In the next section, I'll explore the communication patterns that power microservices, when to use synchronous communication, when to use asynchronous messaging, and the trade-offs that every engineering team must understand before choosing either approach.

Communication Between Services: Where Most Systems Start to Break

Once you move beyond a single application, the real work begins. At this point, I usually tell engineers that the architecture is no longer the hard part. The hard part is communication.

Because now, instead of functions calling functions, you have services calling services over the network. And the network, unlike in-process memory, is unreliable by default.

I have seen systems that looked perfectly clean on paper fall apart in production simply because communication patterns were chosen without enough thought. A simple user request ends up bouncing between services, and when something goes wrong, nobody can immediately tell where the failure started.

This is where you start to feel the difference between synchronous and asynchronous communication, not as theory, but as lived experience.

Synchronous Communication: Simple, But Fragile

Synchronous communication is usually where teams begin. It feels natural because it mirrors how monoliths work. One service call another, waits for a response, and continues execution.

For example, in a simple order flow:

The Order Service receives a request, then calls the Payment Service to confirm payment before finalizing the order.

On the surface, this is straightforward. It is easy to reason about, easy to implement, and easy to debug in small systems.

But I have learned that simplicity in distributed systems can be deceptive.

Because now the Order Service is no longer just responsible for orders. It is also dependent on the availability, latency, and stability of the Payment Service. If the Payment Service slows down, the entire order flow slows down. If it fails, the order flow fails with it.

At scale, this creates a chain reaction. One slow service can degrade the entire system experience, even if everything else is working perfectly.

This is the first place I usually see teams underestimate the cost of microservices.

Asynchronous Communication: Where Systems Start to Breathe

At some point, teams begin to realize that not everything needs an immediate response. This is usually where asynchronous communication enters the picture.

Instead of calling another service directly, a service publishes an event and continues its work. Other services subscribe to that event and react independently.

A simple example is an order event. When an order is created, the Order Service publishes an OrderCreated event. The Notification Service listens and sends emails. The Analytics Service listens and updates metrics. The Inventory Service adjusts stock.

The interesting part here is not just the architecture, but the shift in thinking. The Order Service no longer cares who reacts to the event or how many services are involved. It only cares that the event is published successfully.

This is where systems start to feel more flexible. They scale better, failures become more isolated, and services stop being tightly dependent on each other.

But I always caution engineers here. Asynchronous communication does not remove complexity. It moves it somewhere else.

Now you have to think about event ordering, duplication, eventual consistency, and debugging across time rather than across a single request flow.

I have seen engineers struggle more with this shift than with microservices themselves.

What Real Systems Usually Look Like

One of the biggest misconceptions in software architecture is that systems are either synchronous or asynchronous.

In reality, most successful systems are a mixture of both.

A ride booking request might look like this:

The payment verification remains synchronous because the passenger needs an immediate answer.

The notifications and analytics updates become asynchronous because they do not need to block the booking process.

This approach allows the system to remain responsive while reducing unnecessary dependencies.

Over time, I have found that this hybrid model is usually the most practical approach for production systems.

The goal is not to eliminate synchronous communication.

The goal is to reserve it for situations where immediate feedback is genuinely required.

The moment services begin communicating over a network, failures become inevitable.

Not possible.

Not likely.

Inevitable.

And that realization changes how you design systems.

In the next section, I'll explore why production-ready microservices must be designed with failure in mind from day one, and the resilience patterns that help APIs continue functioning even when parts of the system are failing.

Designing for Failure: Because Production Doesn't Care About Your Architecture

One of the most important lessons I learned about distributed systems is that failure is not an edge case.

It is a feature of the environment.

When engineers first move from a monolith to microservices, they often focus on service boundaries, deployment strategies, and communication patterns. Those things matter, but they are not what keeps systems running in production.

What keeps systems running is how they behave when things go wrong.

And things will go wrong.

A service will crash.

A network connection will drop.

A database will become unavailable.

A third-party provider will experience an outage.

A message broker will become overloaded.

None of these scenarios are unusual. In fact, if your system runs long enough, every single one of them will happen eventually.

The question is not whether failure will occur.

The question is how your system responds when it does.

I often tell engineers that the difference between a development environment and a production environment is simple:

Everything works in development.

Production introduces reality.

The First Mistake: Assuming Every Request Will Succeed

Imagine a ride-hailing platform.

A passenger requests a ride.

The Ride Service calls the Payment Service to validate the passenger's payment method before confirming the booking.

Everything works perfectly during testing.

The Payment Service responds within milliseconds.

The booking is confirmed.

The passenger receives a notification.

Everyone is happy.

Now imagine the same flow in production.

The Payment Service becomes temporarily unavailable.

Maybe the service is being deployed.

Maybe the database is under heavy load.

Maybe there is a network issue between services.

Whatever the cause, the result is the same.

The Ride Service is waiting for a response that never arrives.

The ride booking fails.

Not because the Ride Service is broken.

Not because the passenger did anything wrong.

But because the system assumed every dependency would always be available.

That assumption is one of the fastest ways to create fragile systems.

Timeouts: Teaching Systems When to Stop Waiting

One of the simplest resilience patterns I use is the timeout.

A timeout defines how long a service is willing to wait before giving up on a request.

Without a timeout, a service can wait indefinitely for a dependency that may never respond.

This sounds harmless until dozens, hundreds, or thousands of requests begin piling up.

Eventually, the service runs out of resources and becomes unhealthy itself.

What started as a problem in one service has now spread to another.

I have seen incidents where a single slow dependency caused an entire system to become unresponsive simply because requests were allowed to wait forever.

A timeout acts as a boundary.

It allows a service to fail quickly and preserve resources rather than waiting endlessly for something outside its control.

A failed request is unfortunate.

A system-wide outage is much worse.

Retries: Giving Temporary Failures a Second Chance

Not every failure is permanent.

Sometimes a service is restarting.

Sometimes a network connection briefly drops.

Sometimes a dependency experiences a temporary spike in traffic.

These situations often resolve themselves within seconds.

This is where retries become useful.

Instead of immediately failing, the service attempts the request again.

At first glance, retries seem like an obvious improvement.

But I have learned that retries are dangerous when implemented carelessly.

Imagine a dependency that is already struggling under heavy load.

Now imagine thousands of services retrying requests simultaneously.

Instead of helping the dependency recover, the retries make the problem worse.

This is why I typically pair retries with exponential backoff.

Rather than retrying immediately, each attempt waits progressively longer before trying again.

The goal is not to overwhelm the failing service.

The goal is to give it room to recover.

Circuit Breakers: Preventing Failure from Spreading

One of my favorite resilience patterns is the circuit breaker.

The name comes from electrical systems.

When an electrical circuit experiences a fault, the breaker trips and temporarily stops the flow of electricity to prevent further damage.

The same idea applies to software.

Imagine a Payment Service that has been failing continuously for several minutes.

Without protection, every incoming ride request continues attempting payment verification.

The result is predictable.

Resources are wasted.

Latency increases.

More services become affected.

Instead, the circuit breaker detects repeated failures and temporarily stops sending requests.

When the circuit is open, requests fail immediately rather than waiting for an already unhealthy dependency.

After a configured period, the system can attempt a small number of test requests to determine whether the dependency has recovered.

If recovery is successful, the circuit closes and normal traffic resumes.

This pattern has saved countless systems from cascading failures.

Idempotency: Protecting Against Duplicate Operations

There is one resilience pattern that deserves special attention because it protects something more valuable than infrastructure.

It protects business correctness.

Consider a passenger making a payment for a ride.

The payment request reaches the Payment Service.

The payment succeeds.

Unfortunately, the response never reaches the client because of a network interruption.

The client assumes the request failed and sends it again.

Without protection, the passenger may be charged twice.

I have never met a customer who enjoys discovering duplicate charges.

This is where idempotency becomes essential.

An idempotent operation produces the same outcome regardless of how many times the same request is received.

Instead of processing the payment repeatedly, the service recognizes that it has already handled the request and returns the original result.

In distributed systems, duplicate requests are not unusual.

They are expected.

Retries, network interruptions, message redelivery, and client behavior can all produce duplicates.

Designing for idempotency ensures those duplicates do not become business problems.

Resilience Is Not About Preventing Failure

When engineers first hear about resilience patterns, they often assume the goal is to eliminate failures.

That is impossible.

No architecture, framework, cloud provider, or technology stack can guarantee that failures will never occur.

The purpose of resilience is not to prevent failure.

The purpose is to contain failure.

A resilient system recognizes that dependencies will occasionally become unavailable, requests will occasionally fail, and infrastructure will occasionally behave unpredictably.

Instead of collapsing under those conditions, it continues operating in a controlled and predictable way.

That shift in mindset changed how I design systems.

I stopped asking:

"How do I make sure this never fails?"

And started asking:

"What happens when this fails?"

The answers to that question usually reveal more about a system's production readiness than its architecture diagrams ever will.

Of course, surviving failure is only half the battle.

Even the most resilient system becomes difficult to operate if engineers cannot understand what is happening inside it.

And that brings us to one of the most overlooked aspects of modern software architecture: observability.

Observability: Understanding What Your System Is Actually Doing

The first time I worked on a system with multiple services, I thought logs would be enough.

I was wrong.

A user reported that a payment was successful, but the order was never confirmed. The Payment Service looked healthy. The Order Service looked healthy. The Notification Service looked healthy.

Yet somehow the workflow had failed.

I spent hours jumping between log files trying to piece together what happened.

That experience taught me something important:

Building distributed systems is one challenge.

Understanding them in production is another challenge entirely.

And that is where observability comes in.

Observability is not a monitoring tool.

It is not a dashboard.

It is not a logging library.

Observability is your ability to understand the internal state of a system by examining its outputs.

When a customer says:

"Something isn't working."

Observability helps you answer:

"What happened, where did it happen, and why did it happen?"

Without guesswork.

The Three Pillars of Observability

When I think about observability, I think about three things:

• Logs

• Metrics

• Traces

Each one answers a different question.

Most teams have logs.

Fewer teams have useful logs.

Even fewer teams have traces.

Let's fix that.

Structured Logging: Stop Writing Logs for Humans Only

One of the most common mistakes I see is unstructured logging.

Things like:

Payment failed

or

Error processing order

These logs seem helpful when you're developing locally.

They become almost useless when thousands of requests are flowing through multiple services.

Instead, I prefer structured logs.

A structured log contains context.

{
  "service": "payment-service",
  "orderId": "ord_123",
  "paymentId": "pay_456",
  "userId": "usr_789",
  "status": "failed",
  "reason": "insufficient_funds"
}

Now I can search by:

• orderId

• paymentId

• userId

• status

and immediately find relevant events.

In a NestJS application, I typically use Pino.

import { Logger } from 'nestjs-pino';

@Injectable()
export class PaymentService {
  constructor(
    private readonly logger: Logger,
  ) {}

  async processPayment(orderId: string) {
    this.logger.info({
      orderId,
      action: 'payment_processing_started',
    });

    // payment logic
  }
}

Notice what I'm logging.

Not sentences.

Context.

Machines can search context.

Humans can interpret context.

You need both.

Correlation IDs: Following a Request Across Services

Structured logs alone are not enough.

Imagine a request moving through four services:

Each service generates logs.

The challenge becomes identifying which logs belong to the same user request.

This is where correlation IDs become invaluable.

When a request enters the system, generate a unique identifier.

x-correlation-id:
7f4a1f1b-95a4-4ec8-85ab-bd8e87c43f76

Every service forward that value.

Now every log entry contains:

{
  "correlationId": "ABC123",
  "service": "payment-service",
  "event": "payment_completed"
}

When something goes wrong, I search for the correlation ID and instantly reconstruct the entire request journey.

This single practice has saved me more debugging time than almost any other observability technique.

Implementing Correlation IDs in NestJS

A simple middleware can generate and propagate correlation IDs.

import { v4 as uuid } from 'uuid';

@Injectable()
export class CorrelationIdMiddleware
implements NestMiddleware {

  use(
    req: Request,
    res: Response,
    next: NextFunction,
  ) {

    const correlationId =
      req.headers['x-correlation-id']
      || uuid();

    req['correlationId'] = correlationId;

    res.setHeader(
      'x-correlation-id',
      correlationId,
    );

    next();
  }
}

Every downstream service should preserve and forward the same value.

Think of it as a tracking number for a request.

Metrics: Detecting Problems Before Users Do

Logs tell me what happened.

Metrics tell me whether I should be worried.

For example:

A single failed payment isn't necessarily a problem.

But if payment failures suddenly increase from:

2% to 35%

within five minutes,

I want to know immediately.

Some of the metrics I monitor most often include:

• Request Rate

• Error Rate

• Response Time

• Queue Length

• Database Connection Count

• Memory Usage

A useful mental model is this:

Logs help with investigations.

Metrics help with detection.

Metrics answer:

Is something unusual happening?

before customers start opening support tickets.

Distributed Tracing: The Missing Piece

Let's revisit the earlier scenario.

A customer says:

"My payment succeeded but my order was never confirmed."

Logs can help.

Correlation IDs can help.

But traces provide something even better.

They show the complete request journey.

A trace doesn't just show the path.

It shows timing.

Example:

API Gateway       12ms
Order Service     24ms
Payment Service   980ms
Notification      15ms

Immediately, the bottleneck becomes obvious.

The Payment Service is responsible for most of the latency.

No guessing required.

This is why I consider distributed tracing one of the most valuable investments for any microservices architecture.

OpenTelemetry: My Preferred Starting Point

When teams ask me where to begin with tracing, my answer is almost always the same:

Start with OpenTelemetry.

OpenTelemetry provides a standard way to collect:

• Traces

• Metrics

• Logs

across services.

A minimal setup in NestJS might look like:

import { NodeSDK }
from '@opentelemetry/sdk-node';

const sdk = new NodeSDK();

sdk.start();

In a real system, traces are exported to tools such as:

• Jaeger

• Tempo

• Datadog

• New Relic

Once configured, every request begins producing trace data automatically.

The result is visibility that would be nearly impossible to achieve through logs alone.

What Observability Looks Like in Practice

A mature production environment often looks something like this:

Each tool serves a different purpose.

• OpenTelemetry collects telemetry data

• Jaeger visualizes traces

• Prometheus stores metrics

• Grafana provides dashboards

Together, they provide a complete picture of system behavior.

Not assumptions.

Not guesses.

Evidence.

The Real Goal

The goal of observability is not collecting more data.

I have seen teams collect enormous amounts of logs and metrics while still struggling to diagnose incidents.

The goal is understanding.

When a customer reports a problem, I want answers within minutes, not hours.

I want to know:

• Which service handled the request?

• How long did it take?

• Which dependency failed?

• What happened before the failure?

• What happened after the failure?

The most successful engineering teams I have worked with are not necessarily the teams that experience the fewest failures.

They are the teams that can quickly understand and recover from them.

And that ability begins with observability.

Now that we can see what's happening inside our system, the next challenge is operating microservices responsibly at scale.

Because building services that communicate, recover from failures, and expose useful telemetry is only part of the journey.

Eventually, we need to discuss what separates a collection of services from a production-ready microservices platform.

The Mistakes That Turn Microservices into Distributed Monoliths

One of the reasons I enjoy discussing microservices is because most conversations focus on success stories.

The diagrams are clean.

The services are neatly separated.

Everything appears elegant.

Production rarely looks like those diagrams.

Over the years, I have noticed that many teams do not fail because they chose microservices.

They fail because they unknowingly recreate the same coupling they were trying to escape.

They trade one monolith for a distributed monolith.

And the worst part is that they often do not realize it until the system becomes difficult to evolve.

Mistake #1: Splitting Services Without Clear Boundaries

The first sign of trouble usually appears during service decomposition.

A team takes a monolith and begins creating services:

• User Service

• Order Service

• Product Service

Good so far.

Then:

• Utility Service

• Common Service

• Shared Service

That is usually where I start asking questions.

A service should represent a business capability, not a collection of unrelated helper functions.

When service boundaries are unclear, ownership becomes unclear.

When ownership becomes unclear, complexity follows.

One question I often ask is:

"If this service disappeared tomorrow, what business capability would disappear with it?"

If nobody can answer confidently, the boundary probably needs more thought.

Mistake #2: Sharing the Same Database

One of the most common anti-patterns I see is this:

Technically, the services are separate.

Operationally, they are still coupled.

The database becomes the new monolith.

A schema change from one team can break another team's service.

Independent deployments become difficult.

Autonomy disappears.

Microservices should own their data.

That ownership is what enables true independence.

Mistake #3: Making Everything Synchronous

This usually starts innocently.

A service needs information from another service.

A REST call is added.

Then another.

Then another.

Eventually, a single request looks like this:

The architecture may look organized.

The latency certainly isn't.

One slow dependency can affect the entire chain.

I have seen systems where a single request required seven synchronous service calls before a response could be returned.

That is rarely sustainable.

Mistake #4: Ignoring Observability Until Production

This mistake usually reveals itself during the first serious incident.

A customer reports a problem.

Logs exist.

Metrics exist.

But nobody can explain what actually happened.

At that moment, observability stops feeling optional.

It becomes a necessity.

I strongly prefer introducing structured logging, metrics, and tracing before the first production deployment.

Retrofitting observability later is significantly harder.

Mistake #5: Choosing Microservices Too Early

This is probably my most controversial opinion.

Many systems should remain monoliths.

A well-structured monolith is not a failure.

It is often the simplest and most effective solution.

I would rather maintain a healthy monolith than operate ten unnecessary services.

Microservices introduce operational complexity.

The benefits must justify that complexity.

If they do not, the monolith is usually the better choice.

The Goal Was Never Microservices

If there is one thing I want every engineer reading this to understand, it is this:

Microservices were never the goal.

They were never the destination.

They were never the measure of good architecture.

They are simply one possible answer to a deeper problem.

Over the years, I have seen engineers obsess over architecture diagrams. I have seen teams celebrate the moment they “moved to microservices” as though it represents a level of engineering maturity.

But I have also seen what happens after the excitement fades.

Systems become harder to understand.

Deployments become more delicate.

Debugging becomes slower.

Incidents take longer to resolve.

And slowly, quietly, teams begin to realize that they have not eliminated complexity.

They have relocated it.

What Actually Matters in Production Systems

When I reflect on the systems, I consider truly well-designed, I notice a pattern.

It has very little to do with whether they are monoliths or microservices.

Instead, it comes down to a few practical questions:

Can the system evolve without breaking itself?

Can teams ship changes without fear?

Can failures be understood quickly?

Can issues be isolated instead of spreading?

Can the system recover gracefully when things go wrong?

Those are the real indicators of good architecture.

Not the number of services.

Not the sophistication of the stack.

Not the popularity of the tools.

A System Is Only as Strong as Its Weakest Assumption

One lesson I keep returning to is that most production failures are not caused by broken code.

They are caused by incorrect assumptions.

Assumptions like:

• A dependency will always be available

• A network call will always succeed

• A service will always respond within a reasonable time

• A message will only be processed once

• Data will always be consistent across services

Microservices amplify these assumptions because they force them to cross boundaries.

That is why we spent so much time talking about:

• Timeouts

• Retries

• Circuit breakers

• Idempotency

• Observability

These are not “advanced topics.”

They are survival mechanisms.

What I Tell Engineers Who Want to “Do Microservices Right”

When engineers ask me how to properly adopt microservices, I usually do not start with technology.

I start with questions.

Do you understand your domain well enough to split it safely?

Can your team operate multiple services independently?

Do you have visibility into what happens in production today?

Can you trace a single request across your system without guessing?

Can you recover quickly when something fails?

If the answer to most of these questions is no, then microservices will not solve the problem.

They will simply expose it.

Architecture Is a Reflection of Discipline

One thing I have come to believe strongly is that architecture does not create discipline.

It reveals it.

A well-disciplined team can build a stable monolith.

A well-disciplined team can operate microservices at scale.

An undisciplined team can struggle with both.

The difference is not the architecture.

The difference is the engineering mindset behind it.

The Real Evolution

If I step back and look at the journey we have walked through in this article, it is not really a story about monoliths or microservices.

It is a story about maturity.

We started with a simple system.

We introduced boundaries.

We handled communication.

We prepared for failure.

We added observability.

We discussed production realities.

And finally, we reflected on when not to adopt complexity at all.

That is the real evolution.

Not from monolith to microservices.

But from assumptions to understanding.

Final Thought

If I had to summarize everything in one idea, it would be this:

A great system is not one that avoids failure.

A great system is one that understands failure, contains it, observes it, and recovers from it quickly.

If microservices help you achieve that, use them.

If they do not, you are not obligated to use them.

Because in the end, the goal was never microservices.

The goal was always the same:

To build systems that survive production.

Being able to build a project fast with AI does not make you a senior engineer.

It just means you can move fast.

And speed, on its own, has never been the definition of experience.

Today, you can scaffold an app in a weekend.
Endpoints, database, auth, even tests.
AI will help you get there.

But real engineering does not show up when everything works.

It shows up when things start acting weird.

Engineering Is More Than Writing Code

I recently came across a post by Gergely Orosz where he shared a screenshot from a software architecture book.
The page breaks engineering into craft, production, commerce, and science.

That single diagram explains more about engineering maturity than most job titles ever will.

At the bottom is craft.
That’s where many of us start.
You build things. You ship. You enjoy the act of creation.

With AI, you can do this faster now. Much faster.

But production is different.
Production is when real users show up. When uptime matters. When mistakes cost money.
Commerce is different. That’s when the software has to justify its existence.
Science is different. That’s where deep understanding, patterns, and long-term thinking live.

Most people calling themselves “senior” are still operating in craft.
Speed hides that. AI amplifies it.

Then I noticed a reply under that post from Taha Hussein:

“Engineering isn’t just code.”

That line should slow you down.

Because AI can help you write code.
It cannot teach you what happens when a system starts failing quietly.
It cannot teach you how frontend decisions ripple into backend performance.
It cannot teach you how DevOps trade-offs show up as outages, cost explosions, or security gaps.

That kind of knowledge only comes from being there. From carrying responsibility. From making decisions when there is no clear answer.

AI makes you faster.
Experience makes you careful.

And engineering maturity lives in knowing the difference.

When things work, everyone feels senior

You ship something.
It works.
Then traffic grows.

Suddenly, responses are slow.
Some users complain. Others don’t.
Nothing is fully broken, but nothing feels right.

Now the question is not “can you build?”
It’s “do you know where to look?”

Do you know what part of the system is under pressure?
Do you know what usually fails first?
Do you know what not to touch while debugging?

AI will suggest fixes.
Experience tells you what not to do.

Scaling is not a tutorial problem

Let’s talk about scale.

Not the shiny kind.
The annoying kind.

The kind where you can’t change things freely anymore because people depend on it.

Have you ever scaled a system where a small change could affect thousands of users?

Have you done migrations knowing that rollback might not be possible?

Have you dealt with a situation where half the requests succeed and the other half fail, and nobody can explain why?

These moments don’t care how fast you shipped v1.

They care how well you understand the system.

Senior engineers think beyond their own code, they think system-wide

And this isn’t just backend.

If you change an API response, do you think about how the frontend will break?

Do you think about loading states, retries, weird edge cases users will actually see?

Do you think about what the user experiences when your “small backend change” adds two seconds to a request?

Senior engineers think across the system, not just their corner of it.

Production has a way of humbling everyone

Then there’s production.

Real production.

Have you been on-call before?

Have you woken up to alerts and realized logs are missing, metrics look fine, but users are angry?

Have you pushed a bad deploy and had to fix it while everyone is watching?

Have you felt that quiet pressure of knowing that real money and real trust are on the line?

AI doesn’t feel that.

You do.

What AI accelerates and what it cannot replace

Don’t get me wrong. AI is powerful.

It helps you learn faster.
It helps you explore ideas.
It helps you avoid boilerplate.

Use it.

But don’t lie to yourself about what it replaces.

AI does not replace judgment.
It does not replace responsibility.
It does not replace experience earned from mistakes.

A word for engineers moving fast with AI

If you’re an engineer using AI and shipping fast, that’s a good thing.

Build more.
Break things.
Ask questions.

Just understand this.

Seniority is not about how quickly you can build when things are calm.

It’s about how you think when things are not.

And no tool, no matter how smart, can skip that part for you.

…

If this resonated with you, share it with someone who’s building fast and trying to build right.

I’ll be writing more in this series about AI, engineering maturity, and leadership from real experience.
Follow me here and on LinkedIn to keep the conversation going.

Again, system design is not something to joke with as a backend engineer.
One of the subtle issues you will encounter when dealing with concurrent systems is Dirty Reads.

Let me explain in the simplest way possible.

A Glocal Analogy

Imagine you’re at a buka (local restaurant). You ask the waiter:

“How much is goat meat?”

He quickly checks with the cashier and says:

“₦1,200.”

You start calculating your budget in your head. But before you order, another customer buys goat meat in bulk, and the cashier immediately changes the price to ₦1,500.

You’re stuck; you just made a decision based on a price that wasn’t real anymore.

That’s exactly what a Dirty Read looks like in databases.

What Is a Dirty Read?

A Dirty Read happens when a transaction reads data that has been modified by another transaction but not yet committed.
If that other transaction is later rolled back, the first one has used data that never truly existed.

Think of it as:

• Transaction A updates data (but hasn’t saved it yet).

• Transaction B reads that uncommitted update.

• Transaction A cancels its changes.

• Now Transaction B has used wrong information.

Example in SQL

Here’s a simplified example using a bank account:

-- Transaction A
BEGIN;
UPDATE accounts SET balance = balance - 100 WHERE id = 1;
-- (not committed yet)

-- Transaction B
BEGIN;
SELECT balance FROM accounts WHERE id = 1;
-- Reads the deducted balance (dirty read)

-- Transaction A decides to roll back
ROLLBACK;

Transaction B has now read a balance that doesn’t actually exist.

Why Dirty Reads Are Dangerous

• Banking systems: Customers see money deducted that later reappears.

• Inventory systems: You think stock is reduced, but it’s not.

• Analytics dashboards: Wrong numbers get logged, leading to bad decisions.

In short: your users are interacting with lies.

A Real-World Anecdote from My Work
When I was building a currency exchange platform, we faced this exact problem.

Some users would see their wallet balance deducted after initiating a payout. But if the payout failed due to a downstream issue (like a payment provider timing out), the transaction was rolled back. The balance was restored in the database, but the user had already seen the deducted balance during that short window.

From the system’s point of view, nothing was “lost.” But from the user’s perspective, money had mysteriously disappeared and reappeared. That moment of confusion was enough to cause panic and support tickets.

Transaction B has trusted false data.

How to Fix Dirty Reads

To avoid dirty reads, you need to control how transactions isolate themselves from one another.

1. Use READ COMMITTED isolation level

   • Most databases (Postgres, SQL Server, Oracle) default to this.

   • Ensures only committed changes can be read.

   • In the story I shared earlier, this adjustment stopped users from seeing balances before they were real.

2. Explicit locking

    SELECT balance 
    FROM accounts 
    WHERE id = 1 
    FOR UPDATE;
   

   • Forces other transactions to wait until the first one is done.

3. Trade-offs

   • Stronger isolation = more accuracy.

   • But it can also reduce performance if not managed well.

Wrap-up

Just like you wouldn’t want to budget your buka meal on a wrong price, your systems shouldn’t run on uncommitted data.

Dirty Reads may sound harmless at first, but in production systems (banking, ecommerce, analytics), they can quietly destroy trust and correctness.

As a backend engineer, always check your database’s isolation level and ensure you’re not letting your users interact with lies.

Stay Updated & Connected

I’m writing a series that breaks down common concurrency problems (Race Condition, Dirty Reads, Non-repeatable Reads, Phantom Reads, Deadlocks, and more) in simple language with relatable glocal examples.

• Follow me here for new articles in the series.

• Connect with me on LinkedIn if you love conversations around backend engineering, system design, and real-world problem solving.

• Share this with another engineer who might be struggling with concurrency issues.

Together, let’s make backend engineering less confusing and more practical 🚀.

Have you ever encountered a bug caused by dirty reads without realizing it? I’d love to hear your story in the comments.

But let’s be real, notifications don’t come in just one flavor. You might send:

• Email for official stuff like password resets

• Push alerts to keep users active on mobile

• In-App messages for things users should always see inside the app

If we tried to handle all these directly in one service or processor, the code would quickly become spaghetti. This is where the Strategy Pattern comes in.

Why Strategy?

Think of a strategy as a plug. Each channel (Email, Push, In-App) has its own plug, and the main notification service doesn’t care which one you use. It just picks the right plug and lets it do the work.

This makes it easy to:

• Add new channels later without breaking existing ones

• Swap providers (e.g., Resend → SMTP for email) without rewriting logic

• Keep the code neat and maintainable

Avoiding If/Else Hell

Before Strategy, your code would look something like this:

// notification.processor.ts (bad way)
async process(job: Job<any>): Promise<void> {
  const { channel, payload } = job.data;

  if (channel === 'email') {
    await this.emailProvider.send(payload);
  } else if (channel === 'push') {
    await this.pushProvider.send(payload);
  } else if (channel === 'in-app') {
    await this.inAppRepository.save(payload);
  } else {
    throw new Error(`Unsupported channel: ${channel}`);
  }
}

Problems with this approach:

• Every new channel = edit this file again.

• Swapping a provider = same problem.

• File grows into a giant mess.

• Hard to test and maintain.

Now compare with the Strategy-based approach:

// notification.processor.ts (good way)
async process(job: Job<INotificationJobPayload>): Promise<void> {
  const { channel, payload } = job.data;
  const strategy = this.registry.get(channel);

  if (!strategy) {
    throw new UnsupportedChannelException(channel);
  }

  await strategy.send(payload);
}

Benefits:

• Adding new channels = just add a new strategy class.

• Swapping providers = only touch that strategy.

• Processor stays clean and future-proof.

• Easy to test strategies in isolation.

🔑 Takeaway: The Strategy Pattern prevents “switch/if/else hell” and gives you proper separation of concerns.

Tools Used

Just like in Part 1, here’s the stack behind this part:

• NestJS – framework for structure

• BullMQ – job queue for async processing

• Redis – job storage and retries

• Strategy Pattern – cleanly separating notification channels

Setting Up the Channel Strategies

First, let’s define an enum for our notification channels:

// enums/notification-channel.enum.ts
export enum NotificationChannel {
  EMAIL = 'email',
  PUSH = 'push',
  IN_APP = 'in-app',
  SMS = 'sms',
  WHATSAPP = 'whatsapp'
}

Now, define a simple interface that all channel strategies will follow:

// interfaces/notification-strategy.interface.ts
export interface INotificationStrategy<TPayload = any> {
  send(payload: TPayload): Promise<void>;
}

Define sample interfaces for each of the channels payload:

// strategies/email/interfaces/email-payload.interface.ts
export interface IEmailPayload {
  to: string;
  from?: string;
  subject: string;
  templateName?: string;
  body?: string;
  replacements?: Record<string, any>;
}

// strategies/in-app/interfaces/in-app-payload.interface.ts
export interface IInAppPayload {
  recipientId: string;
  title: string;
  message: string;
  metadata?: Record<string, any>;
}

// strategies/push/interfaces/push-payload.interface.ts
export interface IPushPayload {
  recipientTokens: string[];
  recipientId: string;
  title: string;
  message: string;
  data?: Record<string, string>;
}

Then create separate strategies. Example:

// strategies/email/email.strategy.ts
import { Injectable, InternalServerErrorException, Logger } from '@nestjs/common';
import { INotificationStrategy } from '../../interfaces/notification-strategy.interface';
import { IEmailPayload } from './interfaces/email-payload.interface';

@Injectable()
export class EmailNotificationStrategy
  implements INotificationStrategy<IEmailPayload>
{
  private readonly logger = new Logger(EmailNotificationStrategy.name);

  async send(payload: IEmailPayload): Promise<void> {
    // logic for sending email (Resend, SMTP, etc.)
    this.logger.log(`Sending EMAIL to ${payload.to}: ${payload.subject}`);
  }
}

// strategies/push/push.strategy.ts
import { Injectable, InternalServerErrorException, Logger } from '@nestjs/common';
import { INotificationStrategy } from '../../interfaces/notification-strategy.interface';
import { IPushPayload } from './interfaces/push-payload.interface';

@Injectable()
export class PushNotificationStrategy
  implements INotificationStrategy<IPushPayload>
{
  private readonly logger = new Logger(PushNotificationStrategy.name);

  async send(payload: IPushPayload) {
    // logic for FCM or OneSignal
    this.logger.log(`Sending PUSH to ${payload.recipientTokens.length} devices: ${payload.title}`);
  }
}

// strategies/in-app/in-app.strategy.ts
import { Injectable, InternalServerErrorException, Logger } from '@nestjs/common';
import { INotificationStrategy } from '../../interfaces/notification-strategy.interface';
import { IInAppPayload } from './interfaces/in-app-payload.interface';

@Injectable()
export class InAppNotificationStrategy
  implements INotificationStrategy<IInAppPayload>
{
  private readonly logger = new Logger(InAppNotificationStrategy.name);

  async send(payload: IInAppPayload) {
    // save in DB as in-app notification
    this.logger.log(`Saving IN-APP notification for ${payload.recipientId}`);
  }
}

Strategy Registry

We need a place to register and retrieve the right strategy based on the channel:

// notification-strategy-registry.ts
import { Injectable } from '@nestjs/common';
import { NotificationChannel } from '../enums/notification-channel.enum';
import { INotificationStrategy } from './interfaces/notification-strategy.interface';
import { EmailNotificationStrategy } from './strategies/email/email.strategy';
import { PushNotificationStrategy } from './strategies/push/push.strategy';
import { InAppNotificationStrategy } from './strategies/in-app/in-app.strategy';

@Injectable()
export class NotificationStrategyRegistry {
  private readonly strategies = new Map<NotificationChannel, INotificationStrategy>();

  constructor(
    email: EmailNotificationStrategy,
    push: PushNotificationStrategy,
    inApp: InAppNotificationStrategy,
    sms: SmsNotificationStrategy,
  ) {
    this.strategies.set(NotificationChannel.EMAIL, email);
    this.strategies.set(NotificationChannel.PUSH, push);
    this.strategies.set(NotificationChannel.IN_APP, inApp);
  }

  get(channel: NotificationChannel): INotificationStrategy | undefined {
    return this.strategies.get(channel);
  }
}

Putting It Together in the Worker/Processor

Now the NotificationProcessor can use the registry to dispatch without caring about the internals. Before that, let’s define an interface that unifies the payloads of all the channels.

// interfaces/notification-job.interface.ts
import { NotificationChannel } from 'src/modules/notification/enums/notification-channel.enum';
import { IEmailPayload } from '../strategies/email/interfaces/email-payload.interface';
import { IPushPayload } from '../strategies/push/interfaces/push-payload.interface';
import { IInAppPayload } from '../strategies/in-app/interfaces/in-app-payload.interface';
import { ISmsPayload } from '../strategies/sms/interfaces/sms-payload.interface';

export type INotificationJobPayload =
  | { channel: NotificationChannel.EMAIL; payload: IEmailPayload }
  | { channel: NotificationChannel.PUSH; payload: IPushPayload> }
  | { channel: NotificationChannel.IN_APP; payload: IInAppPayload }
  | { channel: NotificationChannel.SMS; payload: ISmsPayload };

Now we can use that in the NotificationProcessor :

// notification.processor.ts
import { OnWorkerEvent, Processor, WorkerHost } from '@nestjs/bullmq';
import { Logger } from '@nestjs/common';
import { Job } from 'bullmq';

@Processor('notification-queue')
export class NotificationProcessor extends WorkerHost {
  private readonly logger = new Logger(NotificationProcessor.name);

  constructor(private readonly registry: NotificationStrategyRegistry) {
    super();
  }

  @OnWorkerEvent('active')
  onActive(job: Job<INotificationJobPayload>) {
    this.logger.log(`Processing job ${job.name} with id ${job.id}`);
  }

  async process(job: Job<INotificationJobPayload>): Promise<void> {
    const { payload, channel } = job.data;
    const strategy = this.registry.get(channel);

    if (!strategy) {
      throw new BadRequestException(
        `Unsupported notification channel ${channel}`,
      );
    }

    try {
      await strategy.send(payload);
    } catch (err) {
      this.logger.error(`Failed to send via ${channel}: ${err.message}`);
    }
  }
}

Expose Interactable API in the Service

Now in the service layer, we can expose methods that enqueues the notification as a job that a callable from the event listeners.

import { Injectable } from '@nestjs/common';
import { InjectQueue } from '@nestjs/bullmq';
import { Queue } from 'bullmq';
import { NotificationChannel } from 'src/modules/notification/enums/notification-channel.enum';
import { IEmailPayload } from './strategies/email/interfaces/email-payload.interface';
import { IPushPayload } from './strategies/push/interfaces/push-payload.interface';
import { IInAppPayload } from './strategies/in-app/interfaces/in-app-payload.interface';
import { INotificationJobPayload } from './interfaces/notification-job.interface';

@Injectable()
export class NotificationService {
  constructor(
    @InjectQueue('notification-queue') private notificationQueue: Queue,
    private readonly userDeviceService: UserDeviceService,
  ) {}

  async notifyEmail(payload: IEmailPayload) {
    await this.enqueue({ channel: NotificationChannel.EMAIL, payload });
  }

  async notifyPush(payload: IPushPayload) {
    await this.enqueue({ channel: NotificationChannel.PUSH, payload });
  }

  async notifyInApp(payload: IInAppPayload) {
    await this.enqueue({ channel: NotificationChannel.IN_APP, payload });
  }

  private async enqueue(jobPayload: INotificationJobPayload) {
    await this.notificationQueue.add('dispatch', jobPayload, {
      attempts: 3,
      backoff: { type: 'exponential', delay: 5000 },
      removeOnComplete: true,
      removeOnFail: false,
    });
  }
}

The event listeners can then easily call:

await this.notificationService.notifyEmail({
  to: payload.email,
  subject: 'Welcome to our platform',
  templateName: 'welcome',
  replacements: {
    name: payload.firstName
  },
});

await this.notificationService.notifyPush({
  recipientId: user.id,
  title: 'Wallet Credited',
  message: `${formattedAmount} credited. Balance: ${formattedNewBalance}.`,
});

We can as well create a sugar method notifyAll, that uses all the notification channels concurrently. Before we define that, let’s define what the payload for that will look like. This will keep our code strongly typed and easy to extend later.

// intefaces/notification-payloads.ts
import { NotificationChannel } from 'src/modules/notification/enums/notification-channel.enum';
import { IEmailPayload } from '../strategies/email/interfaces/email-payload.interface';
import { IPushPayload } from '../strategies/push/interfaces/push-payload.interface';
import { IInAppPayload } from '../strategies/in-app/interfaces/in-app-payload.interface';
import { ISmsPayload } from '../strategies/sms/interfaces/sms-payload.interface';

export type NotificationPayloads = {
  [NotificationChannel.EMAIL]: IEmailPayload;
  [NotificationChannel.PUSH]: IPushPayload>;
  [NotificationChannel.IN_APP]: IInAppPayload;
  [NotificationChannel.SMS]: ISmsPayload;
};

Finally update the NotificationService with notifyAll sugar.

import { Injectable } from '@nestjs/common';
import { InjectQueue } from '@nestjs/bullmq';
import { Queue } from 'bullmq';
import { NotificationChannel } from 'src/modules/notification/enums/notification-channel.enum';
import { IEmailPayload } from './strategies/email/interfaces/email-payload.interface';
import { IPushPayload } from './strategies/push/interfaces/push-payload.interface';
import { IInAppPayload } from './strategies/in-app/interfaces/in-app-payload.interface';
import { INotificationJobPayload } from './interfaces/notification-job.interface';

@Injectable()
export class NotificationService {
  constructor(
    @InjectQueue('notification-queue') private notificationQueue: Queue,
    private readonly userDeviceService: UserDeviceService,
  ) {}

  async notifyAll(payloads: Partial<NotificationPayloads>) {
    const jobs: Promise<void>[] = [];

    if (payloads.email)
      jobs.push(this.notifyEmail(payloads[NotificationChannel.EMAIL]));
    if (payloads.push)
      jobs.push(this.notifyPush(payloads[NotificationChannel.PUSH]));
    if (payloads['in-app'])
      jobs.push(this.notifyInApp(payloads[NotificationChannel.IN_APP]));

    await Promise.all(jobs);
  }

  // ... existing methods
}

Then, you can call it in the event listeners like:

await this.notificationService.notifyAll({
  [NotificationChannel.EMAIL]: {
    to: user.email,
    subject: 'Wallet Debited',
    templateName: 'wallet-debited',
    replacements: {
      name: user.firstName,
      transaction_date: payload.createdAt.toLocaleDateString(),
      new_balance: formattedNewBalance,
      amount: formattedAmount,
      transaction_reference: payload.reference,          },
  },
  [NotificationChannel.PUSH]: {
    recipientId: user.id,
    title: 'Wallet Debited',
    message: `${formattedAmount} debited. Balance: ${formattedNewBalance}.`,
  },
  [NotificationChannel.IN_APP]: {
    recipientId: user.id,
    title: 'Wallet Debited',
    message: `${formattedAmount} debited. Balance: ${formattedNewBalance}.`,
    metadata: { transactionId: transaction.id },
  },
});

A user’s wallet is credited.
They should instantly get:

• An Email receipt with transaction details.

• A Push Notification on their phone.

• An In-App message when they open the app.

Instead of cluttering your code with if/else or switch, each channel strategy handles its own job, keeping your processor clean and extensible.

What’s Next?

We’ve now decoupled event → queue → worker → channel strategy.

But channels (Email, Push, In-App) still need actual providers:

• Email → Resend or SMTP

• Push → FCM or OneSignal

• SMS → Twilio

In Part 3, we’ll use the Bridge Pattern to connect channels to their providers, so you can switch providers without touching your business logic.

Stay Updated & Connected

If you enjoyed this article and want to follow along as I continue the Notification System series and other backend deep-dives, let’s connect:

• LinkedIn — I share backend and system design insights weekly

• GitHub — Code samples, side projects, and experiments

• X/Twitter — Short takes and tech thoughts

In this series, I’ll Walk you through how I built a flexible, event-driven notification system in NestJS. The system supports multiple channels (Email, Push, and In-App), can plug into different providers (like Resend or SMTP for email, Firebase Cloud Messaging or OneSignal for push), and is built to scale.

Let’s start with the core idea:

Why Event-Driven?

Notifications are essential but building them the wrong way can make your code messy fast. Imagine creating a user account and waiting seconds because the system is busy sending a welcome email. That’s a bad experience.

Let’s see that in action.

The naïve approach

Imagine this user registration flow:

async registerUser(dto: RegisterUserDto) {
  const user = await this.userRepo.save(dto);

  // Naïve: directly send an email here
  await this.emailService.sendWelcome(user.email);

  return user;
}

Looks simple, but now:

• Your UserService is tied to email logic.

• If you later add push notifications or in-app alerts, you’ll clutter this service with multiple channels.

• Failures in email sending can break user registration.

Instead, I used event-driven architecture. With this approach:

• The main app only emits an event (like user.registered).

• A notification service picks up the event and does the heavy lifting in the background.

• Users get their notifications without slowing down the main app.

This makes the system fast, clean, and decoupled.

Tools I Used

• NestJS – The framework tying everything together.

• EventEmitter – To fire events inside the app.

• BullMQ – For background job processing and retries.

Setting Up the Core

Here’s the minimal setup I used to get the event-driven flow working:

1. Install Dependencies

We need the event system (@nestjs/event-emitter), job queue (bullmq), and Redis driver (ioredis) to enable event-driven notifications.

npm install @nestjs/event-emitter bullmq ioredis

2. Enable EventEmitter

This turns on the event system in NestJS, so we can emit and listen to events across the app.

// app.module.ts
import { Module } from '@nestjs/common';
import { EventEmitterModule } from '@nestjs/event-emitter';
import { NotificationModule } from './notification/notification.module';

@Module({
  imports: [
    EventEmitterModule.forRoot(), // enables event system
    NotificationModule,
  ],
})
export class AppModule {}

3. Setup BullMQ Queue

Here we register a queue called notification-queue. This is where notification jobs will be added and processed asynchronously

// notification.module.ts
import { Module } from '@nestjs/common';
import { BullModule } from '@nestjs/bullmq';
import { NotificationListener } from './notification.listener';
import { NotificationProcessor } from './notification.processor';

@Module({
  imports: [
    BullModule.registerQueue({
      name: 'notification-queue',
    }),
  ],
  providers: [NotificationListener, NotificationProcessor],
})
export class NotificationModule {}

4. Emit an Event

When a user registers, we emit an event (user.registered) instead of sending the email directly. This keeps the main flow fast and clean.

// user.service.ts
import { Injectable } from '@nestjs/common';
import { EventEmitter2 } from '@nestjs/event-emitter';

@Injectable()
export class UserService {
  constructor(private eventEmitter: EventEmitter2) {}

  async registerUser(dto: any) {
    const user = await this.createUser(dto);

    // fire event
    this.eventEmitter.emit('user.registered', { userId: user.id });
    return user;
  }

  private async createUser(dto: any) {
    // fake user creation
    return { id: '123', ...dto };
  }
}

5. Event Listener (captures event)

Our listener picks up the user.registered event and hands it off to the notification service, which will queue the job.

// notification.listener.ts
import { OnEvent } from '@nestjs/event-emitter';
import { InjectQueue } from '@nestjs/bullmq';
import { Queue } from 'bullmq';
import { Injectable } from '@nestjs/common';

@Injectable()
export class NotificationListener {
  constructor(private readonly notificationService: NotificationService) {}

  @OnEvent('user.registered')
  async handleUserRegistered(payload: User) {
    await this.notificationService.notifyEmail({
      to: payload.email,
      subject: 'Welcome to our platform',
      templateName: 'welcome',
      replacements: {
        name: payload.firstName
      },
    });
  }
}

6. Service (adds job to queue)

We define what an email payload looks like, then enqueue it with BullMQ. Retry and backoff options ensure reliability.

// email-payload.interface.ts
export interface IEmailPayload {
  to: string;
  from?: string;
  subject: string;
  templateName?: string;
  body?: string;
  replacements?: Record<string, any>;
}


// notification.service.ts
import { Injectable } from '@nestjs/common';
import { InjectQueue } from '@nestjs/bullmq';
import { Queue } from 'bullmq';

@Injectable()
export class NotificationService {
  constructor(
    @InjectQueue('notification-queue') private notificationQueue: Queue,
  ) {}

  async notifyEmail(payload: IEmailPayload) {
    await this.notificationQueue.add('dispatch', jobPayload, {
      attempts: 3,
      backoff: { type: 'exponential', delay: 5000 },
      removeOnComplete: true,
      removeOnFail: false,
    });;
  }
}

7. Process Jobs

The processor actually does the sending. In this foundation, we just log, but later we’ll plug in real providers (SMTP, Resend, FCM, OneSignal, etc.).

// notification.processor.ts
import { OnWorkerEvent, Processor, WorkerHost } from '@nestjs/bullmq';
import { Logger } from '@nestjs/common';
import { Job } from 'bullmq';

@Processor('notification-queue')
export class NotificationProcessor extends WorkerHost {
  private readonly logger = new Logger(NotificationProcessor.name);

  constructor() {
    super();
  }

  @OnWorkerEvent('active')
  onActive(job: Job<IEmailPayload>) {
    this.logger.log(`Processing job ${job.name} with id ${job.id}`);
  }

  async process(job: Job<IEmailPayload>): Promise<void> {
      this.logger.log(`Sending email to user ${job.data.to}`);
  }
}

So, what just happened?

1. Zayd registers with zayd@email.com.

2. UserService saves him and emits user.registered.

3. Notification module listens for that event.

4. The listener, NotificationListener passes it to a service that adds a job to BullMQ notification-queue.

5. The job worker, NotificationProcessor picks it up and logs:

📧 Sending welcome email to user zayd@email.com

6. Zayd gets his notification, and your business logic stays clean. 🎉

Wrap up

That’s the foundation. We now have the skeleton of an event-driven notification system:

• Clean separation of concerns

• Async and fault-tolerant with queues

• Ready for multiple channels

In the next part, I’ll explain how I used the strategy pattern to handle different notification channels (Email, Push, In-App) without making a mess in the codebase.

Stay Updated & Connected

If you enjoyed this article and want to follow along as I continue the Notification System series and other backend deep-dives, let’s connect:

• LinkedIn — I share backend and system design insights weekly

• GitHub — Code samples, side projects, and experiments

• X/Twitter — Short takes and tech thoughts

No? 👉 Check out this post first HTTP beyond the acronym and come back here.

Now, what about the extra S in HTTPS?

Most people just say, “It stands for Secure,” and move on.
But let’s really understand what makes it secure.

Let’s go back to our favorite analogy:

Remember HTTP? That’s your trusted amala (a Nigerian delicacy) delivery guy. He moves between you (the client) and your favorite buka (a local food spot or street restaurant) — basically, your server.

Now here's where it gets interesting.

With HTTP, your guy walks there casually, says your order out loud, picks it up, and walks back.

It works fine, but here’s the risk:

👀 Anyone watching can:

• Hear your order

• Tamper with it

• Or pretend to be you and make a fake request

Now picture HTTPS.

You’re still using the same delivery guy.
But this time, he’s dressed smartly, carrying a secure briefcase, and your message is locked inside.

Only the buka has the right key to open it.

Even if someone intercepts him on the way, all they see is scrambled nonsense like:
“7dfj23&^%$#kld==” instead of “GET me amala and abula.”

Your request stays protected from the moment it leaves you to the moment it reaches the buka. The same protection applies to the response coming back.

That’s the big difference:

✅ HTTP sends plain text that’s easy to read and steal
✅ HTTPS scrambles the message so only the right server can understand

This is why HTTPS is very important for:
🔐 Login pages
💳 Online payments
🧾 Any sensitive data transfer

It’s not just an S.
It’s your shield. It’s your padlock. It’s what keeps your information safe.

So next time someone asks:
“What’s the difference between HTTP and HTTPS?”

Tell them:
“One delivers openly. The other protects your message like a top-secret mission.”

See you in the next!

After you mention it's a protocol (HyperText Transfer Protocol) used to transfer data over the web...

Please, don’t stop there. That’s just the abbreviation.

What you really want to explain is how HTTP works and why it matters in real-world applications.

Now, use this simple analogy to understand that dude once and for all:

Think of HTTP like sending someone to get amala from your favorite buka.

You (the client) are hungry.
The buka (a local food spot or street restaurant) which is regarded as the server has what you want.
But you’re not going yourself, you’re sending a trusted guy (HTTP) to go there, collect it, and bring it back.

Here’s how it plays out:

• You tell him what you want: “GET me amala, abula (a Nigerian soup) and ogunfe (goat meat) 🤤.”

• Or you say: “POST this new meal idea — mixed rice and beans with palm oil stew — into their suggestion box.”

• You could even say: “PUT this correction in my usual order.” or “DELETE my order entirely.”

When he gets back, he gives you a report:

• 200 — All went well. Here’s your food.

• 404 — Buka no get amala today.

• 500 — Kitchen catch fire. Everything scatter.

He also gives you some notes (headers):

• “The buka says next time come early o.”

• “They served it hot 🔥, and they added ponmo (cow skin) for free.”

And the actual food itself? That’s the body of the response. That’s what you really needed.

But here’s the funny thing, the guy forgets you after every trip 😭.
Unless you hand him a token or cookie like “This is from me, Usman. They know me there.”

That’s HTTP: stateless, structured, fast — and when upgraded (HTTP/2, HTTP/3), he now moves faster, delivers multiple orders at once, and dodges Lagos traffic like a pro 😁.

So the next time someone asks you, “What is HTTP?”
Smile.
And tell them: “He’s my trusted delivery guy, carrying my web messages back and forth. Clean, clear, and fast.”

At least I can't be in the interview room with you...

But if this post helps you walk in more confidently, that’s good enough for me.

Check out the next part for HTTPS
See you in the Next!

The problem? The database wasn’t designed to handle real-time tracking from thousands of concurrent users. Instead of celebrating their success, the team scrambles for days, migrating to a better-suited system.

This scenario is more common than you might think. Choosing the right database isn’t just a technical decision—it directly impacts performance, scalability, and even your team's stress levels.

In this article, we'll explore the major database types in a way that's easy to understand. With real-world examples, you'll gain a clear framework for making database decisions that scale with your need

The Database Family Tree

Before diving into details, let's get a bird's-eye view of our options:

• Relational databases: The traditional organized filing cabinets (MySQL, PostgreSQL)

• Non-relational (NoSQL) databases: The flexible specialists, including:

  • Document stores (MongoDB, Couchbase)

  • Key-value stores (DynamoDB, Riak)

  • Wide-column stores (Cassandra, HBase)

  • Graph databases (Neo4j, Amazon Neptune)

• In-memory databases: The speed demons (Redis, Memcached)

Each family exists because it solves specific problems better than the others. None is universally "best"—it's all about matching the right tool to your particular needs.

Relational Databases: The Reliable Workhorses

What They Are

Imagine a collection of spreadsheets (tables) with strict relationships between them. Customer information in one table connects to orders in another via customer IDs. Every piece of data has its designated place, and there are rules to keep everything consistent. Think of it like a library where every book has exactly one correct shelf location, with a detailed catalog system ensuring you can find anything quickly

When They Shine

Relational databases excel when your data is structured, relationships are clear, and data integrity is non-negotiable.

Real-world example: Shopify powers millions of e-commerce stores using MySQL. When a customer places an order, multiple tables update simultaneously: inventory decreases, order history updates, payment processes, and customer information links to the new order. All of these operations must succeed or fail together—something relational databases handle beautifully through transactions.

Another example: Bank of America uses Oracle to process financial transactions. When you transfer money, the bank must guarantee that the amount is deducted from one account and added to another—no exceptions. Relational databases provide ACID guarantees (Atomicity, Consistency, Isolation, Durability) that make this reliability possible.

When They Struggle

Despite their strengths, relational databases face challenges when:

• Data doesn’t fit neatly into tables

• Schema changes frequently

• Horizontal scaling is needed across many servers

• Extremely high write speeds are required

For example: Facebook originally used MySQL for everything but found it couldn't handle the complex, interconnected nature of a social graph at scale. The company eventually developed custom solutions, including graph databases for relationship data, because relational systems struggled with queries like, "Show all posts from friends of friends who live in Lagos Nigeria and like biking."

Popular Options

• PostgreSQL: Open-source, feature-rich, excellent for complex queries

• MySQL: Fast, reliable, and widely supported

• Oracle: Enterprise-grade with advanced features (and a price tag to match)

• Microsoft SQL Server: Deep Windows/Microsoft ecosystem integration

Non-Relational (NoSQL) Databases: The Flexible Specialists

NoSQL databases emerged to solve problems where relational databases struggled. Rather than offering one-size-fits-all solutions, they provide specialized tools for specific data patterns.

Document Stores

What They Are: Collections of JSON-like documents where each document contains all relevant information about an entity.

Real-world example: The New York Times uses MongoDB to store articles and related content. Each article document contains the text, author info, related images, tags, and comments—everything needed to render that article. This approach makes it incredibly fast to retrieve and display complete articles without joining multiple tables.

When They Shine: Content management, catalogs, user profiles, and any scenario where data items make natural "documents."

Key-Value Stores

What They Are: Simple databases that store values associated with keys, similar to a giant dictionary.

Real-world example: Lyft uses Amazon DynamoDB to track driver locations in real-time. With millions of location updates per minute, they need blazing-fast writes and simple lookups by driver ID. The key-value model perfectly matches this pattern.

When They Shine: Session storage, user preferences, shopping carts, and scenarios requiring simple, high-speed lookups.

Wide-Column Stores

What They Are: Databases that store data in columns rather than rows, allowing for massive scalability across distributed systems.

Real-world example: Netflix uses Apache Cassandra to store and process viewer activity data. With 200+ million subscribers generating viewing events constantly, they need something that scales horizontally across hundreds of servers while handling enormous write loads. Cassandra’s column-oriented design makes it possible to add more servers as demand grows.

When They Shine: Time-series data, event logging, and high-volume write scenarios requiring massive scale.

Graph Databases

What They Are: Specialized databases that excel at representing and querying complex relationships.

Real-world example: LinkedIn's connection network runs on graph technology. When you see "3rd-degree connection" or receive suggestions for "people you may know," that’s a graph database efficiently traversing relationship paths that would require expensive joins in a relational database.

When They Shine: Social networks, recommendation engines, fraud detection, and knowledge graphs.

In-Memory Databases: The Speed Demons

What Makes Them Special:
Most databases primarily store data on disk, reading it into memory as needed. In-memory databases flip this model by keeping everything in RAM, making them dramatically faster—often by 100x or more.

Real-World Examples:

• Twitter/X uses Redis to cache user timelines and handle millions of requests per second with sub-millisecond latency. When you refresh your timeline, you're likely seeing data served from Redis rather than hitting their main databases.

• Airbnb uses Redis to power their search autocomplete. When you start typing "New Yo..." and instantly see "New York" suggestions, that's Redis responding in real-time.

When They Shine
In-memory databases excel when:

• Speed is critical (e.g., trading platforms, gaming leaderboards)

• Data access patterns are predictable

• You need simple operations at massive scale

• Caching frequently accessed data would improve performance

When They Struggle
The tradeoffs come with:

• Limited capacity (RAM is expensive compared to disk storage)

• Persistence challenges (though many now offer persistence options)

• Data recovery after crashes (some in-memory data may be lost)

Making the Decision: A Practical Framework

When choosing a database, ask yourself these questions:

• What's your data structure? Tabular with clear relationships → Relational; Nested or variable structure → Document; Simple values → Key-value; Complex relationships → Graph

• What are your consistency needs? Strong consistency for financial or critical data → Relational; Eventual consistency acceptable → Many NoSQL options

• What's your scale? Will you need to scale horizontally across many machines? Many NoSQL databases make this easier.

• What are your query patterns? Complex joins and aggregations → Relational; Simple lookups by ID → Key-value; Relationship traversal → Graph

Hybrid Approaches

Most mature companies use multiple database types together:

Uber combines PostgreSQL for financial data and MySQL for transaction records, with Redis for real-time data like driver locations, and Cassandra for trip data requiring massive scale.
Airbnb uses MySQL for core booking data, Elasticsearch for searching listings, Redis for caching and real-time features, and Apache Druid for analytics.

Case Study: Evolution of Database Architecture

Let’s follow the journey of an imaginary fitness app as it scales:

Stage 1: MVP (1,000 users)
A single PostgreSQL database handles everything: user accounts, workout data, social features
Simple architecture gets the product launched quickly

Stage 2: Growth (50,000 users)
Added Redis for caching frequently accessed data
Still using PostgreSQL as the system of record
Response times improve, but database load during peak hours is concerning

Stage 3: Scaling (500,000 users)
Workout time-series data moved to MongoDB for better handling of variable workout structures
User graph (followers, friends) moved to Neo4j for better performance on social features
PostgreSQL still handles accounts, payments, and subscriptions
Redis expanded for real-time features like "users working out now"

Stage 4: Enterprise (5+ million users)
Time-series workout data migrated from MongoDB to Cassandra for better horizontal scaling
Redis Cluster implemented for distributed caching
PostgreSQL sharded by user region

Conclusion: The Right Tool for the Right Job

Choosing the right database isn’t about finding a perfect solution but understanding the strengths of each option. Relational databases are great for consistency, NoSQL excels with flexible, scalable data, and in-memory databases shine when speed is needed.

The fitness app team learned this the hard way. What began as a simple database soon became a bottleneck as the app grew. Reflecting on their experience, they realized they wouldn’t have faced that painful migration if they had understood their options from the start.

Six months later, their app supported 100,000 users with a hybrid system—PostgreSQL for user data, Redis for real-time features, and a time-series database for workout metrics. Their journey reinforced that there’s no one-size-fits-all database.

Now, with this knowledge, you can make informed choices based on your needs and growth. Start simple, but design with flexibility to scale. The right database lets you focus on building great features, not firefighting performance issues.

While growing as a backend engineer, I faced similar challenges. For example, generating large reports or pulling detailed account statements often led to timeouts. Why? Because I was processing them in the request instead of taking them off to be handled in the background. This is where job/worker queues come in to save the day.

By moving these long-running tasks to the background using a queue, you can free up your system to handle other requests quickly and efficiently. This not only improves performance but also enhances user experience by ensuring the system responds promptly without making them wait.

In this article, I’ll walk you through the concept of job/worker queues, how Redis can be used as the backbone for this system, and how introducing queues into your system design can significantly improve backend performance.

Understanding Queuing in System Design

Before diving into job queues specifically, it’s important to understand the broader concept of queues and their role in system design.

In computer science, a queue is a data structure that works on a first-in, first-out (FIFO) basis. Think of it as a line of people waiting for a service—the first person to arrive is the first to be served. This simple mechanism allows systems to manage and prioritize tasks in an orderly fashion.

There are several types of queues, each suited to different use cases:

• Message Queues: Facilitate communication between distributed systems by allowing asynchronous message exchange without direct dependency.
  Example: Microservices using RabbitMQ or Apache Kafka.

• Job Queues: Offload time-consuming tasks to background workers, preventing them from blocking the main application.
  Example: Sending emails, processing payments or generating reports in the background.

• Task Queues: Manage smaller, immediate tasks that can be processed in parallel.
  Example: Resizing images or updating profiles in bulk.

• Priority Queues: Handle tasks based on priority, ensuring critical tasks are processed first.
  Example: Triggering alerts for system failures before handling routine events.

Each type enhances system performance by decoupling tasks and optimizing processing time.

What Are Job Queues and Why Do You Need Them?

Job queues allow you to offload time-consuming tasks to be processed asynchronously, meaning they run in the background, freeing up your system to handle other requests. This is crucial for tasks like:

• Sending emails

• Generating reports

• Processing payments

• Importing/exporting data

Think of it this way: you don’t want a user waiting on a page while an email is sent or a large report is generated. Instead, you can queue the task and let your system respond immediately, then complete the task later in the background.

More Real-world Scenarios

Imagine the following scenarios where job queues can make a significant difference:

• Large E-commerce Platform During Peak Sales: Imagine a large e-commerce platform where users place orders during peak sales, such as Black Friday. Instead of processing each order and sending a confirmation email immediately, the system can offload tasks like sending confirmation emails, updating inventory, and notifying shipping services to the background. This ensures that the platform remains responsive for users continuing to browse or place orders.

• User Registration with Email Verification: When users register for a new account, a verification email needs to be sent. Without a queue, the system might make users wait while the email is sent. By queuing the email task, the system can respond instantly with a welcome message, while the email is sent in the background, improving the user experience.

• Financial Services App for Generating Account Statements: In a banking or fintech application, users may request detailed account statements that span several months or years. Generating these reports can be resource-heavy, but by queuing the task, users can continue using other features of the app while the report is prepared in the background, reducing system load and ensuring faster response times for other users.

• 

Redis as the Backbone for Job Queues

Redis, an in-memory data structure store, is perfect for building job/worker queues. Why? Because Redis is:

• Fast: Redis stores data in memory, making it quick to enqueue and dequeue jobs.

• Reliable: It supports durability, so queued jobs are not lost even if your system crashes.

• Scalable: Redis can handle a large number of jobs and workers distributed across multiple servers.

Redis supports list operations like LPUSH, LPOP, and BRPOP, making it easy to implement a basic queue. However, using Redis with some purpose-built tools will make your life as a backend engineer even easier.

Now that we understand Redis as the backbone for job queues, let’s explore how tools like BullMQ simplify job management in Node.js

Tools Built on Redis for Job Queues

Let’s look at two popular tools built on Redis that will simplify working with job queues in your backend system:

1. BullMQ (for Node.js)

One of the most powerful and widely used job queue libraries is BullMQ, built on Redis. It’s great for handling tasks asynchronously in Node.js applications. Whether you need to send notifications, resize images, or schedule tasks, BullMQ provides the tools to do it efficiently.

Here’s how BullMQ can improve your system:

• Job retries: Automatically retry failed tasks.

• Delayed jobs: Schedule tasks to be executed later, not immediately.

• Concurrency: Process multiple jobs in parallel, reducing bottlenecks.

const { Queue } = require('bullmq');

// Create a new queue
const reportQueue = new Queue('reportQueue');

// Add a job to the queue
reportQueue.add('generate-report', { userId: 12345 });

With BullMQ, you can create multiple workers to consume tasks from the same queue. This allows you to scale up your system, as workers can run in parallel, processing tasks independently. This setup ensures tasks are completed faster, without overloading your main application.

import { Worker } from 'bullmq';

// Create a worker to process jobs from the queue
const worker1 = new Worker('reportQueue', async job => {
  // process the job here
  console.log(`Worker 1 is processing job ${job.id}`);
});

// Create a second worker to consume from the same queue
const worker2 = new Worker('reportQueue', async job => {
  // process the job here
  console.log(`Worker 2 is processing job ${job.id}`);
});

In this example, worker1 and worker2 both consume tasks from the report queue. If the queue has many jobs, the workers will split the load, improving processing time and efficiency.

2. Bee-Queue (Node.js)

Bee-Queue is another lightweight, Redis-backed job queue for Node.js. It’s fast, with low overhead, making it ideal for simple use cases where you don’t need the extra complexity of tools like Bull.

While it’s simpler, Bee-Queue still gives you the essentials: delayed jobs, job retries, and concurrency. It works well for small-scale applications or services where simplicity is key.

Why Job Queues Improve Backend Performance

Integrating job queues into your system architecture has several advantages:

1. Improved Response Times: Offloading tasks to a queue ensures that your API responds quickly to user requests, while background workers handle the heavy lifting behind the scenes.

2. Scalability: Redis-based job queues allow you to scale horizontally. You can add more workers to handle more jobs as your system grows, without adding pressure on your core application.

3. Failure Handling: Tools like BullMQ allow for job retries and dead-letter queues (DLQ). If a task fails, it can be retried a certain number of times, and if it still fails, it’s moved to a DLQ where it can be reviewed manually.

Designing Your Backend with Job Queues

When building or scaling your backend system, consider when and where you can introduce job queues. Typical scenarios where queues can dramatically improve system performance include:

• Heavy computational tasks: Media processing is CPU-intensive. Queue these tasks so they don’t block your server’s performance. Tasks like report generation or video processing can be offloaded to a queue, ensuring users don’t experience long waits or timeouts.

• Third-party API interactions: API calls to third-party services (e.g., payment gateways) can be unreliable or slow. Queue them to ensure the request goes through without impacting your system’s response time.

• Scheduled tasks: Need to run scheduled tasks like sending daily summaries or alerts? Queues can handle this efficiently, without adding extra complexity to your application logic. Emails, SMS, or push notifications don’t need to be sent instantly. Queue these tasks to avoid slowing down your app.

• Data Import/Export: Handling large files or data exports can take time, and it’s better to let a worker process it in the background.

Conclusion

To summarise, Redis-based job queues, such as BullMQ and Bee-Queue, are vital tools in modern backend system design. They improve performance by handling background tasks asynchronously, reduce system load during peak times, and provide a reliable mechanism for scaling workers.

Incorporating job queues into your system design improves response times, ensures reliability, and allows you to scale your application effortlessly as it grows. Next time you’re faced with a task that could slow down your app, remember—queue it!

Have you encountered long-running tasks that slow down your system? How would implementing job queues benefit your current projects? Let me know in the comments.

Further Reading

If you're interested in diving deeper into implementing BullMQ in an Express.js application, be sure to check out the next post for a detailed guide.

What is Idempotency?

Idempotency is a property of an operation that guarantees the same result, no matter how many times you perform the same operation. In other words, repeating an operation won’t have any additional side effects beyond the first execution.

Still sound like jargon? Let’s make it simpler.

Now, imagine you’re pouring a cup of coffee. Once the cup is full, pouring more coffee doesn’t change the fact that the cup is already full. Whether you try to fill the cup once or multiple times, the result remains the same—the cup is already at capacity.

This is the essence of idempotency: repeating the same action doesn’t alter the outcome after the first successful attempt.

Got it now? Yea… awesome!

Idempotency in API Design

When designing your APIs, especially REST APIs, idempotency is critical for making sure operations are safe and predictable. The HTTP protocol specifies which methods should be idempotent by nature and which ones are not. Here’s what you need to know:

• GET: Retrieving data should always be idempotent. Every time you request the same resource, you should get the same result without changing the state of the server.

• DELETE: Deleting a resource is idempotent because once the resource is deleted, any further DELETE requests should confirm that the resource no longer exists any errors or additional changes.

• PUT: Updating or replacing a resource with PUT is idempotent. If you send the same update multiple times, the resource should remain in the same state after the first successful update.

• POST: POST is typically not idempotent. This method often creates new resources, so if the request is repeated, you might end up with multiple identical resources. However, with proper handling, you can design certain POST operations to behave idempotently.

Why Idempotency Matters

Now that you understand what idempotency is, let’s explore why it matters. Imagine you’re ordering food online. Once you confirm your order, you expect it to be processed only once, no matter how many times you click the “Confirm Order” button. However, In real-world systems, things don’t always go smoothly. Network interruptions or timeouts might cause a client to resend a request, and if your APIs aren’t designed to handle these retries properly, you could run into trouble—duplicate orders, payments processed multiple times, or a messed-up system state.

• Handling Retries Safely: It helps handle network issues where clients might retry requests due to timeouts or connectivity problems. Idempotent APIs ensure no unintended side effects occur due to retries.

• Maintaining Consistency: In modern cloud-based architectures, multiple instances of services may be running simultaneously, and the same request might be processed by different servers. Idempotency ensures that no matter how many instances process the same request, the outcome remains consistent.

• Preventing Data Corruption: By designing idempotent APIs, you reduce the risk of issues like data duplication, corruption, or errors when clients retry requests. This increases the overall reliability of the system, making it more robust and predictable.

Real-World Example of Idempotency

Let’s take a practical example. Consider an API for updating a user’s email address:

PUT /users/123/email
{
  "email": "email23@example.com"
}

In this case, PUT is idempotent. Whether you send this request once or 100 times with the same payload, the user’s email will remain "email23@example.com".

Now, compare this to creating a new user:

POST /users
{
  "name": "John Doe",
  "email": "johndoe@example.com"
}

Each time this POST request is sent, a new user might be created, leading to multiple identical user records. This non-idempotent behaviour can cause problems in systems that don't properly manage retries.

Designing Idempotent APIs

To ensure your API is idempotent, you can use techniques like:

• Idempotency keys: For example, in payment systems, you can generate a unique idempotency key for each transaction. If the client retries the same transaction with the same key, the server can recognize it and return the original response rather than processing it again.

• Proper use of HTTP methods: Always use “PUT" for updating resources and POST for creating resources. This naturally enforces idempotent behaviour for updates.

• Handling edge cases: Even if a method like POST is not idempotent, you can design it to behave safely in retries by checking for duplicate requests or adding constraints that prevent multiple creations of the same resource.

A Real-Life Example: Fixing Duplicate Payment Notifications

To illustrate, let’s revisit a scenario from a product I have worked on. The application had a POST endpoint to handle payment webhook notifications. Due to network issues, the payment provider sometimes resents the same notification, leading to duplicate values given to the user.

Here’s how I tackled the problem:

• Each payment transaction was assigned a unique idempotency key, similar to a unique booking reference.

• The key is included in the provider’s webhook notification

• Before processing the webhook, the application checked whether the idempotency key had already been processed.

If the key was already processed, the system skipped further actions, preventing duplicate values for the user. This ensured that users only received value once, even if the webhook was resent.

Conclusion

Idempotency is a cornerstone of reliable API design. By ensuring that repeated operations don’t produce unexpected side effects, you can make your systems more robust, handle retries safely, and maintain consistency across distributed environments. Whether you’re working with payments, user data, or any critical operation, designing for idempotency is key to preventing common issues like data duplication and corruption.

As you design your APIs, keep idempotency in mind to ensure your system behaves predictably no matter what challenges come its way.

Concurrency involves multiple processes or transactions executing at the same time, potentially interacting with shared data. Imagine a social media platform where several users react to a post simultaneously. Each reaction is an operation that affects the post’s reaction count.

The Lost Update Problem

In simple terms, the update of a client's request is overwritten by the update of another. This is also referred to as RACE CONDITIONS

Let’s consider a simple scenario to illustrate the Lost Update Problem. Suppose you have a post with a reaction count of 100. Two users simultaneously decide to like and dislike the post respectively. Ideally, after these reactions are processed, the count should return to 100. However, without proper concurrency control, one of these reactions might be lost, and the final count might not reflect all reactions accurately. This issue occurs because multiple transactions (or operations) are trying to update the same data at the same time without proper coordination.

The Problem in Action

To better understand the Lost Update Problem, let’s walk through a simple scenario using the example of reacting to a social media post.

Imagine a post that currently has 100 likes. Two clients, A and B, both decide to interact with this post at almost the same time, but with different actions—Client A wants to like the post, while Client B decides to un-like it. Below is a table that shows the steps each client takes:

• Step 1: Client A reads the current like count (x = 100) and increments it by 1 in memory (x = 101). At the same time, Client B reads the same current like count (x = 100) and decrements it by 1 in memory (x = 99).

• Step 2: Client B writes the updated count (x = 99) to the database.

• Step 3: Client A then writes its updated count (x = 101) to the database.

The Result: Instead of the expected behaviour, where the final like count should return to 100, it ends up being 101—Client A’s write operation overwrote Client B’s, effectively losing Client B’s update.

This is a classic example of the Lost Update Problem, where concurrent transactions interfere with each other, leading to incorrect data being written to the database.

Solutions to the Problem

To address the Lost Update Problem, various strategies can be employed, each with its trade-offs. Let’s explore four key approaches:

Using Atomic Operations

Atomic operations ensure that an update is performed as a single, indivisible unit. This means that each reaction is handled independently, and the increment operation is performed without interference from other operations.

Example:

-- Increment the reaction count atomically
UPDATE posts SET like_count = like_count + 1 WHERE id = 1;

Sequelize (MySQL/PostgreSQL) Example:

await Post.update(
  { like_count: Sequelize.literal('like_count + 1') },
  { where: { id: 1 } }
);

Mongoose (MongoDB) Example:

await Post.findByIdAndUpdate(
  1,
  { $inc: { like_count: 1 } },
  { new: true }
);

Pros: Simple and efficient, especially for straightforward updates.

Cons: Limited to basic operations and may not handle complex business logic.

Using Pessimistic Locking

Pessimistic locking involves putting a lock on a record or document that is selected by a query until the transaction is committed or rolled back. It prevents other transactions from updating or deleting the record.

This type of approach uses transactions. In SQL-based databases, this is often done using the FOR UPDATE clause.

Raw SQL Query Example with FOR UPDATE:

BEGIN;

SELECT like_count 
FROM posts 
WHERE id = 1 
FOR UPDATE;

UPDATE posts 
SET like_count = like_count + 1 
WHERE id = 1;

COMMIT;

Sequelize (MySQL/PostgreSQL) Example:

const transaction = await sequelize.transaction();

try {
  const post = await Post.findOne({
    where: { id: 1 },
    lock: true, // Locking the row for update
    transaction
  });

  post.like_count += 1;
  await post.save({ transaction });

  await transaction.commit();
} catch (error) {
  await transaction.rollback();
  throw error;
}

Mongoose (MongoDB) Example: Pessimistic locking is not natively supported in MongoDB as it is in SQL databases, but you can simulate it with a custom locking mechanism.

Types of Pessimistic Locks:

• Exclusive Lock (Write Lock): This type of lock prevents other transactions from both reading and writing to the locked resource. It is typically used when the transaction intends to modify the data.

• Shared Lock (Read Lock): A shared lock allows other transactions to read the data but not to modify it. Multiple transactions can acquire a shared lock on the same resource, but no transaction can change it until all shared locks are released.

In Sequelize ORM, the lock option can be used to specify the type of lock when querying the database. The value passed to this option depends on the database being used:

• For MySQL: LOCK.UPDATE for an exclusive lock and LOCK.SHARE for a shared lock.

• For PostgreSQL: LOCK.UPDATE for an exclusive lock and LOCK.KEY_SHARE for a shared lock.

Pros: Guarantees that no other transactions can interfere.
Cons: Can reduce concurrency and lead to potential deadlocks—a situation where two or more transactions are waiting for each other to release lock

await Post.findOne({
  where: { id: 1 },
  lock: sequelize.Transaction.LOCK.UPDATE, // Exclusive Lock
  transaction
});

Using Serializable Isolation Level

The isolation level in a DBMS determines how transactions interact with each other and the level of visibility they have over one another. Higher isolation levels reduce the chances of concurrency issues but can impact performance.

• Serializable is the highest isolation level, which handles transactions as if they are executed one after another. It effectively prevents concurrency problems like the Lost Update Problem but can reduce throughput due to the strict control.

• Other levels include Repeatable Read, Read Committed, and Read Uncommitted.

• Read Uncommitted is the lowest level, offering maximum performance but introducing more concurrency problems like dirty reads and lost updates.

Raw SQL Example:

SET TRANSACTION ISOLATION LEVEL SERIALIZABLE;

BEGIN;

SELECT like_count 
FROM posts 
WHERE id = 1;

UPDATE posts 
SET like_count = like_count + 1 
WHERE id = 1;

COMMIT;

In this example, the SERIALIZABLE isolation level ensures that no other transaction can read or write to the posts table until the transaction is completed.

Sequelize Example:

const transaction = await sequelize.transaction({
  isolationLevel: Sequelize.Transaction.ISOLATION_LEVELS.SERIALIZABLE
});

try {
  const post = await Post.findOne({
    where: { id: 1 },
    transaction
  });

  post.like_count += 1;
  await post.save({ transaction });

  await transaction.commit();
} catch (error) {
  await transaction.rollback();
  // Handle concurrency conflict (e.g., retry or notify user)
}

In Sequelize, you can set the isolation level by passing the isolationLevel option to the transaction method.

Mongoose (MongoDB) Example:
Mongoose, being built on MongoDB, doesn't directly support transaction isolation levels like traditional relational databases. However, MongoDB offers some degree of isolation by default when using transactions.

Pros: Ensures complete isolation and prevents all other concurrency problems.
Cons: Can significantly impact performance and reduce concurrency.

Using Optimistic Locking

In this approach, multiple transactions proceed without locking resources. It assumes that conflicts are rare and checks for them only before committing changes. If a conflict is detected (e.g., another transaction has modified the data), the transaction is rolled back, and the operation can be retried.

It is typically implemented using a versioning mechanism where each record or document has a version or last update timestamp field. This field is checked during the update, and if it doesn't match the expected value, the transaction is aborted.

Raw SQL Example:

BEGIN;

SELECT like_count, version 
FROM posts 
WHERE id = 1;

-- Assume version = 5
UPDATE posts 
SET like_count = like_count + 1, 
    version = version + 1 
WHERE id = 1 AND version = 5;

COMMIT;

In this example, the UPDATE statement ensures that the version hasn't changed since it was last read. If the version has changed, the update will fail, signalling a conflict.

Sequelize Example:

const post = await Post.findOne({
  where: { id: 1 }
});

const originalVersion = post.version;

post.like_count += 1;

try {
  await post.save({
    where: {
      id: post.id,
      version: originalVersion
    }
  });
} catch (error) {
  // Handle concurrency conflict (e.g., retry or notify user)
}

In Sequelize, optimistic locking can be implemented by manually checking the version field before updating the record. The update will proceed only if the version in the database matches the original version.

Mongoose Example:

const post = await Post.findById(id);

const originalVersion = post.__v; // Mongoose uses __v for versioning by default

post.like_count += 1;

try {
  await post.save(); // Mongoose automatically checks and updates __v
} catch (error) {
  // Handle concurrency conflict (e.g., retry or notify user)
}

Mongoose has built-in support for optimistic locking through versioning (__v field). When you save a document, Mongoose automatically checks the version and throws an error if there's a version mismatch.

Pros:

• One of the key advantages of optimistic locking is that it allows you to gracefully handle concurrency issues. For example, if a conflict is detected, you can notify the client that the data has been updated by someone else. This allows the client to review the changes and decide how to proceed, which enhances the user experience.

• Allows greater concurrency and is suitable for scenarios where conflicts are rare.

Cons: Requires additional logic to handle conflicts and retries.

More Reads

If you’re interested in diving deeper into the topics covered in this article, here are some additional resources:

• https://on-systems.tech/blog/128-preventing-read-committed-sql-concurrency-errors/

• https://www.geeksforgeeks.org/concurrency-problems-in-dbms-transactions/

• https://www.javatpoint.com/dbms-concurrency-control

Conclusion

Concurrency is an important concept that drives the performance of modern applications but also brings challenges such as the Lost Update Problem. By employing strategies like atomic operations, pessimistic locking, isolation levels, and optimistic locking, you can effectively manage these challenges and ensure data accuracy in concurrent environments.

Ultimately, the best solution depends on your application's specific needs and limitations. Knowing these techniques and how they work together is crucial for building strong, high-performance systems.

If there's anything you believe needs a more detailed explanation, or if you spot any errors, feel free to leave a comment. Your feedback is invaluable!

if you found this article helpful, a little love ❤️ would be greatly appreciated.

To ensure secure and dependable storage of uploaded files, integration with cloud storage providers such as Amazon S3, Google Cloud Storage, Cloudinary or Microsoft Azure Blob Storage is often preferred, although the process can seem daunting to you.

Fret not! This article will guide you through integrating with these cloud storage platforms.

Outline

• Let's Get Started

• Prerequisites

• Introducing express-file-wizardry: Your Ally in File Uploads

• Why a New Package?

• Setting up Your Express Application

• Setting up Preferred Cloud Storage

• File Upload Routes and Middleware

• Bringing Everything Together and Testing

• Advanced Features and Customization

• Conclusion

• References and Further Reading

Let's Get Started

Welcome to my comprehensive guide on mastering file uploads and integrating them seamlessly with various cloud storage platforms in your web applications. Whether you're a junior developer navigating the complexities of file uploads or an experienced developer looking to streamline your workflow, this article is here to assist you every step of the way.

After reading this article, you will have the information and self-assurance to incorporate strong file upload features into your applications.

Prerequisites

• Basic knowledge of JavaScript and Node.js - Familiarity with JavaScript and Node.js is essential to understanding the concepts and code examples in this tutorial.

• A Cloud storage platform account - set up any of the cloud storage express-file-wizardry supports

Introducing express-file-wizardry: Your Ally in File Uploads

express-file-wizardry is a middleware for Express.js that empowers developers to effortlessly handle file uploads in their applications. With its intuitive interface and robust feature set, express-file-wizardry abstracts away the complexities of file handling, allowing developers to focus on building innovative solutions without getting bogged down by boilerplate code.

Why a new package?

In a filled environment of options for managing file uploads in Express.js apps, you may question the necessity of a new package. The creation of express-file-wizardry came from a specific need that current solutions didn't fully meet.

1. Flexible Storage Options: express-file-wizardry seamlessly integrates with a wide range of cloud storage platforms, giving you the freedom to leverage the power of your preferred provider.

2. Versatile File Type Support: from popular formats like JPEG, PDF, and MP4 to lesser-known file types, express-file-wizardry ensures that your application can handle uploads with ease, regardless of the file format.

3. Simplified Configuration: With intuitive and straightforward configuration options, express-file-wizardry makes it easy to set up and customize file upload functionality to suit your specific requirements.

STEP 1: Setting up your Express Application

• Install Node.js and npm
  If you haven't already, you'll need to install Node.js and npm (Node Package Manager) on your system. You can download and install them from the official Node.js website: Node.js.

• Create a New Directory for Your Project
  Open the terminal make a new directory and navigate into the directory for your Express.js project. This can be accomplished by using the following command.

    mkdir file-upload-app
    cd file-upload-app
  

• Initialize Your Project
  Initialize your project by running:

    npm init -y
  

• Install Express
  Install Express.js as a dependency for your project:

    npm install express
  

• Create Your Express Application
  Create a new file, for example, app.js, and write your Express application code:

    const express = require('express');
    const app = express();
    const port = 3000;
  
    app.get('/', (req, res) => {
      res.send('Hello World!');
    });
  
    app.listen(port, () => {
      console.log(`Server is running on http://localhost:${port}`);
    });
  

• Run Your Express Application
  Run your Express application using Node.js:

    node app.js
  

  You should see a message indicating that your server is running on http://localhost:3000.

  

• Test Your Application
  Open your API client (Postman or any other one) and navigate to http://localhost:3000. You should see the message "Hello World!" displayed in the response section as shown below.

  

STEP 2: Setting up Preferred Cloud Storage

• Install express-file-wizardry :

    npm install express-file-wizardry
  

• Set a preferred storage type

    const express = require('express');
    const { FileWizardry } = require('express-file-wizardry');
  
    const app = express();
    const port = 3000;
  
    const fileWizardry = new FileWizardry();
    //fileWizardry.setStorageType('memory'); // Storage type can be any of the supported types: memory, disk, cloudinary, amazons3
    fileWizardry.setStorageType('cloudinary', { 
        cloud_name: 'my-cloud', 
        api_key: 'api-key', 
        api_secret: 'api-secret' 
    });
    // fileWizardry.setStorageType('amazons3', { 
        // accessKeyId: 'your-access-key', 
        // secretAccessKey: 'your-secret-key', 
        // region: 'your-region', 
        // bucket: 'your-bucket' 
    // });
    // fileWizardry.setStorageType('disk', { 
        // destination: '/path/to/upload/folder' 
    // });
  
    app.get('/', (req, res) => {
      res.send('Hello World!');
    });
  
    app.listen(port, () => {
      console.log(`Server is running on http://localhost:${port}`);
    });
  

  Understanding the Code:

  1. Importing Dependencies: I start by importing the express framework and the FileWizardry class from the express-file-wizardry package.

  2. Initializing Express Application: Next, I create an instance of the Express application and define the port number (in this case, port 3000) on which the server will listen for incoming requests.

  3. Configuring FileWizardry Storage Type: The FileWizardry class provides methods for configuring the storage type for file uploads. In this example, we're setting the storage type cloudinary with its configuration options, which indicates that uploaded files will be stored temporarily in memory. Alternatively, you can uncomment one of the other storage type configurations (memory, amazons3, or disk) and provide the necessary configuration options to use Amazon S3, or disk storage, respectively.

  4. Defining Routes: We define a simple route handler for the root URL / which sends the response 'Hello World!' when the server receives a GET request to this endpoint.

  5. Starting the Server: Finally, we start the Express server and listen for incoming connections on the specified port. Once the server runs, a message is logged to the console indicating the server's URL.

STEP 3: File Upload Routes and Middleware

After setting up express-file-wizardry to use our preferred cloud storage provider, we will create file upload routes and middleware in our Express.js app. This section will explain how to set routes for managing file uploads and incorporate express-file-wizardry middleware for a seamless process.

• Defining File Upload Routes

    app.post('/upload-signle', fileWizardry.uploadFile({ formats: ['image/jpeg', 'image/png'], fieldName: 'image' }), (req, res) => {
      // File upload successful
      res.status(200).json({ message: 'File uploaded successfully', file: req.file });
    });
  
    app.post('/upload-multi', fileWizardry.uploadFile({ formats: ['image/jpeg', 'image/png'], fieldName: 'images',  multiFile: true }), (req, res) => {
      // File upload successful
      res.status(200).json({ message: 'Files uploaded successfully', file: req.files });
    });
  

  In this example:

  • I defined a POST route /upload-single to handle single file upload using the fileWizardry.uploadFile middleware. This middleware expects a single file with the field name image. This middleware takes an options object specifying the allowed file formats (formats) and the field name (fieldName) under which the file will be uploaded.

  • In contrast, the second route /upload-multi is designed to handle multiple file uploads. Here, the fileWizardry.uploadFile middleware is again employed, but with an added option: multiFile, which is set to true. This indicates that the route can process multiple files simultaneously.

  • Across both routes, I've restricted uploads to only accept JPEG and PNG image files, ensuring that the application remains focused on handling image uploads exclusively.

• Error Handling and Validation

    app.post('/upload-single', fileWizardry.uploadFile({ formats: ['image/jpeg', 'image/png'], fieldName: 'image' }), (req, res) => {
      if (req.fileValidationError) {
          return res.status(400).json({ 
            error: req.fileValidationError.message 
          });
      }
      res.status(200).json({ 
          message: 'File uploaded successfully', 
          file: req.file 
      });
    });
  
    app.post('/upload-multi', fileWizardry.uploadFile({ formats: ['image/jpeg', 'image/png'], fieldName: 'images',  multiFile: true }), (req, res) => {
      if (req.fileValidationError) {
          return res.status(400).json({ 
            error: req.fileValidationError.message 
          });
      }
      res.status(200).json({ 
          message: 'Files uploaded successfully', 
          file: req.files
      });
    });
  

  • Validation Check: After the file upload middleware, a validation check is performed using req.fileValidationError. If a validation error occurs during the file upload process, req.fileValidationError will be populated with details of the error. Every validation has been handled by the express-file-wizardry package.

  • Error Handling: If req.fileValidationError exists, indicating a validation error, the route sends a 400 Bad Request response with an error message describing the validation error.

  • Success Response: If no validation errors occur, the route sends a 200 OK response with a success message and information about the uploaded file(s) req.file or req.files

Advanced Features and Customization

• Max Size Upload Option (Optional)
  Maximum allowed size for each uploaded file, in bytes. If specified, files exceeding this size will be rejected.

• File Deletion
  File deletion is an essential aspect of managing uploaded files, especially when dealing with storage solutions like local disk or cloud storage. Here's how you can implement file deletion in an Express.js application using express-file-wizardry

    const fileWizardry = new FileWizardry();
  
    fileWizardry.deleteFile('cloudinary', {
        public_id: 'cloudinary_file_public_Id',
    });
  
    fileWizardry.deleteFile('amazons3', {
        key: 'my-file',
        Bucket: 'my-bucket',
    });
  
    fileWizardry.deleteFile('disk', {
        path: 'my-file',
    });
  

Bringing Everything Together and Testing

const { config } = require('dotenv');
const express = require('express');
const { FileWizardry } = require('express-file-wizardry');

config();
const app = express();
const port = 3000;

const fileWizardry = new FileWizardry();
fileWizardry.setStorageType('cloudinary', {
    cloud_name: process.env.CLOUD_NAME,
    api_key: process.env.API_KEY,
    api_secret: process.env.API_SECRET,
});

app.get('/', (req, res) => {
    res.send('Hello World!');
});

app.post(
    '/upload-single',
    fileWizardry.upload({ formats: ['image/jpeg', 'image/png'], fieldName: 'image' }),
    (req, res) => {
        try {
            if (req.fileValidationError) {
                return res.status(400).json({
                    error: req.fileValidationError.message || req.fileValidationError,
                });
            }
            res.status(200).json({
                message: 'File uploaded successfully',
                file: req.file,
            });
        } catch (error) {
            console.log(error);
            res.status(500).json({
                message: 'Internal server error',
                error: error.message,
            });
        }
    }
);

app.post(
    '/upload-multi',
    fileWizardry.upload({
        formats: ['image/jpeg', 'image/png'],
        fieldName: 'images',
        multiFile: true,
    }),
    (req, res) => {
        try {
            if (req.fileValidationError) {
                return res.status(400).json({
                    error: req.fileValidationError.message || req.fileValidationError,
                });
            }
            res.status(200).json({
                message: 'Files uploaded successfully',
                file: req.files,
            });
        } catch (error) {
            console.log(error);
            res.status(500).json({
                message: 'Internal server error',
                error: error.message,
            });
        }
    }
);

app.delete('/delete-file/:fileId', async (req, res) => {
    try {
        await fileWizardry.deleteFile(req.params.fileId);

        res.status(200).json({
            message: 'File uploaded successfully',
            file: req.file,
        });
    } catch (error) {
        console.log(error);
        res.status(500).json({
            message: 'Internal server error',
            error: error.message,
        });
    }
});

app.listen(port, () => {
    console.log(`Server is running on http://localhost:${port}`);
});

Start your Express.js application by running the following command in your terminal:

node app.js

• Once the application is running, you can now test your application by sending requests to the defined routes (e.g., /upload-single for single file upload, /upload-multi for multiple file uploads and /delete-file/:fileId for file deletion) using your preferred API client.

  

  

  

  

  

• Monitor the terminal for any logs or errors that may occur while your application is running. If there are any issues, you can debug them based on the error messages displayed.

You can click here to check out the complete code on GitHub.

Conclusion

This article delves into the integration of file upload features in Express.js using express-file-wizardry, addressing the importance of this function in web development, challenges, and the tool's benefits. The guide covers setting up an Express.js app for file uploads, configuration of cloud storage options, defining routes, error handling, security practices, and advanced customization. express-file-wizardry streamlines the file upload process, boosting performance and user experience. It supports local disk, cloud storage like Amazon S3, and third-party services, offering a comprehensive solution. Overall, it enables developers to focus on creating robust apps without the hassle of managing file uploads.

Happy Coding!

References and Further Reading

• Express-file-wizardry Docs

• dotenv Docs