9. System Design Fundamentals

Scalability is a system’s ability to handle increased load by adding resources. Vertical scaling means giving a single machine more power (CPU, RAM), while horizontal scaling means adding more machines and distributing the load. Horizontally scalable systems are often preferred for large-scale applications because they can grow by adding commodity servers rather than relying on ever-larger single machines.

A key distinction is between handling more traffic (throughput) and keeping latency acceptable. Design choices like stateless services, shared-nothing architectures, and efficient use of caches all contribute to scalable systems. Understanding where bottlenecks appear—CPU, memory, network, storage—is critical when designing for growth.

Load balancing distributes incoming traffic across multiple servers or instances so no single node becomes overloaded. A load balancer can operate at different layers (e.g., L4 vs L7) and use different strategies like round-robin, least-connections, or more advanced health-aware algorithms. From the client’s perspective, there is often a single endpoint, but behind it multiple servers share the work.

In addition to distribution, load balancers often handle health checks, removing unhealthy instances from rotation and sometimes performing SSL termination. In cloud environments, load balancers are standard building blocks for resilient and scalable systems. They are closely tied to auto-scaling strategies that add or remove instances as demand changes.

Caching stores the results of expensive operations in faster storage so they can be reused. This might be in-memory caches, distributed cache systems, or even browser caches on the front end. The idea is to avoid redoing work like hitting the database or recalculating complex results when the answer hasn’t changed.

Common caching strategies include in-memory caches per instance, distributed caches like Redis or Memcached, and HTTP-level caching with appropriate headers. Caching introduces its own challenges, especially around invalidation: ensuring the cache stays fresh enough for your correctness requirements. Done well, caching can drastically reduce load on databases and downstream services and improve response times.

Microservices architecture breaks a system into many small, independently deployable services, each focused on a specific capability. Instead of one large monolith, you have many smaller units that communicate over the network—often via REST or message queues. Each microservice can be developed, deployed, and scaled independently, allowing teams to work more autonomously.

The benefits include flexibility, technology diversity, and targeted scaling (only the hot services need extra instances). However, microservices also introduce complexity: network communication, distributed transactions, observability, and deployment management all become more challenging. For many organizations, a gradual evolution from a modular monolith to microservices gives a better balance than jumping straight into a fully distributed design.