How to Build Scalable Cloud Architecture on AWS

Most startups don't have an AWS problem at launch. They have a "works on my machine, works for 200 users" setup — a single EC2 instance, maybe a managed database, and a prayer that the traffic stays manageable.

Then one day it doesn't. A product launch goes better than expected, a social post hits, an enterprise client onboards a thousand users at once. And suddenly the question of whether the architecture can actually scale becomes urgent rather than theoretical.

The gap between an AWS setup that buckles under pressure and one that handles 100,000 concurrent users isn't primarily budget. It's a set of specific architectural decisions — about how compute is managed, how traffic is distributed, how state is handled, and what happens when a single component fails. This post covers those decisions, in the order they matter.

The Three AWS Compute Models (And When Each Is Right)

AWS gives you three fundamentally different ways to run application code. Choosing correctly — or mixing deliberately — is the first architectural decision that determines how your system scales.

For most product startups in 2025, the right answer is a combination of Lambda and ECS/Fargate, not a single choice across the board.

Lambda is the right tool for anything event-driven: processing a file upload when it hits S3, sending a transactional email after a database row is created, running a scheduled data sync at 2 AM. The pay-per-execution model means low-traffic services cost almost nothing, and scaling is automatic — Lambda simply runs more concurrent executions as demand rises.

ECS with Fargate is right for your core application API — the service handling the majority of your user requests. Containers give you a consistent, reproducible runtime that behaves identically in development, staging, and production. Fargate removes the need to manage the underlying EC2 instances, so you define your container's CPU and memory requirements and AWS handles the rest. It's the pragmatic middle ground between Lambda's managed simplicity and EC2's raw control.

EC2 is the right answer when you have workloads that Lambda and containers genuinely can't serve — specific GPU requirements for ML inference, unusual kernel-level access, or steady-state high-utilization workloads where owning the underlying compute is cheaper than Fargate's per-task pricing. For most product startups, this is not the starting point.

The Multi-Tier Architecture That Handles Real Traffic

Regardless of which compute model you choose, a production AWS architecture that scales is built in layers, each layer sitting in its own network segment with its own scaling behaviour.

Tier 1: Traffic distribution (public-facing)

Traffic arrives at an Application Load Balancer (ALB), which distributes incoming requests across your compute resources using configurable routing rules. The ALB handles SSL termination, health checks, and path-based routing (sending /api/* to your API containers, /static/* to S3 or CloudFront).

CloudFront sits in front of everything as a CDN — caching static assets, reducing latency for geographically distributed users, and absorbing the kind of traffic spikes that would otherwise reach your origin servers.

Route 53 manages DNS, with health checks that can automatically fail over to a secondary region if your primary region becomes unavailable.

Tier 2: Compute (private subnets)

Your application containers or Lambda functions run here, in private subnets with no direct internet access. The only way traffic reaches them is through the load balancer. Auto Scaling Groups (for EC2) or ECS service auto scaling (for Fargate) add or remove compute capacity based on CPU usage, request count, or custom metrics.

The critical design rule at this tier: stateless compute. No session data stored on the application instance. Sessions go in ElastiCache (Redis), uploaded files go to S3, queued jobs go to SQS. If any individual container can be terminated and replaced without affecting the user experience, your compute tier scales horizontally without friction.

Tier 3: Data (private subnets)

RDS (PostgreSQL or MySQL) with Multi-AZ deployment provides the relational database layer. Multi-AZ creates a synchronous standby replica in a separate Availability Zone. If the primary instance fails, RDS automatically promotes the standby — typically with under two minutes of downtime — without any manual intervention.

Read replicas reduce load on the primary by directing read-heavy queries to separate instances. ElastiCache (Redis) caches the results of expensive queries so they don't hit the database repeatedly.

Docker and Containers: The Standardisation Layer

If you're running application code on ECS, Docker is how that code gets packaged and deployed. Understanding why containers improve scalability is not just conceptual — it changes how you think about the deployment pipeline.

A Docker container bundles your application code, its dependencies, its runtime, and its configuration into a single, immutable image. That image runs identically on a developer's MacBook, in a CI environment, in staging, and in production. The "it works on my machine" class of bugs largely disappears.

For scaling, this matters because containers are the unit of scaling. When ECS auto scaling decides you need more capacity, it launches more instances of your container image — not more servers, not more installed runtimes, but more copies of the exact same environment. This is deterministic. A new container running your API responds identically to one that's been running for three days.

A few container design rules that matter for production:

One process per container. Don't run your API server and your background job processor in the same container. Run them as separate services. When request volume spikes, scale the API service. When the job queue backs up, scale the job processor. Coupling them eliminates that flexibility.

Immutable images. Don't modify running containers. If a configuration changes, build a new image and deploy it. This keeps your deployment history clean and makes rollbacks reliable — rolling back is just re-deploying the previous image tag.

Secrets out of images. Database passwords, API keys, and environment-specific configuration never go in a Docker image. AWS Secrets Manager or Systems Manager Parameter Store injects them at container startup. An image that ends up in a container registry should contain no sensitive data.

Auto Scaling: The Part Most Teams Get Wrong

Auto scaling is widely implemented and frequently misconfigured. The goal is to add capacity before requests start failing, not after. The gap between those two outcomes is in the scaling policy.

The most common misconfiguration: scaling on CPU utilisation alone with a high threshold (80% or 90%). By the time CPU hits 80%, requests are already queuing and response times have degraded. Scale at 60–70% CPU with a predictive component based on request count if your traffic patterns have recognisable shapes.

A well-configured ECS service auto scaling policy for a web API looks like:

• Target tracking policy on CPU: maintain target CPU at 60%, triggering scale-out when sustained above, scale-in slowly (cooldown 300 seconds) to prevent thrashing
• Target tracking policy on ALB request count per target: maintain a target request count per container based on your measured capacity — if each container handles 200 RPS comfortably, set the target at 150 to leave headroom
• Minimum capacity above zero: at least two tasks running at all times across two Availability Zones, so a single AZ failure doesn't take down the service entirely

Scale-in is almost always more dangerous than scale-out. Terminating a container that's mid-request loses that request. ECS supports connection draining — a grace period during which a de-registered container completes in-flight requests before termination. Set this to at least 30 seconds for web services. For services processing long jobs, set it long enough to complete the longest reasonable job.

What Infrastructure as Code Changes

Setting up an AWS architecture manually through the console produces something that works but that no one fully understands after six months. How was that security group configured? Why does this Lambda have that IAM policy? When was this RDS parameter group last modified?

Infrastructure as Code (IaC) — using Terraform, AWS CloudFormation, or AWS CDK — defines your infrastructure in version-controlled files. Every change is a pull request. The current state of your infrastructure is the code in your repository, not the current state of the console.

For startups, this pays off in three specific ways. First, disaster recovery: if a region fails or an account is compromised, you can recreate the entire infrastructure from the codebase in hours. Second, staging environments: spinning up an identical staging environment is running the same Terraform with different variable values. Third, auditability: the git history of your infrastructure repository is the audit log of every infrastructure change.

The operational overhead of learning IaC is real. The operational overhead of managing un-codified infrastructure at 50 engineers is larger.

If you're planning infrastructure for a product that will need to handle meaningful scale, our cloud and custom software services include architecture design as part of the engagement — defining the AWS setup as code before any application code is written.

The Production Readiness Checklist

Before an application handles real user traffic at scale, these items should be deliberately verified rather than assumed:

High availability

• All critical components (ALB, ECS service, RDS) span at least two Availability Zones
• RDS Multi-AZ enabled with automatic failover tested
• ECS service minimum task count is 2+, distributed across AZs

Scaling

• Auto scaling policies defined and tested under synthetic load
• Connection draining configured on ECS services
• RDS read replicas provisioned if read:write ratio is high

Security

• All application and database resources in private subnets
• Security groups follow least-privilege (web tier accepts only ALB traffic; DB tier accepts only app tier)
• Secrets in AWS Secrets Manager, not environment variables or Docker images
• SSL/TLS enforced at the load balancer level

Observability

• CloudWatch alarms on error rate, latency, and CPU for all services
• Centralised log aggregation (CloudWatch Logs or a log management service)
• Distributed tracing configured for debugging across services
• RDS automated backups enabled with tested restoration procedure

Infrastructure

• All infrastructure defined as code (Terraform or CloudFormation)
• CI/CD pipeline deploys new container images without downtime (rolling or blue/green)
• Deployment rollback procedure defined and tested

FAQ

What AWS services do I need to build a scalable web application? At minimum: Route 53 (DNS), CloudFront (CDN), an Application Load Balancer, ECS with Fargate (compute), RDS with Multi-AZ (relational database), ElastiCache for Redis (caching and sessions), S3 (file storage), and CloudWatch (monitoring). Lambda is added for event-driven workloads. The exact set depends on your application's architecture, but these cover the majority of production web applications.

When should a startup move from a single server to a proper cloud architecture? The signal is usually one of three things: your traffic has become unpredictable enough that a fixed-size server is either over-provisioned most of the time or overwhelmed at peak, you've had a downtime incident that affected users because there was no redundancy, or you're approaching a scale where compliance or enterprise customers are asking about your infrastructure. The cost of re-architecting after you've already grown into the problem is higher than getting it right before it becomes critical.

Is ECS or Kubernetes (EKS) better for a startup? ECS is the right choice for almost all startups. Kubernetes offers power and flexibility at the cost of significant operational complexity — you need dedicated platform engineering time to run it well. ECS on Fargate gives you containerised workloads, auto scaling, and deep AWS integration without that operational overhead. The only reason to choose EKS early is if you have a specific Kubernetes-ecosystem tooling requirement or multi-cloud portability is a hard requirement.

How much does a properly scaled AWS architecture cost? A production-ready setup for a mid-size application — ALB, ECS Fargate (2–4 tasks), RDS Multi-AZ (db.t3.medium), ElastiCache, CloudFront, and associated services — typically runs Rs. 25,000–60,000 per month depending on traffic volume and data transfer. This scales up with usage and down with Reserved Instance or Savings Plans discounts (typically 30–40% off on-demand pricing for steady workloads). [VERIFY: cost estimate based on typical production configuration, verify with AWS Pricing Calculator for specific workloads]

What is the most common mistake teams make with AWS architecture? Storing state on application servers. When you store session data, uploaded files, or user state on a running EC2 instance or container, you can't scale horizontally — every request has to hit the same server. Moving all state external (Redis for sessions, S3 for files, a database for everything else) is the single change that makes horizontal scaling possible. Everything else follows from that.

Building an architecture that won't fall over when it works is not a problem to solve after you've launched. It's a set of decisions to make before you write the first line of application code. If you're scoping a new product or reviewing an existing setup that's showing cracks, reach out to our team for a cloud infrastructure review.

Custom Software & SaaS development · API-first development guide · How to ensure 99.9% uptime for your SaaS

Author: Abdul Rahaman
Last updated: August 2026