Level 122
95% bulls..t
Nov 12, 2025
11
Level 2
Should I only add complexity when there are issues with the Web System itself and start off simple and follow YAGNI and the Evolutionary Architecture?
I was asking Gemini AI how I should add System Design Patterns to a Web Application System. My question was this ->
So adding complexity like a System Design Pattern like "Sequential Convoy" or "Queue Based Load Leveling" or a "CDN Cache" or a "Circuit Breaker" or "Bulk Head" should come from bottlenecks and other conditions like high CPU, RAM, or I/O Disk or Race Conditions or Latency Bugs? Is that correct? A System should start off with that it needs, and then gradually add complexity when it is needed from bottlenecks for example. Explain in detail.
And it said this ->
Yes, that is absolutely correct and summarizes the core principle of pragmatic system design:
Start simple and introduce complexity only when necessary to solve a proven problem. This approach is known as Evolutionary Architecture or the YAGNI (You Ain't Gonna Need It) principle in design patterns.
-----------------------------------------------------
Here is a detailed explanation of why and how this process works:
1. The Principle: Pragmatic, Evolutionary Design
The primary goal of a system designer is not to build the most feature-rich or most complex system, but the simplest system that satisfies all current functional and non-functional requirements.
Why Start Simple?
1) Reduced Initial Overhead: Complex patterns like "Queue-Based Load Leveling" or "Saga" require significant time for setup, configuration, and integration. Doing this prematurely slows down development and time-to-market.
2) Lower Cognitive Load: A simpler system is easier for a development team to understand, debug, and maintain. Every added pattern increases the system's overall complexity tax.
3) Preventing Premature Optimization: You often cannot accurately predict where the real bottlenecks will be. You might implement a complex sharding strategy only to find your actual bottleneck was database connection pooling, not I/O.
4) Flexibility: A simpler system is easier to refactor or pivot than one heavily entrenched with complex, interconnected patterns.
-----------------------------------------------------
2. Triggering the Need for Complexity (The Bottlenecks)
The decision to introduce a pattern must be data-driven, arising from observable performance issues or failure conditions. You correctly identified the primary triggers:
1) Trigger Condition ------ 2) Observable Symptoms ------- 3) System Design Pattern to Address
1) High Latency/Slow Response
2) High CPU usage, slow database queries, long processing times for specific requests.
3) CDN Cache, In-Memory Caching (e.g., Redis), CQRS (to separate read/write).
1) High Load/Traffic Spikes
2) Consistent 100% CPU or RAM utilization on servers, request queue build-up.
3) Load Balancer, Horizontal Scaling (adding more servers), Queue-Based Load Leveling (for asynchronous tasks).
1) I/O Disk Bottleneck
2) Slow read/write speeds, high disk utilization, database server struggling.
3) Database Sharding/Partitioning, Database Replication (to offload reads), Caching.
1) Service Dependency Failure
2) Downstream service (e.g., payment, authentication) fails, causing the entire application to hang or crash (cascading failure).
3) Circuit Breaker (stops requests to a failing service), Bulkhead (isolates resource pools).
1) Race Conditions/Data Inconsistencies
2) Two or more processes try to update the same resource simultaneously, resulting in incorrect data.
3) Distributed Locking, Sequential Convoy (ensuring ordered access), Saga (for distributed transactional integrity).
1) Unpredictable Asynchronous Loads
2) A background process (e.g., image resizing, report generation) suddenly creates massive, unpredictable spikes in system load.
3) Queue-Based Load Leveling (decouples the producer from the consumer).
-----------------------------------------------------
3. How to Implement the Change Strategically
The process of adding complexity should be systematic:
Step 1: Establish Monitoring (Observability)
You cannot fix what you cannot measure. Implement logging, metrics (CPU, RAM, I/O, latency, error rates), and distributed tracing to precisely locate the bottleneck.
Step 2: Identify the Root Cause
Determine why the issue is occurring. Is it a slow database query, a memory leak, or a network timeout? The correct pattern is the one that addresses the root cause. Example: If your problem is slow page load time, the root cause might be a database query that takes 500ms. Applying a CDN (network/edge cache) will not help; applying a Database Cache will.
Step 3: Apply the Targeted Pattern
Introduce the simplest pattern that solves the identified problem, usually starting with the less invasive options first. For Latency: Start with caching before considering complex patterns like CQRS.For Failure: Start with a Circuit Breaker on the problematic external call before moving to a full Bulkhead implementation.
Step 4: Measure and Validate
After implementing the pattern, use your monitoring tools to verify that the bottleneck has been eliminated or significantly mitigated. If the problem persists or shifts, return to Step 2.
In summary, the sophisticated patterns you mentioned are tools to be used on demand, not features to be built by default. A strong system designer is one who can delay complexity until the moment it provides undeniable, measurable value.
So my question is, is it true to start off simple and then add complexity as the system needs it?
I understand that things like DevOps (Jenkins, Ansible, Unit Testing Tools, Code Coverage Tools, Static Code Analysis Tools, Docker, Cluster Management Systems) are a requirement and not something to add when needed. I understand that most Cloud Services (like AWSs EC2, Cloud Formation, ELB, Auto-Scaling, ECS, EKS, IAM, RDS, S3 if using this Storage, etc...), most of these Services are a requirement not added when needed. I understand Monitoring Tools like New Relic or Data Dog and ELK are a requirement. I know about Security which is a topic in itself, but also a requirement.
So maybe most things that are taken from System Design, I am using the following patterns from this website -> https://roadmap.sh/system-design . The patterns found there (if you scroll down the web page through the list of patterns) mostly from Cloud Design Patterns section under "Design and Implementation", "Data Management", and "Messaging", and the Reliability Patterns section under "Availability", "High Availability", and "Resiliency". These patterns are the ones mostly that are going to be added as "Complexity" to the Web Application when needed if bottlenecks arise or other issues like Race Conditions, Latency Bugs, etc... is that correct?
Since I know the Patterns that "are a given" meaning they are more than likely a requirement, are like the Caches, CDNs, Queues, Consistency Patterns, Database Selections, (I am using the website for reference the roadmap.sh), I know these are sometimes more than likely a must.
So are the Patterns taken from the website (roadmap.sh) Cloud Design Patterns section under "Design and Implementation", "Data Management", and "Messaging", and the Reliability Patterns section under "Availability", "High Availability", and "Resiliency", are the ones (more than likely) going to get added over time if needed? It just makes sense to me, to see it this way... is all.
Or can someone explain to me how to add these things over time, or is the quote the AI reply Gemini gave me "the sophisticated patterns you mentioned are tools to be used on demand, not features to be built by default.", and should be added to resolve issues like bottlenecks or failures that the monitoring systems are alerting of, is that correct and accurate?
Side Note - I know that for Portfolio Projects, there should be large complex web applications to impress hiring recruiters, if the competency of the Web Developer can allow of such a thing of course. I am talking in a professional setting where it makes sense to start off small and gradually scale the system out.
Level 2
@snapey Thank you for replying to my question, can you please explain why you say this? Should System Design be built UPFRONT when building a Web Application, instead of relying on bottlenecks and then gradually fixing the bottleneck with a System Design Pattern? I am trying to understand System Design, should I review a System as in:
- Gather Functional Requirements and Non-Functional Requirements
- Design Core Entities - Database Design
- API Design
- Create a High Level Overview
- Deep Dive into the System
I have taken this checklist from a YouTube Video from a Ex-Staff Engineer from Meta. But there are so many Patterns (Sequential Convoy, Queue Based Load Leveling, etc...), I am lost at which ones I should add and leave out...
Should I add to the checklist a section for something like, "review each System Design Pattern and see if it applies to my System and ADD it if it does"? And keep on doing that until it satisfies my System Design? Would that be better?
Level 122
The only way to gain experience is to write code, not to write about writing code.
If you want to be the worst type of academic then carry on. If you want to earn a living and build products that people love then start building. You can correct design issues later.
Level 2
@snapey I understand that writing code will make me better, obviously. But my question was regarding System Design. And you just critiqued my question by saying "95% bul...t" and not helping at all. I am trying to understand System Design that way I can write code at a higher level, as in writing a Sequential Convoy pattern for a Web Service, just an example of writing code. But how can I write code for a Sequential Convoy pattern if I don't know how to implement System Design. So how am I not writing code... You are not helping. What was the point in a leaving a comment if you didn't clarify what you meant.
Level 122
What the hell is Sequential Convoy pattern?
Level 80
@june23 You very much seem to be picking solutions and then looking for problems, rather than looking at problems and solving them. Also known as “premature optimisation”.
Level 50
So my question is, is it true to start off simple and then add complexity as the system needs it?
Complexity is never good. You should never seek to increase complexity. You should try to solve problems instead, and do it in the most simple and robust way.
So yes, you should start simple.
Side Note - I know that for Portfolio Projects, there should be large complex web applications to impress hiring recruiters, if the competency of the Web Developer can allow of such a thing of course.
Not true at all. Create a project that you can actually complete. There's no point in spending thousands of man-hours building a "large" project with 500 controllers if your app has zero users. You'll learn nothing about scalability or reliability because you're never dealing with scale in the real world. You'd be solving imaginary problems in a vaccuum.
Nobody will be impressed if your unfinished app has hundreds of tables and a thousand CRUD routes. But if you've completed and polished a small usable product, that's great.
Juggling process buzz words takes you no closer to actually learning anything. When you say things like "... CDNs, Queues and Caches are more than likely a requirement ...", it's an indication that your thinking is backwards. Those aren't requirements for their own sake, they are solutions to problems.
BTW, the URL you linked is about building large scale systems. If you start building your own system with 99.9999% availability, load-balancing, distributed database, automatic fail-over, and everything else listed from the get-go, you'll spend a lot of money and never complete the project.
Level 2
@jussimannisto Okay I understand that simple is the way to go, but how do I know when I should add another complexity? Is that where the use of tools like New Relic come into play? Like when the Alerts from New Relic are telling me that it found something, I can use a System Design Pattern to solve it? Is that correct?
Level 50
Okay I understand that simple is the way to go, but how do I know when I should add another complexity?
You should never aim to add complexity. That's the opposite of what software engineering is about. Try to build a system and solve its problems.
And stop worrying about buzzwords and product names. There are thousands of SaaS companies pushing their own products. If you have a problem that their app solves, then by all means spend your money, otherwise there's no point in investing time in it.
I can use a System Design Pattern to solve it? Is that correct?
Use your eyes, brain and fingers. "System Design Pattern" means nothing.
Level 14
(like AWSs EC2, Cloud Formation, ELB, Auto-Scaling, ECS, EKS, IAM, RDS, S3 if using this Storage, etc...), most of these Services are a requirement not added when needed - all of that is optional tho, you don't need every AWS service to run a server. The cloud saved us so much money we can almost afford the team to run it mentality
Please or to participate in this conversation.