I recently asked a question on here about System Design, and nobody left a reply, not sure if it was because I added a giant wall of AI Answer and nobody wanted to read all of that, but I am hoping that, that is why nobody answered. This will be my final try at asking this. Since it seems relevant to programming, and I am explaining in my own words this time.
Just so it is clear, I know that for System Design one should:
Gather all Functional Requirements and Non-Functional Requirements
Figure out Capacity Estimates
Design Core Entities or in other words Database Design
API Design
Create High Level Overview
Deep Dive into the actual Architecture
That was a checklist from a YouTube Teacher (Ex Meta Staff Engineer).
There is probably other ways to unravel the core parts of a Web System, all the Functional Requirements and there specific needs, like a Database Cache, or a Queue, or Load Balancer, or a Replication for Databases... I have learned there are two ways of seeing this
Big Design Up Front (BDUF)
YAGNI
Meaning either you create a Perfect System from the start = BDUF!
Or you start with the basics, MVP features, less System Design = YAGNI
But my question is, even if you choose which ever philosophy you choose (in my case I choose YAGNI, but leaning towards a hybrid approach) how do you implement the more complicated patterns, or a better way to ask what I am asking is, when to choose a Complicated pattern like Queue Based Load Leveling or Sequential Convoy or Leader Election or CQRS? Should I keep iterating over a list like this for each Web Service (Timeline Service, Message Service, Post Service as examples) -> https://roadmap.sh/system-design until the Web Service handles all the assumptions like will this Web Service need a Load Balancer starting with N amount of users, for example, or should I implement a Sequential Convoy Pattern since my Messages need to all be in order for some reason. My point I am trying to make, is should I know all of these patterns by heart and know when to use it when the time arises? Or do I look up information when a the System runs into a bottleneck or when I am planning everything up front.
That is my first question... Should I keep iterating over a list of System Design Patterns -> https://roadmap.sh/system-design for a specific Web Service say the Timeline Service for a Social Network, until every corner has been thought of, and then move on to the next Web Service.
My other question to this first question is, should I memorize all System Design patterns so I can implement the Pattern on demand in any specific context, either in BDUF meaning I can work like a machine as I plan the System and throw every System Design Pattern at the Web Service there is until satisfied... or either in YAGNI when a New Relic Alert pops up I know what to do. Or is looking up information okay to do, like on here or Googling stuff up, in a professional setting.
Another question is should I start to add System Design patterns Up Front following the BDUF Philosophy... or should I use the YAGNI Philosophy. Or a Hybrid Approach.
Another question is even with either or or the Hybrid Approach, the correct way to know when to add another System Design Pattern say the Competing Consumers Pattern is by first receiving an alert from an Instrumentation Tool let's say New Relic, and solve that bottleneck or failure from New Relic by using a System Design Pattern. Is that accurate? Or why else would there be a need for implementing a new System Design Pattern let's say 2 years into production if not by the Observability System like New Relic complaining about high Disk I/O... Is that correct or am I mistaken?
I understand that things like DevOps (Automation Servers, Configuration Management Tools, Unit Testing Tools, Code Coverage Tools, Static Code Analysis Tools, Docker, Cluster Management Systems) are a requirement and not something to add when needed. I understand that most Cloud Services (like AWSs EC2, Cloud Formation, ELB, Auto-Scaling, ECS, EKS, IAM, RDS, S3 if using this Storage, etc...), most of these Services are a requirement not added when needed. I understand Monitoring Tools like New Relic or Data Dog and ELK are a requirement. I know about Security which is a topic in itself, but also a requirement. And my point to this is, the tools listed above, besides maybe some Cloud Components like Storage or Load Balancers, etc..., aren't apart of System Design right? They are all required in modern Software Development and should not be "chosen" for.
