@nutickets Web servers like nginx and Apache already log requests? So you could create an ingest from those logs and avoid a database altogether.
Strategy for logging HTTP requests
We are looking into ways to log HTTP requests made to our Laravel API and wanted to see how others are dealing with this: we currently have a middleware set up which put simply, logs the request and response in the primary MySQL database using eloquent, within the terminate method as to avoid any kind of interruption to the request itself.
This approach works well until it doesn't, i.e. when our API gets busy it gets really busy and those writes end up bottlenecking the entire platform. We are looking for a more "scalable" solution.
Couple of thoughts so far:
- Use an alternative MySQL database, to remove the load from the primary. This would be nice because it involves no code changes aside from referencing a new connection name. Maybe a costly solution and what happens if that database cannot handle the traffic? (We'd have to worry about scaling another MySQL instance)
- Push the logs into one of our existing cache stores and then sync that data into the database on a cron to slow/control the write throughput. Potential risk of data loss using this mechanism and still needs some consideration around scaling of the cache store, but feels like a simple implementation
- Use CloudWatch agent to stream logs into CW logs - we already use CW agent for nginx/fpm/laravel error logs so it's a possibility. My main hesitation here is fear of filling up the server, as our API is used to stream fairly sizeable payloads in near realtime to our mobile applications, we'd most likely be talking GBs of data within hours. Also testability of this approach is limited as it cannot be tested (easily) in a local environment.
- Push realtime logs (over http) into X/Y/Z cloud logging service (this could include CW, just without the agent bit). There would be a HTTP call made for every request, which may or may not be a problem, it may be costly though perhaps not, haven't looked too much into this yet.
- A NoSQL solution, maybe DynamoDB or similar? Cost considerations as always here too. both maintenance, testing and actual running/storage costs
Another thing to consider is that these logs are currently viewable in a custom filterable UI wired up to the same eloquent collection that captures them, so I suppose it would be nice if we could utilise that regardless of implementation method, but it's not a dealbreaker as long as there's a way to navigate and filter the request logs based on any of the attributes we store.
Please or to participate in this conversation.