iheb_jabri's avatar

Optimizing Real-Time Polling Architecture with Redis and Laravel WebSockets – Is There a Better Way?

I'm building a live betting platform where I need to poll an external API every second (only supports HTTP pulling). To reduce latency and decouple the client from the API, I designed the following system:

A Node.js service fetches data every second from the external API.

The data is stored in Redis (used as a caching system).

My Laravel backend checks Redis for changes and broadcasts updates to frontend clients using WebSockets (or Pusher).

This approach reduced client latency and removed heavy API calls from the frontend. 👉 I’m looking for feedback: Is this the most stable and scalable way? Would you suggest alternative designs or optimizations to make the system stronger or more efficient?

Thanks for your time and insights!

0 likes
2 replies
OussamaMater's avatar

It really depends on how complex you want it to be. You can always take it a step further, but is it really worth it? That's something you need to consider.

In any case, regarding improvements to your current system: I believe the API doesn't offer webhooks and you have no control over that, but you do have full control over the Node service and your backend. I assume there's currently a scheduled command that checks Redis for changes. Instead of polling, you could notify your backend, using internal webhooks, pub/sub, or similar, whenever the data actually changes.

But again, is it really worth it? Added complexity comes at a cost, and you are probably better off keeping things simple until you truly need the upgrade.

That said, we have done something similar in the past, and I would keep it even simpler by consolidating everything into the same Laravel app. I dont see a need for a separate Node service unless its a business requirement. You could just run a background command as a daemon (similar to how Reverb works), and in case of changes broadcast an event to your frontend.

martinbean's avatar

@iheb_jabri A large betting website here in the UK used to (not sure if they still do) work by calculating odds, storing them as a JSON file, and then clients requested that JSON file from a CDN. It was much faster (and cheaper) than retrieving odds from a database, or having a socket server or whatever that needed to send out events to however many clients were connected at that time.

In your case, I don’t think delivery of odds is the problem, it’s going to be the source of this data. If you’re scraping odds providers’ websites against their terms, or hitting APIs every second, without explicit permission from these providers to scrape their data, then you’re very quickly going to be rate limited or flat-out blocked, especially if you’re hitting them as frequently as every second.

Please or to participate in this conversation.