Laravel Pulse - Multiple Aggregations from a single recorder?

I'm using Laravel Pulse to try and paint a picture of where the traffic is at, across multiple dimensions:

The most requests paths (i.e. example.com/xyz has 1K reqs, example.com/123 has 5K reqs, etc.)
The most requested domains (i.e. example.com has 6K reqs, lorem.com has 10K reqs, etc.)
The most requested API endpoints (i.e. /api/abc has 1K reqs, /api/def has 2K reqs, etc.)

Presentation wise, I'd like to show all of the above on separate cards for maximum visibility.

I know that I could create 3 separate recorders, and use the pulse 'count()' aggregation to achieve the desired results.. however it feels "heavy" to record what is essentially the same data, 3 times, just for the sake of different aggregations.

Is it possible in pulse to use a single record, which captures all of the URI elements, and then use the built in count() aggregation to aggregate on 3 different groupings?

Currently, the implementation with the 3 separate records looks something like:

// PathRecorder.php
$this->pulse->record(
    type: 'path',
    key: json_encode([
        $request->method(),
        $domainAndPath,
        $via
    ], flags: JSON_THROW_ON_ERROR),
    value: $duration,
    timestamp: $startedAt,
)->count();

// DomainRecorder.php
$this->pulse->record(
    type: 'domain',
    key: json_encode([
        $request->method(),
        $domain,
        $via
    ], flags: JSON_THROW_ON_ERROR),
    value: $duration,
    timestamp: $startedAt,
)->count();

// ApiPathRecorder.php
$this->pulse->record(
    type: 'api_paths',
    key: json_encode([
        $request->method(),
        $apiPath,
        $via
    ], flags: JSON_THROW_ON_ERROR),
    value: $duration,
    timestamp: $startedAt,
)->count();

which works but records a lot of data, and If I wish to create more slices, i.e. most requested subdomains, or querystrings, etc etc, then the recorded data per-request starts to get a bit daft.

I'm aware that I could throw the data into a separate table and query that in real-time, but keen to see if Pulses powerful aggregation mechanism can achieve the desired results

LaryAI

1 year ago

Level 58

To achieve multiple aggregations from a single recorder in Laravel Pulse, you can record the data once and then use different groupings to aggregate the data as needed. This approach will help you avoid recording redundant data and keep your implementation efficient.

Here's a solution that demonstrates how to record the data once and then perform different aggregations:

Single Recorder Implementation: Record the data once with all necessary information.

// UnifiedRecorder.php
$this->pulse->record(
    type: 'request',
    key: json_encode([
        'method' => $request->method(),
        'domain' => $domain,
        'path' => $domainAndPath,
        'api_path' => $apiPath,
        'via' => $via
    ], flags: JSON_THROW_ON_ERROR),
    value: $duration,
    timestamp: $startedAt,
)->count();

Aggregation Logic: Use different groupings to aggregate the data based on your requirements.

use Illuminate\Support\Facades\DB;

// Most requested paths
$mostRequestedPaths = DB::table('pulse_records')
    ->select(DB::raw('JSON_UNQUOTE(JSON_EXTRACT(key, "$.path")) as path'), DB::raw('count(*) as count'))
    ->groupBy('path')
    ->orderBy('count', 'desc')
    ->get();

// Most requested domains
$mostRequestedDomains = DB::table('pulse_records')
    ->select(DB::raw('JSON_UNQUOTE(JSON_EXTRACT(key, "$.domain")) as domain'), DB::raw('count(*) as count'))
    ->groupBy('domain')
    ->orderBy('count', 'desc')
    ->get();

// Most requested API endpoints
$mostRequestedApiPaths = DB::table('pulse_records')
    ->select(DB::raw('JSON_UNQUOTE(JSON_EXTRACT(key, "$.api_path")) as api_path'), DB::raw('count(*) as count'))
    ->groupBy('api_path')
    ->orderBy('count', 'desc')
    ->get();

Presentation: Display the aggregated data on separate cards for maximum visibility.

// In your Blade template or frontend code

@foreach ($mostRequestedPaths as $path)
    <div class="card">
        <h3>Path: {{ $path->path }}</h3>
        <p>Requests: {{ $path->count }}</p>
    </div>
@endforeach

@foreach ($mostRequestedDomains as $domain)
    <div class="card">
        <h3>Domain: {{ $domain->domain }}</h3>
        <p>Requests: {{ $domain->count }}</p>
    </div>
@endforeach

@foreach ($mostRequestedApiPaths as $apiPath)
    <div class="card">
        <h3>API Path: {{ $apiPath->api_path }}</h3>
        <p>Requests: {{ $apiPath->count }}</p>
    </div>
@endforeach

By recording the data once and then using different groupings for aggregation, you can efficiently achieve the desired results without duplicating the recorded data. This approach also makes it easier to add more slices in the future without significantly increasing the amount of recorded data per request.

Please or to participate in this conversation.