Help with grouping and sorting on relationships.

The code below works great when I am trying to limit records based on date. However, when the last part happens (the 2nd code snippet below), it is causing issues.

When I groupBy('date'), I lose the dates that I filtered during the first part of the code. If I groupBy('employee'), which is what I want, I get all dates. This tells me that the second part (listed below) is bringing back all data instead of the ones I want.

How do I keep my date filter in the 2nd part?

1st part:

        $employeePunches = $this->employee->with('punches', 'user')
            ->has('punches')
            ->whereHas('punches', function (\Illuminate\Database\Eloquent\Builder $query) use ($startDate, $endDate) {
                return $query->where('date', '>=', $startDate)
                    ->where('date', '<=', $endDate);
            })
            ->withSum('punches as total', 'total_time_converted')
            ->withSum('punches as ptoTotal', 'pto')
            ->withSum('punches as overtimeTotal', 'overtime')
            ->withSum('punches as holidayTotal', 'holiday')
            ->get();

2nd part:

        $employeePunches->transform(function ($employee) {
            $punches = $employee->punches->groupBy('date');
            $employee->setRelation('punches', $punches);
            return $employee;
        });