What is wrong with what you have; it is a relatively short Controller action that organises its information clearly, and passes it to the view?
You could optimise to ensure you are getting todo that are due within the three dates only:
$all_todos = Auth::user()->todos()
->whereBetween(\DB::raw('DATE(due_date)', [Carbon::yesterday(), Carbon::tomorrow()])
->get();
Otherwise, you could try to get clever with the Collection using groupBy but the work required to get the keys computed correctly would not be worth the effort in my opinion.