In my opinion the best option is to do it in a more CRUD way. Then you can use your api with basic crud operations. I think it's better to have 2 routes because they're doing different thinks, and it will be better when setting the roles for the different routes:
Route::get('/department-users/{department}', function() {
return $department->users();
});
Route::get('/users', function() {
return User::all();
});
Route::middleware(['scopes:admin'])->group(function () {
Route::post('/departments', 'DepartmentController@store');
Route::post('/department-members/{department}/store', 'DepartmentMemberController@store');
Route::post('/department-members/{department}/destroy', 'DepartmentMemberController@destroy');
Route::post('/department-managers/{department}/store', 'DepartmentManagerController@store');
Route::post('/department-managers/{department}/destroy', 'DepartmentManagerController@destroy');
});
This way you will only have basic crud operations organized between all the controllers, and you will only have 5 operations max per controller so it will look very clean.
Adam Wathan has a good laracon video explaining this: https://www.youtube.com/watch?v=MF0jFKvS4SI