Get sum of field in belongsTo from many-to-many

Let's say I have a typical many to many relationship between Tasks and Users. A Task belongsToMany Users and a User belongsToMany Tasks. I have the typical tasks, users, and (pivot) task_user tables. I also have a one to many relationship between a User and Teams. A User belongsTo a Team and a Team hasMany Users. The Team also has an integer field/attribute for points. I have these relationships setup in my models, working the way I'd expect.

If I want to display statistics about a specific Task, one thing I might want to do is to get all of the Users that belongTo the Task, then find the sum of the points for the team they belong to. As an example, a specific Task might belong to three Users. Two Users are on Team A which has 3 points and the third User is on Team B which has 5 points.

I'd like to be able to have a method on Tasks called usersTeamsPoints() so I can write something like $tasks->usersTeamPoints and it will return 11. (Users from Team A is 3+3 and the single User from Team B is 5 points.)

I can't quite figure out how to add this to my Tasks model. I've tried some variations on hasManyThrough but I can't seem to get the collection of Teams to add the points.