order comments by amount of points they got

I have 3 tables Gallery, Comments and Votes, my votes table is polymorphic since votes can be added to either gallery or a comment, they look like this:

Comments:

id
user_id
gallery_id
content

Votes:

id
user_id
votable_type
votable_id
status

status can be 'upvote', 'downvote'. Each comment can be either upvoted or downvoted. Amount of points comments has is calculated from count('upvote') - count('downvote').

I want to get all the comments and order them by amount of points, comment with highest amount of points being at top.

How can I do this? so far I have this simple code but it does not do any counting of votes:

Comment::with('votes')->with('owner')
                                            ->where('gallery_id', $gallery->id)
                                            ->paginate(5);