is an aggregate query select count(*) from users, while
User::get()->count();
is a count on the collection of all users (models) fetched from the db.
That said, depending on the size of your table, the difference might be insignificant, but also could be a real issue. Loading thousands of eloquent models into array is going to make the difference.
Then I don't think you should be overly concerned about this. The query builder is technically faster. Realistically though, you won't notice a difference.
If you are truly concerned with performance (many thousands of records) you should think about using a raw statement that only counts the primary key indexed field, but as said, it's only marginally better than Eloquent's implementation (in most cases milliseconds or less difference).
If you have enough records that this actually becomes a concern (say > 100K records), you should try storing this in cache and purging that cache entry every time the table record count is changed.