Hi @ninjstin
It would not be hard I believe but you may miss some very interesting features about the notifications and other goodies these 2 versions are bringing.
Check the release notes to figure out if you need one of these. https://laravel.com/docs/5.4/releases
Updating from 5.2 to 5.3 was more disruptive for me than going from 5.3 to 5.4 (according to my own experience and needs of course)
What provider do you use for your server? Since it uses a 5.6 branch I do not think that upgrading to php >= 5.6.4 would be an issue (unless working in finance...)