Most of your core styling will apply to all screen sizes, so by starting with mobile, these styles can easily cascade (hence Cascading Style Sheets) from mobile to desktop.
Personally, when implementing a design, I find it best to write my html markup to suit the tablet or desktop (depending on the complexity of the project. Most drastic differences are normally between the mobile & tablet). Once I have clean, semantic html markup, I'll then begin to style, normally working with mobile first. If you follow BEM/OOCSS or MVCSS you're probably breaking your styles out into components, eg. header.scss and slider.scss etc. which will certainly make your life easier. I suggest that you use Browserify to structure your JS files & folders much like your styles.
For me, the best advantage with mobile first, is that if a client decides against a responsive website, by taking a mobile first approach, if (and when, they normally change their mind further down the road) a client requests a responsive version further down the road, the bulk of the work has already been done.
Having said all of that, I wouldn't put too much weight on existing Frameworks. Personally I find that Foundation with a good OOCSS approach gives me just enough freedom. Really, the most important thing you need is a good solid grid. The rest you can add on top later.