I'd say it depends on the amount of css/js you are sending over the wire and how much of it is used on the page.
As @tomi said, the loading of a big bundled file will slow down your users first visit. Studies have shown that longer load times results in an increased amount of users abandoning their visit to your site. Especially when they're first time users who don't yet know what your site has to offer.
Let's say on your homepage you're serving a js bundle of which only 10% is actually used on that page. The other 90% will still be loaded.
This could result in losing users who will never get to benefit from that faster, cached, second request because they're already gone.
Another point to watch out for is bundeling up things like jQuery. Every time you change something in your JS your users would be downloading the whole library again because your bundle changed.
Those are some downsides to think about before bundling all your files together.