The biggest downside for me is not the battery life: I’ve actually managed to tweak that to my satisfaction, including suspend life. It’s pretty peachy now, but maybe I don’t have big enough expectations? Details in my review.
The biggest downside is the lack of ports. The framework only has four ports, and that’s kind of a hard limit. If you’re going to have two monitors, the SSD device and power, boom, you’re out of ports already. There’s no room for a 2.5" drive in there, so just converting from my previous laptop (a Purism Librem 13v4), I quickly ran out of ports.
This means dongles and so on, which might not be your cup of tea. The expansion ports are really nice, and there’s a really cool modding community around it, but it also means carrying around half a dozen of those things, so it’s kind of dongle hell all over again, except kind of different. 
Don’t get me wrong: I love my laptop, this is an awesome product. But that’s the downside I find…