just because you don't want to button it, doesn't me you shouldn't be able to. That's not to say that all jackets need to be loose enough to button up over chunky knitwear, but i don't think i would wear chunky knitwear under a jacket that was to small too button. either get a bigger jacket or don' wear that sweater with it.
Still seems to be an awkward and roundabout way to say that it is a bad fit. Especially since a buttoned-up sport coat with a chunky jumper is a contradiction in terms, except in the world of iGents I suppose.