That's definitely what I have in mind when I think about it.
I waffle on the idea a lot, so I'm not vehemently opposed. Indeed, this thread has me looking again and trying to convince Mrs. Rambler to take advantage of the current J. Crew sale so I can maybe piggyback a shirt. Still, there's something about them that has always been iffy to me.
FWIW, I recall you posting a pic on Facebook one Easter when you were rocking the seersucker suit, and I was envious of how well you pulled it off. I generally feel costumey when I don the full suit and prefer to wear it as separates, so you may be a more advanced-level seersuckerer than I am and, therefore, better able to make a shirt work.