Darkside
for starters you need to acquire a navy/blue blazer/odd jacket in wool whether that is flannel (which many including The Manton will poo poo) plain weave (hopsack, fresco) or worsted (wool, wool mit cashmere etc,) This will be very flexible in terms of combination with slacks as well as across seasons. One must be mindful of fulfilling the SecurityGuard™ staple look, which in all seriousness can be artfully accomplished without looking like an ancillary member of the Thin Blue Line brigade.
You dress casually so I would suggest not-blazer to avoid metallic buttons which would limit its use in your case unless I am wrong.
Additional blues - one or two more of varying shades, fabrics, with or without pattern (eg. windowpane) add additional flexibility. Consideration can be given to tweeds (eg. thorn proof or donegal) or faux tweeds (worsteds that approximate the look of tweed but in a worsted usually of lesser weight)
Colours in the brown range (dark, mid, tans etc.) lend well to odd jackets and casualness. A brown herringbone with an overcheck adds some interest. Once again tweeds are ideal for your climate but faux tweeds accomplish the same thing in lesser weights and are more refined if you eschew the tweediness of tweed although as mentioned by others there are softer tweeds like Shetlands in medium weights.
Hounds/dog/puppytooth fabrics can be dressed up or down and are very flexible. They work in earthy tones of browns/tans/greens as well as blues with complimentary colour checks resulting in very good flexibility. Gunclubs as well but I tend to dislike most that are currently in favour because they are in too large patterns and bolder colours that scream "salesman". Take a look at IS' messed up cuff with IMO suck ass colour to know what to avoid.
A grey odd jacket would be normally down the list when acquiring staples but for you might rank higher up as you may wear them with denim and more casual cotton trousers in shades like Robertito posted. Be careful of going too light in colour and not having enough surface texture and too much plainness. A plain worsted light grey is not as flexible as one might think.
More when you provide additional information/comments