that's because its a racist, eliteist, boring sport played by white men.
Golf is really popular in East Asia, too - Japan, South Korea, China.
In Japan, it's not quite as big as it used to be, as companies' expense accounts aren't as inflated as they were a few decades ago. However, it's still very popular, both for business connections (as Pimpernel Smith pointed out) and for relaxation.
When we fly back from Japan after visiting family, it's very common to see Japanese people with either golf clubs, or surfboards, waiting to board the plane. Australians fly to Japan to ski; Japanese fly to Australia to play golf or surf.