I'd always look at Swanny as a seabreeze is usually stronger offshore. Albeit only 1-2km offshore, but pretty good chance Garden Island reads higher than what you'd be actually sailing in?
Correct me if i'm wrong here, but i think the anemometer for Garden island is located on the top of the causeway bridge. This will obviously give a false reading as to the wind speed at ocean level.