The beach at Brighton runs from NNE to SSW, maybe a few degrees closer to N/S.
Due North at Brighton is cross-offshore. You need a little bit of West for Brighton to be benign.
It looks like the houses overlooking the beach form a bowl. The wind hits the bowl and is deflected dead offshore. So if you dick around on the beach or stay too close inshore you are going to have difficulties.
Years ago a kiter was fiddling with a kite on the beach, fully dressed, no leash. A gust lifted him up and deposited him 50m out in the water. A kite was once blown off the beach. The pursuing kiter just missed it at Dendy St, again at Green Pt. By Sandringham it was a km out to sea and accelerating.
More importantly, if you have a mishap and get blown directly downwind the next land from Brighton is at Dromana, 45km away.