Lived in Manly for 8 years
Sailed the Queenscliff Bombie 3 times
1 of the best sails I've had on the eastcoast... and two of the worst !
As Ben said, rig big... and expect to break gear.
It is also possible to sail the queenscliff side of North Steyne (north of the surf club near the pipe)
Sailed that 12 or so times... but it has to be a cranking nor easter... and if it's that windy you're better off making the trek to Long Reef and getting out to Butterbox.
In a big southerly blow Makaha is the go. You can launch off the point in a channel but will need to carry your gear down the hill (through the golf course)
Bit of a hike but much easier than tacking up from the vacant lot... and a lot less further to walk if you break gear