As Swalkington said swell direction is important but I think wave period or frequency plays the biggest part for size. I remember a surf I had at a particular place on a 1.3metre swell that with the average 10 second frequency would have an over head high wave on the sets. That day the frequency was 16 which is pretty unusual and it was double head and a half on the sets. That day cemented in me the importance of wave frequency.
Edit ^^^^ Just read your post Ted and I agree