I was in a similar situation, I ended up moving to seaford which is a much better town than newhaven. Slightly more expensive tho
Certainly the town centre is better but we looked at some shithouse houses there which were in the middle of a vast grim estate and miles from any shops or anything else...