Completely agree, walk around places like Mitcham, St Helier or Colliers wood in the suburbs and they feel very much like Chelsea strong holds and were that way long before Abromvich's reign. This obviously impacts on attendances of the established South London clubs but still for such a large population to not have a local 'big' team is strange IMO.
If you look at outer South London with Sutton, Croydon and Bromley there's a combined population of around 1.5 million and plus the catchment of North Surrey yet the only league team there is Palace who only a mid sized club who generally attract sub 20k crowds.
Bromley is in Kent and I would say is a Charlton area, also Millwall are followed by a few all around SE London.