You can spend £80-100 on a decent stylus and cartridge.
That's about a minimum price for a decent cartridge, and they can go all the way up to quite silly prices.
http://www.stoneaudio.co.uk/?product=koetsu+jade+moving+coil+cartridge
For turntables, the choice is immense, and there is a thriving second hand market, but not much value to be had. Vinyl is still very much alive in the hearts of hifi enthusiasts.
Look for anything with a three point suspended chassis, belt driven platter from a mechanically remote motor, and the ability to change tonearms.
Names I remember from my past obsession include Rega, Thorens, Linn, Oracle, Mitchell... but there are many, many more.
Once you have your turntable, you need to place it on a rigid support, either a shelf or table, most people spike the table legs, to give a solid conection to the floor.
...but you also have to decouple the turntable from the surroundings a little. I use a little blu-tac under the turntable supports.
Much has been written about whether vinyl sounds better than CD, and vice versa.
It's mainly about opinions, but the fact remains, that in the process of digitisation to CD format, some of the sound is thrown away. Science says that all that is thrown away is inaudible, but I think that some music/sounds are felt rather than simply heard with the ears.
As a rule of thumb, CD sounds far better than vinyl on cheaper systems. On more expensive hi-fi vinyl sings.
Oracle Alexandria/Mission 774SM/Ortofon MC15 feeding a Naim 32.5/250/HiCap into a pair of Linn Isobariks.