I messed up my nvidia drivers quite a bit while trying to enable vdpau, to the point openGL was not working anymore and the display was suffering from many bugs. I decided to remove and reinstall the nvidia drivers.
sudo dpkg-reconfigure xserver-xorg
select "nv" for the first screen and hit enter until it's done.
Then you can
sudo /etc/init.d/gdm stop
sudo /etc/init.d/gdm start
and it should let you back in X, without nvidia proprietary drivers.
To clean up completely:
sudo apt-get --purge remove nvidia-glx* nvidia-kernel-common nvidia-settings
sudo rm /etc/init.d/nvidia-*
sudo update-rc.d nvidia-kernel remove
To reinstall a stable version:
sudo apt-get install nvidia-glx-180