I would think that "apt-get remove XFree86" would get rid of most of it as most anything X related should be dependent on that. You might also have to remove base gnome/kde libraries to get the bulk of those windowing systems (like "apt-get remove kdelibs").
Sometimes people install X on the server but don't actually run it on the server. They run the X apps on the server and display them on the X server running on their local machine.