I've just compiled and installed the latest NVIDIA driver on both the 2.6 and 2.4 kernels, and dear god, it's fucking fast.
Found a few caveats with the driver, though. Apparently, you can either have direct rendering (I.E., use OpenGL) using X, or console.
That's right: use nvidia frame buffering (that's nvidiafb.o, I think) and forget getting direct access using X, "Goodbye, gltron. Hello, tiny console fonts."
Also, make sure that your bios allocates an IRQ to the video output, otherwise Bad Things™ happen. (I've had to fsck more times today than in the last 3 months

Finally, I'd suggest that you install the drivers available on nvidia's website. (AMD's x86-64 version available here)