So apparently this works perfectly fine. I just got it to work, and came back to update this post, and it was exactly the same I probably didn't have the screen and N64 grounded together. So make sure they are!
Note: This will only work with revision 3 boards and earlier. Revision 4 boards still have the VDC-NUS chip, but the parts to get the c-sync were not put onto the board. If you can get the parts to fill in the missing ones on the Rev 4 board, then this should work.
References:
http://forums.benheck.com/viewtopic.php ... 9&start=21" onclick="window.open(this.href);return false;
http://www.gamesx.com/rgbadd/rgbn64.htm" onclick="window.open(this.href);return false;
http://pinouts.ru/Game/n64video_pinout.shtml" onclick="window.open(this.href);return false;