Hello Herschel
Unfortunately we have not validated any audio or video cards to work in the 860 so I am unable to provide suggestions. Regarding the memory, you can install 4x2GB DIMMs for a total of 8GB. You have an 800MHz FSB CPU, so the memory speed max is 533MHz. This is the 2GB DIMM that is compatible for your configuration:
F6806
Adding audio and video cards to our servers is a process of trial and error. If no one responds with items they have tested to function then i would suggest searching online for anyone that has tried adding audio/video cards to the 860.
Thanks