High performance wrkstations like your xw 6400 have very specific requirements for installed memory.
Did you purchase guaranteed compatible memory?
This is from the Hardware Service and Technical Reference manual for your workstation. You did not follow the requirements! All of the modules must be the same capacity and type. That is, quite specifically, what you did not do. Correct that and all should work as expected. Four identical PC2-5300F DIMMS from the same manufacturer should give you 32 GB of installed memory. Read the reference guide carefully. Remove the 2GB module and put the 8GB module in slot 1.
MMemory module requirements
Use only industry-standard, registered PC2-5300F DIMMs
From the HP QuickSpecs for the xw6400 the maximum memory listed there is 16GB. Erico is giving good advice that it is best to not mix in different sizes of RAM, or at least to have the two banks of ram with identical sizes and HP certified.
A HP engineer who helps us here, Dan_WGBU, has stated multiple times that the very best way to go is to have all memory slots filled with identical RAM. That way you get the added benefit of the memory controller chipset being able to run the RAM most efficiently, with an extra boost in performance. So, for you the best would be 4 identical 4GB sticks to get to 16GB total. Why not 8GB x 4 slots in your xw6400?
Well, I tried that. You may not know that the xw8600 is certified to run specific 8GB sticks. I did research and found that the memory controller for the xw6600 and the xw8600 is the same, but it is different from that of the xw6400. I purchased some of those 8GB official HP sticks of RAM, and tried them in my xw6600..... worked perfectly even though the xw6600 is not officially certified to run those 8GB sticks. Sometimes parts for one HP workstation will work in another but HP did not take the time to test that configuration deeply enough to officially certify that part for both.
But, when those same HP 8GB sticks were moved to my xw6400 test workstation they would not work at all (presumably due to it having the older type of memory controller).