Tesla K40 in HP Z600 - Code 12 error

[DrNewcenstein]

First things first: I have Googled this matter for about a month, and the results and answers I've gotten thus far compel me to include the following preface:

1. Yes, I know the Nvidia Tesla K40 is intended to be used in servers with giant noisy fans as they have none of their own.

This is not the issue, and there are documented workarounds (i,e, liquid cooling, etc). I do not need to be told this again and again because it is not the issue and absolutely irrelevant to the matter at hand.

2. Yes, I know the Tesla K40 cannot be sufficiently powered by the Z600's default PSU. I do not need to be told this again and again as it is not the issue and is absolutely irrelevant to the matter at hand. I have an external PSU that exceeds the K40s requirements. Yes, this means the side panel of the Z600 is off. It's not the issue.

3. Yes, I'm well aware the Tesla K40 does not have outputs. I knew that before I bought it. I verified it when it arrived. I do not need to be told this again and again. It is not the issue and is irrelevant to the matter at hand.

4. I do use DisplayDriverUninstaller (DDU) to swap out GPUs. This is not the issue.

To the matter at hand: 

I have a later-revision HP Z600 Workstation with dual X5650 Xeons running Win 7x64 Pro and upgraded to 48GB of ECC RAM.

I replaced the fx1800 GPU with a K4000.

The BIOS is 3.61a (IIRC - whatever the latest one is. I ugraded it 2 months ago or more when it actually became relevant to me).

THE ISSUE >> I'm trying to add a Tesla K40, but keep getting Code 12 errors (Cannot find enough free resources). << THE ISSUE

If I replace the K40 with a GTX980, the machine works fine.

If I replace the K40 with a 1080ti, the machine works fine.

If I replace the K40 with a Titan Z, the machine works fine. Keep in mind the Titan Z reads as 2 GPUs, and enough resources are found.

All of these are also powered externally.

If I remove the K4000, leaving only the K40, and access the machine remotely through RDP, Device Manager still gives a Code 12 error.

The K40 requires the PCIe slots to be x16(x16). The BIOS shows Slot 2 and Slot 4 (the two longer slots) as x16(x16).

However, is this a realtime assessment or is it simply telling me what the spec default is? Are these slots being cut down to x16(x8) when both are populated, as with my ASUS Z87 Pro mobo (whose BIOS does give a realtime assessment - showing the actual reading when multiple slots are populated)?

GPU-Z sees it but cannot help as it requires resources to be assigned to read it properly. Exporting the BIOS from the card using GPU-Z causes a Plug-and-Play BSOD. Sometimes, even installing drivers causes BSOD.

Nvidia-smi.exe does nothing because it requires resources be assigned.

All evidence so far points to having two defective Tesla K40s (bought from an Ebay seller with 10K+ transactions and high feedback, but things do happen). I would rather not smash $450+ in GPUs with a sledgehammer, as I'm not going to re-sell broken GPUs.

Before I get to that step, I need to know if the Z600 is capable of running a single Tesla K40 at all. If not, it's the Z600. If so, it's the GPUs.

Do I need Windows Server 2008R2 instead of WIn7x64 Pro?

[DrNewcenstein]

Ok, I have the answer from Nvidia (finally).

I found this thread:
https://devtalk.nvidia.com/default/topic/1039289/cuda-setup-and-installation/k40-setup-on-lenovo-p51...

And after asking if server OEMs install some sort of BIOS lock or other code which limits the Tesla K40 to the OEM's system, I got this response:

It's a system incompatibility. Tesla cards (with a few historical exceptions, e.g. C2075, K20c, K40c) are designed to be purchased and installed (only) in an OEM server system certified for their use. HP does not certify any of their workstations for any current Tesla cards, nor were any ever certified for K40m usage. If you buy a Tesla card believing you can install it in any system you want, you are asking for trouble. It is simply not possible, in the general case, and there is no design intent to make it possible.

There is no documentation to support this configuration. There is no OEM system load or card lock. The resources in question are resources that would be assigned to PCI BAR regions by the system BIOS during the PCI plug-and-play enumeration process. The K40m requires a large complement of resources. A system that cannot or will not assign these resources will cause the cards to be non-functional. There is nothing you can do to fix this (barring modification of user-accessible BIOS settings that modify the BIOS resource assignment behavior).

A server designed to support this card of course has taken these requirements into account in the design of the server, which includes the design of the system BIOS. It's not a "lock" of any sort. In most cases, a PCIE Tesla GPU can be easily enough removed from a supported HP server configuration and placed in a supported Dell server configuration (just to pick a random pair/example), with full expectation that it should work normally.

But your workstation is not a supported configuration for that GPU. There are many statements like this on these forums.

Tesla K40 is an obsolete product. For non-obsolete products, you can find supported server configurations here:

https://www.nvidia.com/en-us/data-center/tesla/tesla-qualified-servers-catalog/

There is no suggestion anywhere that Tesla cards can be placed in any system you want with an expectation of proper behavior.

So now the question becomes "Can I install the BIOS of an HP server on an HP Z600?"

Alternatively, is there a "BIOS hack" I can perform to affect the BAR mapping required? I've done the "headless" mod to the BIOS to effectively enable RDP, and that worked well enough, but I'm not seeing anything that references BAR or large memory addressing.

I do see "Network Server Mode *Disable". At the risk of getting an absolutely idiotic and simplistic response, what *exactly* would changing this to "Enable" do? Would I need an actual Server OS or BIOS to take advantage of this setting? Would this open up large memory addressing like a server, or is the chipset merely incapable of handling this?

[huxleyhound]

Hi,

You will NOT be able to put this card in a Z600. The power supply does not support it and you cannot put a "server" bios on the system. It will not work. You will need something like a Z800 with a 1100w powersupply with both front fans to run it. I have three Z800's with K40's and Nvidia 5800's in them. I have tested the K40 in a Z600 and there is no way it will work.