HP Z840 Clean Install

[BambiBoomZ]

CinéMatica,

Very pleased to have assisted.

This is the first instance I've seen of a single model processor with a particular setting creating a systemic performance deficit. 

It appears that in the recent years during which core counts increased so quickly that BIOS and applications were continuously behind the processor designs, creating these problems.  Innovation also affected the continuity of backwards compatibility, for example, the emergence of NVMe memory.

I agree completely that disabling half the threads of the E5-2699 v3's is not the solution- throwing away all that capability means it would be better to change to two 10 or 12-cores.  The Xeon E5-2683 v4 (16C@ 2.1 / 3.0) with Passmark CPU= 18863 and Single Thread Mark = 1502 is quite good but another CPU possibility offers advantages:  Xeon E5-2690 v4 (14-core@2.6 / 3.5) with Passmark CPU= 21167 and Single Thread Mark = 1911.  The advantages are due to higher clock speeds and newer architecture.

Without knowing the AVID multithreading efficiency- which I suspect is very good, the understanding that AVID GPU acceleration is minimal, and in combination with the idea that so many applications are single-thread oriented, the Xeon E5-2690 v4 would seem a good choice: much higher single-thread (=1911) than the E5-2699 v3 (=1502) and the total calculation density of 21167 as compared to 18863 means that the E5-2690 v4 would, with fewer cores still be producing more clock cycles per unit time.  Also the better STM will have noticeably better performance in other, mostly single-threaded applications, for example 3D CAD and graphic design- Adobe.

Again, an interesting thread and I'd enjoy knowing what happens.

Bonne chance,

BambiBoomZ

HP z620_2 (2017) (R7) > Xeon E5-1680 v2 (8-core@ 4.3GHz) / z420 Liquid Cooling / 64GB DDR3-1866 ECC Reg / Quadro P2000 5GB _ GTX 1070 Ti 8GB / HP Z Turbo Drive M.2 256GB AHCI + Samsung 970 EVO M.2 NVMe 500GB + HGST 7K6000 4TB / Focusrite Scarlett 2i4 sound interface + 2X Mackie MR824 / 825W PSU /> HP OEM Windows 7 Prof.’l 64-bit > 2X Dell Ultrasharp U2715H (2560 X 1440)

[CinéMatica]

Hello BambiBoomZ,

The bad part is that the document you have showed me already existed since  early October 2014.

I admit that I made a failure choosing those Dual 18-Core Xeon E5-2699v3 processors. But actually the vendor of these systems, who does nothing else than building refurbished HP and Dell computers, should have know this.

And it all doesn't end with just replacing the processors. Going for a v4 processor also means that I have to replace my currunt RAM memory that runs at 2133 MHz. The Avid guide says that for all v4 chips you'll need at least 8 DIMM's of 2400 MHz  DDR4 RAM. As this vendor only has 32 GB DIMM's of refurbished 2400 MHz RAM leftover means that I am going the double the memory that I have now. 256 GB, and of course not for free. Ouch!

I find it really weird that from all the processors available for a Z640 and  Z840 I just picked the one that doesn't operate smooth.

Thanks again for all your investigating time and help! I surely keep you posted about what is happening with the system.

Have a very nice day and weekend!

Friendly greetings,

Ivan

[DGroves]

i've personally only seen one 18 cpu dual core system which was not a server so far, and it was using windows 10 (a clean install from win 7)

note that this was not a z series workstation (supermicro board) and it was not using avid (but was video related)

 so far as i know they have never called my old company for any sort of issues with the system 

so disabling hyperthreading may be a issue with avid itself (or another app) if you have time i would still get a small test ssd/drive and do a clean windows 10 install

[BambiBoomZ]

Ciematica,

I think it's possible to forgive anyone that had never heard of the dual  E5-2699 v3 plus hyperthreading problem.  I've never known another instance of a single model of processor with HT enabled, having degraded perormance presumably based on core count. The nearest reference I've heard was years ago, that in gaming, Intel i5-'s would perform better with HT disabled, but that's  the entire list.

As for memory,  you could consider trying the 2133 and I would be surprised if the system seemed sluggish. The CAS / latency on 16GB  modules is better than 32GB so it might not be noticeable. Also, I wonder if it would be at all common- even with large AVID files' processing,  to saturate 8 X 32GB.  There is another factor in the filling of both channel sof RAM .  The module configuration needs to be symmetrical for both CPU's , but  could there be a greater lag in the memory controller distribution addressing more modules? 

An alternative to the exchange requiring 32GB modules might be to return the current 128GB for a refund and then buy used PC4-19200R (DDR4-2400 ECC Registered) made for HP Proliant servers off of Ebay DE.  For example:

https://www.ebay.de/sch/i.html?LH_PrefLoc=2&_sop=2&_osacat=0&_odkw=128GB+%288x16GB%29+PC4-19200R+DDR...

>Qui! Beaucoup d'argent.

In both my HP E5-v2  workstations I use DDR3-1866 ECC Registered that has the HP part number on it and never a failure.

BambiBoomZ 

[CinéMatica]

After once enabling and disabling the Hyper-Trading mode, it now also starts to lag and perform badly with Hyper-Trading de-activated. No context menu can be opened in Avid Media Composer and Sorenson Squeeze Desktop 11.1 Pro. 

What a kind of rare issue is this all?

[BambiBoomZ]

Cinematica,

I am sorry that you're still having the trouble.

Here's some stranger stuff:

I checked the original specification for the z840 and the maximum core count processor is only 8-cores

https://support.hp.com/us-en/document/c04505606

Whereas the the later v4 systems list the E5-2699 v4:.

https://www8.hp.com/h20195/v2/GetPDF.aspx/c04400043.pdf

 There appears to have been only one model of motherboard for the z840, the 2129.

It's possible that the problem will persist as long as there are more than 64 logical cores.  The gradual lag may to the accumulation of thousands of errors.  Consider first purging the system of error logs using CLeaner or similar.  If removing all the error logs restores performance, it may be that as long as the E5-2699 v3's are installed the system will continue to gradually degrade.  

Secondly, consider testing the system with CPU 1 removed and only CPU 0 remaining; thereby reducing the core count below 64.  Follow the system requirement as regards RAM related to CPU 1.  There may need to be a module related to CPU 1 even though the processor itself is not present.   If full performance is restored, then I would suggest using the system with one processor until the replacement processor as are obtained. The seller should agree to send the replacements first so the system can remain in use. I've done this.   When once I had a replacement motherboard (z620) that didn't work, the seller sent another one- that did work, and then I returned the first one. 

Watch for a private message with a couple of other ideas.

BambiBoomZ

[CinéMatica]

Hi BambiBoomZ,

"I checked the original specification for the z840 and the maximum core count processor is only 8-cores

https://support.hp.com/us-en/document/c04505606"

That must have been the first specification page from the Z840 since it's release in September 2014.

"Whereas the the later v4 systems list the E5-2699 v4:.

https://www8.hp.com/h20195/v2/GetPDF.aspx/c04400043.pdf"

Yes, but that v4 is 22-cores instead of the 18 that I have.

The 22-core on the site of my vendor is E5-2696v4 instead of E5-2699v4 from the PDF you've posted.

You are absolutely right! More than 64 logical cores will give problems.

I got some more tips an tricks on the Avid forum.

1. Do not connect a sata SSD for the OS to the SAS controller. (connector group 13) but use the 2 6GB/s Sata ports of the sata raid controller (connector group 14).

The two SSD's are corectly connected to connector group 14.

2. If using a sata SSD disable the raid functionality on the Sata controller and set it to AHCI.

This often restored performance on a windows level.

I disabled the RAID fuction of the SATA controller in the BIOS and set it to AHCI. The sSATA controller is still set to RAID because I have 4 drives in the lower bays in RAID5. After I have done this the system doesn't start, but is Preparing Automatic Repair, followed by Diagnosing your PC. Then I get a blue Automatic Repair screen that tells me that my PC didn't start correctly. I have two options. Restart or Advanced Options. When I press Restart the whole cycle starts over again and again. From The Advanced Option I don't know too much so I leave them as they are. After setting the SATA controller back to RAID, the system boots again.

3. Check if memory modules are all equal. If not check that they are paired correctly accross both CPU's. I have seen refurbishers just throwing in modules that would show up and complete the desired memory size without ever testing speed.

These are OK, I have 8 DIMM's of 16 GB DDR4 2133 MHz RAM. They are in slot 1, 3, 6 and 8 for CPU0 and CPU1 like preferred in the HP and Avid guides.

4. Disable NUMA.

Done this when I started the system for the  very first time.

5. Use DDU to completely remove graphics drivers and manually install the version Avid advices to use with your version of Media composer and Graphics card. After the (re)install set the driver into video editing mode and max performance. On windows 10 this makes a difference.

Done this and installed the latest NVIDIA driver for the Quadro P4000 that is certified for Avid Media Composer 2019.12. This is driver 441.28 (64-bit).

6. Test if your DVD recorder still works. It showing up in the device manager is not enough. Best is to just put a disk in and copy something to a local disk. I have seen broken CD/DVD players/recorders giving the Sata/Sas controllers and therfor Os a bad time. Same applies to a broken card reader but I do not know if you use one. If you find something weird disconnect/replace.

The DVD burner is working like it should. I copied some files to my RAID5 drives without any problems.

7. Disable hyperthreading. The above mentioned high cpu count issue I have read about when trying to fix performance on a z820. Especially quicktime, which never was any good anyway, performed hopeless at the time. Disabling hyperthreading helped to improve performance but it still wasn't great. We repurposed that machine at the time as we couldn't get it to perform even after several bios changes and clean installs. After a few days we gave up. 

And finally also done this!

All these tests and changes do not make any difference. The system is even performing bad with the Hyper-Trading disabled. Sometimes it performs better, but the same amount of times not. It's very fickle!

Thanks for your suggestion on taking one processor out and testing that, but I don't have the time to do all that. Sorry!

The whole system goes back and these evil processors need to beswapped out together with  the RAM-memory. There is no way you can work with this system.

Friendly greetings!

[CinéMatica]

Short update!

The vendor of my system has build an exact copy of the machine I ordered. Only the graphics card is different. They use a NVIDIA Quadro K6000 and I a P4000. But according to them this shouldn't matter. The copy of my machine is snappy and fast. I saw it in a short video they have sent me.

I am advised to remove all the PCIe cards that I have installed, disconnect all 4 HDD'S, the SSD that is not the boot drive and then perform a clean install of Windows 10 Pro 1909.

The document that was mentioned in an earlier post is very old, and only counts for old software that is based on Windows API. The current Windows 10 (64-bit) can manage 255 threads!

When I have done the re-install, I will report back!

[DGroves]

i again recommend you follow my suggestion that once win 10 is installed, you then install one card or application at a time and reboot after each such install

also, the z840 should have the bios reset to FACTORY defaults not a modified "saved" copy

the "Z" series allows you to create a custom default bios profile that is loaded in place of the factory default, you need to do a full cmos reset if you do have modified default bios settings 

to do a full reset, power off system remove pwr cord press pwr button for 5 sec open case remove cmos battery and then press motherboard cmos reset button for 5 sec reinstall cmos batt, pwr cord side panel and power on system and reset time/date

[BambiBoomZ]

Cinematica,

An interesting turn of events. Yes, it's logical that the special problem of 2014 as described in the white paper would have been solved eventually.

If one assumes that the vendor installed Windows and configured the system, it's logical to assume that the differential settings and changes made subsequently between your systems and the replicated system are the reason for the problems.

Before reloading Windows, consider obtaining a detailed description of absolutely every setting in BIOS and other system configuration choices of the replicated system.  Perhaps the seller could send screen shots of the relevant configuration screens. Do not rely on default settings for the reason mentioned of some differential settings. Set a restore point and test the system before any software or added hardware is added except for the GPU.  Ensure that all software and drivers are the latest version.  Make one change at a time and set a new restore point to be able to revert if necessary.

BambiBoomZ