HP ZBOOK Fury Nvidia RTX5000 Ada/16GB, issues on training object detector models

[DragosStan]

ZBOOK FURY 16G10, GPU NVIDIA RTX5000 Ada/16 Gb, Driver Version: 555.85, bought last month, CUDA 12.5/12.1,  primary setup (environment variables) for CUDA 12.1 CuDnn 9.1.1 windows, pytorch 12,3.  During training of yolov8 models for several datasets, epochs 100..300, batch size 4...32  shows a consistent bad behaviour as follows :

For a few epochs training is normal and fast, 3/16 minutes/epoch depending on batch size and dataset characteristics

nvidia-smi shows : GPU temperature 65 ...70 Celsius, Power 60..70 WATT, GPU utilization 40..75%, you can hear that fans are working;

Then for several other epochs training becomes extremelly slow, 1...8 hours/epochs, nvidia-smi shows : GPU temperature 53...63 Celsius, Power ~35 WATT, GPU utilization 100%, fans are not working.

I repeated the experiments for various datasets and yolov8 model, the behaviour is similar, does not matter the number of epochs already occured. Always it is happening that after a few epochs the temperature to drop below 63 Celsius, Power drop at around 35 WATT, utilization to increase at 100%, training time increases hugely.

It looks like  GPU might be thermally throttling to prevent overheating, which would explain the lower temperature, power usage, and utilization during slow epochs.

I want to add that I'm a Computer Vision practitioner with more than 5 years experience and I'm running for the time being the same models/experiments on a Dell Server with A4000/12 Gb with much consistent and constant results during models training, always the temperature is over 70 celsius.

Another remark related to this ZBOOK/GPU is that Windows 11 task manager/performance do not recognise GPU utilization (shows always 0 utilization comparing with nvidia-smi). No other tasks are ran in parallel.

Please let me know which could be possible causes and how to handle this because for the moment my notebook is useless. 

Thanks

Dragos

[BeemerBiker]

I am running 12 systems, CUDA apps, 24/7  but have nothing newer than RTX-2080Ti. I have to run MSI Afterburner to ensure the fans blow enough air to keep the boards cool.  I also prevent the CPU from running in turbo mode as the CPU will quickly overheat.  24/7 like this will ruin a notebook.

Your RTX5000 may not be recognized. If you select 3D for performance the CUDA should show up at or near %100.

if not then there is a problem with the drivers not providing the proper answers.  Alternately, windows is not showing the correct display report.

  If in doubt then use GPUz's sensor display to confirm what nvidia-smi shows

Your cooling system does both the CPU and GPU. 

To prevent the CPU from running in turbo set the % utilization to %99

Enter the phrase 'edit power plan' in the window search box
Then select the 'Advanced' tab

To keep the CPU from overheating , consider
using %99 instead of %100.

Verify CPU speed using this tool (CPU-Z)

Possibly the MSI afterburner can be used to force the GPU to a lower performance to keep from overheating

I have read that the maximum temperature of that GPU is 93c but in a laptop one should not get anywhere near that.