F5F Stay Refreshed Hardware Desktop The 2x3090 device stops functioning when subjected to certain stress levels.

The 2x3090 device stops functioning when subjected to certain stress levels.

The 2x3090 device stops functioning when subjected to certain stress levels.

Pages (2): 1 2 Next
K
KlayDog1
Senior Member
685
06-20-2022, 03:31 AM
#1
My machine has been operating smoothly for months, but a recent diffusion model training session triggers an immediate shutdown. SSH disconnects, power falls below 10 watts, no visual output appears, the GPU shuts off completely, all fans cease spinning, and the power button stops responding—though the RAM display remains active. Interestingly, the RGB on the motherboard continues functioning normally. Only after resetting the power supply does the system recover. It seems the issue isn't limited to GPUs; either the motherboard or PSU is cutting power to most components (except RAM). Other tasks didn’t reproduce this problem. I’ve pushed the GPU to higher loads (over 700W) without success. The UPS reports around 515 watts total. I've updated drivers and tried different CUDA versions, but using a single GPU works fine while dual-GPU causes the failure. I recently replaced the PSU with a Corsair HX1200, yet no improvement was seen. The driver version is 470.141.03 for Ubuntu 22.04 EVGA FTW3 3090 NVidia, and the Corsair HX1200 (previously RM1000x) paired with a 5900X on ASUS AM4 TUF Gaming X570-Plus BIOS 4204. Thank you!
K
KlayDog1
06-20-2022, 03:31 AM #1

My machine has been operating smoothly for months, but a recent diffusion model training session triggers an immediate shutdown. SSH disconnects, power falls below 10 watts, no visual output appears, the GPU shuts off completely, all fans cease spinning, and the power button stops responding—though the RAM display remains active. Interestingly, the RGB on the motherboard continues functioning normally. Only after resetting the power supply does the system recover. It seems the issue isn't limited to GPUs; either the motherboard or PSU is cutting power to most components (except RAM). Other tasks didn’t reproduce this problem. I’ve pushed the GPU to higher loads (over 700W) without success. The UPS reports around 515 watts total. I've updated drivers and tried different CUDA versions, but using a single GPU works fine while dual-GPU causes the failure. I recently replaced the PSU with a Corsair HX1200, yet no improvement was seen. The driver version is 470.141.03 for Ubuntu 22.04 EVGA FTW3 3090 NVidia, and the Corsair HX1200 (previously RM1000x) paired with a 5900X on ASUS AM4 TUF Gaming X570-Plus BIOS 4204. Thank you!

C
catseecoo
Senior Member
662
06-20-2022, 03:31 AM
#2
Hard shutdown would occur immediately, not causing the machine to stop and enter idle mode.
C
catseecoo
06-20-2022, 03:31 AM #2

Hard shutdown would occur immediately, not causing the machine to stop and enter idle mode.

T
Tomcatod
Member
71
06-20-2022, 03:31 AM
#3
Have you attempted to run the training without the 3090 module? This should help verify it's not related to the power supply.
T
Tomcatod
06-20-2022, 03:31 AM #3

Have you attempted to run the training without the 3090 module? This should help verify it's not related to the power supply.

S
Shankit
Junior Member
49
06-20-2022, 03:31 AM
#4
Did you attempt to connect via SSH once it stopped working? It might still be active, and the GPUs could have failed. It looks unclear how to interpret logs after the crash started.
S
Shankit
06-20-2022, 03:31 AM #4

Did you attempt to connect via SSH once it stopped working? It might still be active, and the GPUs could have failed. It looks unclear how to interpret logs after the crash started.

N
NGWessel
Member
160
06-20-2022, 03:31 AM
#5
It works well when you configure it to use just a single GPU.
N
NGWessel
06-20-2022, 03:31 AM #5

It works well when you configure it to use just a single GPU.

L
levoyageur92
Posting Freak
807
06-20-2022, 03:31 AM
#6
Sure, I'll adjust the power output to 300 or 250W for the 3090 and observe the results.
L
levoyageur92
06-20-2022, 03:31 AM #6

Sure, I'll adjust the power output to 300 or 250W for the 3090 and observe the results.

_
_MABGamer_
Junior Member
11
06-20-2022, 03:31 AM
#7
I'll check for any remaining logs after the crash.
_
_MABGamer_
06-20-2022, 03:31 AM #7

I'll check for any remaining logs after the crash.

M
Marcelo_Game
Junior Member
22
06-20-2022, 03:31 AM
#8
It seems there might still be a power supply concern. Nvidia suggests increasing the constant maximum power rating for the 30 series GPU by double to match peak demands. With two 3090s, you'd need around 1400W plus a PSU. The issue could occur before the UPS load meter registers it, possibly due to faster-than-display updates in the Power Meter.
M
Marcelo_Game
06-20-2022, 03:31 AM #8

It seems there might still be a power supply concern. Nvidia suggests increasing the constant maximum power rating for the 30 series GPU by double to match peak demands. With two 3090s, you'd need around 1400W plus a PSU. The issue could occur before the UPS load meter registers it, possibly due to faster-than-display updates in the Power Meter.

P
puppylove_75
Member
56
06-20-2022, 03:31 AM
#9
It appears the fix worked after a short period without lockup. Impressive how they can boost the voltage. I plan to use an even bigger PSU to confirm power limits are fully resolved. This excites me for the 4000W unit needed for the 4000 series cards. I’ll mark this as solved now and follow up with details about the new PSU later. Thanks a lot for your support!
P
puppylove_75
06-20-2022, 03:31 AM #9

It appears the fix worked after a short period without lockup. Impressive how they can boost the voltage. I plan to use an even bigger PSU to confirm power limits are fully resolved. This excites me for the 4000W unit needed for the 4000 series cards. I’ll mark this as solved now and follow up with details about the new PSU later. Thanks a lot for your support!

O
oldgrand2
Member
66
06-20-2022, 03:31 AM
#10
I wouldn't bother upgrading the PSU unless power limiting actually helped; the performance loss should stay near 10%. If you're already running intensive tasks all day, extra time won't cost much and your energy costs will drop.
O
oldgrand2
06-20-2022, 03:31 AM #10

I wouldn't bother upgrading the PSU unless power limiting actually helped; the performance loss should stay near 10%. If you're already running intensive tasks all day, extra time won't cost much and your energy costs will drop.

Pages (2): 1 2 Next