Monitoring AMD, Intel and NVIDIA graphics card usage under Linux


Graphics card resource usage monitoring is quite rare. Most desktop widgets and applications will just show CPU and RAM usage. There are however tools that can be used to monitor AMD, nVidia and Intel graphics cards under Linux. It can be used for fancy graphics or to locate the weakest chain in the GPU rendering process when trying to determine why some tasks are so slow.


In case of Intel integrated GPUs we can use some tools from intel-gpu-tools, like for example:

So when using:
intel_gpu_time someapplication
We can get results like:
user: 0.428928s, sys: 0.071933s, elapsed: 5.725313s, CPU: 8.7%, GPU: 14.4%
intel_gpu_top in action
intel_gpu_top in action

intel_gpu_top can also log to file (check --help for such options).


For the closed source driver we can use nvidia-smi, which will show GPU temperature, fan speed and memory usage. Some fully supported models will have more detailed data displayed (use --help to see extra options, a list of supported models as well as other goodies like a Python library). At the moment the list looks like so:

Supported products:
- Full Support
    - NVIDIA Tesla Line:
            S2050, C2050, C2070, C2075,
            M2050, M2070, M2075, M2090,
            X2070, X2090,
            K10, K20, K20X, K20Xm, K20c, K20m, K20s, K40c, K40m, K40t, K40s, K40st
    - NVIDIA Quadro Line:
            410, 600, 2000, 4000, 5000, 6000, 7000, M2070-Q
            K2000, K2000D, K4000, K5000, K6000
    - NVIDIA GRID Line:
            K1, K2, K340, K520
    - NVIDIA GeForce Line: None
- Limited Support
    - NVIDIA Tesla Line:   S1070, C1060, M1060 and all other previous generation Tesla-branded parts
    - NVIDIA Quadro Line:  All other current and previous generation Quadro-branded parts
    - NVIDIA GeForce Line: All current and previous generation GeForce-branded parts
nvidia-smi in action
nvidia-smi in action

You can also try using nvidia-settings.

For example I've noticed that old Quadro FX 580 had a big FPS drop from 66.37 to 21.3 on Xonotic phoronix benchmarks between High and Ultra quality. Looking at GPU memory usage (512MB) at ultra the GPU was quite likely lacking needed RAM space as whole RAM was used. At high quality there are only few MB of RAM free. Graphics cards with more RAM doesn't show such effect in Xonotic benchmarks and drop FPS more gently with increasing quality.


For the binary closed source driver (fglrx) we can use aticonfig, although it won't give much:

aticonfig --odgc --odgt

For the open (radeon) and closed source drivers we can use more advanced radeontop (also available in some repositories, like Ubuntu).

aticonfig in action
aticonfig in action
radeontop output during benchmarks
radeontop output during benchmarks

There is also a desktop application - radeon-profile which works with open and closed source drivers.

Comment article