Monitoring AMD, Intel and NVIDIA graphics card usage under Linux

Graphics card resource usage monitoring is quite rare. Most desktop widgets and applications will just show CPU and RAM usage. There are however tools that can be used to monitor AMD, nVidia and Intel graphics cards under Linux. It can be used for fancy graphics or to locate the weakest chain in the GPU rendering process when trying to determine why some tasks are so slow.

Intel

In case of Intel integrated GPUs we can use some tools from intel-gpu-tools, like for example:

• intel_gpu_time: just like time, but will show average CPU and GPU usage for the application it started.
• intel_gpu_top: just like top, but for the GPU, not CPU.

intel_gpu_top can also log to file (check --help for such options).

nVidia

For the closed source driver we can use nvidia-smi, which will show GPU temperature, fan speed and memory usage. Some fully supported models will have more detailed data displayed (use --help to see extra options, a list of supported models as well as other goodies like a Python library). At the moment the list looks like so:

Supported products:
- Full Support
    - NVIDIA Tesla Line:
            S2050, C2050, C2070, C2075,
            M2050, M2070, M2075, M2090,
            X2070, X2090,
            K10, K20, K20X, K20Xm, K20c, K20m, K20s, K40c, K40m, K40t, K40s, K40st
    - NVIDIA Quadro Line:
            410, 600, 2000, 4000, 5000, 6000, 7000, M2070-Q
            K2000, K2000D, K4000, K5000, K6000
    - NVIDIA GRID Line:
            K1, K2, K340, K520
    - NVIDIA GeForce Line: None
- Limited Support
    - NVIDIA Tesla Line:   S1070, C1060, M1060 and all other previous generation Tesla-branded parts
    - NVIDIA Quadro Line:  All other current and previous generation Quadro-branded parts
    - NVIDIA GeForce Line: All current and previous generation GeForce-branded parts

You can also try using nvidia-settings.

For example I've noticed that old Quadro FX 580 had a big FPS drop from 66.37 to 21.3 on Xonotic phoronix benchmarks between High and Ultra quality. Looking at GPU memory usage (512MB) at ultra the GPU was quite likely lacking needed RAM space as whole RAM was used. At high quality there are only few MB of RAM free. Graphics cards with more RAM doesn't show such effect in Xonotic benchmarks and drop FPS more gently with increasing quality.

AMD

For the binary closed source driver (fglrx) we can use aticonfig, although it won't give much:

For the open (radeon) and closed source drivers we can use more advanced radeontop (also available in some repositories, like Ubuntu).

There is also a desktop application - radeon-profile which works with open and closed source drivers.