Guidance for monitoring GPUs

Introduction

Prerequisite

To configure GPU monitoring 

Metrics and attributes for out-band monitoring

Matrix of metrics and attributes for in-band monitoring

Introduction:

Entuity supports in-band and out-band monitoring of the AMD, Intel, and Nvidia GPUs in your managed servers. Based on the type of monitoring, some GPUs have specific requirements for being able to poll performance metrics. Prerequisites and configuration instructions for each GPU vendor and monitoring type follow, along with the metrics provided by each monitoring type.

Prerequisite

• If the GPUs that you want to monitor are in a Linux server, ensure that Sudo is installed on the server before attempting to monitor the GPUs. 

To configure GPU monitoring:

All GPU vendors out-band:

To perform out-band monitoring of GPUs from all supported vendors, Entuity uses certain server hardware collectors along with a server hardware monitoring user account. 

To configure out-band monitoring of GPUs from all supported vendors:

• Add the server to Entuity for hardware monitoring as you normally would. No further configuration is required. If the associated server hardware collector can discover GPUs in the server, the GPUs are displayed on server dashboards and dashlets in the UI. 

For more information on which server hardware collectors can discover GPU information for out-band monitoring, see this article.

For more information on adding a server to your network, see this article. 

AMD in-band SSH

To perform in-band monitoring of AMD GPUs through SSH, Entuity uses the amd-smi utility and a server OS monitoring user account. Keep in mind that the OS monitoring user account must have SSH access and must be able to run the amd-smi utility.

To install the amd-smi utility and configure AMD GPU in-band SSH monitoring:

1. Follow the instructions in the AMD SMI documentation to install the amd-smi utility. 
   Note, only the amd-smi CLI is required. 
2. If you haven't already done so, create an OS monitoring user account for the server, as described in this article.
3. Ensure that the OS monitoring user account can run the amd-smi utility.
4. When adding the server to Entuity, scroll to the OS Monitoring section of the Add Asset form, and do the following:
   • Click the Credential field, and in the Local Credentials section of the Credential form that is displayed, enter the OS monitoring account username, password, and SSH key (if used) in the appropriate fields. 
   • In the GPU Connection Method dropdown field, select AMD GPU (SSH).

For more information on adding a server to your network, see this article.

Intel in-band SSH:

To perform in-band monitoring of Intel GPUs through SSH, Entuity uses the xpu-smi utility, the hl-smi utility, and a server OS monitoring user account. Keep in mind that the OS monitoring user account must have SSH access and must be able to run the xpu-smi and/or hl-smi utilities.

Note, the xpu-smi utility runs on Linux and Windows. 

Installing the Intel GPU utilities:

• The xpu-smi utility is included in Intel's XPU Manager. For more information on supported operating system versions and installation instructions for XPU Manager, see the Intel XPU Manager documentation.
• The hl-smi utility is automatically installed when you set up the official Intel Gaudi Driver and Software Stack.

To configure Intel GPU in-band SSH monitoring:

1. If you haven't already done so, create an OS monitoring user account for the managed server, as described in this article for Linux and this article for Windows, based on the OS of the managed server.
2. Ensure that the OS monitoring user account can run the xpu-smi and hl-smi utilities.
3. When adding the server to Entuity, scroll to the OS Monitoring section of the Add Asset form, and do the following:
   • Click the Credential field, and in the Local Credentials section of the Credential form, enter the OS monitoring account username, password, and SSH key (if being used) in the appropriate fields. 
   • In the GPU Connection Method dropdown field, select Intel GPU (SSH).

For more information on adding a server to your network, see this article.

Nvidia in-band SSH:

To perform in-band monitoring of Nvidia GPUs through SSH, Entuity uses the required nvidia-smi utility (which is bundled in the Nvidia Graphics driver), the optional dcgmi utility (with the nv-hostengine service), and a server OS monitoring user account. Keep in mind that the OS monitoring user account must have SSH access and must be able to run the nvidia-smi and dcgmi utilities.

Note, the dcgmi utility is not required but is needed if you want Entuity to discover graphics/SM/tensor/DRAM activity, fabric switches, and SM occupancy for MIG instances. To list MIG instance profile names, the OS monitoring user account requires root or passwordless sudo. Without these privileges, MIG instances are still discovered in Entuity, but without profile names.  

The nvidia-smi utility ships with the Nvidia official display drivers. 

To install the dcgmi utility and configure Nvidia GPU in-band SSH monitoring:

1. Follow the instructions in the Nvidia DCGM documentation to install the dcgmi utility.  
2. If you haven't already done so, create an OS monitoring user account for the managed server, as described in this article for Linux and this article for Windows, based on the OS of the managed server.
3. Ensure that the OS monitoring user account can run the nvidia-smi and dcgmi utilities.
4. When adding the server to Entuity, scroll to the OS Monitoring section of the Add Asset form, and do the following:
   • Click the Credential field, and in the Local Credentials section of the Credential form, enter the OS monitoring account user name, password, and SSH key (if being used) in the appropriate fields. 
   • In the GPU Connection Method dropdown field, select NVIDA GPU (SSH).

For more information on adding a server to your network, see this article.

All GPU vendors in-band Auto:

When you first add a server to Entuity and you plan to monitor the GPUs in that server, Entuity provides the option to automatically detect the GPU vendor types and use the appropriate vendor utilities on the managed server to discover the GPUs for monitoring. 

To perform this automatic in-band monitoring of GPUs through SSH, Entuity uses the required GPU vendor utilities on the server and a server OS monitoring user account. Keep in mind that the OS monitoring user account must have SSH access and must be able to run the required vendor utilities, based on the GPU vendor type.

To install the required GPU utilities and configure automatic GPU in-band monitoring:

1. See the AMD, Intel, and Nvidia sections earlier in this article and follow the instructions in each section to install each of the required GPU vendor utilities on the managed server. 
2. If you haven't already done so, create an OS monitoring user account for the managed server, as described in this article for Linux and this article for Windows, based on the OS of the managed server.
3. Ensure that the managed the OS monitoring user account can run all of the utilities.
4. When adding the server to Entuity, scroll to the OS Monitoring section of the Add Asset form, and do the following:
   • Click the Credential field, and in the Local Credentials section of the Credential form that is displayed, enter the OS monitoring account user name, password, and SSH key (if being used) in the appropriate fields. 
   • In the GPU Connection Method dropdown field, select Auto.

For more information on adding a server to your network, see this article.

Metrics and attributes for out-band monitoring

• Component Name
• Vendor
• Model
• Serial Number
• Part Number
• Component Type
• Version
• Firmware Version
• Slot
• Location
• Description
• GPU Temperature
• Memory Temperature
• Power Usage
• Operating Speed
• Bandwidth Percent
• Memory Speed
• Memory Bandwidth Percent
• Correctable Error Count
• SM Utilization Percent
• SM Activity Percent
• SM Occupancy Percent
• Tensor Core Activity Percent
• HMMA Utilization Percent
• PCIe Raw Tx Bandwidth
• PCIe Raw Rx Bandwidth
• IMMA Utilization Percent
• DMMA Utilization Percent
• Current PCIe Link Speed
• Max Supported PCIe Link Speed
• DRAM Utilization Percent
• Link Tx Throughput
• Link Rx Throughput
• Link Count
• State / State Detail 

Matrix of metrics and attributes for in-band monitoring:

The following table lists the metrics available for each type of monitoring.

Note, this table lists all possible metrics. Based on the driver/utility versions and the GPU vendor type, some of the metrics shown might not be available.