Table of contents

Pre-requisites for installing Watson Studio Local with NVIDIA GPU support

Ensure that your servers meet the requirements for installing Watson Studio Local if you use a GPU NVIDIA operating system.

Graphics Processing Unit system preparation

Watson Studio Local supports GPUs by POWER systems, NVIDIA in Azure, AWS and Softlayer. If you have NVIDIA GPUs, you must perform the following steps before installing Watson Studio Local.

Azure specific pre-install instructions
If you are an Azure user, Azure is the only environment that has very specific kernel version requirements. The default kernel level of a non HPC node will not be compatible with Watson Studio Local:
uname -r 3.10.0-514.28.1.el7.x86_64

To install a supported kernel level, run the following code:

yum install kernel-3.10.0-514.21.1.el7.x86_64

General pre-install instructions

Follow these steps, along with the example code, to pre-install packages, modules and update video drivers for Watson Studio Local pre-installation. For instructions on downloading NVIDIA drivers, jump to the section Download and install NVIDIA GPU driver.

  1. Install pciutils.
    yum install pciutils
  2. Check for the default video driver.
    lsmod | grep -i nouveau
  3. Disable the default nouveau drivers using the following procedure below to a) update grub.conf and b) blacklist.conf (depending on the image you use these might already be disabled. See step 2) and c) reboot.
    1. Update grub to blacklist the nouveau driver by appending rd.driver.blacklist=nouveau nouveau.modeset=0 to the GRUB_CMDLINE_LINUX line as shown below.
      vi /boot/default/grub
       Change : GRUB_CMDLINE_LINUX="console=ttyS0,115200n8 console=tty0
      net.ifnames=0 crashkernel=auto”
       GRUB_CMDLINE_LINUX="console=ttyS0,115200n8 console=tty0
      net.ifnames=0 crashkernel=auto rd.driver.blacklist=nouveau

      Complete the update by running the following command:

      grub2-mkconfig -o /boot/grub2/grub.cfg
    2. Edit/create the file path /etc/modprobe.d/blacklist.conf and append.
      blacklist nouveau
    3. Reboot the system to activate the changes.
  4. Add kernel-tools and kernel-devel packages by installing the version that matches the kernel version.
    yum install kernel-devel-$(uname -r)
    kernel-headers-$(uname -r)
  5. Install gcc.
    yum install gcc
  6. Acquire the dkms package from an external repo.
    rpm -ivh
    yum install dkms

Download and install NVIDIA GPU driver

Follow these steps, along with the examples provided, to download the latest NVIDIA driver:

  1. Download the NVIDIA 8.0 drivers: Download Drivers
    Figure 1. Example of selected driver

    Type: Tesla Series: K-Series Product: Tesla K80 System: Linux 64-bit RHEL CUDA Toolkit: 8.0 Language: US (English)

  2. Install the NVIDIA repo.
    rpm -i nvidia-diag-driver-local-repo-rhel7-384.66-1.0-1.x86_64.rpm
  3. Install the drivers and then reboot. CUDA-enabled NVIDIA 8.0 GPU must be installed on the host operating system compute nodes that have a GPU.
    yum install cuda-drivers
  4. Verify the installation:
    Tip: If the command is slow, persistence mode might be enabled. Disable persistence with the following command:
    nvidia-smi -pm 0
Figure 2. Example of verification results

installation verification results example

You are now ready to install Watson Studio Local and reap the accelerated compute speed of the GPU.

More requirements for POWER with NVIDIA GPUs and device plugin for Kubernetes on POWER

Check the following items described in IBM PowerAI documentation:

  • Operating system and repository setup (RHEL version should be 7.5; do not run the yum update command because it might change the RHEL version)
  • System firmware
  • NVIDIA Components: IBM POWER9 specific udev rules (for POWER9 only)
  1. Download NVIDIA driver 418.39 or higher from NVIDIA Driver Downloads.
    1. Select Product Type: Tesla
    2. Select Product Series: P-Series or V-Series
    3. Select Product: Tesla P100 or Tesla V100
    4. Select Operating System: 'Show All Operating Systems' - 'Linux POWER LE RHEL 7'
    5. Select CUDA Toolkit: 10.0 or higher
    6. Click Search to go do the download link, download nvidia-driver-local-repo-rhel7-*.ppc64le.rpm.
  2. Install the GPU driver:
    rpm -i nvidia-driver-local-repo-rhel7-*.ppc64le.rpm
    yum install cuda-drivers
  3. Verify GPU:
    #verify GPU can be seen
    #verify if device file has been created
    ls /dev/nvidia-uvm
    #If device file not found download utility ( and execute it:
    #verify file is created
    ls /dev/nvidia-uvm
    #verify if device log file exists:
    ls /var/lib/docker/volumes/
    #if the device log file is missing, create the directory `nvidia_driver_xxx.xx`
    cd /var/lib/docker/volumes
    mkdir nvidia_driver_396.44

    If the verification fails, you might need to restart your server.

  4. Download nvidia-docker 2.0 and install it:
    # Add the package repositories
    distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
    curl -s -L$distribution/nvidia-container-runtime.repo | \
       tee /etc/yum.repos.d/nvidia-container-runtime.repo
    # Install the nvidia runtime hook
    yum install -y nvidia-container-runtime-hook
    # Adjust SELINUX Permissions
    sudo chcon -t container_file_t  /dev/nvidia*
    # Test nvidia-smi with the latest official CUDA image
    docker run --rm nvidia/cuda-ppc64le nvidia-smi
  5. Install the NIVIDIA device plugin for Kubernetes:
    git clone && cd k8s-device-plugin
    git checkout v1.11
    kubectl create -f nvidia-device-plugin.yml
    # Test if resource is available on the nodes
    kubectl describe node | grep -Ea "Name:|Capacity:|Allocatable:|"