Skip to Content
DocsAIGPU Setup

GPU Setup

Note

This article has been initially checked against the Ubuntu 26.04 LTS April 2026 release notes. AI/GPU driver and CUDA/ROCm versions still need ongoing validation against vendor support matrices.

The GPU is the key hardware for running most AI workloads. This article provides a detailed guide for configuring the NVIDIA GPU environment on Ubuntu 26.04, including driver installation, CUDA Toolkit, cuDNN, and verification steps. It also briefly covers AMD GPU ROCm configuration.

26.04 Repository Status

The Ubuntu 26.04 repositories now provide the NVIDIA CUDA Toolkit (sudo apt install cuda-toolkit) and AMD ROCm 7.1.0 (sudo apt install rocm). They are not preinstalled by default. For production training, specific PyTorch/TensorFlow versions, or multi-GPU servers, continue to treat the NVIDIA/AMD vendor support matrices and framework release notes as the final source of truth.

NVIDIA Driver Installation

Ubuntu 26.04 provides the convenient ubuntu-drivers tool for NVIDIA driver installation.

Check Hardware

# View NVIDIA GPUs in the system
lspci | grep -i nvidia
 
# Check recommended driver versions
sudo ubuntu-drivers devices

Example output:

== /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0 ==
modalias : pci:v000010DEd00002684sv...
vendor   : NVIDIA Corporation
model    : AD102 [GeForce RTX 4090]
driver   : nvidia-driver-560 - third-party non-free recommended
driver   : nvidia-driver-550 - third-party non-free
driver   : nvidia-driver-555 - third-party non-free
# Automatically install the recommended driver version
sudo ubuntu-drivers autoinstall
 
# Reboot to activate the driver
sudo reboot

Manual Installation of a Specific Version

# View available driver versions
apt list nvidia-driver-* 2>/dev/null | grep -v "Listing"
 
# Install a specific version
sudo apt install -y nvidia-driver-560
 
# Reboot
sudo reboot

Install from NVIDIA Official Repository (Latest Version)

# Add the NVIDIA official repository (Ubuntu 26.04 uses ubuntu2604)
wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2604/x86_64/cuda-keyring_1.1-1_all.deb
sudo dpkg -i cuda-keyring_1.1-1_all.deb
sudo apt update
 
# Install the latest driver
sudo apt install -y nvidia-open
 
# Reboot
sudo reboot

Verify Driver Installation

# Check driver version and GPU status
nvidia-smi

Normal output example:

+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 560.35.03              Driver Version: 560.35.03      CUDA Version: 12.6     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce RTX 4090       Off  | 00000000:01:00.0  On  |                  Off |
|  0%   35C    P8              18W / 450W |     512MiB / 24564MiB  |      0%      Default |
+-----------------------------------------+------------------------+----------------------+
Tip

The “CUDA Version” shown in nvidia-smi output indicates the highest CUDA version supported by the driver, not the actually installed CUDA Toolkit version. You need to install the CUDA Toolkit separately.

CUDA Toolkit Installation

The CUDA Toolkit is NVIDIA’s GPU computing platform and programming model, required by most AI frameworks (PyTorch, TensorFlow, etc.).

How Driver and CUDA Versions Relate

Each CUDA Toolkit requires a minimum NVIDIA driver version, and newer CUDA needs a newer driver. Before installing, rely on the NVIDIA CUDA Toolkit release notes  and the version support matrices from PyTorch  / TensorFlow to pick a combination compatible with all three. A common approach: run nvidia-smi to see the highest CUDA version the driver supports, then choose the CUDA Toolkit and framework wheels accordingly.

Ubuntu 26.04 ships the CUDA Toolkit in its repository. For most local development this is the simplest option:

# Install the CUDA Toolkit from the Ubuntu repository
sudo apt install -y cuda-toolkit
 
# Verify
nvcc --version

Method 2: Install a Specific Version from the NVIDIA Repository

If you need a specific CUDA Toolkit version (for example, to match a particular PyTorch wheel):

# Add the NVIDIA official repository (Ubuntu 26.04 uses ubuntu2604)
wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2604/x86_64/cuda-keyring_1.1-1_all.deb
sudo dpkg -i cuda-keyring_1.1-1_all.deb
sudo apt update
 
# List available CUDA Toolkit versions in the repository
apt-cache search cuda-toolkit
 
# Install a specific version (replace XX-Y with an actual available version, e.g. cuda-toolkit-13-0)
sudo apt install -y cuda-toolkit-XX-Y
Note

Ubuntu 26.04 now ships the NVIDIA CUDA Toolkit in its official repositories, so sudo apt install cuda-toolkit works directly. NVIDIA’s ubuntu2604 repository remains useful when you need a specific CUDA version (for example, to match a particular PyTorch wheel).

Configure Environment Variables

# Add CUDA to PATH
cat >> ~/.bashrc << 'EOF'
 
# CUDA Toolkit
export CUDA_HOME=/usr/local/cuda
export PATH=$CUDA_HOME/bin:$PATH
export LD_LIBRARY_PATH=$CUDA_HOME/lib64:$LD_LIBRARY_PATH
EOF
 
source ~/.bashrc

Verify CUDA Installation

# Check CUDA compiler version
nvcc --version
 
# Example output:
# nvcc: NVIDIA (R) Cuda compiler driver
# Cuda compilation tools, release X.Y, VX.Y.xxx
 
# Run CUDA samples (optional)
# Newer CUDA versions no longer ship samples with the Toolkit; clone them from GitHub
sudo apt install -y cmake build-essential git
git clone https://github.com/NVIDIA/cuda-samples.git
cd cuda-samples
cmake -B build && cmake --build build -j$(nproc)
 
# Run device query (path may vary by version)
./build/Samples/1_Utilities/deviceQuery/deviceQuery
 
# If output shows "Result = PASS", installation is successful

Install Multiple CUDA Versions

# With the NVIDIA repository you can install multiple versions (replace XX-Y with actual versions)
sudo apt install -y cuda-toolkit-XX-Y
sudo apt install -y cuda-toolkit-ZZ-W
 
# View installed versions
ls /usr/local/ | grep cuda
 
# Switch default version (by updating the symlink; replace X.Y with the target version)
sudo rm /usr/local/cuda
sudo ln -s /usr/local/cuda-X.Y /usr/local/cuda
 
# Verify current version
nvcc --version

cuDNN Installation

cuDNN (CUDA Deep Neural Network library) is NVIDIA’s deep learning acceleration library. Both PyTorch and TensorFlow depend on it.

# Ensure the NVIDIA repository has been added
# Install cuDNN for CUDA 12
sudo apt install -y cudnn9-cuda-12
 
# Or install specific packages
sudo apt install -y libcudnn9-cuda-12 libcudnn9-dev-cuda-12
 
# Verify installation
dpkg -l | grep cudnn

Install via tar Package

# Download cuDNN from the NVIDIA developer website
# https://developer.nvidia.com/cudnn-downloads
# Requires a free NVIDIA developer account
 
# Extract to the CUDA directory
sudo tar -xzf cudnn-linux-x86_64-9.x.x.x_cuda12-archive.tar.xz
sudo cp cudnn-linux-x86_64-9.x.x.x_cuda12-archive/include/* /usr/local/cuda/include/
sudo cp cudnn-linux-x86_64-9.x.x.x_cuda12-archive/lib/* /usr/local/cuda/lib64/
sudo ldconfig

Verify cuDNN

# Check if cuDNN can be found
ldconfig -p | grep cudnn
 
# Verify via Python (requires PyTorch to be installed first)
python3 -c "import torch; print('cuDNN version:', torch.backends.cudnn.version()); print('cuDNN enabled:', torch.backends.cudnn.enabled)"

Verify GPU Availability

After installing all components, perform a comprehensive verification.

System-Level Verification

# 1. Check driver
nvidia-smi
 
# 2. Check CUDA
nvcc --version
 
# 3. Check cuDNN
ldconfig -p | grep cudnn
 
# 4. View detailed GPU information
nvidia-smi -q | head -50
 
# 5. Monitor real-time GPU status
watch -n 1 nvidia-smi

PyTorch GPU Verification

# Install PyTorch (inside an activated virtual environment)
# pip install torch installs the wheel for the current CUDA by default;
# older drivers can pin a lower CUDA, e.g. cu126
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu126
 
# Verify GPU availability
python3 << 'EOF'
import torch
 
print(f"PyTorch version: {torch.__version__}")
print(f"CUDA available: {torch.cuda.is_available()}")
print(f"CUDA version: {torch.version.cuda}")
print(f"cuDNN version: {torch.backends.cudnn.version()}")
print(f"GPU count: {torch.cuda.device_count()}")
 
if torch.cuda.is_available():
    print(f"Current GPU: {torch.cuda.get_device_name(0)}")
    print(f"GPU VRAM: {torch.cuda.get_device_properties(0).total_memory / 1024**3:.1f} GB")
 
    # Simple GPU compute test
    x = torch.randn(1000, 1000, device='cuda')
    y = torch.randn(1000, 1000, device='cuda')
    z = torch.mm(x, y)
    print(f"GPU compute test passed! Matrix multiplication result shape: {z.shape}")
else:
    print("Warning: CUDA not available. Please check driver and CUDA installation")
EOF

TensorFlow GPU Verification

# Install TensorFlow
pip install tensorflow
 
# Verify
python3 << 'EOF'
import tensorflow as tf
 
print(f"TensorFlow version: {tf.__version__}")
gpus = tf.config.list_physical_devices('GPU')
print(f"Available GPU count: {len(gpus)}")
 
for gpu in gpus:
    print(f"  GPU: {gpu.name}")
 
if gpus:
    # Simple test
    with tf.device('/GPU:0'):
        a = tf.random.normal([1000, 1000])
        b = tf.random.normal([1000, 1000])
        c = tf.matmul(a, b)
        print(f"GPU compute test passed! Matrix multiplication result shape: {c.shape}")
EOF

ROCm (AMD GPU) Overview

ROCm (Radeon Open Compute) is AMD’s open-source GPU computing platform, similar to NVIDIA’s CUDA.

Supported AMD GPUs

ROCm primarily supports the following AMD GPU series:

  • Radeon RX 7900 XTX / 7900 XT (RDNA 3)
  • Radeon RX 7800 XT / 7700 XT (RDNA 3)
  • Radeon PRO W7900 / W7800
  • Instinct MI300X / MI250X / MI210 (data center)

Install ROCm

Ubuntu 26.04 is the first release where you can install ROCm with a single apt command. Prefer the package shipped in the 26.04 repository:

# Method 1 (recommended): use the ROCm packaged in the Ubuntu 26.04 repository
# Full runtime stack
sudo apt install -y rocm
 
# Or just the libraries and headers for development
# sudo apt install -y rocm-dev
 
# Add user to render and video groups
sudo usermod -aG render,video $USER
 
# Reboot
sudo reboot
Note

The Ubuntu 26.04 repository ships ROCm 7.1.0. If you need a newer version, use AMD’s official repository below and follow the current version number from AMD’s official docs.

# Method 2: install a newer version from AMD's official repository
# Check the current version at https://repo.radeon.com/rocm/apt/ and replace <version> below
# Until AMD ships an Ubuntu 26.04 (resolute) repo, the 24.04 (noble) repo can be used as a fallback
wget https://repo.radeon.com/rocm/rocm.gpg.key -O - | \
  gpg --dearmor | sudo tee /etc/apt/keyrings/rocm.gpg > /dev/null
 
echo "deb [arch=amd64 signed-by=/etc/apt/keyrings/rocm.gpg] https://repo.radeon.com/rocm/apt/<version> noble main" | \
  sudo tee /etc/apt/sources.list.d/rocm.list
 
sudo apt update
sudo apt install -y rocm
sudo usermod -aG render,video $USER
sudo reboot

Verify ROCm

# View AMD GPU information
rocm-smi
 
# Check ROCm version
rocminfo | head -20
 
# Install PyTorch ROCm version
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/rocm6.2
 
# Verify
python3 -c "import torch; print('ROCm available:', torch.cuda.is_available()); print('Device:', torch.cuda.get_device_name(0) if torch.cuda.is_available() else 'N/A')"
Note

AMD ROCm’s software ecosystem is not as mature as NVIDIA CUDA. Some AI tools may not be supported or may have limited support. If your primary use is AI development, NVIDIA GPUs remain the safer choice. However, ROCm is continuously improving, and PyTorch and several major frameworks now have good support.

Common Troubleshooting

Issue 1: nvidia-smi Shows No Output or Errors

# Check if the driver is loaded
lsmod | grep nvidia
 
# If no output, try reloading the driver
sudo modprobe nvidia
 
# Check for Secure Boot issues
mokutil --sb-state
# If Secure Boot is enabled, you may need to sign the driver or disable Secure Boot
 
# Check kernel logs
dmesg | grep -i nvidia
sudo journalctl -b | grep -i nvidia

Issue 2: Driver Version Conflicts

# Remove all NVIDIA drivers
sudo apt purge -y 'nvidia-*'
sudo apt purge -y 'libnvidia-*'
sudo apt autoremove -y
 
# Reboot
sudo reboot
 
# Reinstall
sudo ubuntu-drivers autoinstall
sudo reboot

Issue 3: CUDA Version and PyTorch Mismatch

# Check current CUDA version
nvcc --version
 
# Check PyTorch expected CUDA version
python3 -c "import torch; print(torch.version.cuda)"
 
# If mismatched, install the PyTorch wheel for the corresponding CUDA version
# Default (current CUDA)
pip install torch
 
# CUDA 12.6
pip install torch --index-url https://download.pytorch.org/whl/cu126
 
# CUDA 12.8
pip install torch --index-url https://download.pytorch.org/whl/cu128
 
# Available cuXXX indexes are listed at https://pytorch.org/get-started/locally/

Issue 4: GPU Out of Memory (OOM)

# Check current GPU memory usage
nvidia-smi
 
# Find processes using the GPU
nvidia-smi --query-compute-apps=pid,name,used_memory --format=csv
 
# Free GPU memory (kill the occupying process)
# First confirm the process PID
kill -9 <PID>
 
# Clear PyTorch cache
python3 -c "import torch; torch.cuda.empty_cache()"

Issue 5: Nouveau Driver Conflict

# Nouveau is the open-source NVIDIA driver, which may conflict with the official driver
# Check if nouveau is loaded
lsmod | grep nouveau
 
# If there is output, disable nouveau
sudo bash -c 'cat > /etc/modprobe.d/blacklist-nouveau.conf << EOF
blacklist nouveau
options nouveau modeset=0
EOF'
 
# Regenerate initramfs
sudo update-initramfs -u
 
# Reboot
sudo reboot

Issue 6: Using GPU in Docker Containers

# Install NVIDIA Container Toolkit
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | \
  sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
 
curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
  sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
  sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
 
sudo apt update
sudo apt install -y nvidia-container-toolkit
 
# Configure Docker
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker
 
# Test Docker GPU access
docker run --rm --gpus all nvidia/cuda:12.6.0-base-ubuntu24.04 nvidia-smi
# Tip: nvidia/cuda image tags are organized by CUDA version and base OS;
# pick an existing tag at https://hub.docker.com/r/nvidia/cuda/tags
Tip

After installing NVIDIA drivers, we recommend installing NVIDIA Container Toolkit immediately. This allows you to use GPU acceleration when running AI tools in Docker containers (such as Open WebUI, ComfyUI, etc.) without configuring drivers separately inside the container.

Complete Installation Script

The following script consolidates the entire GPU environment setup process:

#!/bin/bash
# gpu-setup.sh -- Ubuntu 26.04 NVIDIA GPU Environment One-Click Setup
set -e
 
echo "=== 1. Install NVIDIA Driver ==="
sudo apt update
sudo ubuntu-drivers autoinstall
 
echo "=== 2. Install CUDA Toolkit (Ubuntu 26.04 repository) ==="
sudo apt install -y cuda-toolkit
 
echo "=== 3. (Optional) For a newer version from the NVIDIA repository, use the following instead ==="
# wget -q https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2604/x86_64/cuda-keyring_1.1-1_all.deb
# sudo dpkg -i cuda-keyring_1.1-1_all.deb
# sudo apt update
# sudo apt install -y cuda-toolkit-XX-Y   # replace with an actual available version
 
echo "=== 4. Install cuDNN ==="
sudo apt install -y cudnn9-cuda-12
 
echo "=== 5. Configure Environment Variables ==="
if ! grep -q "CUDA_HOME" ~/.bashrc; then
  cat >> ~/.bashrc << 'ENVEOF'
 
# CUDA Environment
export CUDA_HOME=/usr/local/cuda
export PATH=$CUDA_HOME/bin:$PATH
export LD_LIBRARY_PATH=$CUDA_HOME/lib64:$LD_LIBRARY_PATH
ENVEOF
fi
 
echo "=== 6. Install NVIDIA Container Toolkit ==="
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | \
  sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
  sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
  sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt update
sudo apt install -y nvidia-container-toolkit
 
echo ""
echo "=== Installation Complete! ==="
echo "Please reboot and then run nvidia-smi and nvcc --version to verify."
echo "Reboot command: sudo reboot"
# Save and run the script
chmod +x gpu-setup.sh
./gpu-setup.sh
Last updated on
OpenClaw