Ollama on RHEL 9 and Rocky Linux: Enterprise Setup and SELinux Guide

RHEL 9 and Rocky Linux are the dominant choices for enterprise Linux deployments. Running Ollama on these distributions means working within a security-first environment: SELinux is enforcing by default, firewalld replaces UFW, subscription management controls package access, and FIPS compliance may be a requirement. These are not obstacles — they are features that protect production systems. But they require specific configuration that differs significantly from the Ubuntu/Debian instructions you find in most Ollama tutorials.

This guide covers the complete process of deploying Ollama on RHEL 9 and Rocky Linux 9 for enterprise environments. We address SELinux policy configuration, NVIDIA driver installation through the correct RHEL channels, firewalld rules, systemd hardening, integration with Red Hat's subscription model, and enterprise security considerations that a production deployment demands.

NVIDIA Driver Installation on RHEL 9 / Rocky Linux

RHEL and Rocky Linux do not include NVIDIA drivers in their default repositories. You need to add NVIDIA's official RPM repository or, for RHEL, use the Red Hat-certified driver packages.

Rocky Linux 9 — NVIDIA Repository Method

# Enable EPEL (Extra Packages for Enterprise Linux)
sudo dnf install epel-release -y

# Enable the CodeReady Builder repository (needed for dependencies)
sudo dnf config-manager --set-enabled crb

# Add the NVIDIA CUDA repository
sudo dnf config-manager --add-repo   https://developer.download.nvidia.com/compute/cuda/repos/rhel9/x86_64/cuda-rhel9.repo

# Install the NVIDIA driver
sudo dnf module install nvidia-driver:latest-dkms -y

# Or install a specific version
sudo dnf install nvidia-driver-550 -y

# Reboot to load the new kernel modules
sudo reboot

# Verify after reboot
nvidia-smi

RHEL 9 — Red Hat Certified Drivers

# For RHEL 9 with an active subscription:
# Enable the required repositories
sudo subscription-manager repos --enable=rhel-9-for-x86_64-baseos-rpms
sudo subscription-manager repos --enable=rhel-9-for-x86_64-appstream-rpms
sudo subscription-manager repos --enable=codeready-builder-for-rhel-9-x86_64-rpms

# Install EPEL
sudo dnf install https://dl.fedoraproject.org/pub/epel/epel-release-latest-9.noarch.rpm -y

# Add NVIDIA repository
sudo dnf config-manager --add-repo   https://developer.download.nvidia.com/compute/cuda/repos/rhel9/x86_64/cuda-rhel9.repo

# Install the driver with DKMS support
sudo dnf install nvidia-driver nvidia-driver-dkms -y

# Install kernel-devel for module building
sudo dnf install kernel-devel-$(uname -r) kernel-headers-$(uname -r) -y

# Build and load modules
sudo dkms autoinstall
sudo reboot

# Verify
nvidia-smi

Secure Boot Considerations

# If secure boot is enabled, NVIDIA kernel modules need to be signed
# Check secure boot status
mokutil --sb-state

# If enabled, DKMS should handle signing automatically
# If modules fail to load, check:
dmesg | grep -i "module verification failed"

# Generate and enroll a MOK key
sudo mokutil --import /var/lib/dkms/mok.pub
# Set a password when prompted, then reboot
# During boot, the MOK Manager will prompt you to enroll the key

Installing Ollama

# Install Ollama using the official script
curl -fsSL https://ollama.com/install.sh | sh

# Verify the installation
ollama --version
systemctl status ollama

# Check GPU detection
journalctl -u ollama | grep -i gpu

SELinux Configuration

SELinux on RHEL 9 and Rocky Linux is enforcing by default, and it should stay that way in production. Disabling SELinux to make Ollama work is the wrong approach — instead, create proper policies that allow Ollama to operate within SELinux's security framework.

Identifying SELinux Denials

# Check if SELinux is blocking Ollama
sudo ausearch -m avc -ts recent | grep ollama
sudo sealert -a /var/log/audit/audit.log | grep ollama

# Common denials you will see:
# - Access to /dev/nvidia* devices
# - Network port binding on 11434
# - Reading/writing model files in custom directories
# - Memory mapping operations for GPU compute

Creating a Custom SELinux Policy for Ollama

# Install policy development tools
sudo dnf install policycoreutils-python-utils setools-console -y

# Method 1: Generate a policy from audit denials (recommended)
# First, run Ollama and let it fail, then generate policy from the denials
sudo ausearch -m avc -ts today | audit2allow -M ollama_policy

# Review the generated policy before applying
cat ollama_policy.te

# Install the policy module
sudo semodule -i ollama_policy.pp

# Method 2: Create a comprehensive policy manually
cat > ollama_custom.te << 'EOF'
module ollama_custom 1.0;

require {
    type init_t;
    type unreserved_port_t;
    type usr_t;
    type nvidia_device_t;
    class tcp_socket name_bind;
    class chr_file { open read write ioctl map getattr };
    class file { read write create unlink rename open getattr map execute };
    class dir { read write add_name remove_name search getattr };
}

# Allow Ollama to bind to port 11434
allow init_t unreserved_port_t:tcp_socket name_bind;

# Allow access to NVIDIA GPU devices
allow init_t nvidia_device_t:chr_file { open read write ioctl map getattr };

# Allow model file operations
allow init_t usr_t:file { read write create unlink rename open getattr map execute };
allow init_t usr_t:dir { read write add_name remove_name search getattr };
EOF

# Compile and install
checkmodule -M -m -o ollama_custom.mod ollama_custom.te
semodule_package -o ollama_custom.pp -m ollama_custom.mod
sudo semodule -i ollama_custom.pp

# Verify the module is loaded
sudo semodule -l | grep ollama

SELinux File Contexts for Custom Model Directories

# If using a custom model directory, set the correct SELinux context
sudo semanage fcontext -a -t usr_t '/data/ollama/models(/.*)?'
sudo restorecon -Rv /data/ollama/models

# Verify contexts
ls -laZ /data/ollama/models/

# For the Ollama binary
ls -Z /usr/local/bin/ollama

SELinux Port Labeling

# Label Ollama's port for SELinux
sudo semanage port -a -t http_port_t -p tcp 11434

# If the port is already defined in another type:
sudo semanage port -m -t http_port_t -p tcp 11434

# Verify
sudo semanage port -l | grep 11434

Firewalld Configuration

RHEL 9 and Rocky Linux use firewalld instead of iptables or UFW.

# Check firewalld status
sudo firewall-cmd --state

# Create a custom service for Ollama
sudo firewall-cmd --permanent --new-service=ollama
sudo firewall-cmd --permanent --service=ollama --add-port=11434/tcp
sudo firewall-cmd --permanent --service=ollama --set-short="Ollama LLM API"
sudo firewall-cmd --permanent --service=ollama --set-description="Ollama local LLM inference server"

# Add the service to the appropriate zone
# For internal network access only:
sudo firewall-cmd --permanent --zone=internal --add-service=ollama

# Or for the default zone:
sudo firewall-cmd --permanent --add-service=ollama

# Restrict to specific source networks
sudo firewall-cmd --permanent --zone=internal --add-source=10.0.0.0/8
sudo firewall-cmd --permanent --zone=internal --add-service=ollama

# Reload to apply
sudo firewall-cmd --reload

# Verify
sudo firewall-cmd --list-all
sudo firewall-cmd --zone=internal --list-all

Systemd Service Hardening

Enterprise deployments require a more restrictive systemd service configuration than the default.

# Create a hardened override
sudo systemctl edit ollama.service

# Hardened service configuration for enterprise RHEL/Rocky
[Service]
# Run as dedicated user
User=ollama
Group=ollama

# Restrict filesystem access
ProtectSystem=strict
ProtectHome=yes
ReadWritePaths=/usr/share/ollama/.ollama
PrivateTmp=yes

# Network restrictions
RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX

# Device access — NVIDIA GPU only
DevicePolicy=closed
DeviceAllow=/dev/nvidia0 rw
DeviceAllow=/dev/nvidia1 rw
DeviceAllow=/dev/nvidiactl rw
DeviceAllow=/dev/nvidia-uvm rw
DeviceAllow=/dev/nvidia-uvm-tools rw

# Capability restrictions
CapabilityBoundingSet=
NoNewPrivileges=yes

# Memory limits
LimitMEMLOCK=infinity
LimitNOFILE=65535

# Environment
Environment="OLLAMA_HOST=0.0.0.0"
Environment="OLLAMA_FLASH_ATTENTION=1"

# Apply changes
sudo systemctl daemon-reload
sudo systemctl restart ollama

# Verify the service is running with restrictions
systemctl show ollama | grep -E "ProtectSystem|ProtectHome|DevicePolicy"

NVIDIA Container Toolkit for Docker Deployments

# Install the NVIDIA Container Toolkit on RHEL 9 / Rocky Linux
curl -s -L https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo |   sudo tee /etc/yum.repos.d/nvidia-container-toolkit.repo

sudo dnf install nvidia-container-toolkit -y

# Configure Docker (if using Docker)
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

# Configure Podman (if using Podman — default on RHEL 9)
sudo nvidia-ctk cdi generate --output=/etc/cdi/nvidia.yaml

# Verify with Podman
podman run --rm --device nvidia.com/gpu=all   nvidia/cuda:12.4.0-base-ubi9 nvidia-smi

Podman as an Alternative to Docker

RHEL 9 ships with Podman as the default container runtime. Running Ollama in a Podman container is a valid enterprise deployment strategy.

# Run Ollama in Podman with GPU access
podman run -d   --name ollama   --device nvidia.com/gpu=all   -v ollama_data:/root/.ollama:Z   -p 11434:11434   --restart=always   docker.io/ollama/ollama:latest

# The :Z suffix on the volume mount tells Podman to set the
# correct SELinux context automatically

# Pull a model
podman exec ollama ollama pull llama3.1:8b

# Create a systemd service from the Podman container
podman generate systemd --name ollama --new --files
sudo mv container-ollama.service /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable container-ollama

Enterprise Monitoring Integration

# Create a health check script for enterprise monitoring (Nagios/Zabbix/Prometheus)
cat > /usr/local/bin/check_ollama.sh << 'CHECKEOF'
#!/bin/bash
# Nagios-compatible health check for Ollama

OLLAMA_URL="${OLLAMA_URL:-http://localhost:11434}"

# Check API availability
response=$(curl -s -o /dev/null -w "%{http_code}"   --max-time 5 "${OLLAMA_URL}/api/version" 2>/dev/null)

if [ "$response" != "200" ]; then
    echo "CRITICAL - Ollama API not responding (HTTP $response)"
    exit 2
fi

# Check GPU availability
gpu_info=$(nvidia-smi --query-gpu=memory.used,memory.total,temperature.gpu   --format=csv,noheader,nounits 2>/dev/null)

if [ -z "$gpu_info" ]; then
    echo "WARNING - Ollama running but GPU not detected"
    exit 1
fi

mem_used=$(echo "$gpu_info" | cut -d',' -f1 | tr -d ' ')
mem_total=$(echo "$gpu_info" | cut -d',' -f2 | tr -d ' ')
temp=$(echo "$gpu_info" | cut -d',' -f3 | tr -d ' ')

usage_pct=$((mem_used * 100 / mem_total))

if [ "$temp" -gt 85 ]; then
    echo "WARNING - GPU temperature ${temp}C | gpu_mem_pct=${usage_pct}%;80;95 gpu_temp=${temp};80;90"
    exit 1
fi

echo "OK - Ollama running, GPU ${usage_pct}% VRAM, ${temp}C | gpu_mem_pct=${usage_pct}%;80;95 gpu_temp=${temp};80;90"
exit 0
CHECKEOF

chmod +x /usr/local/bin/check_ollama.sh

Frequently Asked Questions

Should I disable SELinux to run Ollama on RHEL 9?

Absolutely not. Disabling SELinux removes a critical security layer that protects your entire system, not just Ollama. The correct approach is to generate a targeted SELinux policy module using audit2allow based on the actual denials your Ollama installation encounters. This takes 10-15 minutes of effort but keeps your system's security posture intact. If you are in a regulated environment (PCI DSS, HIPAA, SOC 2), disabling SELinux may violate your compliance requirements regardless of what application you are trying to run.

Can I use Ollama with RHEL's FIPS 140-2/140-3 compliance mode?

FIPS mode restricts the cryptographic algorithms available on the system. Ollama itself does not perform cryptographic operations that conflict with FIPS requirements — it is doing matrix multiplication for inference. However, if you serve the Ollama API over TLS (through a reverse proxy), ensure your TLS configuration uses FIPS-approved ciphers. The NVIDIA driver and CUDA operations are not subject to FIPS cryptographic restrictions since they are not performing security-relevant cryptographic operations. Test your deployment in FIPS mode before production rollout.

How do I integrate Ollama with Red Hat Identity Management (IdM) for access control?

Ollama has no built-in authentication. For enterprise access control, deploy a reverse proxy (nginx or HAProxy) in front of Ollama and configure it to authenticate against IdM/FreeIPA using SSSD and PAM, or use Keycloak as an authentication gateway with OIDC. Open WebUI provides user management with LDAP integration, which can connect to Red Hat IdM. The proxy approach is more flexible because it works regardless of which frontend you use. Configure the proxy to check group membership in IdM before allowing API access.

RHEL linux selinux ollama Enterprise Rocky Linux