前言
随着互联网设施的建设和云计算的发展，“云桌面”、“云游戏”等概念已逐渐走入人们的视线。无论如何，它都改变了我们理解硬件性能和游戏体验的方式。本系列文章将介绍如何使用Proxmox VE构建一个简单的，支持Windows、Linux的云桌面系统。
认识vGPU
要为VM添加图形功能，一般有三种方法：
- 软件模拟（如“标准VGA”，“VMware SVGA II”）
- PCI Passthrough
- vGPU（mDev / sr-IOV）
性能为 PCI Passthrough > vGPU > 软件模拟
对于vGPU，市面上有三种方案：
| Intel | AMD | NVIDIA |
|---|
| GVT-g | MxGPU | NVIDIA vGPU |
笔者采用的是NVIDIA Tesla P40这款GPU，故本文只介绍NVIDIA vGPU。
vGPU架构
从上图可以看出，NVIDIA vGPU的实现由软硬件协同而成，硬件上有GPU，软件上有NVIDIA vGPU Manager。
vGPU支持的显卡
一般情况下，vGPU只支持数据中心级的Tesla GPU和部分Quadro GPU，大致如下：
| 架构 | 型号 |
|---|
| Maxwell | M6, M10, M60 |
| Pascal | P4, P6, P40, P100, P100 12GB |
| Volta | V100 |
| Turing | T4, RTX 6000, RTX 6000 passive, RTX 8000, RTX 8000 passive |
| Ampere | A2, A10, A16, A40, RTX A5000, RTX A5500, RTX A6000 |
| Ada | L4, L40, RTX 6000 Ada |
不过有热心大神开发了解锁补丁，可以在9，10，20系显卡上解锁vGPU功能，具体链接及教程如下：
{% link https://github.com/mbilker/vgpu_unlock-rs %}
{% link https://gitlab.com/polloloco/vgpu-proxmox %}
支持的平台
根据官方文档，目前支持的平台如下：
- Citrix XenServer
- Linux with KVM（比如Proxmox VE）
- Microsoft Azure Stack HCI
- Microsoft Windows Server（即Hyper-V）
- Nutanix AHV
- VMware vSphere ESXi
vGPU类型
NVIDIA官方介绍如下：
- vCS：NVIDIA 虚拟计算服务器，加速基于 KVM 的基础架构上的虚拟化 AI 计算工作负载。（如
GRID P40-1C） - vWS： NVIDIA RTX 虚拟工作站，适用于使用图形应用程序的创意和技术专业人士的虚拟工作站。（如
GRID P40-1Q） - vPC： NVIDIA 虚拟 PC，适用于使用办公效率应用程序和多媒体的知识工作者的虚拟桌面 (VDI)。（如
GRID P40-1B） - vApp： NVIDIA 虚拟应用程序，采用远程桌面会话主机 (RDSH) 解决方案的应用程序流。（如
GRID P40-1A）
准备vGPU驱动
{% note color:warning 注意 本文中使用的Proxmox VE版本为8.0.3，内核版本为6.2.16-4-pve %}
移除企业源
Proxmox VE默认使用企业源，如果没有订阅密钥是没有权限访问的
echo "deb https://mirrors.tuna.tsinghua.edu.cn/proxmox/debian bookworm pve-no-subscription" >> /etc/apt/sources.list.d/pve-no-subscription.list
rm /etc/apt/sources.list.d/pve-enterprise.list
更新系统
apt update
apt dist-upgrade
reboot
保证内核为最新版本
安装必要工具
apt install -y git build-essential dkms pve-headers mdevctl
其中dkms保证在每次更新内核后会自动编译适应的驱动模块
启用IOMMU
一般系统
编辑/etc/default/grub，找到GRUB_CMDLINE_LINUX_DEFAULT="quiet"，在其后添加：
Intel
intel_iommu=on iommu=pt
AMD
amd_iommu=on iommu=pt
结果应该如下（Intel）：
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt"
ZFS上的系统
编辑/etc/kernel/cmdline，找到root=ZFS=rpool/ROOT/pve-1 boot=zfs，在其后添加：
Intel
intel_iommu=on iommu=pt
AMD
amd_iommu=on iommu=pt
结果应该如下（Intel）：
root=ZFS=rpool/ROOT/pve-1 boot=zfs intel_iommu=on iommu=pt
现在更新引导配置：
proxmox-boot-tool refresh
禁止nouveau驱动
nouveau是一个开源的NVIDIA显卡驱动，它与vGPU驱动冲突，必须禁用它
echo "blacklist nouveau" >> /etc/modprobe.d/blacklist-nouveau.conf
echo "options nouveau modeset=0" >> /etc/modprobe.d/blacklist-nouveau.conf
加载vfio模块
编辑/etc/modules，添加如下内容：
vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd
更新initramfs
update-initramfs -u -k all
重启
reboot
检查IOMMU是否启用
重启后输入：
dmesg | grep -e DMAR -e IOMMU
在我的C602双路服务器上，它输出以下内容：
{% folding open:false %}
root@pve-r720:~# dmesg | grep -e DMAR -e IOMMU
[ 0.000000] Warning: PCIe ACS overrides enabled; This may allow non-IOMMU protected peer-to-peer DMA
[ 0.012034] ACPI: DMAR 0x000000007D3346F4 000158 (v01 DELL PE_SC3 00000001 DELL 00000001)
[ 0.012084] ACPI: Reserving DMAR table memory at [mem 0x7d3346f4-0x7d33484b]
[ 0.591102] DMAR: IOMMU enabled
[ 1.347160] DMAR: Host address width 46
[ 1.347161] DMAR: DRHD base: 0x000000d1100000 flags: 0x0
[ 1.347168] DMAR: dmar0: reg_base_addr d1100000 ver 1:0 cap d2078c106f0466 ecap f020de
[ 1.347170] DMAR: DRHD base: 0x000000dd900000 flags: 0x1
[ 1.347175] DMAR: dmar1: reg_base_addr dd900000 ver 1:0 cap d2078c106f0466 ecap f020de
[ 1.347177] DMAR: RMRR base: 0x0000007f458000 end: 0x0000007f46ffff
[ 1.347178] DMAR: RMRR base: 0x0000007f450000 end: 0x0000007f450fff
[ 1.347180] DMAR: RMRR base: 0x0000007f452000 end: 0x0000007f452fff
[ 1.347181] DMAR: ATSR flags: 0x0
[ 1.347183] DMAR-IR: IOAPIC id 2 under DRHD base 0xd1100000 IOMMU 0
[ 1.347185] DMAR-IR: IOAPIC id 0 under DRHD base 0xdd900000 IOMMU 1
[ 1.347186] DMAR-IR: IOAPIC id 1 under DRHD base 0xdd900000 IOMMU 1
[ 1.347187] DMAR-IR: HPET id 0 under DRHD base 0xdd900000
[ 1.347188] DMAR-IR: x2apic is disabled because BIOS sets x2apic opt out bit.
[ 1.347189] DMAR-IR: Use 'intremap=no_x2apic_optout' to override the BIOS setting.
[ 1.347933] DMAR-IR: Enabled IRQ remapping in xapic mode
[ 2.304979] DMAR: No SATC found
[ 2.304981] DMAR: dmar0: Using Queued invalidation
[ 2.304990] DMAR: dmar1: Using Queued invalidation
[ 2.310738] DMAR: Intel(R) Virtualization Technology for Directed I/O
{% endfolding %}
关键在于DMAR: IOMMU enabled一行，此行表明IOMMU已启用。
安装vGPU驱动
{% note color:warning 注意 本节使用535.54.06版驱动，GRID版本为16.0。 %}
Proxmox VE作为KVM平台，自然需要KVM版的vGPU驱动
目前仅有16.0版的vGPU驱动支持新的6.2内核，15.3、15.1等版本最高支持到PVE 7
很不幸，NVIDIA不会让你随便下载高贵的GRID驱动（fuck-you-nvidia.jpg），你需要在这里注册一个免费的vGPU许可来下载驱动。
{% note color:warning 注意 申请免费许可时，请勿使用@gmail.com、@qq.com等免费邮箱，否则可能会面临人工审核，我的建议是使用自己域名的邮箱。 %}
{% note color:green 提示 觉得注册太麻烦？你也可以去佛西大佬的博客下载 %}
下载完成后，解压zip文件，在Host_Drivers文件夹中找到NVIDIA-Linux-x86_64-535.54.06-vgpu-kvm.run，将其上传到Proxmox VE中
在Proxmox VE中执行以下命令：
./NVIDIA-Linux-x86_64-535.54.06-vgpu-kvm.run --dkms
安装完成后，提示是否要注册DKMS模块，一定要选择Yes，这样在升级内核后，系统会重新编译适应新内核的驱动。
{% note color:warning 注意 由于Proxmox VE使用apt update升级系统时不会升级pve-headers，因此务必在升级系统前安装新的pve-headers，避免造成dkms编译失败。 %}
重启后执行nvidia-smi，在我的双Tesla P40机器上，输出以下内容（无视那些“vgpu”进程）：
执行mdevctl types查看mdev类型：
{% folding open:false %}
root@pve-r720:~# mdevctl types
0000:04:00.0
nvidia-156
Available instances: 0
Device API: vfio-pci
Name: GRID P40-2B
Description: num_heads=4, frl_config=45, framebuffer=2048M, max_resolution=5120x2880, max_instance=12
nvidia-215
Available instances: 0
Device API: vfio-pci
Name: GRID P40-2B4
Description: num_heads=4, frl_config=45, framebuffer=2048M, max_resolution=5120x2880, max_instance=12
nvidia-241
Available instances: 0
Device API: vfio-pci
Name: GRID P40-1B4
Description: num_heads=4, frl_config=45, framebuffer=1024M, max_resolution=5120x2880, max_instance=24
nvidia-46
Available instances: 0
Device API: vfio-pci
Name: GRID P40-1Q
Description: num_heads=4, frl_config=60, framebuffer=1024M, max_resolution=5120x2880, max_instance=24
nvidia-47
Available instances: 0
Device API: vfio-pci
Name: GRID P40-2Q
Description: num_heads=4, frl_config=60, framebuffer=2048M, max_resolution=7680x4320, max_instance=12
nvidia-48
Available instances: 0
Device API: vfio-pci
Name: GRID P40-3Q
Description: num_heads=4, frl_config=60, framebuffer=3072M, max_resolution=7680x4320, max_instance=8
nvidia-49
Available instances: 0
Device API: vfio-pci
Name: GRID P40-4Q
Description: num_heads=4, frl_config=60, framebuffer=4096M, max_resolution=7680x4320, max_instance=6
nvidia-50
Available instances: 0
Device API: vfio-pci
Name: GRID P40-6Q
Description: num_heads=4, frl_config=60, framebuffer=6144M, max_resolution=7680x4320, max_instance=4
nvidia-51
Available instances: 0
Device API: vfio-pci
Name: GRID P40-8Q
Description: num_heads=4, frl_config=60, framebuffer=8192M, max_resolution=7680x4320, max_instance=3
nvidia-52
Available instances: 0
Device API: vfio-pci
Name: GRID P40-12Q
Description: num_heads=4, frl_config=60, framebuffer=12288M, max_resolution=7680x4320, max_instance=2
nvidia-53
Available instances: 0
Device API: vfio-pci
Name: GRID P40-24Q
Description: num_heads=4, frl_config=60, framebuffer=24576M, max_resolution=7680x4320, max_instance=1
nvidia-54
Available instances: 0
Device API: vfio-pci
Name: GRID P40-1A
Description: num_heads=1, frl_config=60, framebuffer=1024M, max_resolution=1280x1024, max_instance=24
nvidia-55
Available instances: 0
Device API: vfio-pci
Name: GRID P40-2A
Description: num_heads=1, frl_config=60, framebuffer=2048M, max_resolution=1280x1024, max_instance=12
nvidia-56
Available instances: 0
Device API: vfio-pci
Name: GRID P40-3A
Description: num_heads=1, frl_config=60, framebuffer=3072M, max_resolution=1280x1024, max_instance=8
nvidia-57
Available instances: 0
Device API: vfio-pci
Name: GRID P40-4A
Description: num_heads=1, frl_config=60, framebuffer=4096M, max_resolution=1280x1024, max_instance=6
nvidia-58
Available instances: 0
Device API: vfio-pci
Name: GRID P40-6A
Description: num_heads=1, frl_config=60, framebuffer=6144M, max_resolution=1280x1024, max_instance=4
nvidia-59
Available instances: 0
Device API: vfio-pci
Name: GRID P40-8A
Description: num_heads=1, frl_config=60, framebuffer=8192M, max_resolution=1280x1024, max_instance=3
nvidia-60
Available instances: 0
Device API: vfio-pci
Name: GRID P40-12A
Description: num_heads=1, frl_config=60, framebuffer=12288M, max_resolution=1280x1024, max_instance=2
nvidia-61
Available instances: 0
Device API: vfio-pci
Name: GRID P40-24A
Description: num_heads=1, frl_config=60, framebuffer=24576M, max_resolution=1280x1024, max_instance=1
nvidia-62
Available instances: 0
Device API: vfio-pci
Name: GRID P40-1B
Description: num_heads=4, frl_config=45, framebuffer=1024M, max_resolution=5120x2880, max_instance=24
0000:42:00.0
nvidia-156
Available instances: 0
Device API: vfio-pci
Name: GRID P40-2B
Description: num_heads=4, frl_config=45, framebuffer=2048M, max_resolution=5120x2880, max_instance=12
nvidia-215
Available instances: 0
Device API: vfio-pci
Name: GRID P40-2B4
Description: num_heads=4, frl_config=45, framebuffer=2048M, max_resolution=5120x2880, max_instance=12
nvidia-241
Available instances: 0
Device API: vfio-pci
Name: GRID P40-1B4
Description: num_heads=4, frl_config=45, framebuffer=1024M, max_resolution=5120x2880, max_instance=24
nvidia-46
Available instances: 0
Device API: vfio-pci
Name: GRID P40-1Q
Description: num_heads=4, frl_config=60, framebuffer=1024M, max_resolution=5120x2880, max_instance=24
nvidia-47
Available instances: 0
Device API: vfio-pci
Name: GRID P40-2Q
Description: num_heads=4, frl_config=60, framebuffer=2048M, max_resolution=7680x4320, max_instance=12
nvidia-48
Available instances: 0
Device API: vfio-pci
Name: GRID P40-3Q
Description: num_heads=4, frl_config=60, framebuffer=3072M, max_resolution=7680x4320, max_instance=8
nvidia-49
Available instances: 0
Device API: vfio-pci
Name: GRID P40-4Q
Description: num_heads=4, frl_config=60, framebuffer=4096M, max_resolution=7680x4320, max_instance=6
nvidia-50
Available instances: 0
Device API: vfio-pci
Name: GRID P40-6Q
Description: num_heads=4, frl_config=60, framebuffer=6144M, max_resolution=7680x4320, max_instance=4
nvidia-51
Available instances: 0
Device API: vfio-pci
Name: GRID P40-8Q
Description: num_heads=4, frl_config=60, framebuffer=8192M, max_resolution=7680x4320, max_instance=3
nvidia-52
Available instances: 0
Device API: vfio-pci
Name: GRID P40-12Q
Description: num_heads=4, frl_config=60, framebuffer=12288M, max_resolution=7680x4320, max_instance=2
nvidia-53
Available instances: 0
Device API: vfio-pci
Name: GRID P40-24Q
Description: num_heads=4, frl_config=60, framebuffer=24576M, max_resolution=7680x4320, max_instance=1
nvidia-54
Available instances: 0
Device API: vfio-pci
Name: GRID P40-1A
Description: num_heads=1, frl_config=60, framebuffer=1024M, max_resolution=1280x1024, max_instance=24
nvidia-55
Available instances: 0
Device API: vfio-pci
Name: GRID P40-2A
Description: num_heads=1, frl_config=60, framebuffer=2048M, max_resolution=1280x1024, max_instance=12
nvidia-56
Available instances: 0
Device API: vfio-pci
Name: GRID P40-3A
Description: num_heads=1, frl_config=60, framebuffer=3072M, max_resolution=1280x1024, max_instance=8
nvidia-57
Available instances: 0
Device API: vfio-pci
Name: GRID P40-4A
Description: num_heads=1, frl_config=60, framebuffer=4096M, max_resolution=1280x1024, max_instance=6
nvidia-58
Available instances: 0
Device API: vfio-pci
Name: GRID P40-6A
Description: num_heads=1, frl_config=60, framebuffer=6144M, max_resolution=1280x1024, max_instance=4
nvidia-59
Available instances: 0
Device API: vfio-pci
Name: GRID P40-8A
Description: num_heads=1, frl_config=60, framebuffer=8192M, max_resolution=1280x1024, max_instance=3
nvidia-60
Available instances: 0
Device API: vfio-pci
Name: GRID P40-12A
Description: num_heads=1, frl_config=60, framebuffer=12288M, max_resolution=1280x1024, max_instance=2
nvidia-61
Available instances: 0
Device API: vfio-pci
Name: GRID P40-24A
Description: num_heads=1, frl_config=60, framebuffer=24576M, max_resolution=1280x1024, max_instance=1
nvidia-62
Available instances: 0
Device API: vfio-pci
Name: GRID P40-1B
Description: num_heads=4, frl_config=45, framebuffer=1024M, max_resolution=5120x2880, max_instance=24
{% endfolding %}
配置vGPU授权服务器
众所周知，vGPU的授权费极其昂贵，不是富哥很难承受的起，很长一段时间内，众多玩家采用伪造PCI ID的形式来伪装成Quadro显卡，从而骗过NVIDIA的检测。但目前已经有成熟的开源vGPU许可服务器fastapi-dls可以直接使用了，官方还提供了Docker，这省了许多事。
安装容器
{% note color:green 提示 建议在虚拟机 / LXC中部署此Docker镜像，PVE中直接安装Docker可能会导致VM断网等问题 %}
# 拉取镜像
docker pull collinwebdesigns/fastapi-dls:latest
# 创建目录
mkdir -p /opt/fastapi-dls/cert
cd /opt/fastapi-dls/cert
# 生成公私钥
openssl genrsa -out /opt/fastapi-dls/cert/instance.private.pem 2048
openssl rsa -in /opt/fastapi-dls/cert/instance.private.pem -outform PEM -pubout -out /opt/fastapi-dls/cert/instance.public.pem
# 生成SSL证书
openssl req -x509 -nodes -days 3650 -newkey rsa:2048 -keyout /opt/fastapi-dls/cert/webserver.key -out /opt/fastapi-dls/cert/webserver.crt
# 创建容器, 1825是授权天数, YOUR_IP处填VM / LXC的IP
docker volume create dls-db
docker run -d --restart=always -e LEASE_EXPIRE_DAYS=1825 -e DLS_URL=<YOUR_IP> -e DLS_PORT=443 -p 443:443 -v /opt/fastapi-dls/cert:/app/cert -v dls-db:/app/database collinwebdesigns/fastapi-dls:latest
自此，vGPU配置完成！
参考资料
NVIDIA Virtual GPU Software Documentation
佛西博客 - 在Proxmox VE 7.2 中开启vGPU_unlock，实现显卡虚拟化 (buduanwang.vip)
PolloLoco / NVIDIA vGPU Guide · GitLab
oscar.krause/fastapi-dls: Minimal Delegated License Service (DLS). This is a mirrored repo from https://git.collinwebdesigns.de/oscar.krause/fastapi-dls. - fastapi-dls - Gitea (publichub.eu)
collinwebdesigns/fastapi-dls - Docker Image | Docker Hub
