ServerOnArm资料收集(入坑教训)

  • 最近入手了OrangePi5Plus(RK3588),纯粹是想体验一下目前的arm架构,顺便搭建一些Android虚拟机,方便进行各种实验。
  • 无奈相关资料实在太少,经过了无数的坑,勉强算走入正轨并成功运行Android虚拟机(带显卡)
  • 下面是整个过程中的一些总结性资料,未认真整理

前言——最惨痛教训

!!!请安装厂商提供的Desktop的系统,不要安装Server版本!!!

!!!请安装厂商提供的Desktop的系统,不要安装Server版本!!!

!!!请安装厂商提供的Desktop的系统,不要安装Server版本!!!

  • 因为系统版本的原因,浪费了一周的时间
  • 根本原因是Desktop的系统自带了显卡驱动,而Server版本并没有,导致显卡无法正常驱动。除非不打算用显卡了,否则必须老老实实安装Desktop版本的系统

一、准备

(一)、资料准备

  • 机器:国产OrangePi 5Plus + eMMC + NVME
  • 系统:准备好厂商提供的 Debian12 Bookworm系统,并安装到 eMMC 中
  • 参考链接1:https://foxi.buduanwang.vip/virtualization/pve/1902.html/
  • 参考链接2:https://pve.proxmox.com/wiki/Install_Proxmox_VE_on_Debian_12_Bookworm
  • 参考链接3:https://www.dsx2020.com/homelab-development-board-arm64-orangepi-5-plus-uses-ssd-m2-to-install-the-pve7-virtual-machine/
  • 参考链接4:https://github.com/jiangcuo/Proxmox-Arm64/wiki/Install-Proxmox-VE-on-Debian-bullseye
  • 参考链接5:https://github.com/jiangcuo/Proxmox-Arm64/wiki/Qemu-VM
  • 参考链接6[WindowsOnArm]:https://zhuanlan.zhihu.com/p/82165601
  • 参考链接7[WindowsOnArm]:http://bbs.risc-v.org.cn/thread-392-1-1.html
  • 参考链接8[RK3588Notes]:https://github.com/jiangcuo/Proxmox-Arm64/wiki/RK3588-NOTE
  • 参考链接9[WOR]:https://worproject.com/guides/how-to-install/on-rockchip
  • 参考链接10[WOR]:https://worproject.com/guides/getting-windows-images
  • 修改root密码,passwd
  • 更新最新组件,apt update & apt upgrade -y

(二)、网络准备

  • 准备网口配置数据

    lan1 enP4p65s0 c0:74:2b:fe:ea:4e
    lan2 enP3p49s0 c0:74:2b:fe:ea:4f

  • 具体操作参考https://foxi.buduanwang.vip/virtualization/pve/1902.html

  • 停止原有 NetworkManager

    systemctl disable NetworkManager
    systemctl stop NetworkManager
    rm /etc/network/interfaces.d/*
    
  • 具体修改后/etc/network/interfaces配置数据如下

    iface enP4p65s0 inet manual
    
    iface enP3p49s0 inet manual
    
    auto vmbr0
    iface vmbr0 inet static
            address 192.168.103.241/22
            gateway 192.168.103.1
            bridge-ports enP4p65s0
            bridge-stp off
            bridge-fd 0
    
    auto vmbr1
    iface vmbr1 inet manual
            bridge-ports enP3p49s0
            bridge-stp off
            bridge-fd 0
    
  • 具体修改后的 /etc/hostname配置如下

  • opi5plus
    
  • 具体修改后 /etc/hosts/ 配置如下

    127.0.0.1       localhost.localdomain localhost
    192.168.103.241 opi5plus.pvetest.com opi5plus
    
    ::1             localhost ip6-localhost ip6-loopback
    fe00::0         ip6-localnet
    ff00::0         ip6-mcastprefix
    ff02::1         ip6-allnodes
    ff02::2         ip6-allrouters
    
  • 其他操作

    #添加DNS
    echo "nameserver 223.5.5.5" >>/etc/resolv.conf
    
    
    
  • 注:原文中的安装gpg证书这一步失败,我改为了使用 wget 命令就能成功

    wget https://mirrors.apqa.cn/proxmox/debian/pveport.gpg -O /etc/apt/trusted.gpg.d/pveport.gpg  
    
  • RK3588笔记https://github.com/jiangcuo/Proxmox-Arm64/wiki/RK3588-NOTE

二、安装Docker的Android虚拟机

标准镜像——未驱动GPU

#安卓11
mkdir -p /mnt/nvme0n1p2/docker/redroid_11/data

docker run -dit \
--restart=always \
--name redroid_11 \
--privileged \
--pull always \
-v /mnt/nvme0n1p2/docker/redroid_11/data:/data \
-p 5511:5555 \
redroid/redroid:11.0.0-latest


#安卓12
mkdir -p /mnt/nvme0n1p2/docker/redroid_12/data

docker run -dit \
--restart=always \
--name redroid_12 \
--privileged \
--pull always \
-v /mnt/nvme0n1p2/docker/redroid_12/data:/data \
-p 5512:5555 \
redroid/redroid:12.0.0-latest

尝试驱动GPUhttps://radxa.com/blog/Redroid-running-on-ROCK-5B/

mkdir -p /mnt/nvme0n1p2/docker/redroid_12/data
mkdir -p /mnt/nvme0n1p2/docker/redroid_13/data
mkdir -p /mnt/nvme0n1p2/docker/redroid_14/data

#横屏chisbread/rk3588-gaming:redroid-firefly
docker run -itd \
--restart=always \
--name redroid_12 \
--privileged \
-v /mnt/nvme0n1p2/docker/redroid_12/data:/data \
-p 5512:5555 \
chisbread/rk3588-gaming:redroid-firefly \
androidboot.redroid_height=1600 \
androidboot.redroid_width=2560 \
androidboot.redroid_dpi=480 \
androidboot.redroid_gpu_mode=host \
--mount=type=bind,source=/dev/mali0,destination=/dev/mali0

#竖屏dobox:rk3588-202303017
docker run -itd \
--restart=always \
--name redroid_13 \
--privileged \
-v /mnt/nvme0n1p2/docker/redroid_13/data:/data \
-p 5513:5555 \
registry.cn-hangzhou.aliyuncs.com/whsyf/dobox:rk3588-202303017 \
androidboot.redroid_gpu_mode=host \
--mount=type=bind,source=/dev/mali0,destination=/dev/mali0

#registry.cn-hangzhou.aliyuncs.com/whsyf/dobox:rk3588-202303017
#shangzebei/rk3588 androidboot.redroid_gpu_mode=mali


#进入Redroid容器查看安卓日志
docker exec -it redroid_12 sh
logcat

尝试容器虚拟安卓https://wiki.t-firefly.com/zh_CN/ROC-RK3588S-PC/usage_aic.html

ARM GPU讨论https://github.com/istoreos/istoreos/issues/1121

rk3588-云手机-dockerhttps://blog.csdn.net/zcg121314/article/details/128994290

甲骨文ARM利用docker镜像运行安卓云手机 http://bbs.apkplus.com/?t/128.html

GPU驱动https://github.com/ChisBread/rk3588-gaming-step-by-step/blob/main/gpu-envs.md

在 RK3399 上运行开源的 GPU 驱动https://dev.t-firefly.com/thread-100733-1-1.html

尝鲜 ARM Mali 开源 Vulkan 驱动 panvkhttps://zhuanlan.zhihu.com/p/480270449

Manjaro ARM Orange Pi 5 Plushttps://github.com/manjaro-arm/opi5-plus-images

# step0. 内核编译参数(也许不是必选)
#编译内核时关闭 CONFIG_DRM_IGNORE_IOTCL_PERMIT
# step1. 下载GPU固件到/lib/firmware
sudo wget https://github.com/tsukumijima/libmali-rockchip/tree/master/firmware/g610/mali_csffw.bin -O /lib/firmware/mali_csffw.bin
# step2. 安装环境依赖 
sudo apt install build-essential cmake meson git python3-mako libexpat1-dev bison flex libwayland-egl-backend-dev libxext-dev libxfixes-dev libxcb-glx0-dev libxcb-shm0-dev libxcb-dri2-0-dev libxcb-dri3-dev libxcb-present-dev libxshmfence-dev libxxf86vm-dev libxrandr-dev libwayland-dev libx11-xcb-dev
# step3. 编译安装libdrm (覆盖原有的旧版) 
git clone https://gitlab.freedesktop.org/mesa/drm
mkdir drm/build
cd drm/build
meson
sudo ninja install
# step4. 编译安装wayland-protocols
git clone https://gitlab.freedesktop.org/wayland/wayland-protocols
mkdir wayland-protocols/build
cd wayland-protocols/build
git checkout 1.24
meson
sudo ninja install
# 注:出现Pkg-config binary for machine 0 not found. Giving up.
# 解决办法:apt install pkg-config
# step5. 编译安装panfork (安装到/opt/panfrost)
git clone https://gitlab.com/panfork/mesa
mkdir mesa/build
cd mesa/build
meson -Dgallium-drivers=panfrost -Dvulkan-drivers= -Dllvm=disabled --prefix=/opt/panfrost
sudo ninja install
# step1. 下载GPU固件到/lib/firmware
sudo wget https://github.com/tsukumijima/libmali-rockchip/tree/master/firmware/g610/mali_csffw.bin -O /lib/firmware/mali_csffw.bin
# step2. 添加ppa
sudo add-apt-repository ppa:liujianfeng1994/panfork-mesa
# step3. 使用中科大ppa反代(可选)
sudo sed -i -e "s@http.*://ppa.launchpadcontent.net@https://launchpad.proxy.ustclug.org@g" /etc/apt/sources.list.d/*
sudo apt update
# step4. 原地升级
sudo apt dist-upgrade
# step5. 安装32位环境(可选)
sudo apt install -y libegl-mesa0:armhf libgbm1:armhf libgl1-mesa-dri:armhf libglapi-mesa:armhf libglx-mesa0:armhf
#检查linux模块是否能支持android
apt install linux-modules-extra-`uname -r`
modprobe binder_linux devices="binder,hwbinder,vndbinder" #进程通信模块
modprobe ashmem_linux #内存共享模块
mkdir -p /mnt/nvme0n1p2/docker/redroid_1/data

docker run -itd --rm --privileged \
--pull always \
-v /mnt/nvme0n1p2/docker/redroid_1/data:/data \
-v /dev/net/tun:/dev/tun \
-v /dev/mali0:/dev/mali0 \
-p 5501:5555 \
--name redroid_1 \
registry.cn-hangzhou.aliyuncs.com/whsyf/dobox:rk3588-202303017

docker exec -it redroid_1 sh

logcat

Linux GPU初始化失败

dmesg | grep -i mali

执行结果如下:

image

尝试驱动GPU

mkdir -p /mnt/nvme0n1p2/docker/redroid_13/data

docker run -itd \
--restart=always \
--name redroid_13 \
--privileged \
-v /dev/mali0:/dev/mali0 \
-v /mnt/nvme0n1p2/docker/redroid_13/data:/data \
-p 5513:5555 \
shangzebei/rk3588 androidboot.redroid_gpu_mode=mali

添加防火墙规则 iptables

#临时开放端口
iptables -A INPUT -p tcp --dport 5500:5599 -j ACCEPT

#永久开放端口
apt-get install iptables-persistent
netfilter-persistent save

三、编译内核

参考文章:

https://blog.seeflower.dev/archives/203/

https://blog.csdn.net/u013833472/article/details/129485916

四、安装PVE

  • 输入安装命令
export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

apt update && apt install -y proxmox-ve

一、准备

(一)、资料准备

  • 机器:国产OrangePi 5Plus + eMMC + NVME
  • 系统:准备好厂商提供的 Debian12 Bookworm系统,并安装到 eMMC 中
  • 参考链接1:https://foxi.buduanwang.vip/virtualization/pve/1902.html/
  • 参考链接2:https://pve.proxmox.com/wiki/Install_Proxmox_VE_on_Debian_12_Bookworm
  • 参考链接3:https://www.dsx2020.com/homelab-development-board-arm64-orangepi-5-plus-uses-ssd-m2-to-install-the-pve7-virtual-machine/
  • 参考链接4:https://github.com/jiangcuo/Proxmox-Arm64/wiki/Install-Proxmox-VE-on-Debian-bullseye
  • 参考链接5:https://github.com/jiangcuo/Proxmox-Arm64/wiki/Qemu-VM
  • 参考链接6[WindowsOnArm]:https://zhuanlan.zhihu.com/p/82165601
  • 参考链接7[WindowsOnArm]:http://bbs.risc-v.org.cn/thread-392-1-1.html
  • 参考链接8[RK3588Notes]:https://github.com/jiangcuo/Proxmox-Arm64/wiki/RK3588-NOTE
  • 参考链接9[WOR]:https://worproject.com/guides/how-to-install/on-rockchip
  • 参考链接10[WOR]:https://worproject.com/guides/getting-windows-images
  • 修改root密码,passwd
  • 更新最新组件,apt update & apt upgrade -y

(二)、网络准备

  • 准备网口配置数据

    lan1 enP4p65s0 c0:74:2b:fe:ea:4e
    lan2 enP3p49s0 c0:74:2b:fe:ea:4f

  • 具体操作参考https://foxi.buduanwang.vip/virtualization/pve/1902.html

  • 停止原有 NetworkManager

    systemctl disable NetworkManager
    systemctl stop NetworkManager
    rm /etc/network/interfaces.d/*
    
  • 具体修改后/etc/network/interfaces配置数据如下

    iface enP4p65s0 inet manual
    
    iface enP3p49s0 inet manual
    
    auto vmbr0
    iface vmbr0 inet static
            address 192.168.1.222/22
            gateway 192.168.1.1
            bridge-ports enP4p65s0
            bridge-stp off
            bridge-fd 0
    
    auto vmbr1
    iface vmbr1 inet manual
            bridge-ports enP3p49s0
            bridge-stp off
            bridge-fd 0
    
  • 具体修改后的 /etc/hostname配置如下

  • opi5plus
    
  • 具体修改后 /etc/hosts/ 配置如下

    127.0.0.1       localhost.localdomain localhost
    192.168.1.222 opi5plus.pvetest.com opi5plus
    
    ::1             localhost ip6-localhost ip6-loopback
    fe00::0         ip6-localnet
    ff00::0         ip6-mcastprefix
    ff02::1         ip6-allnodes
    ff02::2         ip6-allrouters
    
  • 其他操作

    #添加DNS
    echo "nameserver 223.5.5.5" >>/etc/resolv.conf
    
    
    
  • 注:原文中的安装gpg证书这一步失败,我改为了使用 wget 命令就能成功

    wget https://mirrors.apqa.cn/proxmox/debian/pveport.gpg -O /etc/apt/trusted.gpg.d/pveport.gpg  
    
  • RK3588笔记https://github.com/jiangcuo/Proxmox-Arm64/wiki/RK3588-NOTE

二、安装Docker的Android虚拟机

标准镜像——未驱动GPU

#安卓11
mkdir -p /mnt/nvme0n1p2/docker/redroid_11/data

docker run -dit \
--restart=always \
--name redroid_11 \
--privileged \
--pull always \
-v /mnt/nvme0n1p2/docker/redroid_11/data:/data \
-p 5511:5555 \
redroid/redroid:11.0.0-latest


#安卓12
mkdir -p /mnt/nvme0n1p2/docker/redroid_12/data

docker run -dit \
--restart=always \
--name redroid_12 \
--privileged \
--pull always \
-v /mnt/nvme0n1p2/docker/redroid_12/data:/data \
-p 5512:5555 \
redroid/redroid:12.0.0-latest

尝试驱动GPUhttps://radxa.com/blog/Redroid-running-on-ROCK-5B/

mkdir -p /mnt/nvme0n1p2/docker/redroid_12/data
mkdir -p /mnt/nvme0n1p2/docker/redroid_13/data
mkdir -p /mnt/nvme0n1p2/docker/redroid_14/data

#横屏chisbread/rk3588-gaming:redroid-firefly
docker run -itd \
--restart=always \
--name redroid_12 \
--privileged \
-v /mnt/nvme0n1p2/docker/redroid_12/data:/data \
-p 5512:5555 \
chisbread/rk3588-gaming:redroid-firefly \
androidboot.redroid_height=1600 \
androidboot.redroid_width=2560 \
androidboot.redroid_dpi=480 \
androidboot.redroid_gpu_mode=host \
--mount=type=bind,source=/dev/mali0,destination=/dev/mali0

#竖屏dobox:rk3588-202303017
docker run -itd \
--restart=always \
--name redroid_13 \
--privileged \
-v /mnt/nvme0n1p2/docker/redroid_13/data:/data \
-p 5513:5555 \
registry.cn-hangzhou.aliyuncs.com/whsyf/dobox:rk3588-202303017 \
androidboot.redroid_gpu_mode=host \
--mount=type=bind,source=/dev/mali0,destination=/dev/mali0

#registry.cn-hangzhou.aliyuncs.com/whsyf/dobox:rk3588-202303017
#shangzebei/rk3588 androidboot.redroid_gpu_mode=mali


#进入Redroid容器查看安卓日志
docker exec -it redroid_12 sh
logcat

尝试容器虚拟安卓https://wiki.t-firefly.com/zh_CN/ROC-RK3588S-PC/usage_aic.html

ARM GPU讨论https://github.com/istoreos/istoreos/issues/1121

rk3588-云手机-dockerhttps://blog.csdn.net/zcg121314/article/details/128994290

甲骨文ARM利用docker镜像运行安卓云手机 http://bbs.apkplus.com/?t/128.html

GPU驱动https://github.com/ChisBread/rk3588-gaming-step-by-step/blob/main/gpu-envs.md

在 RK3399 上运行开源的 GPU 驱动https://dev.t-firefly.com/thread-100733-1-1.html

尝鲜 ARM Mali 开源 Vulkan 驱动 panvkhttps://zhuanlan.zhihu.com/p/480270449

Manjaro ARM Orange Pi 5 Plushttps://github.com/manjaro-arm/opi5-plus-images

# step0. 内核编译参数(也许不是必选)
#编译内核时关闭 CONFIG_DRM_IGNORE_IOTCL_PERMIT
# step1. 下载GPU固件到/lib/firmware
sudo wget https://github.com/tsukumijima/libmali-rockchip/tree/master/firmware/g610/mali_csffw.bin -O /lib/firmware/mali_csffw.bin
# step2. 安装环境依赖 
sudo apt install build-essential cmake meson git python3-mako libexpat1-dev bison flex libwayland-egl-backend-dev libxext-dev libxfixes-dev libxcb-glx0-dev libxcb-shm0-dev libxcb-dri2-0-dev libxcb-dri3-dev libxcb-present-dev libxshmfence-dev libxxf86vm-dev libxrandr-dev libwayland-dev libx11-xcb-dev
# step3. 编译安装libdrm (覆盖原有的旧版) 
git clone https://gitlab.freedesktop.org/mesa/drm
mkdir drm/build
cd drm/build
meson
sudo ninja install
# step4. 编译安装wayland-protocols
git clone https://gitlab.freedesktop.org/wayland/wayland-protocols
mkdir wayland-protocols/build
cd wayland-protocols/build
git checkout 1.24
meson
sudo ninja install
# 注:出现Pkg-config binary for machine 0 not found. Giving up.
# 解决办法:apt install pkg-config
# step5. 编译安装panfork (安装到/opt/panfrost)
git clone https://gitlab.com/panfork/mesa
mkdir mesa/build
cd mesa/build
meson -Dgallium-drivers=panfrost -Dvulkan-drivers= -Dllvm=disabled --prefix=/opt/panfrost
sudo ninja install
# step1. 下载GPU固件到/lib/firmware
sudo wget https://github.com/tsukumijima/libmali-rockchip/tree/master/firmware/g610/mali_csffw.bin -O /lib/firmware/mali_csffw.bin
# step2. 添加ppa
sudo add-apt-repository ppa:liujianfeng1994/panfork-mesa
# step3. 使用中科大ppa反代(可选)
sudo sed -i -e "s@http.*://ppa.launchpadcontent.net@https://launchpad.proxy.ustclug.org@g" /etc/apt/sources.list.d/*
sudo apt update
# step4. 原地升级
sudo apt dist-upgrade
# step5. 安装32位环境(可选)
sudo apt install -y libegl-mesa0:armhf libgbm1:armhf libgl1-mesa-dri:armhf libglapi-mesa:armhf libglx-mesa0:armhf
#检查linux模块是否能支持android
apt install linux-modules-extra-`uname -r`
modprobe binder_linux devices="binder,hwbinder,vndbinder" #进程通信模块
modprobe ashmem_linux #内存共享模块
mkdir -p /mnt/nvme0n1p2/docker/redroid_1/data

docker run -itd --rm --privileged \
--pull always \
-v /mnt/nvme0n1p2/docker/redroid_1/data:/data \
-v /dev/net/tun:/dev/tun \
-v /dev/mali0:/dev/mali0 \
-p 5501:5555 \
--name redroid_1 \
registry.cn-hangzhou.aliyuncs.com/whsyf/dobox:rk3588-202303017

docker exec -it redroid_1 sh

logcat

Linux GPU初始化失败

dmesg | grep -i mali

执行结果如下:

image

尝试驱动GPU

mkdir -p /mnt/nvme0n1p2/docker/redroid_13/data

docker run -itd \
--restart=always \
--name redroid_13 \
--privileged \
-v /dev/mali0:/dev/mali0 \
-v /mnt/nvme0n1p2/docker/redroid_13/data:/data \
-p 5513:5555 \
shangzebei/rk3588 androidboot.redroid_gpu_mode=mali

添加防火墙规则 iptables

#临时开放端口
iptables -A INPUT -p tcp --dport 5500:5599 -j ACCEPT

#永久开放端口
apt-get install iptables-persistent
netfilter-persistent save

三、编译内核

参考文章:

https://blog.seeflower.dev/archives/203/

https://blog.csdn.net/u013833472/article/details/129485916

四、安装PVE

  • 输入安装命令
export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

apt update && apt install -y proxmox-ve