Adventure migrating to Proxmox
I've begun descending into this madness of containerization and virtualization. I did a tiny bit of it at work for a test machine and omg was it a HUGE help. This is all because of LXC containers. "oops... uhhh... screw it" kill the container and start over which only takes about 30seconds to 1 minute. The resource overhead of a LXC container is so small compared to a VM. < insert real metrics here: container, VM, bare metal>
Yes baremetal is a point of comparison. I listened to a video from either Level1Tech or TechTechPotato, i forget who, talking to a VMware person. He mentioned that the VMs in some cases would perform better than baremetal because windows no longer had to deal with drivers because the hypervisor did everything instead. I believe it. The VM would not have to deal with waiting for hardware to initialize. There are lots of little other things that add up. Of course I'm not dealing with 100-core servers and actual server motherboards that actually take a minutes just to get to the bootloader to just start up but you get the picture. For VMs vs Containers, I'm still learning. Containers have much less overhead. They simulate the conceptual environment instead of simulating the hardware. For VMs you can pass hardware along to the VM using things like IOMMU and SR-IOV. Concepts I am aware of but haven't fully delved into yet.
The mental exercise of migrating some of the servers between nodes is making me realize what the purpose of a Storage Area Network / SAN is for now. LTT said recently in a video about how a SAN is a role, not a piece of hardware. That comment made me more perceptive of it and I'm seeing why it exists now. or at least I think I know what it exists now.
For example, take hypervisor A and B. These are badass servers with lots of cores and ram and storage. They are working together in a cluster. Virtual machine, VM, is working on A but something happens and A goes down. If it was replicated across B as well then B can continue the work. The problem is that this means you have two copies of your VM sitting on systems. One on A and one on B. If you add a third system, SAN, that only handles storage, then your VM can reside on that computer instead. Or maybe instead of having the whole VM on the SAN, you put the large dataset your VM was working with.
A applicable example for me is Jellyfin. I can put all of the media on the SAN, then that means I can move the Jellyfin container between my two servers. Lets say that I don't need very much horsepower at the moment so I shut the second server off entirely and move all services to one server to save power. Cool! Of course that doesn't work in my case as I would need 3 servers then. The SAN would just be the same as a NAS + networking fun.
Here is my bigger challenge. Minimal downtime while I learn how to migrate all services to being hypervised, containers, etc. Starting original layout:
- Ubuntuserver
- Mumble - High Priority, daily affected avg: 6 users
- Web
- Jellyfin - Medium Priority, daily affect avg: 3users
- Nextcloud
- various games
- openvpn client
- QBittorrent
- Rosie
- Zoneminder (maybe Frigate later)
- Home assistant
- zwave-js-ui
Hardware changes: Adding 1tb NVME drive to each server. Adding ECC ram to 64GB.
That's it for now. will need a seperate post (:
list of links to not lose
- https://gist.github.com/crundberg/a77b22de856e92a7e14c81f40e7a74bd
- https://forum.proxmox.com/threads/usb-passthrough-to-a-container-lxc.101741/
- https://medium.com/@konpat/usb-passthrough-to-an-lxc-proxmox-15482674f11d