Migrating VMs inside a proxmox cluster is easy. But what to do when you need to migrate a VM to another proxmox cluster? With a bit of command line, the rest is easy as long as you’re using ZFS. Even for the largest VMs the amount of down time required is mimimal.
#Example scenario #
#Source cluster host: PVEA
#Source cluster VM ID: 199 (WARNING! make sure your IDs don't overlap on clusters otherwise you may overwrite your data)
#Destination cluster host PVEB
#on PVEA take snapshot and call it "migrate"
#confirm you're migrating the right one
#send data. This command may run very long time, but it's ok because the source VM keeps running
PVEA>zfs send rpool/data/vm-199-disk-0@migrate | ssh PVEB zfs recv rpool/data/vm-199-disk-0
#when it finishes shutdown VM ID 199
#on PVEA take snapshot and call it "migrate2"
#send incremental data
PVEA>zfs send -i rpool/data/vm-199-disk-0@migrate rpool/data/vm-199-disk-0@migrate2 | ssh PVEB zfs recv rpool/data/vm-199-disk-0
#check that you got the data (this transfer should be super fast)
PVEB>zfs list -t snapshot
#send config over
PVEA>scp 199.conf PVEB:/etc/pve/qemu-server/199.conf
#on PVEB start VM ID 199 (and optionaly delete snapshots)
GSM community edition comes with an artificial restriction to the amount of RAM your VM may use. There is a way to remove it
- Boot up using SystemRescueCd or similar
- mount /dev/sda1 /mnt
- vi /mnt/grub/grug.cfg
- remove all references to mem= (you can also remove CPU limits while at it)
The words “email” and “security” have never mixed, but some things are just too ridiculous to be left as they are.
There are several ways to spoof email, each with their own countermeasures:
- Sender email address -> DKIM and SPF
- Sender email display name -> Solution below
While researching ways to fix this, I came across different methods, ranging from database lookups of valid names to looking for suspicious patterns in the display name. None of those methods get to the root of the problem which is that
senders email display name should never have been a “field” that existed in the first place.
Whose idea was it to create a field where anyone can type in “John Smith” as their name, and to top it all off, the recipients mobile device happily says “you have email from John Smith”. Really? From John Smith? Are you sure? No? Not even little bit sure? Is that because anyone can type in anything they want and there is nothing to verify it against? Then why would you show such information as if it’s the truth? No thanks.
To reverse this horrible mistake, the solution is obvious. Let’s remove email display name from existence and display only email addresses (which can still be spoofed, but at least there are countermeasures in place for anyone trying to get away with it)
/^FROM:.<(.@.)>/ REPLACE From: <$1>
/^REPLY-TO:.<(.@.)>/ REPLACE Reply-to: <$1>
postmap -r header_checks
Piping SMTP commands directly into nc doesn’t work very well. What’s needed is a pause before each command. Something like this
cat "mail.txt" |while read L; do sleep "1"; echo "$L"; done | "nc" -C -v "smtp.example.com" "25"
Most modern apps rely on SSL pinning to make sniffing SSL traffic through proxy more difficult. This is great security-in-depth practice, but it’s a real pain when trying to inspect app’s traffic as a part of vulnerability assessment or penetration test. Luckily there if Frida.
- Run frida-server https://www.frida.re/docs/android/
- frida-ps -Uai #find the target’s application identified ex: com.company.myapp
- download file bypass.js into current directory
- frida -U -f com.company.myapp -l bypass.js –no-pause
With increased security, it’s getting trickier to intercept HTTPs traffic send by Android Apps. For that reason most methods rely on rooting Android. But what to do when you don’t have a rooted Android on hand?
Use Android Emulator from Android Studio:
- Go to “Settings->Wireless & Networks->More”
- Go to “Cellular Network Settings”
- Go to “Access Point Names”
- Edit Proxy and Port fields
- Install root cert from BurpSuite or whatever tool you are using to intercept the traffic
- Install and run your APK
When you need to protect an application against XSS and other nasty attacks, but you can’t modify the source code, ModSecurity can save the day.
- Install apache
- Install ModSecurity
- Setup apache as a proxy with the following configuration
#SecRuleRemoveById 999999 whitelist any rules here
- Turn on /etc/modsecurity/modsecurity.conf
5. Turn on CRS blocking in /etc/modsecurity/crs/crs-setup.conf
6. Watch /var/log/apache2/modsec_audit.log for false positives and tweak rules accordingly
Show raw device names
#example output may look something like this
Export to file
dd if=/dev/zvol/rpool/data/vm-100-disk-1 of=/file.raw
Convert to HyperV
qemu-img convert -f raw /file.raw -O vhdx -o subformat=dynamic /file.vhdx
Mount file.vhdx on HyperV and start
One liner method:
Forget about the export and feed the raw device directly to qemu-img
qemu-img convert -f raw /dev/zvol/rpool/data/vm-100-disk-1 -O vhdx -o subformat=dynamic /file.vhdx
After a lot of fiddling around with settings and hardware, I finally have a stable Proxmox 5.1 ATI GPU pass-through system. What helped me was this helpful article to finally get all the bugs ironed out. I did have to make several tweaks for my system. I’m running an Intel system with ATI Radeon GPU.
- Ensure VT-d is supported and enabled in the BIOS
- Enable IOMMU on the host
- append the following to the GRUB_CMDLINE_LINUX_DEFAULT line in /etc/default/grub
- Save your changes by running
- Blacklist NVIDIA & Nouveau kernel modules so they don’t get loaded at boot
echo "blacklist nouveau" >> /etc/modprobe.d/blacklist.conf
echo "blacklist nvidia" >> /etc/modprobe.d/blacklist.conf
echo "blacklist radeon" >>
- Save your changes by running
- Add the following lines to /etc/modules
- Reboot the host
- Create your Windows VM using the UEFI bios hardware option (not the deafoult seabios) but do not start it yet. Use VirtIO. Modify /etc/pve/qemu-server/<vmid>.conf and ensure the following are in the file. Create / modify existing entries as necessary.
- Install Windows
- Mount second ISO (virtio-win*.iso)
- Load IO driver from d:\viostore\w10\amd64\viostore.inf
- After install be sure to enable Remote desktop.
- Pass through the GPU.
- Modify /etc/pve/qemu-server/<vmid>.conf and add
hostpci0: <device address>,x-vga=on,pcie=1. Example
- Passthrough USB keyboard and mouse
- I find it best to passthrough specific USB ports rather than device IDs. That way I can hotplug different devices to specific ports later without having to reboot.
Blue screening when launching certain applications
AMD drivers setup application and/or Windows boot would consistently blue screen on me with the following error:
The fix as outlined here was to create /etc/modprobe.d/kvm.conf and add the parameter “options kvm ignore_msrs=1”
echo "options kvm ignore_msrs=1" > /etc/modprobe.d/kvm.conf
Same fix can be applied at runtime with
echo 1 > /sys/module/kvm/parameters/ignore_msrs
Update 4/9/18: Blue screening happens to Windows 10 1803 as well with the error
System Thread Exception Not Handled
The fix for this is the same – ignore_msrs=1
Device Manager -> Human Interface Devices -> Microsoft Hardware USB Keyboard -> Power Management -> Uncheck “Allow the computer to turn off this device to save power”
Other guides require setting up vfio.conf. With my hardware it was not required. It’s probably needed for a nVidia card though.
- Determine the PCI address of your GPU
and look for your card. Usually 01:00.0 & 01:00.1. You can omit the part after the decimal to include them both in one go – so in that case it would be 01:00
- Run lspci -n -s <PCI address> to obtain vendor IDs. Example :
lspci -n -s 01:00
01:00.0 0300: 10de:1b81 (rev a1)
01:00.1 0403: 10de:10f0 (rev a1)
- Assign your GPU to vfio driver using the IDs obtained above. Example:
echo "options vfio-pci ids=10de:1b81,10de:10f0" > /etc/modprobe.d/vfio.conf