Category Archives: Solved

Migrating ZFS backed VMs between Proxmox Clusters

Migrating VMs inside a proxmox cluster is easy. But what to do when you need to migrate a VM to another proxmox cluster? With a bit of command line, the rest is easy as long as you’re using ZFS. Even for the largest VMs the amount of down time required is mimimal.

####################################
#Example scenario #
####################################
#Source cluster host: PVEA
#Source cluster VM ID: 199 (WARNING! make sure your IDs don't overlap on clusters otherwise you may overwrite your data)
#Destination cluster host PVEB

#on PVEA take snapshot and call it "migrate"

#confirm you're migrating the right one
PVEA>zfs list


#send data. This command may run very long time, but it's ok because the source VM keeps running
PVEA>zfs send rpool/data/vm-199-disk-0@migrate  | ssh PVEB zfs recv rpool/data/vm-199-disk-0

#when it finishes shutdown VM ID 199
#on PVEA take snapshot and call it "migrate2"

#send incremental data
PVEA>zfs send -i rpool/data/vm-199-disk-0@migrate rpool/data/vm-199-disk-0@migrate2 | ssh PVEB zfs recv rpool/data/vm-199-disk-0

#check that you got the data (this transfer should be super fast)
PVEB>zfs list -t snapshot

#send config over
PVEA>scp 199.conf PVEB:/etc/pve/qemu-server/199.conf 

#on PVEB start VM ID 199 (and optionaly delete snapshots)
#done!

Fixing email display name spoofing

The words “email” and “security” have never mixed, but some things are just too ridiculous to be left as they are.

There are several ways to spoof email, each with their own countermeasures:

  • Sender email address -> DKIM and SPF
  • Sender email display name -> Solution below

While researching ways to fix this, I came across different methods, ranging from database lookups of valid names to looking for suspicious patterns in the display name. None of those methods get to the root of the problem which is that

senders email display name should never have been a “field” that existed in the first place.

Whose idea was it to create a field where anyone can type in “John Smith” as their name, and to top it all off, the recipients mobile device happily says “you have email from John Smith”. Really? From John Smith? Are you sure? No? Not even little bit sure? Is that because anyone can type in anything they want and there is nothing to verify it against? Then why would you show such information as if it’s the truth? No thanks.

To reverse this horrible mistake, the solution is obvious. Let’s remove email display name from existence and display only email addresses (which can still be spoofed, but at least there are countermeasures in place for anyone trying to get away with it)

vi /etc/postfix/header_checks
/^FROM:.<(.@.)>/ REPLACE From: <$1>
/^REPLY-TO:.<(.@.)>/ REPLACE Reply-to: <$1>
postmap -r header_checks
postfix reload


SSL Sniffing with Android x86 and frida

  1. Install Android x86 (I used 8.1r3)
  2. Configure with virtwifi
  3. Long press to access advanced menu’s proxy settings
  4. Proxy traffic to BurpSuite or similar
  5. Install frida-server on Android x86
    1. follow https://www.frida.re/docs/android/
    2. frida-ps -aiH 192.168.x.x #find the target’s application identified ex: com.company.myapp
    3. download file bypass.js into current directory
    4. frida -H 192.168.x.x -f com.company.myapp -l bypass.js –no-pause
  6. burpsuite should now see plain text requests coming in

Using Frida to Bypass SSL Pinning on Android

Most modern apps rely on SSL pinning to make sniffing SSL traffic through proxy more difficult. This is great security-in-depth practice, but it’s a real pain when trying to inspect app’s traffic as a part of vulnerability assessment or penetration test. Luckily there if Frida.

  1. Run frida-server https://www.frida.re/docs/android/
  2. frida-ps -Uai #find the target’s application identified ex: com.company.myapp
  3. download file bypass.js into current directory
  4. frida -U -f com.company.myapp -l bypass.js –no-pause
  5. done

HTTPs Inspection of Android APK

With increased security, it’s getting trickier to intercept HTTPs traffic send by Android Apps. For that reason most methods rely on rooting Android. But what to do when you don’t have a rooted Android on hand?

Use Android Emulator from Android Studio:

  1. Go to “Settings->Wireless & Networks->More”
  2. Go to “Cellular Network Settings”
  3. Go to “Access Point Names”
  4. Edit Proxy and Port fields
  5. Install root cert from BurpSuite or whatever tool you are using to intercept the traffic
  6. Install and run your APK

WAF Proxy with ModSecurity and Apache

When you need to protect an application against XSS and other nasty attacks, but you can’t modify the source code, ModSecurity can save the day.

  1. Install apache
  2. Install ModSecurity
  3. Setup apache as a proxy with the following configuration
    <Location />
    ProxyPass https://www.test.com/
    ProxyPassReverse https://www.test.com/
    
    #SecRuleRemoveById 999999 whitelist any rules here
    </Location>
    
    
  4. Turn on /etc/modsecurity/modsecurity.conf
SecRuleEngine On

#SecRuleEngine DetectionOnly

5. Turn on CRS blocking in /etc/modsecurity/crs/crs-setup.conf

SecDefaultAction "phase:1,log,auditlog,deny"
SecDefaultAction "phase:2,log,auditlog,deny"

#SecDefaultAction "phase:1,log,auditlog,pass"
#SecDefaultAction "phase:2,log,auditlog,pass"

6. Watch /var/log/apache2/modsec_audit.log for false positives and tweak rules accordingly

Convert Proxmox .raw to HyperV .vhdx

Show raw device names

zfs list 
#example output may look something like this
rpool/data/vm-100-disk-1

Export to file

dd if=/dev/zvol/rpool/data/vm-100-disk-1 of=/file.raw

Convert to HyperV

qemu-img convert -f raw /file.raw -O vhdx -o subformat=dynamic /file.vhdx

Mount file.vhdx on HyperV and start

One liner method:

Forget about the export and feed the raw device directly to qemu-img

qemu-img convert -f raw /dev/zvol/rpool/data/vm-100-disk-1 -O vhdx -o subformat=dynamic /file.vhdx

 

Proxmox ATI GPU Passthrough Guide

After a lot of fiddling around with settings and hardware, I finally have a stable Proxmox 5.1 ATI GPU pass-through system.   What helped me was this helpful article to finally get all the bugs ironed out.   I did have to make several tweaks for my system.   I’m running an Intel system with ATI Radeon GPU.

  1. Ensure VT-d is supported and enabled in the BIOS
  2. Enable IOMMU on the host
    1. append the following to the GRUB_CMDLINE_LINUX_DEFAULT line in /etc/default/grub
      intel_iommu=on
    2. Save your changes by running
      update-grub
  3. Blacklist NVIDIA & Nouveau kernel modules so they don’t get loaded at boot
    1. echo "blacklist nouveau" >> /etc/modprobe.d/blacklist.conf
      echo "blacklist nvidia" >> /etc/modprobe.d/blacklist.conf
      echo "blacklist radeon" >>
      /etc/modprobe.d/blacklist.conf
    2. Save your changes by running
      update-initramfs -u
  4. Add the following lines to /etc/modules
    vfio
    vfio_iommu_type1
    vfio_pci
    vfio_virqfd
  5. Reboot the host
  6. Create your Windows VM using the UEFI bios hardware option (not the deafoult seabios) but do not start it yet.  Use VirtIO.  Modify /etc/pve/qemu-server/<vmid>.conf and ensure the following are in the file. Create / modify existing entries as necessary.
    bios: ovmf
    machine: q35
    cpu: host,hidden=1
    numa: 1
  7. Install Windows
    1. Mount second ISO  (virtio-win*.iso)
    2. Load IO driver from d:\viostore\w10\amd64\viostore.inf
    3. After install be sure to enable Remote desktop.
  8. Pass through the GPU.
    1. Modify /etc/pve/qemu-server/<vmid>.conf and add
      hostpci0: <device address>,x-vga=on,pcie=1. Example

      hostpci0: 01:00,x-vga=on,pcie=1
  9. Passthrough USB keyboard and mouse
    1. I find it best to passthrough specific USB ports rather than device IDs.  That way I can hotplug different devices to specific ports later without having to reboot.
  10. Done.

Troubleshooting

Blue screening when launching certain applications

AMD drivers setup application and/or Windows boot would consistently blue screen on me with the following error:

kmode_exception_not_handled

The fix as outlined here was to create /etc/modprobe.d/kvm.conf and add the parameter “options kvm ignore_msrs=1”

echo "options kvm ignore_msrs=1" > /etc/modprobe.d/kvm.conf

Same fix can be applied at runtime with

echo 1 > /sys/module/kvm/parameters/ignore_msrs

Update 4/9/18: Blue screening happens to Windows 10 1803 as well with the error

System Thread Exception Not Handled

The fix for this is the same – ignore_msrs=1

Frezing keyboard/mouse:

Device Manager -> Human Interface Devices -> Microsoft Hardware USB Keyboard -> Power Management -> Uncheck “Allow the computer to turn off this device to save power”

Notes:

Other guides require setting up vfio.conf.  With my hardware it was not required.  It’s probably needed for a nVidia card though.

  1. Determine the PCI address of your GPU
    1. Run
      lspci -v

      and look for your card.  Usually 01:00.0 & 01:00.1. You can omit the part after the decimal to include them both in one go – so in that case it would be 01:00

    2. Run lspci -n -s <PCI address> to obtain vendor IDs. Example :
      lspci -n -s 01:00
      01:00.0 0300: 10de:1b81 (rev a1)
      01:00.1 0403: 10de:10f0 (rev a1)
  2. Assign your GPU to vfio driver using the IDs obtained above. Example:
    echo "options vfio-pci ids=10de:1b81,10de:10f0" > /etc/modprobe.d/vfio.conf