Category Archives: Guide

Migrating ZFS backed VMs between Proxmox Clusters

Migrating VMs inside a proxmox cluster is easy. But what to do when you need to migrate a VM to another proxmox cluster? With a bit of command line, the rest is easy as long as you’re using ZFS. Even for the largest VMs the amount of down time required is mimimal.

#Example scenario #
#Source cluster host: PVEA
#Source cluster VM ID: 199 (WARNING! make sure your IDs don't overlap on clusters otherwise you may overwrite your data)
#Destination cluster host PVEB

#on PVEA take snapshot and call it "migrate"

#confirm you're migrating the right one
PVEA>zfs list

#send data. This command may run very long time, but it's ok because the source VM keeps running
PVEA>zfs send rpool/data/vm-199-disk-0@migrate  | ssh PVEB zfs recv rpool/data/vm-199-disk-0

#when it finishes shutdown VM ID 199
#on PVEA take snapshot and call it "migrate2"

#send incremental data
PVEA>zfs send -i rpool/data/vm-199-disk-0@migrate rpool/data/vm-199-disk-0@migrate2 | ssh PVEB zfs recv rpool/data/vm-199-disk-0

#check that you got the data (this transfer should be super fast)
PVEB>zfs list -t snapshot

#send config over
PVEA>scp 199.conf PVEB:/etc/pve/qemu-server/199.conf 

#on PVEB start VM ID 199 (and optionaly delete snapshots)

Sniffing Android x86 HTTPs traffic with BurpSuite

The task of sniffing traffic from an app on Android x86 ranges from trivial, to very complicated. The difference comes down to the various anti-sniffing techniques employed by the app and or the Android OS itself. This guide will walk you through the process starting with the most trivial scenario, all the way to the most complex.

Scenario 1: ProxyDroid

Before diving into the complexities of sniffing encrypted HTTPs traffic, it’s always good to do a sanity check to make sure that the packets are flowing into BurpSuite

  1. Install ProxyDroid app on Android. In case you are wondering why not use built in proxy settings, it’s because some apps ignore the built in proxy setting. By using ProxyDroid, it eliminates the guesswork and every single request is proxied
  2. Set host and port in ProxyDroid to match settings in BurpSuite
  3. Sniff HTTP traffic, it should be successful
  4. Sniffing HTTPs traffic will fail at this point, but you should at least see connection attempt errors showing up in BurpSuite event log. Those prove that it’s trying. See Scenario 2 on how to fix this.

Scenario 2: Simple HTTPs

When the application established HTTPs connection, the minimum it needs is a valid SSL certificate to prevent man in the middle attacks. By installing BurpSuite root certificate on Android, most applications will accept BurpSuite proxy as a trustworthy server.

  1. Download BurpSuite cert to Android by visiting http://burpsuiteproxyip:8080
  2. Steps to install it depend on exact Android version and sometimes the certificate has to be converted from der format to cer
  3. Sniff HTTPs traffic in BurpSuite
    1. If it works, congratulations!
    2. If you get other errors try Scenario 3

Scenario 3: HTTPs protected with pinning

Some applications aren’t satisfied with any old valid certificate, they are pinned to a specific certificate. Luckily there is a way to bypass this

  1. Android: Install frida-server on Android in /data/local/tmp
  2. Android: Run frida-server -l IPofAndroid (ex.
  3. Windows: Run frida-ps -aiH IpofAndoid to find out application identifier ex:
  4. Windows: download bypass.js into current directory
  5. Windows: frida -H IpofAndroid -f -l bypass.js –no-pause (this will start the app)
  6. Windows: Watch HTTPs requests come in in plain text into BurpSuite, if you’re still gettting errors, especially “Remote host terminated the handshake” in BurpSuite and “CERTIFICATE_VERIFY_FAILED: self signed certificate in certificate chain” in logcat chances are you need to go deeper and look at Scenario 4 (if the app is build on Flutter)

Scenario 4: HTTPs inside Flutter

  • Android: find /data/ | grep
  • Android: upload for analysis in Ghidra
  • Ghidra: search for “ssl_client” string
  • Ghidra: from here find the function that contains “ssl_client”,“ssl_server” and takes long,long,char* as it’s 3 arguments. In my version I had a single match, so I there was no trial and error required to narrow things down.
  • Ghidra: get the function’s signature by copying the first several bytes
  • Ghidra: If you have trouble, here is a more detailed write up , just note the differences between ARM and x86, but the process is very similar.
  • Windows: incorporate the bytes into flutter-addr.js script
  • Windows: frida -H IpOfAndroid -f -l flutter-addr.js –no-pause
  • Windows: you should see the application start and then as the requests are being sent you should see
    • Disabling SSL validation
    • Retval: 0x0
    • If you don’t see retval 0x0 try playing around with “add(0x1)” to address
  • Windows: you should also see plan text requests coming into BurpSuite now

Isolating Untrusted Devices at Switch Level

What do you do when you have a questionable device that you’d like to connect to your switch in order to get it online, but you don’t fully trust it and want to isolate it from your other devices? Or what if you don’t trust any of your devices to talk to each other but you still want to give them internet access? The textbook solution is to create separate VLANs and firewall them from each other at the router. This works, but there is just too much configuration for something so simple. An easier approach is to segregate them at the switch level. That way you can skip all the complexities of setting up a VLAN aware router. Sounds great, what’s the catch? The catch is that you do need a managed switch which supports IPv4 ACLs. Luckily, 1Gbps managed switches are not that expensive these days. In my setup I used and old Dell Powerconnect 6224. Similar syntax will apply to Cisco or HP switch. The general idea is that a managed switch has firewall-like features, so let’s use them.

Let’s look at a specific scenario:

  • LAN gateway IP
  • Port 1 and 2 with devices that have to be segregated from each other and from other devices on the same VLAN
access-list InternetOnlyACL permit ip any
access-list InternetOnlyACL deny ip any
access-list InternetOnlyACL permit ip any any

interface ethernet 1/g1
switchport access vlan 10
ip access-group InternetOnlyACL in 1

interface ethernet 1/g2
switchport access vlan 10
ip access-group InternetOnlyACL in 1

The following line is optional

access-list InternetOnlyACL permit ip any

What it does is make sure that the device can still ping the LAN gateway (important for some devices which verify internet connectivity by pinging LAN gateway) but for most they will happily connect to the internet without ever being able to ping the gateway IP (you just won’t see the first hop in your traceroute). This setup works with both DHCP and static IP assignments.

Caveat: This method blocks funny business on Layer 3 (IP protocol),but it will do absolutely nothing for you if the malicious device starts messing with your network at Layer 2 (ex ARP spoofing). For that you need to configure additional security features on your switch. This is just another layer of protection.

Fixing email display name spoofing

The words “email” and “security” have never mixed, but some things are just too ridiculous to be left as they are.

There are several ways to spoof email, each with their own countermeasures:

  • Sender email address -> DKIM and SPF
  • Sender email display name -> Solution below

While researching ways to fix this, I came across different methods, ranging from database lookups of valid names to looking for suspicious patterns in the display name. None of those methods get to the root of the problem which is that

senders email display name should never have been a “field” that existed in the first place.

Whose idea was it to create a field where anyone can type in “John Smith” as their name, and to top it all off, the recipients mobile device happily says “you have email from John Smith”. Really? From John Smith? Are you sure? No? Not even little bit sure? Is that because anyone can type in anything they want and there is nothing to verify it against? Then why would you show such information as if it’s the truth? No thanks.

To reverse this horrible mistake, the solution is obvious. Let’s remove email display name from existence and display only email addresses (which can still be spoofed, but at least there are countermeasures in place for anyone trying to get away with it)

vi /etc/postfix/header_checks
/^FROM:.<(.@.)>/ REPLACE From: <$1>
/^REPLY-TO:.<(.@.)>/ REPLACE Reply-to: <$1>
postmap -r header_checks
postfix reload

Using Frida to Bypass SSL Pinning on Android

Most modern apps rely on SSL pinning to make sniffing SSL traffic through proxy more difficult. This is great security-in-depth practice, but it’s a real pain when trying to inspect app’s traffic as a part of vulnerability assessment or penetration test. Luckily there if Frida.

  1. Run frida-server
  2. frida-ps -Uai #find the target’s application identified ex:
  3. download file bypass.js into current directory
  4. frida -U -f -l bypass.js –no-pause
  5. done

Proxmox ATI GPU Passthrough Guide

After a lot of fiddling around with settings and hardware, I finally have a stable Proxmox 5.1 ATI GPU pass-through system.   What helped me was this helpful article to finally get all the bugs ironed out.   I did have to make several tweaks for my system.   I’m running an Intel system with ATI Radeon GPU.

  1. Ensure VT-d is supported and enabled in the BIOS
  2. Enable IOMMU on the host
    1. append the following to the GRUB_CMDLINE_LINUX_DEFAULT line in /etc/default/grub
    2. Save your changes by running
  3. Blacklist NVIDIA & Nouveau kernel modules so they don’t get loaded at boot
    1. echo "blacklist nouveau" >> /etc/modprobe.d/blacklist.conf
      echo "blacklist nvidia" >> /etc/modprobe.d/blacklist.conf
      echo "blacklist radeon" >>
    2. Save your changes by running
      update-initramfs -u
  4. Add the following lines to /etc/modules
  5. Reboot the host
  6. Create your Windows VM using the UEFI bios hardware option (not the deafoult seabios) but do not start it yet.  Use VirtIO.  Modify /etc/pve/qemu-server/<vmid>.conf and ensure the following are in the file. Create / modify existing entries as necessary.
    bios: ovmf
    machine: q35
    cpu: host,hidden=1
    numa: 1
  7. Install Windows
    1. Mount second ISO  (virtio-win*.iso)
    2. Load IO driver from d:\viostore\w10\amd64\viostore.inf
    3. After install be sure to enable Remote desktop.
  8. Pass through the GPU.
    1. Modify /etc/pve/qemu-server/<vmid>.conf and add
      hostpci0: <device address>,x-vga=on,pcie=1. Example

      hostpci0: 01:00,x-vga=on,pcie=1
  9. Passthrough USB keyboard and mouse
    1. I find it best to passthrough specific USB ports rather than device IDs.  That way I can hotplug different devices to specific ports later without having to reboot.
  10. Done.


Blue screening when launching certain applications

AMD drivers setup application and/or Windows boot would consistently blue screen on me with the following error:


The fix as outlined here was to create /etc/modprobe.d/kvm.conf and add the parameter “options kvm ignore_msrs=1”

echo "options kvm ignore_msrs=1" > /etc/modprobe.d/kvm.conf

Same fix can be applied at runtime with

echo 1 > /sys/module/kvm/parameters/ignore_msrs

Update 4/9/18: Blue screening happens to Windows 10 1803 as well with the error

System Thread Exception Not Handled

The fix for this is the same – ignore_msrs=1

Frezing keyboard/mouse:

Device Manager -> Human Interface Devices -> Microsoft Hardware USB Keyboard -> Power Management -> Uncheck “Allow the computer to turn off this device to save power”


Other guides require setting up vfio.conf.  With my hardware it was not required.  It’s probably needed for a nVidia card though.

  1. Determine the PCI address of your GPU
    1. Run
      lspci -v

      and look for your card.  Usually 01:00.0 & 01:00.1. You can omit the part after the decimal to include them both in one go – so in that case it would be 01:00

    2. Run lspci -n -s <PCI address> to obtain vendor IDs. Example :
      lspci -n -s 01:00
      01:00.0 0300: 10de:1b81 (rev a1)
      01:00.1 0403: 10de:10f0 (rev a1)
  2. Assign your GPU to vfio driver using the IDs obtained above. Example:
    echo "options vfio-pci ids=10de:1b81,10de:10f0" > /etc/modprobe.d/vfio.conf