Category Archives: Tools

Convert Proxmox .raw to HyperV .vhdx

Show raw device names

zfs list 
#example output may look something like this
rpool/data/vm-100-disk-1

Export to file

dd if=/dev/zvol/rpool/data/vm-100-disk-1 of=/file.raw

Convert to HyperV

qemu-img convert -f raw /file.raw -O vhdx -o subformat=dynamic /file.vhdx

Mount file.vhdx on HyperV and start

One liner method:

Forget about the export and feed the raw device directly to qemu-img

qemu-img convert -f raw /dev/zvol/rpool/data/vm-100-disk-1 -O vhdx -o subformat=dynamic /file.vhdx

 

Proxmox ATI GPU Passthrough Guide

After a lot of fiddling around with settings and hardware, I finally have a stable Proxmox 5.1 ATI GPU pass-through system.   What helped me was this helpful article to finally get all the bugs ironed out.   I did have to make several tweaks for my system.   I’m running an Intel system with ATI Radeon GPU.

  1. Ensure VT-d is supported and enabled in the BIOS
  2. Enable IOMMU on the host
    1. append the following to the GRUB_CMDLINE_LINUX_DEFAULT line in /etc/default/grub
      intel_iommu=on
    2. Save your changes by running
      update-grub
  3. Blacklist NVIDIA & Nouveau kernel modules so they don’t get loaded at boot
    1. echo "blacklist nouveau" >> /etc/modprobe.d/blacklist.conf
      echo "blacklist nvidia" >> /etc/modprobe.d/blacklist.conf
      echo "blacklist radeon" >>
      /etc/modprobe.d/blacklist.conf
    2. Save your changes by running
      update-initramfs -u
  4. Add the following lines to /etc/modules
    vfio
    vfio_iommu_type1
    vfio_pci
    vfio_virqfd
  5. Reboot the host
  6. Create your Windows VM using the UEFI bios hardware option (not the deafoult seabios) but do not start it yet.  Use VirtIO.  Modify /etc/pve/qemu-server/<vmid>.conf and ensure the following are in the file. Create / modify existing entries as necessary.
    bios: ovmf
    machine: q35
    cpu: host,hidden=1
    numa: 1
  7. Install Windows
    1. Mount second ISO  (virtio-win*.iso)
    2. Load IO driver from d:\viostore\w10\amd64\viostore.inf
    3. After install be sure to enable Remote desktop.
  8. Pass through the GPU.
    1. Modify /etc/pve/qemu-server/<vmid>.conf and add
      hostpci0: <device address>,x-vga=on,pcie=1. Example

      hostpci0: 01:00,x-vga=on,pcie=1
  9. Passthrough USB keyboard and mouse
    1. I find it best to passthrough specific USB ports rather than device IDs.  That way I can hotplug different devices to specific ports later without having to reboot.
  10. Done.

Troubleshooting

Blue screening when launching certain applications

AMD drivers setup application and/or Windows boot would consistently blue screen on me with the following error:

kmode_exception_not_handled

The fix as outlined here was to create /etc/modprobe.d/kvm.conf and add the parameter “options kvm ignore_msrs=1”

echo "options kvm ignore_msrs=1" > /etc/modprobe.d/kvm.conf

Update 4/9/18: Blue screening happens to Windows 10 1803 as well with the error

System Thread Exception Not Handled

The fix for this is the same – ignore_msrs=1

Frezing keyboard/mouse:

Device Manager -> Human Interface Devices -> Microsoft Hardware USB Keyboard -> Power Management -> Uncheck “Allow the computer to turn off this device to save power”

Notes:

Other guides require setting up vfio.conf.  With my hardware it was not required.  It’s probably needed for a nVidia card though.

  1. Determine the PCI address of your GPU
    1. Run
      lspci -v

      and look for your card.  Usually 01:00.0 & 01:00.1. You can omit the part after the decimal to include them both in one go – so in that case it would be 01:00

    2. Run lspci -n -s <PCI address> to obtain vendor IDs. Example :
      lspci -n -s 01:00
      01:00.0 0300: 10de:1b81 (rev a1)
      01:00.1 0403: 10de:10f0 (rev a1)
  2. Assign your GPU to vfio driver using the IDs obtained above. Example:
    echo "options vfio-pci ids=10de:1b81,10de:10f0" > /etc/modprobe.d/vfio.conf

Reset Windows Passwords with chntpw

My go-to tool for resetting forgotten windows passwords used to be  Offline NT password recovery

It still works fine, but I discovered that the underlying tool that does the resets is available on other distributions including Ubuntu Live CD (after apt-get) and it’s fully baked into my favorite System Rescue CD

That tool is chntpw

Even though the manual page says it’s for Windows NT/2000, so far it works on more modern windows too (local users):

Windows XP
Windows Vista
Windows 7
Windows 8
Windows 8.1
Windows 10

OpenBSD BGP Router Benchmarking

I had an Open BSD 5.9 router that started to loss packets with moderate traffic levels (sometimes as low as 300 Mbs).  The router had 1 Gbs upstream interfaces so I figured that the fault was with packets per second limits.   What I needed to do is figure out was way to have the router forward a large amount of packets per second and see how it holds up

I booted up a spare Linux server and hooked it up to one end of the router.  Downloaded and compiled a UDP packet generator and ran the following

./snd -l DestinationIP:1234 -m 1000 -n 1000000000

DestinationIP is any IP that routes through the router.  Interestingly the because UDP is connectionless protocol, the IP doesn’t even have to belong to an actual server – much less to a server you control.  As such port 1234 can be any number.   What’s happening is that you’re essentially pointing a UDP fire hose to flow through your router into a non existent destination.

On the router I watched the stats while decreasing -m until 28 (-m is the packet size and smaller size results in higher PPS)

#On Router
systat ifstat

I got only 50K PPS before packet loss appeared.  This is way too low for the hardware that I was using ( decent server model circa 2015 )

Reading other people’s benchmark results, it appeared that upgrading to the latest OpenBSD 6.4 may help.   After the upgrade there was a huge increase.  I am now getting:

Lab test: 1400K PPS (very close to theoretical maximum for 1 Gbs connection with 84 byte packets)
Real world: 680K PPS

Just the upgrade of the OS increased the PPS performance by more than 13x.   This without even trying additional tweaks found here: How to receive a million packets per second

pingdiff: Bandwidth Estimation using ICMP

Previously I found that to estimate bandwidth of an IP network link it’s possible to do it based on a formula that takes into account the difference between a small ping and a large ping. (see pingb)

Now I updated that algorithm to look for the smallest ping replies.   This works better because network links are inherently noisy and there is large variation between pings.   By doing the measurement on the minimum values over many tries, the estimation becomes more accurate.

#!/bin/bash

#
# pingdiff : bandwidth estimation through ICMP echo
# ver : 0.1
#
# (C) 2018 Tomas Florian
#
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.



# Usage
# pingdiff IP MAXPAYLOAD
#
# Experiment with MAX payload size between 100 and 64000 the bigger you can specify
# the more accurate the estimate will be
# The longer you let the loop running the more accurate the results will be
# in my experience the accuracy is +/- 50%
# Surprisingly the accuracy is not affected by link saturation as much as I would expect
# Ex. 10Gbps link shows 4Gbps whether saturated or not

#Errata
# bad error handling (ex Small ping > Big Ping, ping timeouts etc)

#optional configure these by running against 127.0.0.1 to get ping overhead
#LOCALHOST_OVERHEAD_SMALL=0.011
LOCALHOST_OVERHEAD_SMALL=0
#LOCALHOST_OVERHEAD_BIG=0.071
LOCALHOST_OVERHEAD_BIG=0

PAYLOAD_SMALL=1



if [ -z "$1" ]; then
 echo "usage: pingdiff IP PAYLOAD"
 exit 1
fi



if [ -z "$2" ]; then
 echo "usage: pingdiff IP PAYLOAD"
 exit 1
fi



IP=$1
PAYLOAD=$2

echo "IP=$IP, PAYLOAD=$PAYLOAD"

echo "" > big
echo "" > small

#for i in `jot 999`;
while [ 1 ]
do

#echo $i

BIG=`ping -s$PAYLOAD -c1 $IP | grep time | grep -o time=.* | cut -d'=' -f2 | cut -d' ' -f1`
sleep 0.5
SMALL=`ping -s$PAYLOAD_SMALL -c1 $IP | grep time | grep -o time=.* | cut -d'=' -f2 | cut -d' ' -f1`
sleep 0.5
#BIG=`ping -s$PAYLOAD -c1 $IP | grep time | grep -o time=.* | cut -d'=' -f2 | cut -d' ' -f1`
#sleep 0.5

BIG=`echo "$BIG - $LOCALHOST_OVERHEAD_BIG" | bc`
SMALL=`echo "$SMALL - $LOCALHOST_OVERHEAD_SMALL" | bc`



echo $BIG >> big
echo $SMALL >> small



SMALL_MIN=`cat small | grep [0-9].* | awk '{if(min==""){min=max=$1}; if($1>max) {max=$1}; if($1<min) {min=$1}; total+=$1; count+=1} END {print min}'`
BIG_MIN=`cat big | grep [0-9].* | awk '{if(min==""){min=max=$1}; if($1>max) {max=$1}; if($1<min) {min=$1}; total+=$1; count+=1} END {print min}'`



PAYLOAD_DIFF=`echo "$PAYLOAD - $PAYLOAD_SMALL" | bc`

#echo "SMALL_MIN=$SMALL_MIN"
#echo "BIG_MIN=$BIG_MIN"

ROUNDTRIPDIFF=`echo "$BIG_MIN - $SMALL_MIN" | bc`
PINGSPERSEC=`echo "1000 / $ROUNDTRIPDIFF" | bc`

KBPS=`echo "$PINGSPERSEC * $PAYLOAD_DIFF * 8 * 2 / 1000" | bc`
MBPS=`echo $KBPS / 1000 | bc`

echo "$MBPS Mbps (SMALL_MIN=$SMALL_MIN,BIG_MIN=$BIG_MIN)"

done

Beyond ping: Detect packet drop within TCP session

What do you do if a particular website or a particular web service is crawling and yet pings, mtr traceroutes come back perfect?   Or what if you can’t ping at all because of firewall in the way?   Is something throttling this specific connection on the remote end?

The way to look under the hood is to look at the specific TCP session and see how it’s performing.   This is where wireshark comes to the rescue with the following metrics:

  • Packet Loss:   Filter by criteria “tcp.analysis.lost_segment”
  • Up/down throughput:  Go to statistics -> Conversations -> scroll all the way right for “bps” stats
  • Latency: Select TCP packet in question -> Expand TCP SEQ/ACK analysis -> look for RTT to ACK

 

Howto Run Hyper-V 2016 Core without Domain Controller

Hyper-V offers a free version.  The catch is that it is the core Hyper-V without the Windows interface. That’s fine because none of the other hypervisors such as XenServer or ESXi have a graphical interface running on the hypervisor itself either. The trouble is that Microsoft makes working with Hyper-V without a GUI very tricky, unless you join it to a domain. In my opinion joining a hypervisor to a domain is undesirable. Either you have to run a domain controller as a VM creating a weird chicken-and-egg problem, or alternatively you have to run the domain controller as a separate physical host – who in this day and age wants to do that though?

The solution to all this is to jump through couple extra hoops and run Hyper-V without domain controller.

  1. Pick a management machine.  Let’s call it MANAGE01.  Preferably Windows 10 Pro.  In my tests I didn’t have it joined to a domain.  Add user called Admin with password xyz.
  2. Install Hyper-V 2016 on server.  At the end of the installation create a new user called Admin with password xyz (important that username and password matches exactly with step 1).   Change host name to HYPER-V01  (Optional: enable Remote desktop and enable pings)
  3. On MANAGE01 do these steps:
    1. edit hosts file and add entry 1.2.3.4 HYPER-V01
    2. Open Powershell with admin privileges,
    3. Start-Service WinRM
    4. winrm set winrm/config/client ‘@{TrustedHosts=”HYPER-V01″}’
    5. Stop-Service WinRM
    6. Open Hyper-v Manager, and `connect to server`
    7. Enter HYPER-V01
  4. Done

For multiple hosts the command is

winrm set winrm/config/client '@{TrustedHosts="HYPER-V01,HYPER-V02"}'

To get replication you need to do the following:

  • netsh advfirewall firewall add rule name=”Open Port 443″ dir=in action=allow protocol=TCP localport=443
  • Install self signed SSL certificates

Password expiry can be a problem when running without DC.  For that reason it’s best to disable password expiry on all hosts

NET accounts /MAXPWAGE:UNLIMITED

 

Greedy Search Engines – All the same

I fell in love with Google search back when people still used WebCrawler.  But that’s because back then Google was the underdog.   I still like using Google.   Google just works, but once in a while I check out what others are doing because I’m rooting for the next uderdog.  Now that Google is a giant behemoth, making their ads harder to spot, and ratcheting up their tracking, I figured it’s about time to switch.  But where to?

Supposedly Bing is the closest contender.   While I have no love for Microsoft, I was desperate.  So I tried it.   It wasn’t the first time.   I try this once a year or so just to see what’s out there.   Last time I tried Bing was kind of ok except for that giant background picture that slows everything to a crawl when I’m working over remote desktop.   This time I was horrified.  The first screen of results were pure ads.   There wasn’t a single real result until I scrolled down past the first screen.   The bottom of the screen wasn’t any better.  If this is the strategy Microsoft is using to catch up with Google Search then good luck to them – they are screwed.  Or are they?   After all I bet the sheeple who get Bing as default on their Windows 10 don’t notice the tiny little word “ad” beside each of those ads and are just fine whatever they were fed.   After all it was close enough…. and perhaps even helpful.  This is not for me …and that’s why Ad Block Plus exists.

Introducing FreeMyBits.com

FreeMyBits helps you get your data back under your control.

There are many ways your organization can lose control of its precious data. You may have been locked-in by vendor or service provider, you may have outgrown a system or you need to downsize a legacy behemoth, you may have lost a key employee, you may have been hacked or your perhaps obsolescence simply caught up with you.

Whatever the reason is, FreeMyBits helps small and medium business like yours to get control back. Their team consists of specialists with proven enterprise expertise.  FreeMyBits can handle both mainstream systems and obscure legacy contraptions. They succeed where others struggle.

FreeMyBits also provides planning and consulting services to minimize the risk of lock-in.

Script for Verifying Authoritative Name Servers

As organizations tighten their web server security, attackers are looking for new weak links.  One of these weak links are DNS hosting companies and domain registrars.   Even though your own security may be top notch, are you sure that your domain registrar is equally bullet proof?   If the hacker can convince your registrar to change name servers to those under the hacker’s control, the hacker can start doing all kinds of nasty stuff.

Other than shopping around for DNS hosting companies and registrars that have strong security practices including (but not limited to) two-factor authentication, what else can be done?

Monitoring; If you can quickly detect that someone has made an unauthorized change to your name servers or other DNS records, you can minimize the damage by acting quickly and taking control back away from them.

Below is a script that will check your DNS records against a while list of allowed entries:

<?php

//Change this:
//Domain you wish to verify
$domain = "www.somedomain.com";
//Valid DNS hosts for your domain (detect change at registrar)
$authWhitelist= array("ns1.somednshost.com","ns2.somednshost.com");
//Valid A records (detect change at DNS host)
$aRecordWhitelist= array("11.22.33.44","55.66.77.88");

require_once 'Net/DNS2.php';

function lookupAuthorities($domain,$server)
{
 $serverIP = gethostbyname($server);

 $resolver = new Net_DNS2_Resolver( array('nameservers' => array($serverIP)) );

 $resp = $resolver->query($domain, 'A');
 //print_r($resp);

 $servers= array();

 foreach($resp->authority as &$record)
 {
 //keep for later
 $name = $record->name;

 array_push($servers,$record->nsdname);
 }

 $result = (object) [
 'name' => $name,
 'servers' => $servers,
 ];

 return $result;
}

function lookupAuthoritativeNameservers($domain)
{
 $authorities =lookupAuthorities($domain,"a.root-servers.net");

 if($authorities->name != $domain)
 {
 $authorities = lookupAuthorities($domain,$authorities->servers[0]);
 }
 return $authorities->servers;

}

function lookupARecords($domain,$authNameserver)
{
 $serverIP = gethostbyname($authNameserver);

 $resolver = new Net_DNS2_Resolver( array('nameservers' => array($serverIP)) );

 $resp = $resolver->query($domain, 'A');
 //print_r($resp);

 $servers= array();


 foreach($resp->answer as &$record)
 {


 array_push($servers,$record->address);
 }


 return $servers;

}



$authServers = lookupAuthoritativeNameservers($domain);

$authDiff=array_diff($authServers,$authWhitelist);

$aRecords = lookupARecords($domain,$authServers[0]);

$aRecordDiff = array_diff($aRecords,$aRecordWhitelist);


if(count($authDiff)>0 || count($aRecordDiff)>0)
{
 echo ($domain . " Fail (See offending entries below)\n");
 if(count($authDiff)>0) print_r($authDiff);
 if(count($aRecordDiff)>0) print_r($aRecords);

}
else
{
 echo($domain . " Pass\n");
}


?>

Script will return “Pass” if everything is ok.  If any of the results is not specified on the white list script will show “Fail” including a list of all offending entries.

You can either run this as a cron job and include an email notification by tweaking this script, or you can check it periodically by a tool such as PRTG.