OpenBSD BGP Router Benchmarking

I had an Open BSD 5.9 router that started to loss packets with moderate traffic levels (sometimes as low as 300 Mbs).  The router had 1 Gbs upstream interfaces so I figured that the fault was with packets per second limits.   What I needed to do is figure out was way to have the router forward a large amount of packets per second and see how it holds up

I booted up a spare Linux server and hooked it up to one end of the router.  Downloaded and compiled a UDP packet generator and ran the following

./snd -l DestinationIP:1234 -m 1000 -n 1000000000

DestinationIP is any IP that routes through the router.  Interestingly the because UDP is connectionless protocol, the IP doesn’t even have to belong to an actual server – much less to a server you control.  As such port 1234 can be any number.   What’s happening is that you’re essentially pointing a UDP fire hose to flow through your router into a non existent destination.

On the router I watched the stats while decreasing -m until 28 (-m is the packet size and smaller size results in higher PPS)

#On Router
systat ifstat

I got only 50K PPS before packet loss appeared.  This is way too low for the hardware that I was using ( decent server model circa 2015 )

Reading other people’s benchmark results, it appeared that upgrading to the latest OpenBSD 6.4 may help.   After the upgrade there was a huge increase.  I am now getting:

Lab test: 1400K PPS (very close to theoretical maximum for 1 Gbs connection with 84 byte packets)
Real world: 680K PPS

Just the upgrade of the OS increased the PPS performance by more than 13x.   This without even trying additional tweaks found here: How to receive a million packets per second

pingdiff: Bandwidth Estimation using ICMP

Previously I found that to estimate bandwidth of an IP network link it’s possible to do it based on a formula that takes into account the difference between a small ping and a large ping. (see pingb)

Now I updated that algorithm to look for the smallest ping replies.   This works better because network links are inherently noisy and there is large variation between pings.   By doing the measurement on the minimum values over many tries, the estimation becomes more accurate.


# pingdiff : bandwidth estimation through ICMP echo
# ver : 0.1
# (C) 2018 Tomas Florian
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# GNU General Public License for more details.
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <>.

# Usage
# pingdiff IP MAXPAYLOAD
# Experiment with MAX payload size between 100 and 64000 the bigger you can specify
# the more accurate the estimate will be
# The longer you let the loop running the more accurate the results will be
# in my experience the accuracy is +/- 50%
# Surprisingly the accuracy is not affected by link saturation as much as I would expect
# Ex. 10Gbps link shows 4Gbps whether saturated or not

# bad error handling (ex Small ping > Big Ping, ping timeouts etc)

#optional configure these by running against to get ping overhead


if [ -z "$1" ]; then
 echo "usage: pingdiff IP PAYLOAD"
 exit 1

if [ -z "$2" ]; then
 echo "usage: pingdiff IP PAYLOAD"
 exit 1



echo "" > big
echo "" > small

#for i in `jot 999`;
while [ 1 ]

#echo $i

BIG=`ping -s$PAYLOAD -c1 $IP | grep time | grep -o time=.* | cut -d'=' -f2 | cut -d' ' -f1`
sleep 0.5
SMALL=`ping -s$PAYLOAD_SMALL -c1 $IP | grep time | grep -o time=.* | cut -d'=' -f2 | cut -d' ' -f1`
sleep 0.5
#BIG=`ping -s$PAYLOAD -c1 $IP | grep time | grep -o time=.* | cut -d'=' -f2 | cut -d' ' -f1`
#sleep 0.5


echo $BIG >> big
echo $SMALL >> small

SMALL_MIN=`cat small | grep [0-9].* | awk '{if(min==""){min=max=$1}; if($1>max) {max=$1}; if($1<min) {min=$1}; total+=$1; count+=1} END {print min}'`
BIG_MIN=`cat big | grep [0-9].* | awk '{if(min==""){min=max=$1}; if($1>max) {max=$1}; if($1<min) {min=$1}; total+=$1; count+=1} END {print min}'`


#echo "BIG_MIN=$BIG_MIN"


KBPS=`echo "$PINGSPERSEC * $PAYLOAD_DIFF * 8 * 2 / 1000" | bc`
MBPS=`echo $KBPS / 1000 | bc`



Beyond ping: Detect packet drop within TCP session

What do you do if a particular website or a particular web service is crawling and yet pings, mtr traceroutes come back perfect?   Or what if you can’t ping at all because of firewall in the way?   Is something throttling this specific connection on the remote end?

The way to look under the hood is to look at the specific TCP session and see how it’s performing.   This is where wireshark comes to the rescue with the following metrics:

  • Packet Loss:   Filter by criteria “tcp.analysis.lost_segment”
  • Up/down throughput:  Go to statistics -> Conversations -> scroll all the way right for “bps” stats
  • Latency: Select TCP packet in question -> Expand TCP SEQ/ACK analysis -> look for RTT to ACK


Howto Run Hyper-V 2016 Core without Domain Controller

Hyper-V offers a free version.  The catch is that it is the core Hyper-V without the Windows interface. That’s fine because none of the other hypervisors such as XenServer or ESXi have a graphical interface running on the hypervisor itself either. The trouble is that Microsoft makes working with Hyper-V without a GUI very tricky, unless you join it to a domain. In my opinion joining a hypervisor to a domain is undesirable. Either you have to run a domain controller as a VM creating a weird chicken-and-egg problem, or alternatively you have to run the domain controller as a separate physical host – who in this day and age wants to do that though?

The solution to all this is to jump through couple extra hoops and run Hyper-V without domain controller.

  1. Pick a management machine.  Let’s call it MANAGE01.  Preferably Windows 10 Pro.  In my tests I didn’t have it joined to a domain.  Add user called Admin with password xyz.
  2. Install Hyper-V 2016 on server.  At the end of the installation create a new user called Admin with password xyz (important that username and password matches exactly with step 1).   Change host name to HYPER-V01  (Optional: enable Remote desktop and enable pings)
  3. On MANAGE01 do these steps:
    1. edit hosts file and add entry HYPER-V01
    2. Open Powershell with admin privileges,
    3. Start-Service WinRM
    4. winrm set winrm/config/client ‘@{TrustedHosts=”HYPER-V01″}’
    5. Stop-Service WinRM
    6. Open Hyper-v Manager, and `connect to server`
    7. Enter HYPER-V01
  4. Done

For multiple hosts the command is

winrm set winrm/config/client '@{TrustedHosts="HYPER-V01,HYPER-V02"}'

To get replication you need to do the following:

  • netsh advfirewall firewall add rule name=”Open Port 443″ dir=in action=allow protocol=TCP localport=443
  • Install self signed SSL certificates

Password expiry can be a problem when running without DC.  For that reason it’s best to disable password expiry on all hosts



Solved: Outlook 2016 to Exchange 2010 setup

  • Tried to do automatic setup in Outlook 2016 and it failed (kept prompting me for password during autodiscovery), So I thought I will do manual setup instead
  • Manual setup failed with “Log onto Exchange ActiveSync mail Server (EAS): The server cant be found”  Which is weird because Microsoft Remote Connectivity Analyzer succeeded.  Also, setting up the account on Android Phone which uses Active Sync worked.  So why is it complaining that Active Sync is broken?    Turns out that EAS is special type of ActiveSync that’s only compatible with  So that’s a dead end
  • Went back to auto discovery and this time I filled it out with Name, Email address, and Password.  When it prompted me for the password the second time, I changed the username to DOMAIN\username and filled in the same password.  
  • It worked

I guess the proper way is to map DOMAIN\username to, but this does work as a workaround.

Veeam Endpoint Backup Free

Veeam Endpoint Backup is my new favorite backup tool for workstations and laptops.   It is a great alternative to Drive Image XML and to Ghost.


  • Reliable background scheduler
  • Both image based and file level restore
  • Efficient differential backups
  • No server required needed, just point at a drive or network share


  • Have to register at Veeam to download it, but since the registration is free, that’s not a big deal


Mount VMDK containing GPT partition on Linux

Normally, to mount VMDK on Linux, you can follow these steps 

But what if the VMDK in question contains GPT with NTFS inside?  How do you find the offset and mount it?

#first losetup the raw disk
losetup /dev/loop0 ./file-flat.vmdk

#to find out offset do this
parted /dev/loop0
unit s
#find the start offset of the partition you're looking for and multiply by 512 (mine was 135266304 - yours will likely differ)

#use the offset to mount the actual partition
losetup -o 135266304 /dev/loop1 /dev/loop0

#now mount it
mount.ntfs-3g /dev/loop1 /mnt/vmdk

Greedy Search Engines – All the same

I fell in love with Google search back when people still used WebCrawler.  But that’s because back then Google was the underdog.   I still like using Google.   Google just works, but once in a while I check out what others are doing because I’m rooting for the next uderdog.  Now that Google is a giant behemoth, making their ads harder to spot, and ratcheting up their tracking, I figured it’s about time to switch.  But where to?

Supposedly Bing is the closest contender.   While I have no love for Microsoft, I was desperate.  So I tried it.   It wasn’t the first time.   I try this once a year or so just to see what’s out there.   Last time I tried Bing was kind of ok except for that giant background picture that slows everything to a crawl when I’m working over remote desktop.   This time I was horrified.  The first screen of results were pure ads.   There wasn’t a single real result until I scrolled down past the first screen.   The bottom of the screen wasn’t any better.  If this is the strategy Microsoft is using to catch up with Google Search then good luck to them – they are screwed.  Or are they?   After all I bet the sheeple who get Bing as default on their Windows 10 don’t notice the tiny little word “ad” beside each of those ads and are just fine whatever they were fed.   After all it was close enough…. and perhaps even helpful.  This is not for me …and that’s why Ad Block Plus exists.

Troubleshooting vMotion “failed to resume” [Solved]

I got the following error when trying to vMotion a VM from one SAN to another.

The VM failed to resume on the destination during early power on.

First I tried looking at ESXi host /var/log/vmkernel.log but in there all I got was

 Migration considered a failure by the VMX.  It is most likely a timeout, but check the VMX log for the true error.

Because this failure happened so late in the process (around 72%)  I realized that it’s probably failing during resume.  It turns out that during storage migration the machine gets suspended for a very short period of time and has to start back up (with the new disk attached underneath).  Well this made me look at the VM’s own vmware.log … sure enough I found the smoking gun there

[msg.loader.biosfd] Could not open bios.440.rom (No such file or directory).

That’s when I realized that when I set this up long ago, I was forced to use a workaround of including this special bios.440.rom file along with the VM to make it compatible with the OS.  Sure enough, so many months later I forgot about it.  After I copied the file across to the destination manually, the migration succeeded without a problem.


FreeMyBits helps you get your data back under your control.

There are many ways your organization can lose control of its precious data. You may have been locked-in by vendor or service provider, you may have outgrown a system or you need to downsize a legacy behemoth, you may have lost a key employee, you may have been hacked or your perhaps obsolescence simply caught up with you.

Whatever the reason is, FreeMyBits helps small and medium business like yours to get control back. Their team consists of specialists with proven enterprise expertise.  FreeMyBits can handle both mainstream systems and obscure legacy contraptions. They succeed where others struggle.

FreeMyBits also provides planning and consulting services to minimize the risk of lock-in.