All posts by Tomas

Reset Windows Passwords with chntpw

My go-to tool for resetting forgotten windows passwords used to be  Offline NT password recovery

It still works fine, but I discovered that the underlying tool that does the resets is available on other distributions including Ubuntu Live CD (after apt-get) and it’s fully baked into my favorite System Rescue CD

That tool is chntpw

Even though the manual page says it’s for Windows NT/2000, so far it works on more modern windows too (local users):

Windows XP
Windows Vista
Windows 7
Windows 8
Windows 8.1
Windows 10

OpenBSD BGP Router Benchmarking

I had an Open BSD 5.9 router that started to loss packets with moderate traffic levels (sometimes as low as 300 Mbs).  The router had 1 Gbs upstream interfaces so I figured that the fault was with packets per second limits.   What I needed to do is figure out was way to have the router forward a large amount of packets per second and see how it holds up

I booted up a spare Linux server and hooked it up to one end of the router.  Downloaded and compiled a UDP packet generator and ran the following

./snd -l DestinationIP:1234 -m 1000 -n 1000000000

DestinationIP is any IP that routes through the router.  Interestingly the because UDP is connectionless protocol, the IP doesn’t even have to belong to an actual server – much less to a server you control.  As such port 1234 can be any number.   What’s happening is that you’re essentially pointing a UDP fire hose to flow through your router into a non existent destination.

On the router I watched the stats while decreasing -m until 28 (-m is the packet size and smaller size results in higher PPS)

#On Router
systat ifstat

I got only 50K PPS before packet loss appeared.  This is way too low for the hardware that I was using ( decent server model circa 2015 )

Reading other people’s benchmark results, it appeared that upgrading to the latest OpenBSD 6.4 may help.   After the upgrade there was a huge increase.  I am now getting:

Lab test: 1400K PPS (very close to theoretical maximum for 1 Gbs connection with 84 byte packets)
Real world: 680K PPS

Just the upgrade of the OS increased the PPS performance by more than 13x.   This without even trying additional tweaks found here: How to receive a million packets per second

pingdiff: Bandwidth Estimation using ICMP

Previously I found that to estimate bandwidth of an IP network link it’s possible to do it based on a formula that takes into account the difference between a small ping and a large ping. (see pingb)

Now I updated that algorithm to look for the smallest ping replies.   This works better because network links are inherently noisy and there is large variation between pings.   By doing the measurement on the minimum values over many tries, the estimation becomes more accurate.

#!/bin/bash

#
# pingdiff : bandwidth estimation through ICMP echo
# ver : 0.1
#
# (C) 2018 Tomas Florian
#
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.



# Usage
# pingdiff IP MAXPAYLOAD
#
# Experiment with MAX payload size between 100 and 64000 the bigger you can specify
# the more accurate the estimate will be
# The longer you let the loop running the more accurate the results will be
# in my experience the accuracy is +/- 50%
# Surprisingly the accuracy is not affected by link saturation as much as I would expect
# Ex. 10Gbps link shows 4Gbps whether saturated or not

#Errata
# bad error handling (ex Small ping > Big Ping, ping timeouts etc)

#optional configure these by running against 127.0.0.1 to get ping overhead
#LOCALHOST_OVERHEAD_SMALL=0.011
LOCALHOST_OVERHEAD_SMALL=0
#LOCALHOST_OVERHEAD_BIG=0.071
LOCALHOST_OVERHEAD_BIG=0

PAYLOAD_SMALL=1



if [ -z "$1" ]; then
 echo "usage: pingdiff IP PAYLOAD"
 exit 1
fi



if [ -z "$2" ]; then
 echo "usage: pingdiff IP PAYLOAD"
 exit 1
fi



IP=$1
PAYLOAD=$2

echo "IP=$IP, PAYLOAD=$PAYLOAD"

echo "" > big
echo "" > small

#for i in `jot 999`;
while [ 1 ]
do

#echo $i

BIG=`ping -s$PAYLOAD -c1 $IP | grep time | grep -o time=.* | cut -d'=' -f2 | cut -d' ' -f1`
sleep 0.5
SMALL=`ping -s$PAYLOAD_SMALL -c1 $IP | grep time | grep -o time=.* | cut -d'=' -f2 | cut -d' ' -f1`
sleep 0.5
#BIG=`ping -s$PAYLOAD -c1 $IP | grep time | grep -o time=.* | cut -d'=' -f2 | cut -d' ' -f1`
#sleep 0.5

BIG=`echo "$BIG - $LOCALHOST_OVERHEAD_BIG" | bc`
SMALL=`echo "$SMALL - $LOCALHOST_OVERHEAD_SMALL" | bc`



echo $BIG >> big
echo $SMALL >> small



SMALL_MIN=`cat small | grep [0-9].* | awk '{if(min==""){min=max=$1}; if($1>max) {max=$1}; if($1<min) {min=$1}; total+=$1; count+=1} END {print min}'`
BIG_MIN=`cat big | grep [0-9].* | awk '{if(min==""){min=max=$1}; if($1>max) {max=$1}; if($1<min) {min=$1}; total+=$1; count+=1} END {print min}'`



PAYLOAD_DIFF=`echo "$PAYLOAD - $PAYLOAD_SMALL" | bc`

#echo "SMALL_MIN=$SMALL_MIN"
#echo "BIG_MIN=$BIG_MIN"

ROUNDTRIPDIFF=`echo "$BIG_MIN - $SMALL_MIN" | bc`
PINGSPERSEC=`echo "1000 / $ROUNDTRIPDIFF" | bc`

KBPS=`echo "$PINGSPERSEC * $PAYLOAD_DIFF * 8 * 2 / 1000" | bc`
MBPS=`echo $KBPS / 1000 | bc`

echo "$MBPS Mbps (SMALL_MIN=$SMALL_MIN,BIG_MIN=$BIG_MIN)"

done

Beyond ping: Detect packet drop within TCP session

What do you do if a particular website or a particular web service is crawling and yet pings, mtr traceroutes come back perfect?   Or what if you can’t ping at all because of firewall in the way?   Is something throttling this specific connection on the remote end?

The way to look under the hood is to look at the specific TCP session and see how it’s performing.   This is where wireshark comes to the rescue with the following metrics:

  • Packet Loss:   Filter by criteria “tcp.analysis.lost_segment”
  • Up/down throughput:  Go to statistics -> Conversations -> scroll all the way right for “bps” stats
  • Latency: Select TCP packet in question -> Expand TCP SEQ/ACK analysis -> look for RTT to ACK

 

Howto Run Hyper-V 2016 Core without Domain Controller

Hyper-V offers a free version.  The catch is that it is the core Hyper-V without the Windows interface. That’s fine because none of the other hypervisors such as XenServer or ESXi have a graphical interface running on the hypervisor itself either. The trouble is that Microsoft makes working with Hyper-V without a GUI very tricky, unless you join it to a domain. In my opinion joining a hypervisor to a domain is undesirable. Either you have to run a domain controller as a VM creating a weird chicken-and-egg problem, or alternatively you have to run the domain controller as a separate physical host – who in this day and age wants to do that though?

The solution to all this is to jump through couple extra hoops and run Hyper-V without domain controller.

  1. Pick a management machine.  Let’s call it MANAGE01.  Preferably Windows 10 Pro.  In my tests I didn’t have it joined to a domain.  Add user called Admin with password xyz.
  2. Install Hyper-V 2016 on server.  At the end of the installation create a new user called Admin with password xyz (important that username and password matches exactly with step 1).   Change host name to HYPER-V01  (Optional: enable Remote desktop and enable pings)
  3. On MANAGE01 do these steps:
    1. edit hosts file and add entry 1.2.3.4 HYPER-V01
    2. Open Powershell with admin privileges,
    3. Start-Service WinRM
    4. winrm set winrm/config/client ‘@{TrustedHosts=”HYPER-V01″}’
    5. Stop-Service WinRM
    6. Open Hyper-v Manager, and `connect to server`
    7. Enter HYPER-V01
  4. Done

For multiple hosts the command is

winrm set winrm/config/client '@{TrustedHosts="HYPER-V01,HYPER-V02"}'

To get replication you need to do the following:

  • netsh advfirewall firewall add rule name=”Open Port 443″ dir=in action=allow protocol=TCP localport=443
  • Install self signed SSL certificates

Password expiry can be a problem when running without DC.  For that reason it’s best to disable password expiry on all hosts

NET accounts /MAXPWAGE:UNLIMITED

 

Solved: Outlook 2016 to Exchange 2010 setup

  • Tried to do automatic setup in Outlook 2016 and it failed (kept prompting me for password during autodiscovery), So I thought I will do manual setup instead
  • Manual setup failed with “Log onto Exchange ActiveSync mail Server (EAS): The server cant be found”  Which is weird because Microsoft Remote Connectivity Analyzer succeeded.  Also, setting up the account on Android Phone which uses Active Sync worked.  So why is it complaining that Active Sync is broken?    Turns out that EAS is special type of ActiveSync that’s only compatible with outlook.com.  So that’s a dead end
  • Went back to auto discovery and this time I filled it out with Name, Email address, and Password.  When it prompted me for the password the second time, I changed the username to DOMAIN\username and filled in the same password.  
  • It worked

I guess the proper way is to map DOMAIN\username to username@domain.com, but this does work as a workaround.

Veeam Endpoint Backup Free

Veeam Endpoint Backup is my new favorite backup tool for workstations and laptops.   It is a great alternative to Drive Image XML and to Ghost.

Pluses:

  • Reliable background scheduler
  • Both image based and file level restore
  • Efficient differential backups
  • No server required needed, just point at a drive or network share

Minuses

  • Have to register at Veeam to download it, but since the registration is free, that’s not a big deal

 

Mount VMDK containing GPT partition on Linux

Normally, to mount VMDK on Linux, you can follow these steps 

But what if the VMDK in question contains GPT with NTFS inside?  How do you find the offset and mount it?

#first losetup the raw disk
losetup /dev/loop0 ./file-flat.vmdk

#to find out offset do this
parted /dev/loop0
unit s
print
#find the start offset of the partition you're looking for and multiply by 512 (mine was 135266304 - yours will likely differ)

#use the offset to mount the actual partition
losetup -o 135266304 /dev/loop1 /dev/loop0

#now mount it
mount.ntfs-3g /dev/loop1 /mnt/vmdk

Greedy Search Engines – All the same

I fell in love with Google search back when people still used WebCrawler.  But that’s because back then Google was the underdog.   I still like using Google.   Google just works, but once in a while I check out what others are doing because I’m rooting for the next uderdog.  Now that Google is a giant behemoth, making their ads harder to spot, and ratcheting up their tracking, I figured it’s about time to switch.  But where to?

Supposedly Bing is the closest contender.   While I have no love for Microsoft, I was desperate.  So I tried it.   It wasn’t the first time.   I try this once a year or so just to see what’s out there.   Last time I tried Bing was kind of ok except for that giant background picture that slows everything to a crawl when I’m working over remote desktop.   This time I was horrified.  The first screen of results were pure ads.   There wasn’t a single real result until I scrolled down past the first screen.   The bottom of the screen wasn’t any better.  If this is the strategy Microsoft is using to catch up with Google Search then good luck to them – they are screwed.  Or are they?   After all I bet the sheeple who get Bing as default on their Windows 10 don’t notice the tiny little word “ad” beside each of those ads and are just fine whatever they were fed.   After all it was close enough…. and perhaps even helpful.  This is not for me …and that’s why Ad Block Plus exists.

Troubleshooting vMotion “failed to resume” [Solved]

I got the following error when trying to vMotion a VM from one SAN to another.

The VM failed to resume on the destination during early power on.

First I tried looking at ESXi host /var/log/vmkernel.log but in there all I got was

 Migration considered a failure by the VMX.  It is most likely a timeout, but check the VMX log for the true error.

Because this failure happened so late in the process (around 72%)  I realized that it’s probably failing during resume.  It turns out that during storage migration the machine gets suspended for a very short period of time and has to start back up (with the new disk attached underneath).  Well this made me look at the VM’s own vmware.log … sure enough I found the smoking gun there

[msg.loader.biosfd] Could not open bios.440.rom (No such file or directory).

That’s when I realized that when I set this up long ago, I was forced to use a workaround of including this special bios.440.rom file along with the VM to make it compatible with the OS.  Sure enough, so many months later I forgot about it.  After I copied the file across to the destination manually, the migration succeeded without a problem.