All posts by Tomas

WAF Proxy with ModSecurity and Apache

When you need to protect an application against XSS and other nasty attacks, but you can’t modify the source code, ModSecurity can save the day.

  1. Install apache
  2. Install ModSecurity
  3. Setup apache as a proxy with the following configuration
    <Location />
    ProxyPass https://www.test.com/
    ProxyPassReverse https://www.test.com/
    
    #SecRuleRemoveById 999999 whitelist any rules here
    </Location>
    
    
  4. Turn on /etc/modsecurity/modsecurity.conf
SecRuleEngine On

#SecRuleEngine DetectionOnly

5. Turn on CRS blocking in /etc/modsecurity/crs/crs-setup.conf

SecDefaultAction "phase:1,log,auditlog,deny"
SecDefaultAction "phase:2,log,auditlog,deny"

#SecDefaultAction "phase:1,log,auditlog,pass"
#SecDefaultAction "phase:2,log,auditlog,pass"

6. Watch /var/log/apache2/modsec_audit.log for false positives and tweak rules accordingly

Convert Proxmox .raw to HyperV .vhdx

Show raw device names

zfs list 
#example output may look something like this
rpool/data/vm-100-disk-1

Export to file

dd if=/dev/zvol/rpool/data/vm-100-disk-1 of=/file.raw

Convert to HyperV

qemu-img convert -f raw /file.raw -O vhdx -o subformat=dynamic /file.vhdx

Mount file.vhdx on HyperV and start

One liner method:

Forget about the export and feed the raw device directly to qemu-img

qemu-img convert -f raw /dev/zvol/rpool/data/vm-100-disk-1 -O vhdx -o subformat=dynamic /file.vhdx

 

Proxmox ATI GPU Passthrough Guide

After a lot of fiddling around with settings and hardware, I finally have a stable Proxmox 5.1 ATI GPU pass-through system.   What helped me was this helpful article to finally get all the bugs ironed out.   I did have to make several tweaks for my system.   I’m running an Intel system with ATI Radeon GPU.

  1. Ensure VT-d is supported and enabled in the BIOS
  2. Enable IOMMU on the host
    1. append the following to the GRUB_CMDLINE_LINUX_DEFAULT line in /etc/default/grub
      intel_iommu=on
    2. Save your changes by running
      update-grub
  3. Blacklist NVIDIA & Nouveau kernel modules so they don’t get loaded at boot
    1. echo "blacklist nouveau" >> /etc/modprobe.d/blacklist.conf
      echo "blacklist nvidia" >> /etc/modprobe.d/blacklist.conf
      echo "blacklist radeon" >>
      /etc/modprobe.d/blacklist.conf
    2. Save your changes by running
      update-initramfs -u
  4. Add the following lines to /etc/modules
    vfio
    vfio_iommu_type1
    vfio_pci
    vfio_virqfd
  5. Reboot the host
  6. Create your Windows VM using the UEFI bios hardware option (not the deafoult seabios) but do not start it yet.  Use VirtIO.  Modify /etc/pve/qemu-server/<vmid>.conf and ensure the following are in the file. Create / modify existing entries as necessary.
    bios: ovmf
    machine: q35
    cpu: host,hidden=1
    numa: 1
  7. Install Windows
    1. Mount second ISO  (virtio-win*.iso)
    2. Load IO driver from d:\viostore\w10\amd64\viostore.inf
    3. After install be sure to enable Remote desktop.
  8. Pass through the GPU.
    1. Modify /etc/pve/qemu-server/<vmid>.conf and add
      hostpci0: <device address>,x-vga=on,pcie=1. Example

      hostpci0: 01:00,x-vga=on,pcie=1
  9. Passthrough USB keyboard and mouse
    1. I find it best to passthrough specific USB ports rather than device IDs.  That way I can hotplug different devices to specific ports later without having to reboot.
  10. Done.

Troubleshooting

Blue screening when launching certain applications

AMD drivers setup application and/or Windows boot would consistently blue screen on me with the following error:

kmode_exception_not_handled

The fix as outlined here was to create /etc/modprobe.d/kvm.conf and add the parameter “options kvm ignore_msrs=1”

echo "options kvm ignore_msrs=1" > /etc/modprobe.d/kvm.conf

Update 4/9/18: Blue screening happens to Windows 10 1803 as well with the error

System Thread Exception Not Handled

The fix for this is the same – ignore_msrs=1

Frezing keyboard/mouse:

Device Manager -> Human Interface Devices -> Microsoft Hardware USB Keyboard -> Power Management -> Uncheck “Allow the computer to turn off this device to save power”

Notes:

Other guides require setting up vfio.conf.  With my hardware it was not required.  It’s probably needed for a nVidia card though.

  1. Determine the PCI address of your GPU
    1. Run
      lspci -v

      and look for your card.  Usually 01:00.0 & 01:00.1. You can omit the part after the decimal to include them both in one go – so in that case it would be 01:00

    2. Run lspci -n -s <PCI address> to obtain vendor IDs. Example :
      lspci -n -s 01:00
      01:00.0 0300: 10de:1b81 (rev a1)
      01:00.1 0403: 10de:10f0 (rev a1)
  2. Assign your GPU to vfio driver using the IDs obtained above. Example:
    echo "options vfio-pci ids=10de:1b81,10de:10f0" > /etc/modprobe.d/vfio.conf

Reset Windows Passwords with chntpw

My go-to tool for resetting forgotten windows passwords used to be  Offline NT password recovery

It still works fine, but I discovered that the underlying tool that does the resets is available on other distributions including Ubuntu Live CD (after apt-get) and it’s fully baked into my favorite System Rescue CD

That tool is chntpw

Even though the manual page says it’s for Windows NT/2000, so far it works on more modern windows too (local users):

Windows XP
Windows Vista
Windows 7
Windows 8
Windows 8.1
Windows 10

OpenBSD BGP Router Benchmarking

I had an Open BSD 5.9 router that started to loss packets with moderate traffic levels (sometimes as low as 300 Mbs).  The router had 1 Gbs upstream interfaces so I figured that the fault was with packets per second limits.   What I needed to do is figure out was way to have the router forward a large amount of packets per second and see how it holds up

I booted up a spare Linux server and hooked it up to one end of the router.  Downloaded and compiled a UDP packet generator and ran the following

./snd -l DestinationIP:1234 -m 1000 -n 1000000000

DestinationIP is any IP that routes through the router.  Interestingly the because UDP is connectionless protocol, the IP doesn’t even have to belong to an actual server – much less to a server you control.  As such port 1234 can be any number.   What’s happening is that you’re essentially pointing a UDP fire hose to flow through your router into a non existent destination.

On the router I watched the stats while decreasing -m until 28 (-m is the packet size and smaller size results in higher PPS)

#On Router
systat ifstat

I got only 50K PPS before packet loss appeared.  This is way too low for the hardware that I was using ( decent server model circa 2015 )

Reading other people’s benchmark results, it appeared that upgrading to the latest OpenBSD 6.4 may help.   After the upgrade there was a huge increase.  I am now getting:

Lab test: 1400K PPS (very close to theoretical maximum for 1 Gbs connection with 84 byte packets)
Real world: 680K PPS

Just the upgrade of the OS increased the PPS performance by more than 13x.   This without even trying additional tweaks found here: How to receive a million packets per second

pingdiff: Bandwidth Estimation using ICMP

Previously I found that to estimate bandwidth of an IP network link it’s possible to do it based on a formula that takes into account the difference between a small ping and a large ping. (see pingb)

Now I updated that algorithm to look for the smallest ping replies.   This works better because network links are inherently noisy and there is large variation between pings.   By doing the measurement on the minimum values over many tries, the estimation becomes more accurate.

#!/bin/bash

#
# pingdiff : bandwidth estimation through ICMP echo
# ver : 0.1
#
# (C) 2018 Tomas Florian
#
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.



# Usage
# pingdiff IP MAXPAYLOAD
#
# Experiment with MAX payload size between 100 and 64000 the bigger you can specify
# the more accurate the estimate will be
# The longer you let the loop running the more accurate the results will be
# in my experience the accuracy is +/- 50%
# Surprisingly the accuracy is not affected by link saturation as much as I would expect
# Ex. 10Gbps link shows 4Gbps whether saturated or not

#Errata
# bad error handling (ex Small ping > Big Ping, ping timeouts etc)

#optional configure these by running against 127.0.0.1 to get ping overhead
#LOCALHOST_OVERHEAD_SMALL=0.011
LOCALHOST_OVERHEAD_SMALL=0
#LOCALHOST_OVERHEAD_BIG=0.071
LOCALHOST_OVERHEAD_BIG=0

PAYLOAD_SMALL=1



if [ -z "$1" ]; then
 echo "usage: pingdiff IP PAYLOAD"
 exit 1
fi



if [ -z "$2" ]; then
 echo "usage: pingdiff IP PAYLOAD"
 exit 1
fi



IP=$1
PAYLOAD=$2

echo "IP=$IP, PAYLOAD=$PAYLOAD"

echo "" > big
echo "" > small

#for i in `jot 999`;
while [ 1 ]
do

#echo $i

BIG=`ping -s$PAYLOAD -c1 $IP | grep time | grep -o time=.* | cut -d'=' -f2 | cut -d' ' -f1`
sleep 0.5
SMALL=`ping -s$PAYLOAD_SMALL -c1 $IP | grep time | grep -o time=.* | cut -d'=' -f2 | cut -d' ' -f1`
sleep 0.5
#BIG=`ping -s$PAYLOAD -c1 $IP | grep time | grep -o time=.* | cut -d'=' -f2 | cut -d' ' -f1`
#sleep 0.5

BIG=`echo "$BIG - $LOCALHOST_OVERHEAD_BIG" | bc`
SMALL=`echo "$SMALL - $LOCALHOST_OVERHEAD_SMALL" | bc`



echo $BIG >> big
echo $SMALL >> small



SMALL_MIN=`cat small | grep [0-9].* | awk '{if(min==""){min=max=$1}; if($1>max) {max=$1}; if($1<min) {min=$1}; total+=$1; count+=1} END {print min}'`
BIG_MIN=`cat big | grep [0-9].* | awk '{if(min==""){min=max=$1}; if($1>max) {max=$1}; if($1<min) {min=$1}; total+=$1; count+=1} END {print min}'`



PAYLOAD_DIFF=`echo "$PAYLOAD - $PAYLOAD_SMALL" | bc`

#echo "SMALL_MIN=$SMALL_MIN"
#echo "BIG_MIN=$BIG_MIN"

ROUNDTRIPDIFF=`echo "$BIG_MIN - $SMALL_MIN" | bc`
PINGSPERSEC=`echo "1000 / $ROUNDTRIPDIFF" | bc`

KBPS=`echo "$PINGSPERSEC * $PAYLOAD_DIFF * 8 * 2 / 1000" | bc`
MBPS=`echo $KBPS / 1000 | bc`

echo "$MBPS Mbps (SMALL_MIN=$SMALL_MIN,BIG_MIN=$BIG_MIN)"

done

Beyond ping: Detect packet drop within TCP session

What do you do if a particular website or a particular web service is crawling and yet pings, mtr traceroutes come back perfect?   Or what if you can’t ping at all because of firewall in the way?   Is something throttling this specific connection on the remote end?

The way to look under the hood is to look at the specific TCP session and see how it’s performing.   This is where wireshark comes to the rescue with the following metrics:

  • Packet Loss:   Filter by criteria “tcp.analysis.lost_segment”
  • Up/down throughput:  Go to statistics -> Conversations -> scroll all the way right for “bps” stats
  • Latency: Select TCP packet in question -> Expand TCP SEQ/ACK analysis -> look for RTT to ACK

 

Howto Run Hyper-V 2016 Core without Domain Controller

Hyper-V offers a free version.  The catch is that it is the core Hyper-V without the Windows interface. That’s fine because none of the other hypervisors such as XenServer or ESXi have a graphical interface running on the hypervisor itself either. The trouble is that Microsoft makes working with Hyper-V without a GUI very tricky, unless you join it to a domain. In my opinion joining a hypervisor to a domain is undesirable. Either you have to run a domain controller as a VM creating a weird chicken-and-egg problem, or alternatively you have to run the domain controller as a separate physical host – who in this day and age wants to do that though?

The solution to all this is to jump through couple extra hoops and run Hyper-V without domain controller.

  1. Pick a management machine.  Let’s call it MANAGE01.  Preferably Windows 10 Pro.  In my tests I didn’t have it joined to a domain.  Add user called Admin with password xyz.
  2. Install Hyper-V 2016 on server.  At the end of the installation create a new user called Admin with password xyz (important that username and password matches exactly with step 1).   Change host name to HYPER-V01  (Optional: enable Remote desktop and enable pings)
  3. On MANAGE01 do these steps:
    1. edit hosts file and add entry 1.2.3.4 HYPER-V01
    2. Open Powershell with admin privileges,
    3. Start-Service WinRM
    4. winrm set winrm/config/client ‘@{TrustedHosts=”HYPER-V01″}’
    5. Stop-Service WinRM
    6. Open Hyper-v Manager, and `connect to server`
    7. Enter HYPER-V01
  4. Done

For multiple hosts the command is

winrm set winrm/config/client '@{TrustedHosts="HYPER-V01,HYPER-V02"}'

To get replication you need to do the following:

  • netsh advfirewall firewall add rule name=”Open Port 443″ dir=in action=allow protocol=TCP localport=443
  • Install self signed SSL certificates

Password expiry can be a problem when running without DC.  For that reason it’s best to disable password expiry on all hosts

NET accounts /MAXPWAGE:UNLIMITED

 

Solved: Outlook 2016 to Exchange 2010 setup

  • Tried to do automatic setup in Outlook 2016 and it failed (kept prompting me for password during autodiscovery), So I thought I will do manual setup instead
  • Manual setup failed with “Log onto Exchange ActiveSync mail Server (EAS): The server cant be found”  Which is weird because Microsoft Remote Connectivity Analyzer succeeded.  Also, setting up the account on Android Phone which uses Active Sync worked.  So why is it complaining that Active Sync is broken?    Turns out that EAS is special type of ActiveSync that’s only compatible with outlook.com.  So that’s a dead end
  • Went back to auto discovery and this time I filled it out with Name, Email address, and Password.  When it prompted me for the password the second time, I changed the username to DOMAIN\username and filled in the same password.  
  • It worked

I guess the proper way is to map DOMAIN\username to username@domain.com, but this does work as a workaround.