Category Archives: Networking

When ping size matters

Over the couple of days I’ve been fighting with sluggish internet performance from my office. The first thing I do, is the same as you, I start pinging. There is 0% packet loss and latency is a steady 10 ms. So where is the sluggishness coming from? I test the whole network stack from top to bottom and then back up again, router, vpn, switch, virtual network … everything checks out at 0% packet loss. Why are things so slow when ping is showing they should be fine?

I only figured it out when I started to play around with packet size. As I started increasing it, packet loss showed up

  • 10 bytes ~ 0% packet loss
  • 100 bytes ~ 5% packet loss
  • 1000 bytes ~ 30% packet loss
  • 10000 bytes ~ 80% packet loss

Now it made sense, I wasn’t seeing the packet loss because little packets sneak in on time. It was only when I start doing real work with large packets that the slow down occurs

Looking back, sometimes it’s best to keep things simple. I got side tracked because I initially blamed a weird hypervisor NIC driver glitch, that I was fighting with a couple years ago, but this time it was a “good” old ISP issue.

Isolating Untrusted Devices at Switch Level

What do you do when you have a questionable device that you’d like to connect to your switch in order to get it online, but you don’t fully trust it and want to isolate it from your other devices? Or what if you don’t trust any of your devices to talk to each other but you still want to give them internet access? The textbook solution is to create separate VLANs and firewall them from each other at the router. This works, but there is just too much configuration for something so simple. An easier approach is to segregate them at the switch level. That way you can skip all the complexities of setting up a VLAN aware router. Sounds great, what’s the catch? The catch is that you do need a managed switch which supports IPv4 ACLs. Luckily, 1Gbps managed switches are not that expensive these days. In my setup I used and old Dell Powerconnect 6224. Similar syntax will apply to Cisco or HP switch. The general idea is that a managed switch has firewall-like features, so let’s use them.

Let’s look at a specific scenario:

  • LAN gateway IP 192.168.1.1/24
  • Port 1 and 2 with devices that have to be segregated from each other and from other devices on the same VLAN
access-list InternetOnlyACL permit ip any 192.168.1.1 0.0.0.0
access-list InternetOnlyACL deny ip any 192.168.1.0 0.0.0.255
access-list InternetOnlyACL permit ip any any

interface ethernet 1/g1
switchport access vlan 10
ip access-group InternetOnlyACL in 1

interface ethernet 1/g2
switchport access vlan 10
ip access-group InternetOnlyACL in 1

The following line is optional

access-list InternetOnlyACL permit ip any 192.168.1.1 0.0.0.0

What it does is make sure that the device can still ping the LAN gateway (important for some devices which verify internet connectivity by pinging LAN gateway) but for most they will happily connect to the internet without ever being able to ping the gateway IP (you just won’t see the first hop in your traceroute). This setup works with both DHCP and static IP assignments.

Caveat: This method blocks funny business on Layer 3 (IP protocol),but it will do absolutely nothing for you if the malicious device starts messing with your network at Layer 2 (ex ARP spoofing). For that you need to configure additional security features on your switch. This is just another layer of protection.

Fixing email display name spoofing

The words “email” and “security” have never mixed, but some things are just too ridiculous to be left as they are.

There are several ways to spoof email, each with their own countermeasures:

  • Sender email address -> DKIM and SPF
  • Sender email display name -> Solution below

While researching ways to fix this, I came across different methods, ranging from database lookups of valid names to looking for suspicious patterns in the display name. None of those methods get to the root of the problem which is that

senders email display name should never have been a “field” that existed in the first place.

Whose idea was it to create a field where anyone can type in “John Smith” as their name, and to top it all off, the recipients mobile device happily says “you have email from John Smith”. Really? From John Smith? Are you sure? No? Not even little bit sure? Is that because anyone can type in anything they want and there is nothing to verify it against? Then why would you show such information as if it’s the truth? No thanks.

To reverse this horrible mistake, the solution is obvious. Let’s remove email display name from existence and display only email addresses (which can still be spoofed, but at least there are countermeasures in place for anyone trying to get away with it)

vi /etc/postfix/header_checks
/^FROM:.<(.@.)>/ REPLACE From: <$1>
/^REPLY-TO:.<(.@.)>/ REPLACE Reply-to: <$1>
postmap -r header_checks
postfix reload


Using Frida to Bypass SSL Pinning on Android

Most modern apps rely on SSL pinning to make sniffing SSL traffic through proxy more difficult. This is great security-in-depth practice, but it’s a real pain when trying to inspect app’s traffic as a part of vulnerability assessment or penetration test. Luckily there if Frida.

  1. Run frida-server https://www.frida.re/docs/android/
  2. frida-ps -Uai #find the target’s application identified ex: com.company.myapp
  3. download file bypass.js into current directory
  4. frida -U -f com.company.myapp -l bypass.js –no-pause
  5. done

HTTPs Inspection of Android APK

With increased security, it’s getting trickier to intercept HTTPs traffic send by Android Apps. For that reason most methods rely on rooting Android. But what to do when you don’t have a rooted Android on hand?

Use Android Emulator from Android Studio:

  1. Go to “Settings->Wireless & Networks->More”
  2. Go to “Cellular Network Settings”
  3. Go to “Access Point Names”
  4. Edit Proxy and Port fields
  5. Install root cert from BurpSuite or whatever tool you are using to intercept the traffic
  6. Install and run your APK

WAF Proxy with ModSecurity and Apache

When you need to protect an application against XSS and other nasty attacks, but you can’t modify the source code, ModSecurity can save the day.

  1. Install apache
  2. Install ModSecurity
  3. Setup apache as a proxy with the following configuration
    <Location />
    ProxyPass https://www.test.com/
    ProxyPassReverse https://www.test.com/
    
    #SecRuleRemoveById 999999 whitelist any rules here
    </Location>
    
    
  4. Turn on /etc/modsecurity/modsecurity.conf
SecRuleEngine On

#SecRuleEngine DetectionOnly

5. Turn on CRS blocking in /etc/modsecurity/crs/crs-setup.conf

SecDefaultAction "phase:1,log,auditlog,deny"
SecDefaultAction "phase:2,log,auditlog,deny"

#SecDefaultAction "phase:1,log,auditlog,pass"
#SecDefaultAction "phase:2,log,auditlog,pass"

6. Watch /var/log/apache2/modsec_audit.log for false positives and tweak rules accordingly

OpenBSD BGP Router Benchmarking

I had an Open BSD 5.9 router that started to loss packets with moderate traffic levels (sometimes as low as 300 Mbs).  The router had 1 Gbs upstream interfaces so I figured that the fault was with packets per second limits.   What I needed to do is figure out was way to have the router forward a large amount of packets per second and see how it holds up

I booted up a spare Linux server and hooked it up to one end of the router.  Downloaded and compiled a UDP packet generator and ran the following

./snd -l DestinationIP:1234 -m 1000 -n 1000000000

DestinationIP is any IP that routes through the router.  Interestingly the because UDP is connectionless protocol, the IP doesn’t even have to belong to an actual server – much less to a server you control.  As such port 1234 can be any number.   What’s happening is that you’re essentially pointing a UDP fire hose to flow through your router into a non existent destination.

On the router I watched the stats while decreasing -m until 28 (-m is the packet size and smaller size results in higher PPS)

#On Router
systat ifstat

I got only 50K PPS before packet loss appeared.  This is way too low for the hardware that I was using ( decent server model circa 2015 )

Reading other people’s benchmark results, it appeared that upgrading to the latest OpenBSD 6.4 may help.   After the upgrade there was a huge increase.  I am now getting:

Lab test: 1400K PPS (very close to theoretical maximum for 1 Gbs connection with 84 byte packets)
Real world: 680K PPS

Just the upgrade of the OS increased the PPS performance by more than 13x.   This without even trying additional tweaks found here: How to receive a million packets per second

pingdiff: Bandwidth Estimation using ICMP

Previously I found that to estimate bandwidth of an IP network link it’s possible to do it based on a formula that takes into account the difference between a small ping and a large ping. (see pingb)

Now I updated that algorithm to look for the smallest ping replies.   This works better because network links are inherently noisy and there is large variation between pings.   By doing the measurement on the minimum values over many tries, the estimation becomes more accurate.

#!/bin/bash

#
# pingdiff : bandwidth estimation through ICMP echo
# ver : 0.1
#
# (C) 2018 Tomas Florian
#
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.



# Usage
# pingdiff IP MAXPAYLOAD
#
# Experiment with MAX payload size between 100 and 64000 the bigger you can specify
# the more accurate the estimate will be
# The longer you let the loop running the more accurate the results will be
# in my experience the accuracy is +/- 50%
# Surprisingly the accuracy is not affected by link saturation as much as I would expect
# Ex. 10Gbps link shows 4Gbps whether saturated or not

#Errata
# bad error handling (ex Small ping > Big Ping, ping timeouts etc)

#optional configure these by running against 127.0.0.1 to get ping overhead
#LOCALHOST_OVERHEAD_SMALL=0.011
LOCALHOST_OVERHEAD_SMALL=0
#LOCALHOST_OVERHEAD_BIG=0.071
LOCALHOST_OVERHEAD_BIG=0

PAYLOAD_SMALL=1



if [ -z "$1" ]; then
 echo "usage: pingdiff IP PAYLOAD"
 exit 1
fi



if [ -z "$2" ]; then
 echo "usage: pingdiff IP PAYLOAD"
 exit 1
fi



IP=$1
PAYLOAD=$2

echo "IP=$IP, PAYLOAD=$PAYLOAD"

echo "" > big
echo "" > small

#for i in `jot 999`;
while [ 1 ]
do

#echo $i

BIG=`ping -s$PAYLOAD -c1 $IP | grep time | grep -o time=.* | cut -d'=' -f2 | cut -d' ' -f1`
sleep 0.5
SMALL=`ping -s$PAYLOAD_SMALL -c1 $IP | grep time | grep -o time=.* | cut -d'=' -f2 | cut -d' ' -f1`
sleep 0.5
#BIG=`ping -s$PAYLOAD -c1 $IP | grep time | grep -o time=.* | cut -d'=' -f2 | cut -d' ' -f1`
#sleep 0.5

BIG=`echo "$BIG - $LOCALHOST_OVERHEAD_BIG" | bc`
SMALL=`echo "$SMALL - $LOCALHOST_OVERHEAD_SMALL" | bc`



echo $BIG >> big
echo $SMALL >> small



SMALL_MIN=`cat small | grep [0-9].* | awk '{if(min==""){min=max=$1}; if($1>max) {max=$1}; if($1<min) {min=$1}; total+=$1; count+=1} END {print min}'`
BIG_MIN=`cat big | grep [0-9].* | awk '{if(min==""){min=max=$1}; if($1>max) {max=$1}; if($1<min) {min=$1}; total+=$1; count+=1} END {print min}'`



PAYLOAD_DIFF=`echo "$PAYLOAD - $PAYLOAD_SMALL" | bc`

#echo "SMALL_MIN=$SMALL_MIN"
#echo "BIG_MIN=$BIG_MIN"

ROUNDTRIPDIFF=`echo "$BIG_MIN - $SMALL_MIN" | bc`
PINGSPERSEC=`echo "1000 / $ROUNDTRIPDIFF" | bc`

KBPS=`echo "$PINGSPERSEC * $PAYLOAD_DIFF * 8 * 2 / 1000" | bc`
MBPS=`echo $KBPS / 1000 | bc`

echo "$MBPS Mbps (SMALL_MIN=$SMALL_MIN,BIG_MIN=$BIG_MIN)"

done

Beyond ping: Detect packet drop within TCP session

What do you do if a particular website or a particular web service is crawling and yet pings, mtr traceroutes come back perfect?   Or what if you can’t ping at all because of firewall in the way?   Is something throttling this specific connection on the remote end?

The way to look under the hood is to look at the specific TCP session and see how it’s performing.   This is where wireshark comes to the rescue with the following metrics:

  • Packet Loss:   Filter by criteria “tcp.analysis.lost_segment”
  • Up/down throughput:  Go to statistics -> Conversations -> scroll all the way right for “bps” stats
  • Latency: Select TCP packet in question -> Expand TCP SEQ/ACK analysis -> look for RTT to ACK