Slow VMWare performance iscsi tgt and ceph [Solved]

After a lot of head scratching and googling I finally discovered why my ceph performance was so slow compared to NFS when using iscsi tgs on my gateway.   I was getting only 0.1 MB/s compared to 90 MB/s that I was getting through NFS.  It turns out that ESXi had hardware acceleration (VAAI) turned on for it’s iSCSI initiator – apparently it’s something that isn’t compatible with tgt.  To turn it off I followed these steps

Turning off VAAI

I didn’t even have to reboot, or reload any configuration.  The effect was immediate jump in performance back to normal.