After a lot of head scratching and googling I finally discovered why my ceph performance was so slow compared to NFS when using iscsi tgs on my gateway. I was getting only 0.1 MB/s compared to 90 MB/s that I was getting through NFS. It turns out that ESXi had hardware acceleration (VAAI) turned on for it’s iSCSI initiator – apparently it’s something that isn’t compatible with tgt. To turn it off I followed these steps
I didn’t even have to reboot, or reload any configuration. The effect was immediate jump in performance back to normal.