At the CUUG presentation I was asked if I could provide steps on setting up Ceph iSCSI. Sebastien Han’s blog has very good write up on the subject.
Here is a copy of the presentation I gave at CUUG.
After a lot of head scratching and googling I finally discovered why my ceph performance was so slow compared to NFS when using iscsi tgs on my gateway. I was getting only 0.1 MB/s compared to 90 MB/s that I was getting through NFS. It turns out that ESXi had hardware acceleration (VAAI) turned on for it’s iSCSI initiator – apparently it’s something that isn’t compatible with tgt. To turn it off I followed these steps
I didn’t even have to reboot, or reload any configuration. The effect was immediate jump in performance back to normal.
I received an urgent call from a client. The client’s computer got hit by the VVV virus AKA TeslaCrypt. All of their files were renamed to .vvv and when trying to open them, a message said that the only way to get them back was to pay $500 USD before a specific date, or $1000 USD after. The payment had to be in bitcoins. First question I asked: “Do you have a backup?” Client said “No”. Having read up about ransomware like CryptoLocker, I knew this was bad news. Normally once the files are encrypted, it’s game over. Only way out is backup. Nevertheless, I dove into researching a possible solution – just in case.
There were 4 possible methods that I tried:
- Restore from windows shadow copy. This didn’t work in this version of TeslaCrypt. The criminals learned to wipe the shadow copies before encrypting the files.
- Undelete files (someone claimed that when the files are encrypted a copy is made and original deleted). This didn’t work, I could not find a single deleted file that matched it’s encrypted counterpart. All I found were files that were legitimately deleted by the user long before the infection. This kind of makes sense, because surely the encryption process will overwrite the data rather than creating a copy. But anything is worth a try when you are desperate.
- Decrypt using key.dat (where the decryption key is stored). This didn’t work either because this was newer version of TeslaCrypt. In the old version the criminals were stupid enough to leave the decryption key right on the infected system. In the new version, they hold the key on a server they control.
- Decrypt using a tool called teslacrack.py. This eventually ended up working – but more on that later.
Running out of time, and faced with the slim chance of recovery, my client felt he had no choice other than to pay the ransom. The criminals are running an organized business, and this gives it the impression that they may actually deliver the decryption key after payment in order to preserve their reputation. After all, if people have bad experience after sending a payment, the word will spread and no one will ever send them any money no matter how desperate they are. On the other hand if the payment and decryption are very quick and easy, people will see it as an easy way out. This, combined with how easy it is for the criminal to give out the decryption key (just a single click of a button) it is in the criminal’s best interest to deliver the decryption key promptly. All signs pointed to a well organized setup. The criminals even have even had a working web support and billing system where you can get answers to your questions regarding on how to submit your Bitcoin payment. The response time to a general question ended up being about 4 hours – which is not bad. Actually better than support at many large enterprises. Still, there are so many things that could go wrong. Maybe the criminals are just plain evil and see this as a onetime money grab. Maybe they could not care less about their reputation. Maybe the criminals get shut down just before having the chance to send you the key. Maybe the criminals screw up and lose the key, so even though they might have wanted to give it to you, they simply don’t have it. Maybe the criminals are stupid and don’t use business sense for deciding how to behave. You are doomed if you send payment, and you are doomed if you don’t. I was glad that it wasn’t me in this situation. I would have no idea what to do next.
My client on the other hand had to make that decision – and quickly. He decided to pay. Trouble was that the only accepted method of payment was through bit Bitcoin (BTC). Because he did not have $500 USD in BTC, he had to buy Bitcoins first. Even though it’s becoming easier and easier to buy Bitcoin, it’s still quite a process. You either have to go through 2-5 day verification process or you have to fully trust the Bitcoin vendor. Neither of those options was good because we were running out of time and we were not in the mood to blindly “trust people”. In the end we were lucky because now there is a Bitcoin ATM at Café Blanca in downtown Calgary. There you can buy and sell Bitcoins with cash instantly. You insert cash and the bitcoins appear in your Bitcoin wallet 2 seconds later. My client ended up gradually putting in about $700 CAD in cash to purchase 1.1 Bitcoins which the criminals demanded in order to cover the $500 USD fee. With 1.1 bitcoins ready, we went back to the criminal’s web payment portal. Just as we were reading the payment instructions for the 3rd time to be extra sure there is no mistake, we noticed that the criminals raised the cost to 1.25 bitcoins. We had to top up the bitcoin wallet and proceeded to make the payment. We sent the payment and included the transaction ID which proved receipt by the criminals. Then we waited. 10 minutes, nothing. 1 Hour – nothing. 5 Hours – nothing. 1 day – nothing. That was it. The criminals got the money and we’ll never the key. The files were still encrypted and useless.
During all the waiting, I resumed trying my hand at decryption. To my amazement teslacrack.py method actually worked. Not only it worked, but once I figured out the right tools and the right steps, the actual computation time to crack the key was only 30 seconds. This was amazing because normally these methods take weeks if you are extremely lucky. It turns out that in addition to a mountain worth of luck on my side, the criminals made a mistake in the way they encrypted the files. If the criminals didn’t make that mistake, no amount of luck would have helped. Cracking the key would take millions of years. It is also interesting to note that the criminals encrypted different sets of files using 2 different keys meaning that even if they let you decrypt one set of files, the other set of files would still be encrypted (presumably to force you to pay an additional fee). Furthermore the key for one set of files was cracked in 30 seconds, the second key took more than 5 days with no success. Luckily, the first set contained the critical files, and the client was not too concerned about the second set, so we stopped there. By the way, eventually the link that took us to the criminal’s billing/support site went blank. We’re never going to see that key or the $700+ CAD.
- Backup, backup, backup, backup. There is not going to be a second chance like this again. A mistake like this is not going to show up again in their next version of ransomware. In fact there are other variants of ransomware that are already impossible to crack. Did I mention backup?
- When dealing with criminals never expect a favorable outcome. If you’re sending any money for ransom, consider it lost the moment you hit send. What’s more, be prepared for escalation. Once the criminals know you are willing to send money they may come back at you with asking for more. They will either just ask for more even though you already send exactly what they asked for, or you may find out that after part of your files are decrypted, the other part is still encrypted.
How to recover:
Step 1) Take backup and work from the backup
- This is important because even though your files are encrypted and apparently useless, your day will get considerably worse if you lose them somehow. What if the ransomware detects you are trying to decrypt and deletes everything right now and then?
Step 2) Install programs:
- Python 2.7 (32 bit because it must match library below)
- Pycrypto-2.6.win32-py2.7 library (I ended up using this because I had trouble compiling the library from source)
- Msieve150_win32 (I also tried optimized versions for Intel processors and CUDA, but reading about possible bugs didn’t give me confidence, so try them, and if they work then fine, but don’t forget to have version 150 as your fail safe)
Step 3) Identify the AES key that you will try to crack
- Place some vvv files in the same folder as teslacrack.py and run teslacrack.py
- You will get something like this:
Cannot decrypt ./IMG_1111.JPG.vvv, unknown key Software has encountered the following unknown AES keys, please crack them first using msieve: 346FA15D6F7106A05553587E67AD068EBF0CE65C9ECBA74BAE144661AB502CEFFEBCFA9FBB3CDFD9E4043B3402F970051E55063D96C94AB66B443A0F9D088A23 found in ./IMG_1111.JPG.vvv Alternatively, you can crack the following Bitcoin key(s) using msieve, and use them with TeslaDecoder: E82E090D9A73DC4E93201BC56394544493EFD0DD2631F588C4083F006C1CD419F096F1D6E646AA0DE8D0230CB18D009B231DEA6EF7CAFED03C6C53830E51074A found in ./IMG_1111.JPG.vvv
Step 4) Crack the AES key
Run msieve with the AES key found in step 3 (notice 0x in front):
msieve -v -e 0x346FA15D6F7106A05553587E67AD068EBF0CE65C9ECBA74BAE144661AB502CEFFEBCFA9FBB3CDFD9E4043B3402F970051E55063D96C94AB66B443A0F9D088A23
You should see something like this
random seeds: 4e305a00 6fb1837c factoring 2746299090781689070444389534863512001481868057180589068197106350690661 98749363149686207988485815608295042086154388677331498764717631003373547132362899 7155 (154 digits) searching for 15-digit factors P-1 stage 1 factor found searching for 20-digit factors P-1 stage 2 factor found searching for 25-digit factors P-1 stage 2 factor found commencing quadratic sieve (33-digit input) using multiplier of 3 using VC8 32kb sieve core sieve interval: 4 blocks of size 32768 processing polynomials in batches of 51 using a sieve bound of 4909 (341 primes) using large prime bound of 196360 (17 bits) polynomial 'A' values have 4 factors sieving in progress (press Ctrl-C to pause) 696 relations (302 full + 394 combined from 2196 partial), need 437 696 relations (302 full + 394 combined from 2196 partial), need 437 sieving complete, commencing postprocessing begin with 2498 relations reduce to 1007 relations in 2 passes attempting to read 1007 relations recovered 1007 relations recovered 24 polynomials attempting to build 696 cycles found 696 cycles in 1 passes distribution of cycle lengths: length 1 : 302 length 2 : 394 largest cycle: 2 relations matrix is 341 x 696 (0.1 MB) with weight 11065 (15.90/col) sparse part has weight 11065 (15.90/col) filtering completed in 1 passes matrix is 341 x 405 (0.0 MB) with weight 5168 (12.76/col) sparse part has weight 5168 (12.76/col) commencing Lanczos iteration memory use: 0.0 MB lanczos halted after 7 iterations (dim = 330) recovered 63 nontrivial dependencies commencing quadratic sieve (69-digit input) using multiplier of 23 using VC8 32kb sieve core sieve interval: 12 blocks of size 32768 processing polynomials in batches of 17 using a sieve bound of 209771 (9278 primes) using large prime bound of 19508703 (24 bits) using trial factoring cutoff of 24 bits polynomial 'A' values have 9 factors sieving in progress (press Ctrl-C to pause) 9427 relations (4473 full + 4954 combined from 52519 partial), need 9374 9427 relations (4473 full + 4954 combined from 52519 partial), need 9374 sieving complete, commencing postprocessing begin with 56992 relations reduce to 13757 relations in 2 passes attempting to read 13757 relations recovered 13757 relations recovered 11859 polynomials attempting to build 9427 cycles found 9427 cycles in 1 passes distribution of cycle lengths: length 1 : 4473 length 2 : 4954 largest cycle: 2 relations matrix is 9278 x 9427 (1.3 MB) with weight 274633 (29.13/col) sparse part has weight 274633 (29.13/col) filtering completed in 3 passes matrix is 8448 x 8512 (1.2 MB) with weight 244661 (28.74/col) sparse part has weight 244661 (28.74/col) commencing Lanczos iteration memory use: 1.2 MB lanczos halted after 135 iterations (dim = 8443) recovered 61 nontrivial dependencies p1 factor: 3 p1 factor: 5 p6 factor: 418819 p8 factor: 10304417 prp13 factor: 8162073202471 prp14 factor: 84794311049579 prp19 factor: 3135407003350317697 prp25 factor: 2560807722929541167424011 prp26 factor: 19683723106610479028057093 prp45 factor: 387847886921773814156469727175786645600806381 elapsed time 00:00:37
Depending how lucky you are, this process will run for anywhere from minutes to weeks.
Step 5) Unfactor based on prime factors from step 4
- take the 10 factors at the very end of the file and feed them into unfactor-ecdsa.py. If any factors are listed multiple times, repeat them also.
unfactor-ecdsa.py ./IMG_1111.JPG.vvv 3 5 418819 10304417 8162073202471 84794311049579 3135407003350317697 2560807722929541167424011 19683723106610479028057093 387847886921773814156469727175786645600806381 Found AES private key: b'\x6d\xb8\x64\x76\x72\x31\xc4\xff\xfc\x22\x48\x20\xa5\xbc\xcd\x6c\x4c\x30\x2f\xc3\x4d\xd6\xfa\x23\x4b\x4b\x9e\x0c\x1d\xaf\xec\x07' (6DB864767231C4FFFC224820A5BCCD6C4C302FC34DD6FA234B4B9E0C1DAFEC07)
Step 6) Use AES private key to decrypt all your files
- Edit teslacrack.py and add AES private key to the list of keys at the beginning of the file
- Run teslacrack.py C:\
- It will decrypt every vvv file that was encrypted by this specific key
Step 7) Backup, backup, backup. Next time this will NOT work.
Sometimes it’s useful to convert dynamic PHP, ASP.NET, Java, or CGI pages to static HTML sites. This comes in handy when trying to create a copy for disaster recovery. It is also good way to stop hackers from hacking your PHP site that’s obsolete, unpatched or otherwise insecure. I find wget is the quickest way to get the job done:
wget -l10 –mirror -p -e robots=off –convert-links –html-extension http://thesiteyouaremirroring.com
What’s uncloud? Solutions that deliver cloud like empowerment without giving up privacy and control.
Owncloud: Works just like Dropbox, One Drive, G-drive etc, except all the data is stored on a server that you control.
- Easy to setup
- Free (there is also a paid Pro version)
- Open source
You can read more about it on Owncloud’s website
This is a new utility that let’s you paste base64 encoded text from your clipboard directly to a file.
If you’ve ever found your self in a SSH console with no way to do file transfer back to your workstation, this utility is for you. Normally in such a situation, you’d have to base64 encode the file, copy it across to your workstation, and decode it there. I thought that was too many steps, so I simplified the process for my self by making this tool.
Note: This is old version 0.1 – you can get the new version 0.2 here
Download for Windows (64-bit, portable as long as you have .Net framework 4.5)
Phone verification is becoming wide spread for free account sign ups on sites like Yahoo, Gmail, Craig’s list, Facebook etc. I hate giving out my real info just to test something with a burner account. In the past, the solution was as simple as Googling “receive free sms” and going to one of dozen’s sites that will receive the verification code. I knew that it was too easy to last. Sure enough, the numbers from those sites are now black listed.
I really didn’t want to give out my primary cell phone to Yahoo. So I reached went for the next best thing. My VoIP trunk provider has the capability to receive SMS on a VoIP number and forward it to my primary phone, or even as an email. I entered that in, and to my surprise yahoo still rejected it. That got me thinking. How can they be blacklisting my own private number? No one else registered with it before.
How it works:
With the numbers being ported all over the place, how can they know whether a particular number is a VoIP number? The short answer is that they don’t know. They take a guess based on the information that’s publicly available (or available for small fee) through sources such as these:
Local calling guide is actually quite accurate (and free). Where it messes up, is as soon as a number is ported. It will only show the name of the telco that originally registered the number, all subsequent ports are ignored. When I looked up my numbers that failed to verify, sure enough they all came back as belonging to a VoIP company. Same thing when I tried to look up many “receive free sms” service numbers. They all showed up as VoIP.
So now what? I still don’t want to give Yahoo my cell phone
… There isn’t a fool proof way, but some of these may work
- Port a “real” number to VoIP. It costs me about $12/first year to have a secondary VoIP number – which of course doesn’t work that great now days. Porting a real number to VoIP will work, but it’s little bit more time and money. I figure about $50/first year (get a sim, port it, cancel original service, keep it running as VoIP). Not that bad, but a little too much trouble for what I’m using it for.
- Sign up with a service that provides verification codes for burner phone numbers. They do the above steps for you in mass scale as a service. I haven’t tried it though. If they listed their numbers in advance I could look them up to see if they show up as VoIP. For this to work they must have a proper setup using “real” phone numbers. Also it’s hard to believe that they wouldn’t get black listed in a hurry. Maybe there is one that works, but chances are if it works, now it won’t work several months later.
- Give up for now (until something better comes along)… this is kind of what I ended up doing. I still didn’t give them a real number; I ended up reviving one of my old yahoo accounts. I used it instead.
While trying to create a new database in MS SQL , I kept getting a 1807 error, complaining that MS SQL can’t obtain exclusive lock on database ‘model’
Create failed for Database ‘x’. (Microsoft.SqlServer.Smo)
An exception occurred while executing a Transact-SQL statement or batch. (Microsoft.SqlServer.ConnectionInfo)
Could not obtain exclusive lock on database ‘model’. Retry the operation later.
CREATE DATABASE failed. Some file names listed could not be created. Check related errors. (Microsoft SQL Server, Error: 1807)
To solve this:
- run sp_who2 query
- find out which process holds a lock on ‘model’ database
- kill that process using :
Notes regarding killing processes:
- Be careful killing the following types of processes AWAITING COMMAND, CHECKPOINT SLEEP, LAZY WRITER, LOCK MONITOR, SIGNAL HANDLER, it’s probably a bad idea killing them forcefully
- Instead of killing processes it sometimes helps to restart the following services / applications
- SQL Management Studio
- Appassure Agent service
- SQL Server VSS Writer service
Strong IT security is a losing proposition. If you want to develop a system that’s connected to the internet that is unhackable, you’ve lost the game already. There is no such thing. If someone wants to hack you they will succeed. The only thing you can do is inch your way up the ladder and eliminate the lowest hanging fruit. The strategy becomes being faster than your friend rather than trying to outrun the bear. Other than that get ready to be hacked. Your goal should be to get hacked less frequently than your competitors, and when you’re hacked you want to be sure it’s an inconvenience rather than a business ending catastrophy.
Preventing the severity of the attack is where the tables turn. You as an owner of your data, have an upper hand in restoring control. Just as you had no foolproof protection against getting hacked, your adversary faces the same problem if their goal is to wipe your data or your business operation off the face of the earth. They will not succeed if you don’t want them to. Your secret weapon is physical control. As long are you retain physical control over your data, you may lose many battles, but you will ultimately win the war.
When hackers penetrate a single system, they call that “owning the system”. However, that is overstating their accomplishment a bit. They should really claim that they “temporary control” the system. Of course, that doesn’t have the same ring to it. They don’t “own” anything if they can be locked out by you the owner. Yes, they can do all kinds of damage while on the system. Yes, they can cause down time. Yes, it can be a huge disaster. Yes, it sucks to be hacked. But as long as you retain physical control over your system you’ll always have the upper hand in the long run.
Why physical control? What makes physical control so special? It’s because no matter what kind of games and silly wars happen at the OS or application level, you have a secret secret weapon that hackers don’t. It’s a weapon that’s unstoppable. It’s a weapon that makes all the games stop. You have physical control. Only you can unplug a network cable – they can never plug it back in. Only you can shutdown a machine – they can never start it. Only you can do a offline bare-metal restore and they can’t do a thing about it. Only you can put your your offsite backups in your briefcase, they can hack all night and they are not going to touch those.
This is why public cloud computing is problematic. Public cloud takes away the only truly effective weapon you had. You have 0 physical control in public cloud. You can’t unplug cables, you can’t carry away your backups in a briefcase, and you can’t restore elsewhere. You’re not the owner anymore. With public cloud, you’ve outsourced the battle. The battle is now between your cloud provider and the hacker. Will the cloud fight as hard for you as you would fight for yourself? Probably not. In fact I guarantee it. The cloud will point their finger at you and call it a day.
If you’re going to use the cloud, your physical infrastructure needs to be the root that controls it all. Don’t think of a cloud as a way to “keep your stuff safe”, think of it as a pace where you temporarily place your stuff to get a specific job done. It could be there one day, gone the next – just like a cloud. You should always remain the physical owner of your data and of your key infrastructure.