I would like to welcome LinkedIn to the not-so-exclusive club of major web sites that have experienced major password leaks. Like any other major leak it is hard to visit any forum or tech blog without seeing some mention of it. And like any other leak my inbox is starting to fill up with press requests for comments.

But what is interesting here is that there’s nothing interesting here. It’s the same thing we have seen so many times in the past and surely will continue to see.

One thing that highlights this, brought up to me by blogger Johnvey Hwang, is that 93% of the passwords in my Top 10,000 passwords list appear in the LinkedIn hashes dump. Here it is in his words:

I was curious as to what percentage of the most common passwords were present in this dump, as a proxy for gauging the password choices for a supposedly more professional population. A quick search led me to security guy Mark Burnett, who maintains a list of the top 10,000 most used passwords across the internet. He admits to some skew caused by a significant amount of sourcing from adult websites, but I don’t think it really matters.

The fact that such a large number of the LinkedIn passwords appear on the top 10,000 list certainly does help validate my data but more importantly it shows that despite all we have learned, very little has ever changed.

Here are some other interesting facts Johnvey discovered about the list:

• 7,142 of the most common passwords were present
• 546 of the most common passwords were not present
• 2,312 of the most common passwords were too short for LinkedIn’s 6 character minimum

I think that 93% is an amazing number, yet again, the biggest story here is that nothing really has changed.

Sidenote:

I personally have three LinkedIn accounts that I maintain. None of those three passwords appear on the list. Apparently the list is not complete, but the question now is what criteria put those particular passwords on the list.

Password security has always been a hot issue but events in the last few years have made it an even more pressing issue to a greater number of people. When I hear receptionists in a doctor’s office sharing strategies for creating secure passwords I know this is now beyond the realm of network administrators and security professionals.

But one thing I have noticed is that many people don’t truly understand why one password can be so much stronger than another so I thought I would walk through the process of cracking a password. In this case, I decided to use as an example the very password that (until I wrote this) I use for the admin account on this blog.

So like I said in the title, my password is 4.hub.route.edu.

That isn’t the best password I have come up with but it is still fairly strong. It is 15 characters long, contains a number, letters, and some periods. It took me just a couple logins to actually memorize that password. The word components are fast to type because we are trained to type in whole words. And there are four parts, each one ending with a period. The repetition of the period helps the memory process.

Chances are that no one would be able to go to my admin page (which itself is protected by a different password) and just guess that, no matter how much they knew about me and no matter how many of my other passwords they knew because I have never used that password anywhere else. As of writing this article, I can do a Google search for “4.hub.route.edu” and there will be no results.

But the real risk isn’t someone being able to keep trying to guess my password via the admin page, the real risk is someone finding a new 0-day exploit that allows them to dump the users table in my database and get the hash of my password (which happens to be $P$9YCJ/QwbFcgbo7OtfWGYYE8sVJBxtF/). If someone can get your hash, they can now try millions of password combinations without you ever knowing it.

Cracking a password hash is a lot like trying keys in a lock. A hash is a string of characters derived from your password that is calculated in such a way that it is nearly impossible to work backwards to discover the original password so it is relatively safe to store. When you log in to a system, it will run the password you enter through this same complex formula and the result should be the same.

So when I first created my password on this blog I entered 4.hub.route.edu. WordPress ran it through these formulas and came up with the hash $P$9YCJ/QwbFcgbo7OtfWGYYE8sVJBxtF/ which it saved it in the database. The next time I log in, I enter my 4.hub.route.edu password, WordPress runs the same formula on that password and it comes up with $P$9YCJ/QwbFcgbo7OtfWGYYE8sVJBxtF/ which matches the hash it has stored so it knows that I am using the correct password even though WordPress never stored my actual password. Now what is special about these formulas is that it is extremely rare that any two passwords will create the exact same hash (a concept known as collision).

So if someone is able to obtain my hash, they can’t directly get my password from that, but they can try millions or even billions of different passwords and run each one through the formula until they find one that produces that exact same hash. It is a lot like having a lock, you can’t easily create a key from it but you can try a bunch of keys until you find the one that works.

Now when it comes to passwords there are actually hundreds of trillions of possible passwords someone might choose. Even with a cluster of powerful computers it could take decades to try every possible password. Fortunately for hackers, most people aren’t that clever with their passwords. There are a number of strategies they use that can drastically reduce the number of passwords you need to test to crack a password. Below is that strategy

1. Hash Lookup

First, an attacker will check to see if someone else has cracked the password before, using either a local database or an online database such as onlinehashcrack.com or hash-database.net or one of the hundreds of other similar sites. In the past few years there have been many large sites that have been hacked and their passwords leaked. If you password was ever one of these, chances are it will appear in one of these databases. Likewise, if you select a common password that others may also be using, it also might be on this list.

In the case of WordPress, the hashes are created using PHPASS but for the sake of this example, let’s just assume they use MD5 hashes like many other systems use. The MD5 hash for my password 4.hub.route.edu is 7914881ba9b78fa307db6ef0db675e29. You can search any online databases for my hash and you will not find it listed anywhere (at least at the time of writing). If your password is one that you have never used before and others likely have not used, you should be safe (try googling one of your passwords, you may be surprised how many results you get).

If your password hash does not appear in one of these databases, there are also rainbow tables which are massive databases of precomputed hashes consisting of every possible password up to 8-10 characters in length, depending on the algorithm. If your password is less than eight characters long, your password surely will be cracked at this stage. However, you will not find 7914881ba9b78fa307db6ef0db675e29 in any of those databases so I am safe so far.

The lesson here is to never use a password less than ten characters long. Never use the same password on multiple systems. Don’t try to be clever with your password, that never works (NCC-1701 is a very common password).

2. The Word List

Since most passwords consist of dictionary words or something similar, checking every word in a dictionary or a specialized wordlist http://svn.isdpodcast.com/wordlists/ is a quick way to find a weak password. Most hackers will use lists of the most common passwords such as this because chances are very high that someone will be using one of those passwords. It normally doesn’t take more than a minute to go through even a gigantic list of words.

In my case, even a Google search for my password turns up nothing so even if you had the massive list of words that Google has indexed you still wouldn’t be able to crack my password.

Considering this, you can see why so many systems simply probihit any password that is a dictionary word.

3. Rules and Patterns

If a dictionary or wordlist check fails, the next step is to try some of the common (albeit innefective) tricks people use to make a password more complex. If you asked me what I thought was the most common password pattern I would say a proper noun (such as a name) followed by 2-3 numbers. So it would be smart for a hacker to take each word in a wordlist and add ever possible number from 1 through 999. If that doesn’t work, you could try reversing each word or doing simple substitutions like using the number 3 instead of the letter e. It really does not take much effort for a cracking program to try hundreds of different patterns.

For example, a dictionary word may be “password” so a rules-based attack my try PASSWORD, dRowssap, P@SSW0RD, p@ssW0rd, dr0Wss@p, passwordpassword, @ssW0rdp, dp@ssW0r, p@9sW0rd, 1p@ssW0rd, p@$$W0rd, ppp@ssW0rd, 1p@ssW0rd, and thousands of other variants of the word. Depending on the number of rules and the size of the wordlist, this step may take only five to ten minutes and will crack a great number of passwords.

If an attacker has sufficient processing power, another effective strategy is to try two dictionary words together with various delimiters between them (such as dashes or periods). If you had a wordlist of 100,000 words and tried every combination of two words that means you would have ten billion possible combinations. Trying different delimiters between the words would make it a little bit harder but not much.

You probably wouldn’t want to try three-word combinations because that would take you up to a quadillion (1,000,000,000,000,000) possible combinations which would not be an effective strategy. In the case of my password there is a number and three other words that would likely appear in a dictionary but testing for four-word combinations would mean there are 100 quintillion (100,000,000,000,000,000,000) possible passwords, so the odds are my password would still be pretty safe.

The lesson here is that a strong password is not a matter of being clever, it is a matter of beating the numbers. Passwords should always contain three or more words or other sequences.

4. Brute Force

If a password hash doesn’t show up in a database or hasn’t been cracked before, does not show up in a list of common passwords or dictionary words (even after trying hundreds of common variants), the only method left is to simply brute-force the password. This means trying every possible combination of letters until you find the password. It would be like trying to crack a simple bicycle lock, you would start with 000 and try 001, 002, 003, and so on until you got to 999.

In the case of passwords you would need to try every combination of lowercase letters, uppercase letters, numbers, and punctuation symbols. In other words, imagine a bicycle lock where each dial contains abcdefghijklmnopqrstuvwxyz ABCEDFGHIJKLMNOPQRSTUVWXYZ0123456789`~!@#$%^&*()_-+={[}]|:;"'.?/ and there are eight or more dials. This is why so many systems require that you use a variety of characters because using different types of characters is like making each dial larger. And making your password longer is like adding more dials.

Now brute-force attacks are much smarter nowadays using techniques such as mask-based attacks. These types of attacks basically use knowledge about passwords to make the brute-force process much smarter. For example, if you look at this chart http://xato.net/img/UpperCaseLettersLarge.jpg you will see that uppercase letters are very likely to show up in position 1 but are extremely rare after position 8. Knowing this, it would be more effective to not even bother looking for uppercase letters after the first few characters. Now if you look at the distribution of all character sets in this graph http://xato.net/img/CharacterDistributionByPositionLarge.jpg you can see that much can be done to optimize the brute-force process. Nevertheless, these rules become less and less effective the longer and more complex your password gets.

The big secret here is if you can force a hacker to have to use a brute-force attack and you have a password that is at least 15 characters long, chances are that you have won. Eventually computing power will catch up so that even 15 characters might be enough but the good thing is that these numbers grow exponentially so a 16-character password is almost 100 times stronger than a 15-character password and a 17-character is more than 9,000 times stronger!

So What Makes a Password Strong?

Your password must be something very unique and one that you have never used before. In fact it should be so unique that if you did a Google search for it, there would never be any results. You can’t just take a word and dress it up a bit, you need 3-4 words or other sequences to make a password strong. And finally it has to be long. It helps to throw in some numbers and pumctuation but most importantly it has to be long.

Some of you out there know that I have been collecting passwords for quite some time. Since 1998 to be exact. Originally I did it just to have big wordlists for password cracking, then I started gathering them for research on my Perfect Passwords book, finally it became like a big ball of string where you just do it because it makes no sense to stop now. My list currently contains about 6 million unique username/password combinations (not counting those from public lists from Gawker, RockYou, and others).

So I thought that some people might be interested in how I collect these passwords. Note that all of these passwords have already been made public and can easily be found by anyone. There are no passwords on my list that have not already been made public. Also note that so far I have never shared this list with anyone.

1. I use tools such as Athena, which does massive Google searches for and collects passwords in the format “http://user:password@example.com/members”. This tool can easily gather 200,000 combos in a day but the majority of these are already in my database. I run this about once a month.
2. I have a script that nightly leeches from a huge list of well-known password sharing web sites.
3. I use a number of Google alerts that watch for common keylogger log formats. This is just one of many that I use. There are a surprisingly huge number of these logs that can be found via Google, although it is sometimes difficult to parse the passwords from the content.
4. I use Google alerts to watch for SQL database dumps of forum and other common software databases.
5. I also use Google alerts to look for passwords on pastebin.com and other related sites.
6. I use a script that grabs all the Google alerts as RSS feeds and parses out URLs, then another script visits each site and leeches the passwords.
7. I use RSS feeds from filestube.com to watch for and download password lists that might show up on a number of file sharing sites.
8. I use RSS feeds from various torrent searches that I put into uTorrent to download automatically.
9. I use a number of IRC bots that hang out in a large number of IRC channels where password sharing happens. These aren’t as effective as they once were but I still use them occasionally.
10. I use a script to automatically download posts from various Usenet newsgroups, although most of those are just spam nowadays.
11. I visit a number of public and private hacking-related forums to get wordlists and hacked passwords. I often pay for VIP memberships (usually the lifetime ones) so that I can access premium content areas. Leeching from forums has to be done manually, because you often have to comment on posts to be able to download the lists, but occasionally I will spend half a day leeching from these forums. Some forums will let you subscribe to posts and will include the entire post contents in the email. This bypasses the often-used “hide hack” and I can just use another script to save that inbox to local files.
12. I use various FTP search engines to watch for interesting filenames that might show up on FTP sites.
13. In the past I have used various P2P networks (such as LimeWire) to search for files but those don’t produce many results nowadays.
14. Every once in a while someone will send me a big dump of their own lists they have collected.

As these scripts collect data, it is all dumped into a directory on my hard drive and regularly I run program I wrote that parses all the data looking for password is common formats.

Here are some examples of what the program recognizes:

http://www.example.com/members/ L:user1 P:password1
http://www.example.com/members login:user1 password:password1
http://www.example.com/members user: user1 pass:  password1
Login: user1 passw:password1
L:user1 P:password1
username:user1 password:password1
http://www.example.com/members L: user1 P:  password1
username = user1  password= password1
u=user1 p=password1
username    user1  password    password1
login id: user1 password: password1

It grabs the username/password combos and saves them into text log file. After a while these files accumulate and I merge them into my master database. In the database I perform cleanup steps such as removing passwords from well-known password hackers (such as pr0test) and other junk that might appear. I also strip domain names off usernames that are email addresses.

What is interesting about all this is how difficult it is to find new username/passwords combos that aren’t already on my list. These scripts can easily collecting 100,000 unique username/password combos every day, but only a few thousand of those are not already on my list.

After 12+ years of collecting passwords, I have found a few interesting facts:

• Although my list contains about 6 million username/password combos, the list only contains about 1,300,000 unique passwords.
• Of those, approximately 300,000 of those passwords are used by more than one person; about 1,000,000 only appear once (and a good portion of those are obviously generated by a computer).
• The list of the top 20 passwords rarely changes and 1 out of every 50 people uses one of these passwords.

There are a few flaws with my list that I should point out:

• Many of these passwords have been cracked from hashes so a good percentage of them would by nature be crackable, skewing the statistics some.
• These passwords are largely dominated by passwords from adult web sites, which are the ones mostly publicly shared. This results in a higher percentage of adult-related and obscene passwords.
• These passwords are usually from web sites that often do not enforce strong passwords policies that a private organization might. This is bad because this data doesn’t truly reflect all passwords, but on the other hand it shows the kind of passwords users will select if a password policy is not enforced.
• My scripts only grab usernames and passwords between 3 and 30 characters long, all others are thrown out.
• None of the passwords contain a colon, because that is the delimiter used to separate usernames and passwords in the combo lists my scripts generate.

So that is how I collect my passwords, maybe someday I will share the list itself.

Incidentally, the one tool I really wish I had time to build is either a proxy server or a Greasemonkey script that will automatically parse and log usernames and password combos from web pages that you visit. That would be extremely helpful!

 Update (4/25/12): Google has recently changed things that resulted in breaking several of the tools listed here. Now I collect many of my passwords using google alerts and custom searches turned into RSS feeds and automatically added into a private WordPress blog via AutoBlogged. Before each post is added it runs through a tool I have developed (which I will share eventually) that returns just the username/password combos. I can then use the RSS feed from that private blog as a raw combos list to merge into my master list.

It still amazes me that after all the education over the years that there are still so many poor password policies out there. Anyone who has ever filled out a web form likely has run into these overly complex and frustrating passwords policies.

But sometimes a password policy is an indication of a bigger problem. For example, today I was setting up an account and entered a very strong password and was presented with the following error message:

Apparently what caused the error is that I used a period in my password and this policy only allows for numbers and letters. But the bigger question here is why doesn’t the policy allow for numbers? Why does the password have to start with a letter? And why is there a limit of 20 characters?

The reasons why these concern me is that they sound more like technical limitations rather than being motivated by strict password security. Normally when you store a password, you first create a hash of the passsword and then store the hash. The nice thing about hashes is that being hexidecimal values you don’t have to worry about the security risks of special metacharacters and symbols. You also don’t care about maximum length because the hash is always a fixed length, whether your password is 10 characters or 100 characters.

However, if you are worrying about non-alphanumeric characters and the password length, that suspiciously sounds like they are storing the password itself, not a hash of the password. Being a healthcare orginazation this is a big deal. And although we like to think that most big companies have security teams that prevent things like this, the recent announcement that Sony stored passwords in plaintext tells us otherwise.

Secret Questions
A little lower on the page I ran across another problem, they are letting the users select the secret question as shown here:

The problem with this is that most users are not qualified to come up with quality secret questions. At best you will see questions asking for their favorite color (how many colors are there really?), their dog’s name (just look on their Facebook profile), or where do they live. To make matters worse, a shockingly high number of people will actually put the answer as the question itself, as a hint. If you disagree with me on this and have a site that lets users set their own secret questions, check your database and you will be surprised how bad they can be and how often they reveal the answer in the question itself. This is one of those little secrets that hackers have known for years.

Finally, the worst offense is the way in which this site lets you recover your password:

If you click on the Forgot Password link, you are shown the above form which lets you set a new password if you know the secret question. The problem with this is that this makes the secret question as powerful as the password itself, because just knowing the answer lets you set a new password. Normally, answering a secret question will initiate a process whereby the registered email account receives a message that a password reset was initiated and that they need to click a link to finish the process.

The reason we do this is because secret questions are not secure. The information in a secret question is something that is easily discoverable, has a limited number of possible answers, and is a fact that will never change. We can only partially compensate for these problems by sending an e-mail notification to the user and requiring a click through.

In this particular case, the user sets their own secret question which has a high chance of being insecure and then all that is needed to set a new password is to be able to guess that answer, no email access required.

The sad thing is this company has already appeared in this database once before, I hope it doesn’t happen again, especially not with my medical information.

Many people tell me they are surprised with how much effort I put into hardening Windows Server 2003–the last hardening document I wrote for a client was 112 pages long. That’s not 112 pages of writing, policy, and how-to’s, that’s 112 pages of nothing but settings. The process itself involves the modification, removal, or locking down of over 5,000 Registry keys and system files. Continue reading “Does Windows Server 2003 Even Need Hardening?” »

I noticed that Schneier wrote a bit on choosing passwords and gets into some detail on how to secure a password based on some of the techniques used to crack passwords.

His specific advice is:

“…if you want your password to be hard to guess, you should choose something not on any of the root or appendage lists. You should mix upper and lowercase in the middle of your root. You should add numbers and symbols in the middle of your root, not as common substitutions. Or drop your appendage in the middle of your root. Or use two roots with an appendage in the middle.”

While I certainly do agree with the validity of this advice, if you are an administrator, I wouldn’t recommend telling users to “drop their appendages in the middle of their roots.” Here’s some more practical advice: tell them to choose long passwords. Continue reading “Long passwords are strong passwords” »