One day I wrote down a list of 25 thoughts on risk management and testing.  Eventually I’ll weave these ideas into articles, but for now, just a <ul>

* Everyone contributes risks

* No risk exists without probability and impact

* No impact is infinite

* All impacts are measurable

* Risks drive level of design

* Risks drive test plans

* Testing directly affects design

* TDD is not 100%. The decision heuristic is risk.

* Exploratory testing approach is publicly known so customers and decision makers can know what has been tested.

* The word “requirements” promotes simplistic thinking. We develop features up until the point at which a person or team decides to deploy. Some features are never implemented. Listing as requirements makes people think in binary mode instead of order of importance to the organization.

* Measure progress based on running features tested. Compare sum of the weights completed with the total targeted feature set weight.

* Features all have a “weight” relative to other features within the same project.

* A feature without a running test is analogous to “works on my computer.”

* In all projects there is a point at which the set of claims tests are a more accurate statement of what the software does than the list of features (or requirements for you old schoolers.)

* Some features like “easy to learn” are not testable claims. Are they?

* A goal of gathering and writing features is to have them written as claims.

* The best types of claims have inputs and outputs because there is less interpretation in the process from documenting the feature to writing the code and tests.

* Tests are either named or described in a way that relates them to a feature.

* The build process generates a list of all test results for a project. The tests for a project can exist in multiple places (web app, shared component, project specific component)

* Risks have the greatest focus of the project team. Not the “requirements” document. It is a fallacy that software development is about creating a 100% complete list of requirements and then coding to that list.

* Risks are the most managed “thing.” Later in the development process, issues become the primary management tool.

* A project has a list of tests and you are able to click a link to view the actual source code of the test.

* Each test has a history of pass/fail.

* When a developer completes an issue, they add a comment in the issue message that can be run by the reactor and will send a result to the nosy list.

* A daily email is sent of verification results. Project name -> className.testName of failures with link.

In what appears to be some form of risk management strategy, these login forms introduce a different risk to the websites that use the default behaviors – in RoR sites, this is often either AuthLogic or RestfulAuthentication.

Both are broken in a big way.

First, the bug.  And you might not even recognize it is a bug because the behaviour is so wide spread, so ingrained, and so “sensible”.

On almost every login form, if I submit either the wrong username OR the wrong password.  I get the same error message.  Usually it’ll be something brief like the following.

Notice the very important word in this message.  OR.  Because the error message is the same if I enter the wrong user name OR the wrong password, I don’t know which piece of data I entered incorrectly.

I’m guessing virtually everyone reading this is now thinking “well yeah, if you tell them which piece is wrong, someone can keep guessing until they get it right.”

Before we get into that, what are the risks being mitigated in this process?  You don’t want someone to hack into one of your user’s accounts, right?

Right.

But what else?  How about the risk that someone, a new user, comes back to your site and has forgotten which password they are using?  Maybe you forced them to use some complex password.  Maybe not.  In any case, maybe they want to use your site, but it isn’t really that important.  As web app builders, we’re constantly thinking about lowering the barriers to join and begin using a web application, right? Why would we add a barrier here.  It is possible that a user with difficulties signing in might just leave.  Never to come back.  Never to pay you for your service.

All because you were not kind in telling him/her why the credentials they gave on login didn’t work.

So, that’s a risk.  And for a startup, this is a significant risk.  You need every new user you can get.  Especially those people that have taken the time to actually create an account.

But let’s get back to the “common sense” reason for not telling the user more about why their credentials didn’t work.

You may notice on the signin form above, that link to “Forgot“.  If you click that link and enter a user name (email address in this case), the system will almost always tell you if that user name exists.  Using the above site, it’ll look like this.

So, lemme just repeat the problem.

Login forms almost never tell you if your user name is wrong

Compared with

Forgot password forms almost always tell you if your user name is wrong

So let’s get back to that risk that is supposedly being mitigated by not telling the form submitter what piece of data is wrong.  We’re trying to keep people from guessing user names, right?  And yet we provide a form for doing exactly that!

The risk is not mitigated in the slightest by the error message.  Not one bit.

Now, I’m not going to go into how this process should be done.  For two reasons.  One, I’m out of time for writing this morning.  And Two, because I want people to disagree with me so I can understand their thought process.  It is possible I’m missing something here.  Unlikely, but possible.

Password requirements have gotten completely out of hand. And the effect is LESS security rather than more. For instance, I use Rackspace hosting. For their admin site, they require my password to contain at least one capital letter, one number, one symbol, and to be at least 8 characters long. I called to complain about the ridiculous nature of these requirements and the customer service rep told me I could “just save that password with my browser” so I wouldn’t have to remember it.

Um.

HWAT?!?

They want their system to be secure, so they crank up the difficulty of their password scheme and then they outsource that security to the security of my laptop, which doesn’t have a password. That can’t possibly make sense to anyone. The system would no longer be secure.

It’s like, no matter how many studies come out that strong passwords are not the problem, the “security experts” (used incredibly loosely) continue to argue for “stronger” passwords.

At what point do we start learning squirrel?

Every once in a while, I come across a smart business with smart password requirements. For example, litmusapp requires 4 characters. That’s it. 4. Lousy. Characters. Are they crazy? How can that possibly be secure?

Probably, because they’re smart. Most likely they have systems in place that guard against brute force hacks. I’ll probably write more about that at some point, but I’m too lazy right now.

In the meantime, don’t be dumb and don’t be Dilbert’s boss. Be cool, like the Litmus people.