Credit Cards Failing Open

Most consumers are aware that when you close a credit card account, it’s not really closed. For “convenience” reasons, recurring subscription charges such as your cable bill will continue to be approved. You can kind of see where the credit card companies are coming from, but it’s a pretty weak argument. The cable company just needs to notify me that the credit card on file is no longer valid, and I’ll update my information. Problem solved.

But that credit card weirdness is nothing compared to the one I’m about to describe.

Before we do that, let’s take a moment to discuss the design principle of failing securely. The general idea is that if a security mechanism fails, it should fail closed. If your firewall crashes, it should block all traffic, not allow all the packets through. If the power source to your card key system is interrupted, it shouldn’t unlock all the doors. If the connection between your application server and your LDAP directory is severed, subsequent authentication requests should be rejected, not approved. This is not rocket science.

So back to credit cards. I had a conversation last night with an old friend who related a bizarre situation they had encountered during the QA process for one of their web applications. One of their tests involved repeatedly attempting a credit card transaction using a canceled/expired American Express card. Here’s what they saw in their logs, paraphrased by me:

Attempt 1: Denied
Attempt 2: Denied
Attempt 3: Denied
 .
 .
 .
Attempt 49: Denied
Attempt 50: Denied
Attempt 51: Approved

What the…? Approved? That can’t be right. So they ran the test again. Every time, after multiple consecutive rejected attempts, the transaction would inexplicably go through. The threshold wasn’t always 50, but the general pattern was consistent — keep trying and eventually it’ll work. Clearly, this had to be a bug in the code, but a deep-dive into the guts of the application turned up nothing. The application security group got American Express on the phone to see if they had any insight on this odd behavior. The answer? They didn’t concede the failure was on their end, despite log data showing the successful authorization codes.

My gut instinct would be that the application requesting the transactions wasn’t failing securely (e.g. network connection to AmEx timed out, so just approve the transaction). But that explanation wouldn’t account for authorization codes coming back.

So what in the world is going on here? Why would the system behave this way? Is it by design? I can’t think of a single legitimate use case for failing open like this. If this is actually a design decision by the credit card companies, I have no doubt that someone in our audience knows the rest of the story.

Jake | October 30, 2008 4:02 pm

a-hem:

“There’s no possible way /anyone/ (human) would /ever/ try inputting the same code 50 times, so we must have a machine in an infinite loop or something. We can make it shut up if we just give it an approval and move on. It’s clogging our tubes!”

Wim Remes | October 30, 2008 5:15 pm

Hey Chris,

nice write-up and pretty scary … I can see some nasty side-effects to this behaviour. However, in the article you mention some examples to illustrate fail-secure. The doors aren’t a good example, in Disaster Recovery the primary rule is “people first”. A failing card system can never lock people inside a room because there are sensitive documents in there, hence doors tend to fail-safe instead of fail-secure. There’s gray areas between the black and white but doors are a sensitive matter in the security equation and in most cases they fail “safe” instead of “secure”.

me | October 30, 2008 5:24 pm

When you buy a metro ticket in New York with your credit card you need to enter your 5 digit postal code as some kind of security measure. I still have an Austrian credit card and of course the system rejects the 4 digit postal code to which the card is registered. If I however enter the wrong code enough times (4 times or so), the system suddenly accepts the card.

Andy | October 30, 2008 9:15 pm

I’ll try and dig it up if I can.

Chris Eng | October 31, 2008 12:38 am

@Wim: Good point on the door locks. Most systems I’ve encountered though, you can always exit, you only need the card key to enter. In some large buildings you have to press a button on the wall to get out, but not swipe a card. Either way, if egress is card-controlled, then clearly the “people first” rule makes sense.

@me: Interesting. The plot thickens!

Nate | October 31, 2008 11:06 am

Chris, I’m not sure where your friend was submitting the cards for verification. On a retailer website? To a card processor? There are numerous parties in the chain, and it’s unclear where your friend was accessing the system. You need more details.

Chris Eng | October 31, 2008 11:35 am

@Nate: They are the merchant, so I’d guess they were submitting the approval request to a card processor. I don’t know for sure though, it was a brief convo. The comment by “me” does seem to suggest it’s a design issue though, perhaps with a lower threshold for a card-present transaction such as the Metrocard system.

Larry Seltzer | November 2, 2008 9:38 am

You say the Amex logs showed the successful attempts, but did they show the repeated failures? Perhaps that’s all they saw in which case the fault is probably on the merchant end or some intermediary.

None | November 4, 2008 9:31 am

Sounds much like the Android ToS. If the system fails to communicate, it assumes it is authorized.

3.7 Optimistic Delivery. To the extent that total unpaid Market Products are not material to total Market sales, the Market may choose to deliver the Developer’s Products to users without prior payment authorization when the Market Payment Processor cannot communicate with the user’s payment provider. If payment authorization fails, Market Payment Processor will send an email to the user requesting a different billing instrument. If the user never pays, Developer will not get paid for the previously delivered Products and Payment Processor will send a message to Developer in its settlement account that download has occurred but the user was not charged due to authorization failure.

Christopher Gilbert | November 6, 2008 8:58 am

Was the expired credit card account actually billed, or did the requested transaction succeed in response only?

Curiosity compels me to to ask if the credit card service specifies a request limit on this sort of repeated failure. I could imagine that a TOS would require that an application throttle itself for such repeated failures; if the application ignored these rules, sending an “authorized” response without carrying out the transaction may be a quick way to entice the owner to fix the code.

Brian Utterback | November 6, 2008 10:13 am

Recall that originally, the credit card system didn’t use authorizations at all. The credit card was checked manually against a list of canceled cards, and then it was assumed authorized, at least for all transactions below some threshold. Above that amount, the authorization was a manual phonecall.

Given that there are thousands of legitimate transactions for each fraudulent one, and there is a high cost in good will to needlessly denying a transaction, is it any wonder that a system failure would result in “Fail open”?

Please Post Your Comments & Reviews

Your email address will not be published. Required fields are marked *

*

RSS feed for comments on this post