Luckily, the majority of customer funds were kept in cold storage. None-the-less the exchange chose to halt all services for the past three days to avoid operating as a fractional reserve. The purpose of the email was to reveal this bug and the subsequent event caused by its exploitation with customers.
Also read: Ripple Labs Enters US Banking
Justcoin has been forced to impose a partial hold on 23% of all XRP held on the exchange. Deposits are disabled and will hit accounts after deposits are re-opened. Trading is still active for remaining XRP that is not on-hold, but expect delays on withdrawals as funds move from cold storage.
At first glance, this sounds similar to the transaction malleability farce used by legendary failed Bitcoin Exchange, Mt. Gox. Justcoin claims Ripple has known about the bug for two months and remained silent. The hack involves exploiting a seldom implemented part of a Ripple transaction to send a partial payment and receive credit for the full amount.
tfPartialPayment is a feature of Ripple transactions that allow you to attempt to make a payment even if your wallet does not have the full amount. Until late September, it was listed in the official Ripple wiki for application in returning funds, arbitration, and currency conversion and made no mention that the amount sent may differ from the amount received.
However, as of October 1st the new Ripple wiki clearly states “If the tfPartialPayment flag is set, this is not the amount actually received.” How convenient.
The exploit that was used to hack Justcoin was due to how tfPartialPayment fit inside the transaction data structure. Ripple and Stellar were communicating the amount sent in transactions using an Amount field. In a normal transaction, Amount will be the total funds transacted. If the tfPartialPayment flag is set the transaction Amount field still displays the same value. However, a separate field is added to the transaction, DeliveredAmount – the actual amount of funds transacted.
The consequence, allegedly, is that hackers sent deposit transactions for large amounts, e.g. 100,000, to Justcoin. They set the tfPartialPayment flag to something like .0001. The transaction would be perfectly valid, and any client unaware of this behavior in the protocol would likely not be checking for the DeliveredAmount field – since it was never documented until a week ago. The transaction Amount field says “100,000” but the DeliveredAmount is only .0001. The hacker gets credit for 100,000 but only deposits .0001. Then they make a small withdrawal, check the balance on the hotwallet address and drain as much as they can.
On October 9th, the day after the Justcoin hack, Github history shows that the client received a patch from Ripple Labs developer Vahe Hovhannisyan. The patch added additional code to check for the DeliveredAmount field.
This code only affects clients. What about server side? That update was on August 5th. Over two months ago.
Justcoin’s claim that Ripple has known about the bug for two months comes from a post by Medium writer Andreas Brekken. In it, he talks about the findings that led him to reporting it to Stellar. Stellar responded by correcting the bug and communicating with their users. tfPartialPayments is no longer supported by Stellar. He also shows evidence that RippleTrade.com (Ripple’s very own website!) was affected by the same bug until October 9th, the day of the client patch.
At the time of this article, nothing could be found by Ripple alerting users of the potential exploit or even acknowledging the flaw existed. Anyone visiting their documentation for the first time would think that tfPartialPayment “is not a bug, it’s a feature.”
It’s fair to say that cryptocurrency is young, and you should only risk what you are willing to lose… But a part of the responsibility of being a leader and experiencing growing pains is owning them and making it right. Through their actions, Justcoin seems to understand that.
When Gox failed, a finger was pointed at known and documented functionality and used as a scapegoat for failed leadership and poor programming. Ripple has a very big gap to explain. The time between the initial fix to gatewayd, the changes in their documentation, the changes in their client code, is large.
Someone should ask them if they think it would have been possible for someone viewing their server-side fix to become inspired to check for the same bug in the client-side code. Or maybe just for the folks who hadn’t made the most recent updates.
This event is another example of the power of a decentralized network over centralized authorities. Even with cutting edge technology, centralization stacks the interests of the provider against the interests of the payer. Ninja patching your currency to fix a poorly documented feature and then updating the documentation without issuing some statement looks like an attempt to escape accountability.
Silence is golden, except when it’s other people’s gold on the line.
Update: Stellar users are reporting tracking the location of the stolen Justcoin coin’s here
Update 10/12/14: As noted below in the comments JoelKatz shows that he made the change in a section of the comments making mention of partial payments on the 28th of January:
Also there is a community response on the XRPtalk forum thread to Justcoin’s email saying tfPartialPayment is a feature they did not understand.
Update 10/13/2014: A discussion of the medium article on the Ripple Forums.
Posters continue to make claims that only Justcoin was affected. Still no explanation as to how this was a well documented feature yet Ripple Labs missed it themselves until the day after the Justcoin hack.
From Ripple Labs:
We’d like to clarify and correct misinformation about the partial payments feature and Justcoin’s recent issues. First and foremost in this case, there is no vulnerability in the Ripple network, nor was it hacked. Justcoin simply miscredited a deposit.
In the event funds are sent to the wrong person, are sent unsolicited, or need to be returned for some other reason, the person returning the funds shouldn’t pay the currency conversion cost. Importantly, the partial payments feature allows the person returning the funds to send less than the amount specified. Without this feature, returning funds would be difficult, possibly requiring many attempts to guess the market rate or making many small payments. Partial payments were first documented in July 2012 (before Ripple went live), and are currently documented in the Ripple Wiki and Developer Portal.
Justcoin did not implement partial payments correctly. The exchange falsely credited a non-KYC’d user for a deposit, and then allowed the user to illegitimately withdraw the funds from its hot wallet. For every transaction, an exchange needs to ensure the total of user balances plus the new deposit matches the balance of its Ripple cold and hot wallets. If these balances don’t match, the exchange should stop processing the transaction.
Ripple Labs has engaged Justcoin in ongoing discourse about its lack of risk and compliance controls. As demonstrated by this incident, a non-KYC’d user can steal with little fear of being identified and owning the consequences.
As soon as we learned of Justcoin’s incident, we emailed gateways and exchanges integrated with Ripple to directly warn them of the possibility of incorrectly implementing partial payments.
The Ripple protocol is open-source and free for anyone to use. Ripple Labs makes its best effort to provide documentation of features, but cannot be responsible for anyone incorrectly implementing them. Gateways and exchanges are responsible for implementing risk and compliance controls, including KYC and prudent hot wallet limits, and reporting illegal activity to law enforcement.
Update 10/29/14: Justcoin’s technical view on what happened: https://medium.com/@abrkn/partial-payments-ripple-stellar-vulnerability-in-the-wild-29aaefd8a7ac
Images from Shutterstock.
Last modified: October 29, 2014 21:24 UTC