First, wow this is both incredible and crazy! Both the China-side hacks and your side's anti-hack. Mind. Blown.
Second, would have it been cheaper to manufacture somewhere more trustworthy (another country?) instead of spending all this time/money on your anti-hack systems?
This wasn't our device. There was a big, reputable company behind the device. We were ordering a number of those and they would be shipped to us directly from China.
Also, we were basically locked in due to the magnitude of investment in the software we have developed for the device.
Fortunately this only lasted for few months until it was dealt with. It was quite new back then (a decade ago) and it was a surprise for everybody I guess.
> Second, would have it been cheaper to manufacture somewhere more trustworthy (another country?) instead of spending all this time/money on your anti-hack systems?
I'd like to know this too. Has the West completely lost the ability to mass produce microchips at even a reasonable cost for financial applications?
America has fabs, both old and leading edge, but ask industry giants like Gemalto to even bother to manufacture chips anywhere outside of Taiwan, assemble the final product outside of China.
They will never do that, because they look for the cheapest solution.
The bigger the company, the less it cares about things other than cost. This is why Mediatek and Broadcom can usurp the market of network SoCs, while making products with atrociously bad support. I personally dealt with both, and say that they wholely match their popular culture image.
I don't know how it is with USA, but for Russia, the military doesn't care that their chips had frequency measured in kilohertz, and had sizes measured in square sentimetres, for as long as they get them made inside the country.
Gemalto manufactures and assembles many, many products in Europe. The entire European security cluster won't use anything else. I know because I played a part in smart card development there. And all of that was from Europe. Wafers, chips, holograms, mag stripes...
Not exactly, PCB assembly is super cheap everywhere thanks to propagation of chipshooters, what makes the cost go up is logistics - what do you do after you populate the board for your part? Ship it across the world, or to another factory behind the corner?
That largely depends on the size of your board and the total number you want to ship. As soon as you reach full truck loads or full container loads that additional shipping cost is marginal on a board level.
You are right in a certain way. What increases with distance is lead time and as result inventory. Both of which are off set by volume. Cross-border adds customs issues. Complexity is not that much of a problem nowadays, maybe it never was. You are right that all of this oblyakes sense with large numbers, small scale production is better done locally as a rule of thumb.
> for Russia, the military doesn't care that their chips had frequency measured in kilohertz, and had sizes measured in square sentimetres
Well, if your chips are bigger and slower, you will need more chips and mounts/packaging to place them. If you need more chips then the weight of missile/plane/tank will be increased and available space decreased.
So at the end, the 'uncaring' military will receive a weapon which is worse than competition.
I think people should know how stark the differences for assembly in the US are vs outside of it. Something that costs, at low prototype volumes mind you, $20 in China for a dozen boards or so, would run hundreds of dollars in the US and still take the same amount of time. As it scales up, the ratio might improve, but these aren't like 10%-20% differences.
Nationalistic flamewar is not welcome here; neither are political or ideological flamewar. We ban accounts that use HN primarily for those things. Please read https://news.ycombinator.com/newsguidelines.html and follow the rules when posting here.
at first I had the same thought. but i have to question how securely the same manufacturing could be done in a US plant.
the US employee base has its fair share of desperate, ethically challenged individuals. and plenty of incentives to make a quick buck could be offered here too. idk.
OTOH, given US law enforcement's low efficiency what are the chances of being caught? and what if it's merely corporate espionage?
finally, the US is an open society with strong personal freedom guarantees builtin. what if the perpetrator has ties to a foreign country and simply leaves the US after they've installed the vulnerabilities?
You are underestimating the FUN of playing anti-anti-^N-hacks. I have had the privilege to be paid to so anti-anti-^N-hacking on a firewall thingy in the past and it was a challenge and a joy!
Honestly if it weren't programming I feel like this is a movie-worthy story. To me it sounds so thrilling like a spy movie plot but am I just imagining that or was it actually this crazy / cool / integral to way bigger moving parts / things like I'm assuming? Regardless kudos. Your story definitely started my day on a happy note, thanks!
We had MasterCard end-to-end test auditor on site. This is the first time ever you get to do a transaction with real transaction system with real credit card.
Due to requirements we opted to have the only large meeting room to have outside our secure zone. This created an issue as we had no network access from there and in the end we decided to use slow GPRS terminal for the test.
The end-to-end test starts with offline transactions which by their very nature are quite fast (it is negotiated between terminal and card).
But then we went to online transaction and it finished instantly too.
The auditor, bewildered, proclaimed the test failed as he assumed it was incorrectly processed offline instead of going online. But then I pointed out to the printout to show ARQC (basically says it was certified online).
Now, the real discussion started. The terminal was very slow taking quite few seconds to establish GPRS and then even more for the SSL handshake so the auditor said it was not possible to make it work.
How it worked was that I have completely gutted OpenSSL and had entire cryptographic state stored locally (safely, using internal HSM) so the SSL session could be optimistically re-established without another handshake even after TCP connection was closed. The first message the terminal sends is already encrypted transaction message, there is no SSL handshake. I wrote an application to terminate the connection in our data center so that it stored the states of each connection in the database. The entire handshake was only done if the first message could not be decrypted successfully.
The operating system was single-threaded with no multitasking of any kind. This meant that all applications on this device did their operations sequentially. Send network message, print something, display something, etc.
I wrote a cooperative multitasking functionality into the application (using coroutines) so that it could work on multiple tasks at the same time (like talking to network and printing).
I then have segregated all data on the printouts so that it can start printing without having to already have response from network. Hopefully if everything went right, the response would come before it even came to that place on the printout effectively looking as if it was done in zero time.
FWIW, the described technique (or something roughly equivalent) is now standardized as 0-rtt early data in TLS 1.3. (you still need 1-rtt for TCP, unless you can combine this with tcp fast open, or run TLs over UDP)
But am I wrong to have my hackles raised by a) the roll-your-own security nature of this, b) the reliance on a single developer's single stack implementation as what guarantees the integrity of the system? It seems like there are a lot of assumptions baked in.
I, too, would love to see a more detailed write-up--if there's a big idea here (almost a unikernel thought), it deserves to be shared and tried by fire.
When I worked in telecom (a while ago) the manufacturing was shifted from China to Thailand/Other SE Asia due to this. The Thai companies weren't as efficient, but were much more open and honest when problems would arise., plus they didn't blatantly steal tech
See, the article showed that even largest companies are not completely immune to the problem. This was decade ago and payment card industry, not exactly national security matters.
The PLA can lean on factory managers very effectively but they're not going to be interested in small time stuff like credit card numbers. The sort of sophisticated criminal gang doing something like this will have fewer coercive tools at its disposal and I'd imagine would target lower level employees with bribes.
All big and security-responsible companies issue their employees special phones and laptops when they go on business trips to countries like China or Russia and these are quarantined immediately after they return. They get wiped, X-rayed, disassembled and checked, including any accessory (chargers, mice, etc.).
The more critical the field, the more you have to treat those devices as untrusted before attaching them to your trusted zone.
> They get wiped, X-rayed, disassembled and checked, including any accessory (chargers, mice, etc.).
Given how sophisticated these attacks can be, I'd think they'd issue disposable equipment to be destroyed on return, like a cheap netbook or something. I don't see how you could trust an individual viewing a simple X-ray scan to detect some extra microchip the size of a signal conditioning coupler.
Procedures change, as attacks get more sophisticated the next step could be disposable devices. But an attack like the one described in this article won't be mitigated by having a disposable device. On the other hand having your laptop "hijacked" while on a business trip will most likely involve some extra PCB or components that are a little more obvious that something that's "built in".
Then again many companies or public institutions would find it hard to justify shredding each week maybe tens of laptops and phones that still have to be good enough to work on. Basically they still have to be a "standard issue" device with your company's software stack, config, etc.
I'm sure someone can find a good compromise between security and wastefulness.
The obvious solution would be to earmark some number of dedicated "china laptops" that will never be trusted with anything important, and get reused everytime someone has to visit China. If they get backdoored... who cares? They can only spy on things that were getting spyed on anyway due to being in China.
Does the security team have any motivation to reduce waste? Unless you're finding hacks in your devices already I see no reason to think they aren't just making you jump through hoops because it's funny.
The post-trip inspection is not so that the device can be reused, it's so you can (try to) find out if it was compromised. A $3000 laptop is not a significant cost compared to the airfare, hotel bills, etc.
But it's useful to know when/if you're being targeted.
One of the companies named in thr Bloomberg article does. They just deatroy your laptop if it was in the hands of customs without your supervision for any length. US customs explicitly included, which is kind of wierd if you ask me.
That’s perfectly expected. It’s not a stretch of the imagination to think border checks are abused for industrial espionage. If it gives your country a major advantge nothing is off limits these days.
My employer - doing lots of interesting engineering in among others the marine and aerospace sectors - certainly do that; we're based in Europe.
Basically, any device brought to the US and a number of other countries are issued for one-time use; if they leave our custody for even an instant, they are to be scrapped.
> All big and security-responsible companies issue their employees special phones and laptops when they go on business trips to countries like China or Russia
From what I understand, Boeing does this when employees visit France.
They have had problems with men in nice suits going through laptops stored hotel safes.
Sure, we were discussing someone saying their devices came with extra PCBs inside (it's a bit hard to follow but scroll up to the original comment, currently first on the page).
> We would be getting products from China with added boards to beam credit card information.
>> Trying to guess the contents of a box that you cannot open sounds a bit like madness.
>>> Use X-ray? or whatever can penetrate the exterior shell
2 different types of attacks, 2 different types of responses.
Yes, irrespective of country where its manufactured, if there are compliance requirements around an un-openable box, then some process becomes required.
But I think the GP's question is: "Whether it would be cheaper" - in the sense whether such an expensive QA process could have been averted by having a more trustworthy partner. One whom you're not on a race hack after hack.
The point is that if the devices are sensitive with compliance requirements then you must be able to verify them irrespective of who you hired to manufacture them.
You cannot just trust the word of a contractor on this because it's your ass on the line.
The point is that the process was to assure the device wasn't tampered AFTER shipped from manufacturer. Nobody thought it could already have been modified so early in the process. This is the eternal cat and mouse game. When I started in IT in 90s it was assumed that company network was quite safe and you didn't always need passwords, maybe for critical resources only.
I would think that, logically, and as illustrated, "the device wasn't tampered AFTER shipped from manufacturer" means after YOU have shipped it to customers. The anti-tampering system is to prevent modifications in the field.
The manufacturer shipped the device to us from China. We were already customer. The device would already have been locked. We would customize it some more (injecting cryptographic keys, application, placing our labels on the device) and then send them to merchants. The merchants were never customers, they would get it on loan from us. This was the only way to do it as the device could not be re-used with other acquirer so it only functioned as long as the merchant had valid merchant account with us.
Second, would have it been cheaper to manufacture somewhere more trustworthy (another country?) instead of spending all this time/money on your anti-hack systems?