221 points | by userbinator13 hours ago
The thing with SD cards is that there are wide variances in quality, even among the top name-brand manufacturers. You can get the normal crappy cards that everyone buys which are likely to last not much more than 2-3 years under very occasional use all the way up to industrial cards with 10 year warranties and reliability matching the best enterprise SSDs. And everything in between.
The point is: If you buy the cheapest thing on Amazon, you're going to get exactly what you paid for. For my part, I have been using "High Endurance" cards in my Pis for several years as normal OS and data disks. They are not actually much more expensive than "normal" cards. I have not had any of them fail yet.
Memory cards and ssds are famously obtuse like this. There are the branded versions (custom made for your Switch, for your Xbox,etc) which are exactly the same as their cheaper counterparts. Sandisk itself recently started a "creator" line of cards and ssds which are, again, exactly the same but more expensive. Samsung had Evo Plus which was different than Evo+ but same as Evo Select. Go figure!
Sometimes looking at specs helps but usually not - I.e. The "up to" speed marks are famously irrelevant for real time usage, and brand name companies will offer any number of cards with same specs and tags (u2/v3/class10/mark5/whatever) at varying price points. And then there's the WD Red saga where drives branded "NAS" were completely inappropriate for NAS usage.
I ran a photography business a while back and always bought extreme pro because I just couldn't risk it, but honestly, I felt like a bit of a sucker. It's hard to know when you're actually buying real quality and endurance increase.
And nearly always the cheap cards seem to fail due to firmware shortcomings rather than hardware faults. Ie. the built in wear levelling and error recovery is bad.
Considering firmware is effectively free once written, the fact the cheap cards still don't have good firmware leads me to believe it might be deliberate market segmentation. Ie. "we need to make the cheap cards glitch and fail often, or nobody would buy our expensive cards".
If so, it really raises moral questions about how many peoples wedding pictures have been gobbled simply so some executive can get his christmas bonus...
There’s certainly a very visible increase in price and decrease in capacity but it’s certainly interesting when you get sd cards with a proper datasheet vs. customer level devices
https://www.digikey.com/en/products/filter/memory-cards/501?...
Unfortunately SLC costs disproportionately more compared to MLC, TLC, and now QLC -- the actual die cost is only 3x for SLC compared to TLC of the same capacity, but the prices are closer to 10x or more.
The single biggest thing you can do to improve the reliability of your embedded system is to use eMMC’s built-in hardware partitioning.
- Each hardware partition is a separate block device. A firmware update cannot corrupt the device by overwriting a partition table.
- There are two small boot partitions, and they can be made permanently read-only after programming, thus preventing corruption of your bootloader. You can also use the other one read-write for your uboot environment.
- You can easily have two OS partitions for A/B firmware updates. In addition to mounting them readonly, temporary write protection can be enabled on a per-partition basis and disabled when needed for fw updates.
- If you can’t afford the capacity hit from pSLC, I believe it can be enabled on a per-partition basis. (Don’t quote me on this, it could be wrong).
All these settings can be configured with either mmc-utils or u-boot. In volumes, programming houses can take care of this for you. (You’ll have to list all the registers out very specifically in an Excel spreadsheet)
The downside is that calculating all the correct register values is not a simple process, and you’ll have to spend a bit of time reading the eMMC spec.
I also have some really small USB drives from around 2 decades ago, of course SLC flash (the only type of NAND flash at the time), and can confirm that their data is still intact. They've gone through probably a few dozen full-drive writes, while SLC flash back then was rated for 10 years retention after 100K cycles.
They are indeed more expensive but nothing that a programmer salary would notice even if you bought 10 of them. I am wondering whether I should buy another pair and use it as a metadata mirrored vdev on my media NAS. Haven't decided yet.
Also datashits clearly say it's SLC and not TLC: https://static6.arrow.com/aropdfconversion/55056b1a2a966f599...
Could also just be software updates or other things causing this but there should be some component of decreasing performance caused by wear on flash storage.
A lot of - especially older or mid/low range - phones have cheap eMMC storage which is signifcantly worse at wear leveling than the higher end UFS storage.
Which is shocking really - the phones should switch the eMMC to RAW flash mode (ie. no wear levelling), and then write an actually-smart wear levelling algorithm that runs in the OS.
The OS has far better info for wear levelling anyway - it has more visibility into read-write patterns, it has more RAM to store more state, it can cron background scrubs and reorganisation to idle periods, it can be patched if a bug is found which only manifests after years, etc.
Unfortunately, as far as I'm aware, most eMMC's can't be put into any kind of RAW mode anyway.
There's usually trends to look for in regards to that third factor. The lower the # of bits per cell, the higher probability the voltage level is still working the right range. Which is why so much flash is still SLC or pSLC capable. Usually this is more industrial. Then you have entirely different technologies altogether. NVRAM/FRAM/MRAM are various terms for extremely high (or infinite) read/write technologies while still being non-volatile (keeps its data with power off). I don't know how much of a drop in replacement those are. I think LTT had one of those on a flash drive a while back https://youtu.be/oJ5fFph0AEM, but it's so low capacity it'll probably be useless.
It may be possible to hack something up with a MR5A16A. It's a whole 4 MB but has unlimited endurance and over 20 years of endurance. It looks like it has more of an SRAM interface than NAND, but should be capable of saturating a USB high speed link. The drive would likely cost $75? TBH if there was a market it may be a fun project.
If you sacrifice some endurance you can go up to 1Gb per device which might be interesting. But the cost scales.
Accelerated stability testing is fraught with potential issues, and any output is intentionally conservative.
An issue with estimating lifespan on new products is that they'll expose them to more extreme conditions, but those more extreme conditions may trigger (exponentially faster) higher order reactions that are relative non-issues at regular conditions.
Then you have things like activation energy requirements for a reaction that just might not be met at regular conditions, but happen at higher temperatures.
And an IC is quite the soup of molecules in varying combinations unlike a straightforward solution.
Samsung still screwed up with the planar TLC flash used in the infamous 840 EVO SSD, which had a real-world retention measured in months. Their "fix" was to issue a firmware update that continuously rewrites data in the background, but of course this has no effect if the drive isn't always powered.
https://forum.acelab.eu.com/viewtopic.php?t=8735
https://goughlui.com/2024/07/20/salvage-tested-an-elderly-fo...
https://www.digikey.com/en/product-highlight/a/atp/advancedm...
I'm cynically guessing it's down to a relatively simple workaround in the SD card firmware that fixes this very common corruption on power loss while writing. Or maybe that combined with a 0.1 cent capacitor.
What's worse is that these things are connected to the internet (with VNC installed!), and they don't do updates...
The system is awesome, but I VERY quickly moved it to a separate VLAN.
https://www.bunniestudios.com/blog/on-microsd-problems/
https://www.bunniestudios.com/blog/2013/on-hacking-microsd-c...
Just plug in four microsd cards and the hardware takes care of the mirroring and rebuilding when dead card removed+replaced
Be sure to use cards from different batches or even makes so they don't all fail at the same time.
Most pro DSLRs have two SD card slots and will also do it (at the firmware level) if you toggle the appropriate option (usually the options are RAID 1 or a unionfs-like "use both SDs" option).
I RAIDed a bunch of cheap USB 2.0 flash drives on a hub with MDRAID as a learning tool back in the day.
It was horrendously unreliable. USB wasn’t a good choice for storage back then, and I’m convinced the hub had issues. This would work much better now.
I did, however, get to watch the blinkin lights, learn how to recover from failures, and discover quite a few gotchas.
1) All microSD cards have a high failure rate.
2) Cheap cards sourced directly from China are usually fake and will fail a format test. I tried multiple times and could never find a reliable source.
3) MS Bitlocker is a security measure that also seems to serve as a reliability test. Cards with physical memory reliability issues seem more prone to write failures with Bitlocker. It's better to know about this up front.
4) If your data is really important, always make a backup copy of any card.
5) Physically, microSD cards are fairly durable and even water resistant thanks to being encased in epoxy.
I have one on my watch strap and take it wherever I go, even the shower and pool. Just make sure it is dry before plugging it in.
You can buy genuinely reliable microSD cards, if you really need them. Delkin, Innodisk, Swissbit and others make industrial microSD cards with SLC flash, ECC, SMART and power loss protection. Capacities are small and you'll pay a steep premium, but they legitimately have endurance and MTBFs that compete with conventional SSDs.
You can get "industrial" versions of these cards that are intended to be used as BOM items for products. This means you get a stable supply and they won't change the part number out from under you (which happens even with reputable suppliers when you buy as a consumer). When they're about to EOL the part, you get a heads-up multiple months in advance and a "last time buy" opportunity.
In my experience, these aren't "better" performance-wise. You just "know" what you're getting, have an opportunity to qualify it, and you won't get surprise changes.
Of course, the price is jacked up as a result.
Generally speaking, it's usually better to make the customer responsible for the SD card if you have to use one at all.
Big problem in action cam related circles, where lots of people have broken recordings due to fake cards.
Yes, but there is a significant difference --- I can always return it to Amazon postage free if the capacity is fake and it won't format properly. Presumably, Amazon extracts the appropriate weight in flesh from the seller.
The way retail typically works in the USA, the vendor assumes all the risk --- any problems get unrolled backwards. You can sell a fake product on Amazon --- but it will likely cost you --- not Amazon (who controls the money) nor the end consumer if he is astute.
Basically, Amazon acts like an escrow service.
Amazon and other retailers have played this game for a very long time and they aren't in it to lose money.
In my experience, the thing most commonly faked with memory cards is the capacity --- for example, an 8Gb card altered to appear 64Gb, until it is formatted.
If you want extra assurance, buy Amazon branded cards. In this case, Amazon assumes almost all the risk so there is little incentive for fakery.
A lot of cheap cards from China can't even pass this basic test.
On Amazon? I highly doubt it. The threshold for “too many returns” is probably rarely hit. MicroSD’s are super cheap - people don’t return things that are cheap because they don’t consider it worth their time. Then these companies go and pay people for good reviews and just keep flipping inconsistent/garbage products.
If punishment were a real thing that companies had to deal with then this wouldn’t continue to be a problem. It’s been this way for many, many years. Amazon has no equal competitor, there’s nowhere else for folks to go and people rarely look at the specific vendor they’re buying from. As far as they’re concerned it’s all Amazon.
I’ve bought more off target.com than Amazon and I read at least one other commenter here who does the same. Walmart.com also has a wide range of products.
Even sd cards. Which are still going strong.
Remember, Amazon holds the cash for all sales. The cost of returns is extracted from vendor disbursements. A fake card sold for $10 may cost the vendor $20 if it gets returned.
If “the system worked” i.e. companies get appropriately punished for selling bad products, then this issue wouldn’t still be so widespread. It’s basically a feature of Amazon now. People just assume they’re going to randomly get junk. It’s baked into our expectations at this point.
I suspect the real issue with memory cards is that most people probably don't check them. The solution is --- don't be like most people.
“One vendor in particular interested me; it was literally a mom, pop and one young child sitting in a small stall of the mobile phone market, and they were busily slapping dozens of non-Kingston marked cards into Kingston retail packaging. They had no desire to sell to me, but I was persistent; this card interested me in particular because it also had the broken “D” logo but no Kingston marking.”
Oh and compute read/write times while doing so.
If the read back data doesn’t validate, or you can’t write the amount expected, or the read/write rates don’t match expected? faaaaaaake
Pretty accurate too. I personally never had an actual fake Sandisk SD card from Amazon. I bought probably 50 of them in the space of 6 months at one point. Other brands were not great.
USB flash drives though? Literally all trash. I tried at least 5 different ones before I just gave up.
No idea what the market looks like now however.
I recently bought a very expensive Sandisk Extreme UHS-II V90 card from Amazon. It passed without any issue when doing a full capacity check, but was still fake because they were using slower (150mb/s vs 300mb/s) flash.
The average user[0] would never know, because it was definitely “faster” than other cards and maxed out the UHS-I reader in my MacBook Pro. I returned it and bought from my local camera store (Henry’s) and the performance difference was very obvious.
[0]: I guess you could argue that the average user wouldn’t be buying a $200 V90 card, but I still think you could fall victim to this if you didn’t explicitly own a dedicated UHS-II reader.
You mean a simple reformat with fat? Or what’s in your opinion the best test when getting a new SD card?
Read back data and compare against the same pseudorandom data with a same fixed seed.
Or just make a random data file the size of SD card and write/read back compare, if you don't care about having to store the big file and also testing for potential local disk corruption at the same time.
It's also possible to write 64bit address of each 8 byte block to every such block, and avoid pseudorandom generator, and potentially have more insight to what happened when the block ends up mapped to unexpected location.
If you ever tried running uSD card in SDR104 mode, you'd notice that they tend to heat up way more than in lower speed interface modes. So for longetivity, I guess it's better to run them at HS mode (which is just 50MHz), or at least lower the SDR104 bus frequency from 200MHz to something lower.
Do you have better advice?
* cpr the article:
> [manufacturer] doesn’t publish any detailed specifications about the cards’ internal workings [...] I shouldn’t have to go this far in hardware reverse-engineering to just ask a simple question of what Flash SanDisk used in their high-endurance card
--
About details in the article, instead: it states that 3D NAND should be more reliable (3D has less interference than planar) - but actually I read that 3D NAND can be more fragile physically, and that the bits-per-cell value remains crucial (the less the better)...
I personally buy SD cards and USB memory sticks from bhphotovideo, whose customers (pro and enthusiast photographers), care greatly about not losing their images, so B&H watches their suppliers closely. My 2c.
64 GiB microSD still costs ~$14 locally, but the same thing can be had for $4 from China without any warranty.
They stick endless limited warranty on it and increase the price so much that it's economical for them to just replace a few cards that people will actually bother to return, and still make profit. And warranty will tell you nothing about quality.
In fact on many products you can explicitly choose to pay more for longer warranty.
In my opinion, manufacturers aren’t obligated to disclose every technical detail beyond what a typical spec sheet would cover, such as the specific type of flash used. It’s incredibly impressive that someone as talented as the author can figure it out independently, but I don’t see why there’s a need for the constant tone of frustration or an “us vs. the company” mindset throughout the article.
SSD and MicroSD/thumb drive and even HD manufacturers have a damn hide by being so secretive about their devices—of course it's never the manufactures who suffer the burden of data loss, it's the customer.
What's desperately needed are open-source manufacturers who will publish the necessary data.
This problem isn't new, sleazebag Kodak knowingly released shoddy unstable color stock in the 1950s and decades later precious family and wedding photos had faded to nothing.
Let that be a lesson, this solid-state storage shit hasn't been around long enough yet to know whether we'll be seeing a repeat of that Kodak fuckup.
Basic fundamental technical details are regularly missing: e.g., bits per cell.
Most companies are quite happily to lie, obfuscate and omit to the hilt if they can: nearly every labelling regulation is a patch over some fuckery. The relationship is often pretty adversarial, especially at retail.
Doubly so in computer memory devices, which is an industry particularly filled with flimflam and chancers.
Some write endurance and retention figures will be ok. A 1TB flash is useless if, the moment you wrote 1TB, you cannot read it anymore or it gives erroneous values.
The author needed to reverse engineer what could have been on the spec sheet...