from veeesix@lemmy.ca to selfhosted@lemmy.world on 07 Feb 21:52
https://lemmy.ca/post/60059065
cross-posted from: beehaw.org/post/24650125
Because nothing says “fun” quite like having to restore a RAID that just saw 140TB fail.
Western Digital this week outlined its near-term and mid-term plans to increase hard drive capacities to around 60TB and beyond with optimizations that significantly increase HDD performance for the AI and cloud era. In addition, the company outlined its longer-term vision for hard disk drives’ evolution that includes a new laser technology for heat-assisted magnetic recording (HAMR), new platters with higher areal density, and HDD assemblies with up to 14 platters. As a result, WD will be able to offer drives beyond 140 TB in the 2030s.
Western Digital plans to volume produce its inaugural commercial hard drives featuring HAMR technology next year, with capacities rising from 40TB (CMR) or 44TB (SMR) in late 2026, with production ramping in 2027. These drives will use the company’s proven 11-platter platform with high-density media as well as HAMR heads with edge-emitting lasers that heat iron-platinum alloy (FePt) on top of platters to its Curie temperature — the point at which its magnetic properties change — and reducing its magnetic coercivity before writing data.
#selfhosted
threaded - newest
Um thanks but tell us about 2026?
Shrimp platters.
Whoops, sorry, the oceans are hostile to life now. No more shrimp platters. Try again next time.
Can somebody do anything with a normal consumer in mind these days? 😭
Not until somebody shuts off the investor money faucet for AI. Then they’ll come crawling back — although inevitably not until after they go whining to all the world’s governments about wanting a bailout.
But hey, look at the bright side. We’ve already had the cryptocurrency mining boom and bust, and “AI” boom and soon to be bust. There’s still time for some idiot to invent the next tech scam fad which will conveniently require a shitload of hardware for no recognizably useful purpose.
And don’t forget the part where, whether they get a bailout or not, they’ll still have to double the prices of everything to make up for all the money they lost on that stupid AI bubble exploding in their face (which all of us are somehow to blame for, obviously, which is why we have to pay them back for it)
Ahem… Whining? Wanting? Try instructing. They own the governments so they will just tell them to do it, and it will be done.
Does data take up less room when it’s being used by AI?
No, and it’s by design.
You’re gonna lease a tablet and use cloud-based storage services and like it.
The dystopia is here.
Back to the 70s and early 80s…
Yeah, adding all the surveillance technology developed in the last 40 years, so you dont dare to take your eyes out of the display, for example.
140 TB is a whole heck of a lot of movies and TV shows
It’s about the storage I have in my server right now - using 15 drives ☠️
It’s about half of mine, with about 30 drives. Whatcha running?
I’m running a TrueNAS build which has just grown in time. Started off at 5x8TB drives, then added 5x16TB drives and just last week added another 5x26TB drives (that was costly ☠️). It’s all running in a very cheap case using an old threadripper machine I had (2950x), which thankfully supports ECC (128GB purchased years ago before the sillyness).
Normal consumers can install jellyfin. At some point they’ll make downloading a crime, they wouldn’t hurt people to have a decent collection of stuff ready for that day.
That fuck you mean? You can use these drives for any purpose you want.
Well, that’s a target market right now. Intel GPUs are doing better than expected, I think, thanks to all the big corporations abandoning “normal consumers”.
This would be a bitch to have to rebuild in a raid array. At some point a drive can get TOO big. And this is looking to cross that line.
I was thinking the same. I would hate to toast a 140 TB drive. I think I’d just sit right down and cry. I’ll stick with my 10 TB drives.
This is not meant for human beings. A creature that needs over 140 TB of storage in a single device can definitely afford to run them in some distributed redundancy scheme with hot swaps and just shred failed units. We know they’re not worried about being wasteful.
Rebuild time is the big problem with this in a RAID Array. The interface is too slow and you risk losing more drives in the array before the rebuild completes.
Realistically, is that a factor for a Microsoft-sized company, though? I’d be shocked if they only had a single layer of redundancy. Whatever they store is probably replicated between high-availability hosts and datacenters several times, to the point where losing an entire RAID array (or whatever media redundancy scheme they use) is just a small inconvenience.
True, but that’s going to really be pushing your network links just to recover. Realistically, something like ZFS or a RAID-6 with extra hot spares would help reduce the risks, but it’s still a non trivial amount of time. Not to mention the impact to normal usage during that time period.
Network? Nah, the bottleneck is always going to be the drive itself. Storage networks might pass absurd numbers of Gbps, but ideally you’d be resilvering from a drive on the same backplane, and SAS-4 tops out at 24 Gbps, but there’s no way you’re going to hit that write speed on a single drive. The fastest retail drives don’t do more than ~2 Gbps. Even the Seagate Mach.2 only does around twice that due to having two head actuators.
100%. But the post i was responding to was talking about recovering a failed array from other copies, not locally.
Fairly significant factor when building really large systems. If we do the math, there ends up being some relationships between
Basically, for a given risk acceptance and total system size there is usually a sweet spot for disk sizes.
Say you want 16TB of usable space, and you want to be able to lose 2 drives from your array (fairly common requirement in small systems), then these are some options:
The more drives you have, the better recovery speed you get and the less usable space you lose to replication. You also get more usable performance with more drives. Additionally, smaller drives are usually cheaper per TB (down to a limit).
This means that 140TB drives become interesting if you are building large storage systems (probably at least a few PB), with low performance requirements (archives), but there we already have tape robots dominating.
The other interesting use case is huge systems, large number of petabytes, up into exabytes. More modern schemes for redundancy and caching mitigate some of the issues described above, but they are usually onlu relevant when building really large systems.
tl;dr: arrays of 6-8 drives at 4-12TB is probably the sweet spot for most data hoarders.
I’d imagine they are using ceph or similar.
You have disk level protection for servers. Server level protection for racks. Rack level protection for locations. Location level protection for datacenters. Probably datacenter level protections for geographic regions.
It’s fucking wild when you get to that scale.
This is for like, Smaug but if he hoarded classic anime and the entirety of Steam or something. Lol
Yeah I’m running 16s and that’s pushing it imo
It doesn’t really matter, the current limitations are not so much data density at rest, but getting the data in and out at a useful speed. We breached the capacity barrier long ago with disk arrays.
SATA will no longer be improved, we now need u.2 designs for data transport that are designed for storage. This exists, but needs to filter down through industrial application to get to us plebs.
640K ought to be enough for anybody.
I don’t get how a single person would have that much data. I fit my whole life from the first shot I took on a digital camera in 2001… Onto a 4TB drive.
…and even then, two thirds of it is just pirated movies.
Amateur 😀
But seriously I probably have close to 100 TB of music, TV shows, movies, books, audiobooks, pictures, 3d models, magazines, etc.
I need a home for my orphaned podman containers /s
I think this is better targeted to small and medium businesses.
if you run this as a NAS you could easily have all your budd s obsesses files in one place without needing complex networking.
This ONLY works at an insane scale. This will never hit the consumer market.
Also what current consumer level application could require of storage 140TB. That would be some advanced level data hoarding or smth.
@Korkki @just_another_person I see 4k HDR blue ray movie rips these days on the order of 50GB (edit: eg, Eddington.2025.MULTi.VFF.2160p.DV.HDR.BluRay.REMUX.HEVC-[BATGirl]: 77.73G).
Which is too rich for my blood (I'm still watching on 1080p screens over here), but for someone with the right kind of home theater.. that's only ~280 movies on a 14TB drive. Lots of movie collections, even in the olden days of physical VHS and DVDs, span 1,000+ movies.
14TB or 140TB? The latter is what’s being talked about, so that’s more like 2800 movies. Which more than covers that 1000+ movie criteria.
@Zorque I'm saying that 14TB will only fit 280 (or more likely, less) of those ultra-hq movies, so 140TB (or, in the lead up to that, 100TB, since they're talking about 5+ years or more before they even get close to 140TB) is reasonable for a 1,000-2,000 movie collection. Obviously I'm being loose with numbers, but the fact that one single movie can consume almost 80GB.. well, you can start to understand consumer demand for 100+TB drives.
The failure rate is going to be absolute INSANE as well.
Holy fuck can you imagine how long it would take to re-stripe a failed drive in a z2 array 😭
Not a clue. Care to eli5?
When you are running a server just to store files (a NAS) you generally set it up so multiple physical hard disks are joined together into an array so if one fails, none of the data is lost. You can replace a failed drive by taking it out and putting in a new working drive and then the system has to copy all of the data over from the other drives. This process can take many hours to run even with the 10-20 TB drive you get today, so doing the same thing with 140 TB drive would take days.
@SmoothLiquidation @Telorand They also claim up to 8x speed improvements with HAMR. Obviously that remains to be seen, but if they could roughly match capacity improvements, that would keep restriping in the same ballpark.
My Z2 had à drive failure recently, with 4To drives. Took me almost 3 days to re-silver the array 😅. fortunately had a hot spare setup, so it started as soon as it failed, but now a second drive is showing signs of failing soon, so I had to pay the AI tax (168€) to get one asap (arriving Monday), as well as a second one, cheaper (around 120€), but which won’t arrive until the end of April.
Acronyms, initialisms, abbreviations, contractions, and other phrases which expand to something larger, that I’ve seen in this thread:
4 acronyms in this thread; the most compressed thread commented on today has 14 acronyms.
[Thread #72 for this comm, first seen 8th Feb 2026, 00:30] [FAQ] [Full list] [Contact] [Source code]
I wonder why current consumer HDD’s don’t have NVME connectors on them. Like I know speeding up the bus isn’t going to make the spinning rust access faster but the cache ram would probably benefit from not being capped at 550MBps
I just hope smaller sized drives become cheaper. The word “hope” is doing a lot of heavy lifting here.
Ten years from now…
Amazon search: “hard drive”
Result: 4TB $198
BARGAIN!
I think ten years from now you’ll be hard pressed to find anyone even wasting their time on something so small.
so you say, but people still collect “antique” hardware.
Well, retro etc. but I wouldn’t consider this to be that. There’s no inherent value of a run-of-the-mill drive with merely lower storage capacity. And certainly not worth a premium.
it’s not antique yet. i still have my 5.25" diskettes with quest for glory 2 on them and they’re almost antique. i think the usb drive that reads them still works. give them another couple years.
do HDDs work better than SSDs in space? because of the cosmic rays and shit? or something about intermittent power? no, really, this is a real problem that they could be already solving, one i know jack shit about.
Doesn’t this sound awfully similar to the Mini disc technology? The discs were only writable when heated by a laser. They were pretty impressive for the time… But not very fast. Especially when writing.
Probably still with only 1 year warranty…
And if it breaks at 10 months and they take another 2 to send your replacement back, well, they no longer need to send one that actually works this time either
Okay cool, cool, so does this mean ridiculous data centers will use these things, and then can I get another 4TB RED for my NAS so I can fit my whole life on a mirrored total of 8TB without paying 8x what it’s worth, please?
Thaaaaanks…
Is there a Lemmy community for trading surplus hardware yet?
I have a pile of HDDs and servers that I no longer use. I’ve transitioned almost all mine to 20tb+. I might have 8 or 10 4tb REDs laying around. They’re old, probably have thousands of power on hours in the smart data though.
Are you in Europe by any chance? :)
Ah, no. Sorry. Midwest, USA.
No apologies needed, I’m not even OP :) it was just a long shot :D
8TB? That’s my ideal RAM configuration lol. ;-)
I’d put this in a mirror configuration tbh.
Question: Are failures due to issues on a specific platter? Meaning, could a ZRAID theoretically use specific platters as a way to replicate data and not require 140TB of resilvering on a failure?
IIRC, HDDs have some reserved sectors in case some go bad. But in practice, once you start having faulty sectors it’s usually a sign that the drive is dying and you should replace it ASAP.
I think if you know drive topology you can technically create partitions on platter level, but I don’t really see a reason why you’d do it. If the drive is dying you need to resilver the entire drive’s content to a new disk anyway.
@Fmstrat @veeesix Since there's two very diffrent questions there.. The first, "where do the failures happen?": anywhere. It could be the controller dying (in which case the platters themselves are fine if you replace the board, but otherwise the whole thing is toast). It could be the head breaking. It could be issues with a specific platter. It could be something that affects _all_ the platters (like dust getting inside the sealed area). So basically, it very much depends.
@Fmstrat @veeesix The second, could you do raid across specific platters - yes and no. The drive firmware specifically hides the details of the underlying platter layout. But if you targeted a specific model, you could probably hack something together that would do raid across the platters. But given the answer to the first question, why would you?
Great answers, thank you.
Okay. I want total honesty here. How many of you could actually fill that thing up?
With useful stuff? Never. With random bullshit I think might be useful some day if only I find the time? Easy
… or be able to backup it?
I remember Mac OS X having an issue with its mail app awhile back that would create massive log files continuously that would keep generating until they filled the entire drive. You would have to boot to a recovery partition or such because the OS partition wouldn’t have enough room to expand/boot and remove them and fix the issue.
Imagine having 130 terabytes of invisible log files
Archive.org, Anna’s archive, Jan 6 footage, Epstein files, there’s plenty to back up.
Whats the point when the prices for 4-8TB disks are stable the last 5 years? (I think that they are getting higher even…)
The point is that 8TB are too small, and not enough for my anime.
The point is the need for more and more data storage is never going to stop.
Does the increased density mean that the speed also goes up? It would be nice if a 7200 RPM drive could finally saturate SATA3 bandwidth.