scrubbles@poptalk.scrubbles.tech
on 26 Apr 19:38
nextcollapse
They’ve been anti-open source for a while, they clearly don’t see a profit motive without killing off their open source side. Anyone selfhosting or into open source should consider MinIO dead, and migrate. Hopefully someone forks it.
people did, and then proceeded to do nothing with it.
I don’t like minio’s moves here or the way they communicated it but they weren’t wrong when they said the community was not contributing in a significant way.
etchinghillside@reddthat.com
on 26 Apr 20:16
nextcollapse
Sadly I only recognized it as “the thing you put in your docker compose for integration tests”.
+1 to Garage being great! I used it for a personal project and it worked really well. A lot of S3 data browsing clients also support it natively or just through API compatibility too
S3 isn’t just an AWS thing anymore. It has kind of become the standard object storage protocol, and almost every cloud provider uses it aside from a few the made their own API’s (e.g. Azure Blob storage)
So enlighten me then, save me from my terrible hack that is working fine for me and tell me what it DOES have to do with. I thought S3 was a remote filesystem you can use, essentially Amazon’s proprietary version of webdav where you get a http bucket you can only access with aws proprietary tools. What’s the attraction? Clearly it seems like people love it, and I am getting dunked on for asking an honest question, which feels a bit unhealthy and unpleasant for the self-hosting community.
Am I supposed to be familiar with AWS infrastructure as a prerequisite for being here?
eager_eagle@lemmy.world
on 27 Apr 01:26
nextcollapse
ok, to start with, if you need a POSIX interface to the filesystem, you already have an SSH connection to that server, and don’t need much stability across multiple clients, SSHFS may do just fine. For a homelab, that is likely the case.
now, if you’re hosting a web server that needs data distributed across drives/nodes, data redundancy, and the usage is primarily programmatic, closer to a CDN’s or machine learning pipeline than a single user browsing files; then you want an S3-compatible solution. The S3 API makes it easier to plug it into your application, while allowing you to migrate to a different one - which I’m actually currently doing for a MinIO deployment at work.
SSHFS isn’t POSIX compliant. It doesn’t support hard links, file locking, atomic renames, full support for changing file permissions, umasks, and probably other things.
S3 is designed for being used by applications via API, for example you can easily save and retrieve files from it even with a JavaScript application. It is much more difficult to do the same with sshfs
If instead you use it mounted on a computer, S3 is worse because each time you need to list its contents that’s an API request, if you have hundreds of thousands of files then it’s thousands of API reuqests
Many cloud providers offer S3-compatible storage, so it’s a common protocol to use in applications. There are even some databases like SlateDB that fully rely on object storage for everything. Supporting more API’s is extra work (unless you’re using OpenDAL) so most people pick S3 compatible API’s because they’re the most widely supported across all cloud platforms.
ExLisper@lemmy.curiana.net
on 27 Apr 06:00
collapse
why would you need that?
So you can switch to S3 if needed? Using compatible solutions means you have choice. Choice is good.
hendrik@palaver.p3x.de
on 26 Apr 23:05
nextcollapse
Was pretty much clear since last year. At the latest in December when they switched to “maintenance mode”. And now they archived it.
Out of all these, SeaweedFS is the most scalable. Seaweed’s design is based off some of Facebook’s whitepapers about their warm storage system, and it works especially well for use cases that have a very large number of small files (like images).
I’ve honestly never understood the need for s3 buckets. WebDAV satisfies my needs. I’m sure there are some use cases that require S3, but for the life of me I can’t think of one off the top of my head right now.
Many cloud providers offer S3-compatible storage, so it’s a common protocol to use in applications. There are even some databases like SlateDB that fully rely on object storage for everything. Being able to have local S3 compatible storage is useful if you want the storage of your local machine while still doing so over a widely compatible protocol.
A quick web search shows slatedb supports WebDAV through Rust’s object_store interface, or at least it does at first glance.
WebDAV is a wonderful standard and it is compatible with all kinds of things that seem to be overlooked. S3 has turned into this monster of a thing that’s “owned” by AWS vs a nice usable RFC that anybody can implement and know if it actually changes.
object_store does indeed also support WebDAV among a variety of other protocols, Apache Druid or Apache Pinot probably would be better examples.
My only experience with WebDAV is with Nextcloud and hasn’t been that great because it has been very slow, probably should look into it sometime.
EDIT: Apparently it supports CAS, and even has a locking mechanism
I use an external S3 provider so that I only pay for the storage I use for the services I host. It’s dirt cheap, 0.00002750€ per GiB hour (excluding tax). Self-hosting something like MinIO for your app gives you the option of switching to an external provider later on, and it gives you flexibility in the location of the storage.
rako@tarte.nuage-libre.fr
on 27 Apr 10:14
nextcollapse
S3 is not made for you. It’s made for entities that need to store millions of objects, with thousands of different rules, reading/writing from hundreds of machines without coordination, and with consistent, low latency. Now that some software use that as a storage layer, having an implementation for you is useful
In my experience I’ve mostly seen it used for a local equivalent of S3 to plop in your dev environment. It’s pretty good if your prod depends on S3 and you don’t want to deal with the cost and latency of using actual S3 buckets during development.
Wispy2891@lemmy.world
on 27 Apr 05:40
nextcollapse
Of course their closed source alternative is called AIStor and it is crazy expensive because everyone now needs to pivot to AI
threaded - newest
They’ve been anti-open source for a while, they clearly don’t see a profit motive without killing off their open source side. Anyone selfhosting or into open source should consider MinIO dead, and migrate. Hopefully someone forks it.
people did, and then proceeded to do nothing with it.
I don’t like minio’s moves here or the way they communicated it but they weren’t wrong when they said the community was not contributing in a significant way.
Sadly I only recognized it as “the thing you put in your docker compose for integration tests”.
Garage seems like a viable alternative.
Garage has been great in my homelab. It’s not quite as 1:1 with S3 but it does all the basics with some really nice features.
+1 to Garage being great! I used it for a personal project and it worked really well. A lot of S3 data browsing clients also support it natively or just through API compatibility too
Shit, I am actually building a webtool and thought Minio could be a good part to be a file storage in it. What’s an good alternative?
Edit: I try “garage”
…a hard disk? you can just write data to a file
Not if you want to validate S3 compatibility for an actual future use case or, * can you imagine*, just for the fun of it.
i’ll give you the second case, but nobody should plan for putting stuff on aws with the world as it looks right now…
Genuine question, what are the alternatives not called Azure/GCP?
I use S3 with OVH at my workplace. So it’s not just aws / google.
Practically every other block storage provider offers an S3-compatible API.
S3 isn’t just an AWS thing anymore. It has kind of become the standard object storage protocol, and almost every cloud provider uses it aside from a few the made their own API’s (e.g. Azure Blob storage)
any european ones?
IONOS and ImpossibleCloud for instance are ones we use.
I think Hetzner and OVH also offer S3 buckets.
Scaleway, Exoscale, Cyso, Contabo, UpCloud, and others too
There are numerous other vendors with S3 API-compatibility.
There’s also SeaweedFS that I’ve used as an S3 compatible fileserver
S3 compatibility is nice I guess if you need S3 compatibility but also… why would you need that?
sshfs does everything I need and compatibility is almost native.
Kubernetes storage is the reason I was looking at Minio in the past.
Sshfs has way more overhead and doesn’t do remotely the same thing
SSHFS is a hack and has nothing to do with the proposal of S3 compatible backends
So enlighten me then, save me from my terrible hack that is working fine for me and tell me what it DOES have to do with. I thought S3 was a remote filesystem you can use, essentially Amazon’s proprietary version of webdav where you get a http bucket you can only access with aws proprietary tools. What’s the attraction? Clearly it seems like people love it, and I am getting dunked on for asking an honest question, which feels a bit unhealthy and unpleasant for the self-hosting community.
Am I supposed to be familiar with AWS infrastructure as a prerequisite for being here?
ok, to start with, if you need a POSIX interface to the filesystem, you already have an SSH connection to that server, and don’t need much stability across multiple clients, SSHFS may do just fine. For a homelab, that is likely the case.
now, if you’re hosting a web server that needs data distributed across drives/nodes, data redundancy, and the usage is primarily programmatic, closer to a CDN’s or machine learning pipeline than a single user browsing files; then you want an S3-compatible solution. The S3 API makes it easier to plug it into your application, while allowing you to migrate to a different one - which I’m actually currently doing for a MinIO deployment at work.
SSHFS isn’t POSIX compliant. It doesn’t support hard links, file locking, atomic renames, full support for changing file permissions, umasks, and probably other things.
S3 is designed for being used by applications via API, for example you can easily save and retrieve files from it even with a JavaScript application. It is much more difficult to do the same with sshfs
If instead you use it mounted on a computer, S3 is worse because each time you need to list its contents that’s an API request, if you have hundreds of thousands of files then it’s thousands of API reuqests
SSHFS is very unreliable. At least use NFSv4 or even SMB/CIFS.
Many cloud providers offer S3-compatible storage, so it’s a common protocol to use in applications. There are even some databases like SlateDB that fully rely on object storage for everything. Supporting more API’s is extra work (unless you’re using OpenDAL) so most people pick S3 compatible API’s because they’re the most widely supported across all cloud platforms.
So you can switch to S3 if needed? Using compatible solutions means you have choice. Choice is good.
Was pretty much clear since last year. At the latest in December when they switched to “maintenance mode”. And now they archived it.
https://blog.vonng.com/en/db/minio-is-dead/
Alternatives include Garage, SeaweedFS and RustFS.
Versity S3 Gateway is another option that’s trying to focus on simplicity. github.com/versity/versitygw
Out of all these, SeaweedFS is the most scalable. Seaweed’s design is based off some of Facebook’s whitepapers about their warm storage system, and it works especially well for use cases that have a very large number of small files (like images).
I’ve honestly never understood the need for s3 buckets. WebDAV satisfies my needs. I’m sure there are some use cases that require S3, but for the life of me I can’t think of one off the top of my head right now.
Many cloud providers offer S3-compatible storage, so it’s a common protocol to use in applications. There are even some databases like SlateDB that fully rely on object storage for everything. Being able to have local S3 compatible storage is useful if you want the storage of your local machine while still doing so over a widely compatible protocol.
A quick web search shows slatedb supports WebDAV through Rust’s object_store interface, or at least it does at first glance.
WebDAV is a wonderful standard and it is compatible with all kinds of things that seem to be overlooked. S3 has turned into this monster of a thing that’s “owned” by AWS vs a nice usable RFC that anybody can implement and know if it actually changes.
object_store does indeed also support WebDAV among a variety of other protocols, Apache Druid or Apache Pinot probably would be better examples. My only experience with WebDAV is with Nextcloud and hasn’t been that great because it has been very slow, probably should look into it sometime.
EDIT: Apparently it supports CAS, and even has a locking mechanism
I use an external S3 provider so that I only pay for the storage I use for the services I host. It’s dirt cheap, 0.00002750€ per GiB hour (excluding tax). Self-hosting something like MinIO for your app gives you the option of switching to an external provider later on, and it gives you flexibility in the location of the storage.
S3 is not made for you. It’s made for entities that need to store millions of objects, with thousands of different rules, reading/writing from hundreds of machines without coordination, and with consistent, low latency. Now that some software use that as a storage layer, having an implementation for you is useful
So, it’s mostly used for overengineering?
In my experience I’ve mostly seen it used for a local equivalent of S3 to plop in your dev environment. It’s pretty good if your prod depends on S3 and you don’t want to deal with the cost and latency of using actual S3 buckets during development.
Of course their closed source alternative is called AIStor and it is crazy expensive because everyone now needs to pivot to AI
I hope chainguard keeps maintaining their fork github.com/chainguard-forks/minio