Reliable data storage and access is a big challenge in AI applications and IBM ESS is a service you need to solve the problem. If you are building an AI service and you already use the IBM ecosystem for AI compute requirements, ESS can serve as an excellent software based storage solution. Integration with other platforms such as EMR/Databricks is still a challenge and should be improved.
The team in charge of implementing Scality has to be full stack in order to guarantee the correct functioning of the entire system. However, you have to think very carefully about the balance between servers and disks, perhaps adopting smaller fully populated servers instead of large semi-populated servers, which would mean that over time our disk updates will not have a fully useful life.
As an organization, it took us a while to understand the shift from a traditional black box SAN to software-defined storage, but now we are much more certain of what this means. The time invested and the resources were not very high, thanks on the one hand to the technical support and on the other to the coherence and good development of the platform.
IBM ESS is optimized for AI and Big Data usecases while S3 is a general purpose storage solutions. EMR and Databricks have lakehouse/data warehousing solutions for distributed computing but are more optimized for just the big data pipelining solutions and not essentially for AI usecases, especially for inference, when you need to load model artifacts really quickly.
Due to the nature of our business we require extensive encryption and availability for sensitive customer data. The erasure encoding that Scality provides gives us the assurance that documents are rest are never in a state of being downloaded or available to a casual data thief. This is something that can be found with other vendors but at a fraction of the same cost. Having this kind of performance, availability and redundancy at the cost that Scality provides has made a large difference to our organization.
Keeping sensitive customer data secure is a must for our organization and Scality has great features to make this happen.
We replaced a single SAN with a Scality ring and found performance to improve as we store more and more customer data.
Being able to lose various portions of our Scality ring and allow it to continue to service customers while maintaining high performance has been key to our business.