what are differences between CORTX and Hadoop?

[yanqingfu]

Hadoop also has a great horizontal scalable ability.
What are the differences between CORTX and Hadoop?

[justinzw]

Hi @yanqingfu thanks for asking this question and starting this discussion. I really appreciate it.

I assume you are talking about HDFS - Hadoop File System.

As far as I understand one of the known limitations of HDFS is small file sizes and small datasets. It's tuned for large datasets and large files. In fact a typical file in HDFS should be gigabyte or terabyte in size.
For CORTX we are tuning performance for all file sizes.

I'm not as familiar with HDFS though and it's use cases but I know there are some pros and cons of using it.
Here's an interesting article I found: https://medium.com/@bharvi.vyas123/6-major-hadoop-limitations-with-their-solutions-1cae1d3936e1

But I'd love to hear thoughts from others who are familiar with HDFS.

[johnbent]

I think there are a few differences. One, HDFS is not necessarily designed and optimized for mass capacity systems but is more designed and optimized to be good for map-reduce workloads so there may be various design differences due to this. Additionally, CORTX is S3 and HDFS is file. Here are a few good references talking about the difference between S3 and HDFS:

• https://www.quora.com/How-exactly-S3-is-different-from-HDFS-and-how-S3-data-will-be-processed-in-Hadoop-cluster
• https://databricks.com/blog/2017/05/31/top-5-reasons-for-choosing-s3-over-hdfs.html