Reading data from Hadoop with kerberos authentication

[ExtinctionEvent]

Hi all,

I have the following use case, where I need to load data (parquet) to Kedro from a hdfs source that is authenticated by Kerberos. I was hoping to use the pandas.ParquetDataSet to do the load.

How might I configure my catalog.yml and credentials.yml files to achieve this?

Any pointers welcome.

Thanks

[datajoely]

Hello - I've spoken to some people that have solved this before and it seems that they went with extending a DataSet and using the with krbcontext.krbContext(principal=credentials["your_credentials"]) context manager in the constructor.

So the short answer is that our existing datasets don't support this out of the box, but can be extended to work this way. I'm not familiar with Kerberos, but some searching suggested that the KRB5_CONFIG environment variable may be a way of doing this without customising the datasets.