Isn't the whitelist downpowering pandasai a lot?

[mspronesti]

The purpose of this thread is discussing what are the advantages and disadvantages of a fixed whitelist.

Motivation

Since this feature was integrated into pandasai, a couple of threads cropped up on the discussions/issues sections asking for additional libs in the whitelist, e.g. json+io+base64, scipy, datetime. Needless to say, one can think of several scenarios in which other libs would be needed so just enlarging the whitelist doesn't sound like a solution to me.

Discussion

• What is the reason why we chose to have a whitelist of libs in first place ?
• Can we design a way to allow any lib ?

[gventuri]

Hi @mspronesti, the reasons why we added the whitelist option is primerely 2:

• security: we don't want to risk prompt injection attacks or hallucinations to run malicious code of any kind
• scope: PandasAI is meant to be a tool to deal with data in a conversational way, so ideally we don't expect it to generate any kind of code and execute it. Given the fact that the libraries used for data analysis/data science are pretty much the same, in my opinion it's more a game of whitelisting them and then this issues shouldn't occur anymore

One thing that we could do is letting the user pass its own custom whitelisted libraries in a way that if they experience any kind of error, they can then easily have them imported.

What do you think?

[gventuri]

What I mean is something different:

• we whitelist by default the most well known libraries for data analysis.
• we still let the user pass additional libraries if for any reason the library they need to use has not been whitelisted

The error will let them know which library is missing and the workaround to have the code running. Plus, given we have introduced the cache, no additional call will be done to the LLM.

Let me know your thoughts :)

[mspronesti]

I agree on the cache part, I didn't know about this feature. I guess it's brand new as well :)
As regards the rest of the message, I think I had got it in first place: even if the whitelist covers the most common deps, there might be cases in which the generated code requires 2-3 extra libs and one has to manually pass them and the process becomes iterative because the execution will fail on the first disallowed import.
I understand this is no difficult task on the user part - assuming we add a whitelist param - but I find it not much coherent with the idea that pandasai was supposed to be a "plug and play" lib also usable by non-devs¹, doesn't it ?
Also, if we add a webUI, this solution seems less practicable, in my opinion.
If it is the case, I guess that we might only enrich that list hoping it covers all the scenarios :)

Footnotes

1. assumption based on some previous discussions/issues on this repository. ↩

[mspronesti]

@gventuri This issue seems (to me) to reinforce the point that controlling the dependencies is something not really doable.

[gventuri]

@mspronesti merged an update that should improve the way we handle external libraries. Let's see if it works better now, and how likely are other errors like this to incur.

As of that issue in particular, I guess we should rather figure out a way to pass these extra libraries as additional whitelisted libraries.

Let's monitor and see what happens in the coming weeks. If we need to rollback tho, we need to find another way to enforce the security, as we don't want to expose the user using PandasAI to prompt injections or similar threats.

[gventuri]

@mspronesti it seems after the merge there aren't so many issues with whitelisted packages anymore. What do you think?