Add option to specify system tables and its columns

[hfhbd]

Each database has unique system tables containing system information. At the moment sql-psi does not know about these tables so referrencing to these tables fails with No table found with name.

Sample: DB2 requires a FROM clause, so you have to use the SYSIBM.SYSDUMMY1 for simple selects: SELECT TIME_STAMP FROM SYSIBM.SYSDUMMY1.
Oracle provides a similar dual table.

Same use case for system catalog with PostgreSQL: https://www.postgresql.org/docs/current/catalogs.html

[hfhbd]

Hm, only ignoring the non-available table name does not work when you actually want to use a table and its columns. So I think, we need an option to predefine tables in each dialect and the one of the sql standard: #436. And I am unsure how this could be done in psi.

• We could add a frontend IR tree to finally get rid of psi in sqldelight :D Cleaner, but mostly a rewrite and needs support in sqldelight too

• Alternative: Some kind of table (type) resolver in the dialect used by the parser (not sqldelight) which returns psi elements (lightfiles?) => Try to use QueryResult! Not implemented, because creating PSI Elements manually is hard...

• Ooooor: Add an option to specify a folder that contains real sq files with the SQL DDL. Advantages easy to handle (instead of rewriting the DDL in Kotlin psi classes by just copying it from the DB) and no manual psi element creation... The tables just need to be ignored by sqldelight. Disadvantage: What about ansi information schema? => Store it in sqldelight, like ANSITypeResolver

[hfhbd]

Hmmm, I published a release locally using this change and I added the information schema and the pg_catalog for postgresql. The pg_catalog is actually the "standard library", which contains every postgresql function.

On the one hand, it is very very nice because now all functions are resolvable (in theory sql-psi needs support to understand a function) and don't need a custom hard-coded case in type resolver and a new release, but on the other hand, this file is big.

Really big... 70.000 lines. And it would need more postgresql/SQL features, eg domain, GRANT, OWNER. And some definitions are not even valid sql, although I got the definition from postgres itself. So it even needs manual changes.

=> the original use case was missing system tables: SYSIBM.SYSDUMMY1 and DUAL. If we want to support predefined tables and maybe the ansi information schema, I would focus on tables and views. We even don't need the whole DDL, but only the resulting columns. And parsing the whole sql file just for some table definitions sounds expensive too.

Do you have another idea, beside the other 2 alternatives? Maybe stub indexes https://plugins.jetbrains.com/docs/intellij/stub-indexes.html? I never tried these.

Maybe: What about removing all psi classes from QueryResult? Then we can write the tables in code without psi, we do not need another resource file to include, and we could almost reuse existing logic.

[hfhbd]

Decided to add an option for SQL statements provided as a String which will be parsed, but not added to the actual sources, like dependencies. So you are able to use these tables but sqldelight won't include them in your schema/database, because they are already there.

Sorry for the inner monolog :D

[AlecKazakova]

We also have the same problem to solve for the SQLite JSON extensions since they enable querying from a json_each() function. Though for that one that might be a dialect override that enables using a function invocation in a FROM clause.

I think this is good. It's straightforward and easy to contribute to. As long as it doesn't impact existing codegen I'm cool with it.

[AlecKazakova]

I think a lot of this becomes cleaner if this is a List<File> and you just do the file reads inside of the constructor

[hfhbd]

I tried it first too, but it did not work, because the SQL file is inside the jar of the dialect, and you can't reference a resource inside a jar using a java.io.File (you can read the contents with its URL through).

[AlecKazakova]

got it, that makes sense. Does this even need to be passed into sql-psi? Could we just say it has to follow some specific format system_tables.predefined or whatever and then we can just check if its in the resources directory?

Don't feel ridiculously strongly on this either way

[hfhbd]

How do you resolve the references without sql-psi? And psi needs files..., wait, how does IntelliJ do indexing dependency jars? 🤔
Update: The same way I did, I guess. With VirtualLightFiles, which takes a string 🤦🏻‍♂️ so yeah, you are right. We could put the an sql file into the dialect jar.

[hfhbd]

But this function still needs strings, so the signature is correct. Even if you want to get the sql files in the resources, it have to be implemented in the dialect, because it is in their jar file, not in sql-psi one's.

[AlecKazakova]

it should all be loaded on to the same class loader, so you can reference resources from another jar

classloaders are insane so I am definitely not 100% confident in that

[hfhbd]

Hm, I am about to adopt your comments but honestly, I don't get, what you/we want to change. We can't use File. While this is the most idiomatic type, this won't work with resource files. So we need to pass the content of the file. In the beginning, I just used to use List<String> and named the VirtualLightFile by its index: 1.predefined with a hardcoded suffix. But this makes debugging way harder because the order of the files loaded at runtime by the jar is not deterministic.
So the current idea/implementation is to load the predefined files by the dialect, in the best case using the file name from the resources. sql-psi does not care about loading nor naming etc.

[hfhbd]

If we want some kind of autoloading some ressource files we should either provide an extension or simple move this function to sqldelight dialect api, for example. I would not add it to the sql environment because you could use sql-psi without sqldelight, like I do in my frontend.