Add README content #4
Conversation
|
|
||
| lakeFS Catalog is using lakeFS HadoopFileSystem under the hood to interact with lakeFS. | ||
| In addition, for better performance we configure the S3A FS to interact directly with the underlying storage: |
README.md
Outdated
| conf.set("spark.hadoop.fs.lakefs.secret.key", "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY") | ||
| conf.set("spark.hadoop.fs.lakefs.endpoint", "http://localhost:8000/api/v1") |
There was a problem hiding this comment.
Why not keeping the same pattern <your-lakefs-endpoint>?
README.md
Outdated
| conf.set("spark.sql.catalog.lakefs", "org.apache.iceberg.spark.SparkCatalog") | ||
| conf.set("spark.sql.catalog.lakefs.catalog-impl", "io.lakefs.iceberg.LakeFSCatalog") | ||
| conf.set("spark.sql.catalog.lakefs.warehouse", "lakefs://") | ||
| ``` |
There was a problem hiding this comment.
Maybe add a note explaining why we need all of these? the last 2 don't seem trivial
There was a problem hiding this comment.
Added some more comments - though the configuration should be familiar for iceberg users
README.md
Outdated
| ### Create a table | ||
|
|
||
| Let's create a table called `table1` under `main` branch and namespace `name.space.` |
There was a problem hiding this comment.
| Let's create a table called `table1` under `main` branch and namespace `name.space.` | |
| Let's create a table called `table1` under `main` branch and namespace `name.space` |
|
|
||
| ```sql | ||
| CREATE TABLE lakefs.myrepo.main.name.space.table1 (id int, data string); |
There was a problem hiding this comment.
id and string are too confusing examples. It took me a minute to understand it's not something I need to fetch from somewhere. Can we do something stupid like animal and color?
| | 2 | data2| | ||
| +----+------+ | ||
| ``` |
There was a problem hiding this comment.
Do you think we should discuss merge? It will come up soon, shouldn't we explain about what supported and what ain't?
There was a problem hiding this comment.
Will add information about merge but no example as we don't want to go into the realms of lakefs / clients.
No description provided.