Skip to content

Commit

Permalink
Update READMEs
Browse files Browse the repository at this point in the history
  • Loading branch information
fxpineau committed Mar 11, 2024
1 parent f03b518 commit 1958772
Show file tree
Hide file tree
Showing 2 changed files with 53 additions and 27 deletions.
36 changes: 22 additions & 14 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,25 +2,34 @@

# `votable` or `VOTLibRust`

Library to read/write [VOTables](https://www.ivoa.net/documents/VOTable/)
in Rust and to convert them in JSON, YAML, TOML and back to XML.
Library to build, read and write [VOTables](https://www.ivoa.net/documents/VOTable/)
in Rust and to convert them efficiently back and forth in JSON, YAML, TOML, XML-TABLEDATA, XML-BINARY and XML-BINARY2
while preserving all elements (except comments) and their order.

[![](https://img.shields.io/crates/v/votable.svg)](https://crates.io/crates/votable)
[![](https://img.shields.io/crates/d/votable.svg)](https://crates.io/crates/votable)
[![API Documentation on docs.rs](https://docs.rs/votable/badge.svg)](https://docs.rs/votable/)
[![BenchLib](https://github.com/cds-astro/cds-votable-rust/actions/workflows/libtest.yml/badge.svg)](https://github.com/cds-astro/cds-moc-rust/actions/workflows/libtest.yml)


VOT Lib Rust is used in:
VOT Lib Rust is used in:
* [VOTCli](https://github.com/cds-astro/cds-votable-rust/tree/main/crates/cli)
convert VOTables from the command line;
* [VOTWasm](https://github.com/cds-astro/cds-votable-rust/tree/main/crates/wasm)
read/write and convert VOTables in Web Browsers.
* [Aladin Lite V3](https://github.com/cds-astro/aladin-lite)

## Status

This library is in an early stage of development.
We are (reasonably) open to changes in the various format, e.g.:
The code contains all main functionalities:
* supports BINARY, BINARY2 formats
* supports CDATA in rows, StAX mode for streaming
* supports MIVOT block parsing

But it is far from been as clean and documented as I would like.
If you want to use this crate in a Rust project, you should probably have a look
at the VOTCli code.

We are still (reasonably) open to changes, e.g.:
* we could flag attributes with a '@' prefix
* we could use upper case elements tag names
* we could remove the 's' suffix in elements arrays
Expand Down Expand Up @@ -56,10 +65,10 @@ VOTable is an XML based format. Why other formats?

## Design choices and problems

The default provided implementation converting from JSON/YAML/TOML **does not
Although the default provided implementation converting from JSON/YAML/TOML **does not
focus on performances** since we do not use the VOTable FIELDs information but
deserialize each table field in the first succeeding *VOTableValue*
(see `votable::impls::Schema.serialize_seed`).
(see `votable::impls::Schema.serialize_seed`), performances seems to be good nevertheless.

VOT Lib resort heavily on [serde](https://serde.rs/).

Expand Down Expand Up @@ -95,9 +104,8 @@ names (all in camel case).

### WARNINGS

* TOML does not supports `null` (we so far convert `null` values by an empty string).
* The default provided implementation loads all data in memory, so it is not
adapted for large files!
* TOML does not support `null` (we so far convert `null` values by an empty string).
* Conversions from/to TOML/JSON/YAML requires all data to be loaded in memory, it is not adapted for large files.

## Other way to convert from VOTable to JSON

Expand Down Expand Up @@ -561,16 +569,16 @@ votable:
## To-do list
* [ ] Support `CDATA`
* [X] Support `CDATA`
+ [X] Support in `Info`, `Description`, `Link`, `PARAMRef` and `FIELDRef`
+ [ ] Support `CDATA` in `<TD></TD>`
+ [X] Support `CDATA` in `<TD></TD>`
* [ ] Fill the doc for the Rust library (but I so far do not know people interested in such a lib since Rust is not very used in the astronomy community so far, so...)
* [X] Add a check method ensuring that user input VOTableValue (using the API to build a VOTable)
matches the table schema (or automatically converting in the right VOTableValue)
* [ ] Add much more tests!
* [X] Add possibility to convert to/from `TABLEDATA`, `BINARY`, `BINARY2`
+ but still to implement streaming mode in CLI
* [ ] Implements `toCSV` (but not `fromCSV`) in CLI
* [X] Implements `toCSV` (but not `fromCSV`) in CLI
* [ ] Replace `quick_error` by `anyhow` and `thiserror`.
* [ ] Bump `quick_xml` version and:
+ [ ] Implement parsing without copy from `&[u8]` (when full data in men, e.g. in wasm, or for memmapped files)
Expand Down
44 changes: 31 additions & 13 deletions crates/cli/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,10 +21,12 @@ Please, provide us with VOTable examples!

## To-Do list

* [ ] Support `CDATA` in `TD` tags
* [ ] Use the iterator to implement streaming transformations between DATATABLE/BINARY/BINARY2.
* [ ] Also implement streaming conversion to CSV

* [X] Support `CDATA` in `TD` tags
* [X] Use the iterator to implement streaming transformations between DATATABLE/BINARY/BINARY2.
* [X] Also implement streaming conversion to CSV
* [ ] Add commands to modify a VOTable metadata
* [ ] Add commands to select/compute columns and filter rows?

## Install

### From pypi for python users
Expand Down Expand Up @@ -119,20 +121,36 @@ Options:

## Example

### XML/JSON/TOML/YAML convertion

```bash
> time vot xml xml -i VII.vot > xml.1.vot
real 0m0,009s
user 0m0,001s
sys 0m0,009s
# In memory conversion of a VOTable from XML-TABLEDATA to JSON
vot convert --in my_votable.xml --out my_votable.json --out-fmt json
```

> time vot xml toml --pretty -i VII.vot | vot toml json | vot json xml > xml.2.vot
real 0m0,022s
user 0m0,018s
sys 0m0,012s
### Streaming conversion XML-TD, XML-BIN, CML-BIN2 and CSV

> diff xml.1.vot xml.2.vot
```bash
# Streaming conversion of a VOTable from XML-TABLEDATA to XML-BINARY2
vot sconvert --in my_votable.xml --out my_votable.xml.b64 --out-fmt xml-bin
# Streaming conversion from XML to CSV, in parallel, of a single large table
vot sconvert --in my_votable.xml --out my_votable.csv --out-fmt csv --parallel 6
```

### Get metadata

```bash
# Get the structure of a VOTable with virtual identifier for each element
vot get --in my_votable.xml struct --line-width 120
# Get the structure of alarge VOTable till DATA is reached
vot get --in my_votable.xml --early-stop struct --line-width 120
# Get only the colum names of a large table, with a non-ascii separator
vot get --in my_votable.xml --early-stop colnames --separator ''
# Get a field metadata array with selected info
vot get -in my_votable.xml fields-array index,name,datatype,arraysize,width,precision,unit,ucd,description --separator ,
```


## Log messages

You can adapt the `level` of log messages using
Expand Down

0 comments on commit 1958772

Please sign in to comment.