API for the rwp package CLI subcommand

[mickael-menu]

I'm opening a discussion to talk about a new subcommand for rwp to package Readium Web Publications.

It started with an initial proposal from @atomotic to add a command to package audiobooks (See the Slack thread):

Generate an empty readium manifest inside an audio directory 
$ rwp audiobook init -source {audio_directory}

Package an audiobook with an already existent manifest
$ rwp audiobook package -source {audio_directory} -output {output}.audiobook

Generate a new audiobook
$ rwp audiobook new -source {audio_directory} \
    -output {output}.audiobook \
    -id {identifier - autogen uuid/ulid}
    -cover {cover}.jpg \
    -title {title} \
    -date {date} \
    -creator {creator}

I like the idea of offering a two-steps generation to allow modifications of the generated manifest, while also having a simpler command that combines the two other. I think we can go further:

• There's an opportunity to make a more general command that would support all Readium Web Publication packages (Divina, LCPDF, etc.)
• The metadata flags (e.g. -cover) could be used with the init subcommand as well to pre-fill the generated manifest.

Subcommands

rwp package init

Generate a Readium Web Publication Manifest for the given publication (directory or an archive (ZIP, RAR, etc.)).

Examples

# Generate an audiobook manifest in the current directory.
$ cd my_audiobook
$ rwp package init --profile audiobook

# Generate an EPUB manifest from the given publication.
# The profile is auto-detected from the EPUB and the EPUB is exploded in the current directory.
$ rwp package init books/mobydick.epub
$ ls manifest.json

Parameters and flags

• (optional) publication A path to the publication (directory, or an archive (ZIP, RAR, etc.)).
  • If not provided, the tool assume that the current working directory is inside an exploded publication.
• (maybe optional) -profile epub|audiobook|divina|pdf Profile and rules for the generated manifest (see https://readium.org/webpub-manifest/profiles/).
  • This could be auto-detected thanks to the mediatype API.
• (optional) -output Customize the output directory where the generated package will be created.
  • If not provided, it will be the current directory.
• Metadata flags (see below).

What does it do?

1. The profile is auto-detected using the publication argument.
2. If the publication is an archive, it is unarchived to output.
3. If it is a directory, its content is copied to output (unless publication = output).
4. A new Manifest object is created using the profile for template.
   • If profile is epub, the manifest is parsed from the OPF using the epub.Parser APIs.
   • If profile is pdf, the manifest is parsed from the PDF file using the pdf.Parser APIs.
   • If profile is audiobook or divina, a basic Manifest object is created, and all the audio/image files in output are added to the readingOrder in alphabetical order.
5. If metadata flags are provided (see section below), then the Manifest object is modified.
6. The Manifest object is serialized to output/manifest.json.

rwp package build

Package a Readium Web Publication from a directory containing a manifest.

Examples

# Generate an audiobook package in the current directory.
$ cd my_audiobook
$ rwp package build
$ ls my_audiobook.audiobook

# Generate an EPUB package from the given directory.
$ rwp package build books/mobydick
$ ls mobydick.epub

Parameters and flags

• (optional) publication A path to the publication directory.
  • If not provided, the tool assume that the current working directory is the publication.
• (optional) -output Customize the output package file.
  • If not provided, it will be:
    • Next to the publication directory.
      • (Or inside the current working directory? Might be more UNIX-like)
    • Or inside the publication directory, if it is the current directory.

What does it do?

1. Figure out the profile from the publication/manifest.json file.
2. Archive the publication to output using the right file extension for the profile.

Additionally, we could have some verification steps to check that all the files are properly set in the reading order, or that it is valid according to the profile, etc.

rwp package new

Generate and package a new Readium Web Publication for the given publication.

This is basically package init + package build together.

Examples

# Generate an audiobook package in the current directory.
$ cd my_audiobook
$ rwp package new --profile audiobook
$ ls my_audiobook.audiobook

# Generate an EPUB package from the given publication.
# The profile is auto-detected from the EPUB
$ rwp package new books/mobydick.epub
$ ls mobydick.epub

Parameters and flags

• (optional) publication A path to the publication (directory, or an archive (ZIP, RAR, etc.)).
  • If not provided, the tool assume that the current working directory is inside an exploded publication.
• (maybe optional) -profile epub|audiobook|divina|pdf Profile and rules for the generated manifest (see https://readium.org/webpub-manifest/profiles/).
  • This could be auto-detected thanks to the mediatype API.
• (optional) -output Customize the output directory where the generated package will be created.
  • If not provided, it will be the current directory.
• Metadata flags (see below).

What does it do?

1. Figure out the profile from the publication/manifest.json file.
2. Archive the publication to output using the right file extension for the profile.

Additionally, we could have some verification steps to check that all the files are properly set in the reading order, or that it is valid according to the profile, etc.

Metadata flags

Both package init and package new can take additional metadata flags to preset (or override) some of the defaults in the manifest.json. To be more consistent, we should use the same names as in the Metadata manifest object.

• -title Set the metadata.title property.
• -identifier Set the metadata.identifier property. No autogeneration.
• -cover Provide an image cover which will be added to links with a cover relation.
• -author Add an author contributor. This flag can be added multiple times.

@atomotic If we follow-up on some of these ideas, I don't expect you to do everything in your audiobook PR. For example you could only support the audiobook profile for these commands. Of course if you want to contribute more, it's always welcome!

[atomotic]

Good, I like the idea of the generic --profile argument.
To proceed, I could make an initial PR with a complete audiobook package, then we can add the cli later or adjust it.

The basic API should be something like these:

func NewAudiobook(in, out string, metadata *Metadata, identifier string, metafromid3 bool) Audiobook
func (audiobook *Audiobook) MetadatafromID3() (*Metadata, error)
func (audiobook *Audiobook) AddTrack(track string, pos int) error
func (audiobook *Audiobook) AddCover(cover string) error
func (audiobook *Audiobook) WriteManifest() error
func (audiobook *Audiobook) Package() error

I add to the discussion:

• tests. If I want to add some fixtures, or mp3, is good a git submodule?
• mp3 re-enconding. I can investigate further on ffmpeg bindings for Go. Or otherwise we can take the path of using a static ffmpeg build and distribute it in the packaging (or docker), and then execute it with os/exec (not very nice, I know)

[mickael-menu]

Yes for test fixtures a Git submodule is good. We want to avoid having a large repository and Git LFS has some (pricing) issues with GH actions.

There's one we already use in the repo for EPUB files (https://github.com/readium/readium-test-files). I guess we could reuse it for MP3s.

For ffmpeg, I would be in favor of using Go bindings, but having them enabled with a build flag. This way we can still build a small version of rwp without ffmpeg or CGO.

[atomotic]

I look at some ffmpeg binding in Go. The most clear API (also most stars) seems to me u2takey/ffmpeg-go but all this are actually wrappers on ffmpeg binaries, that needs to be available on OS.

A minimal example to convert bitrates:

package main

import (
	"log"

	ffmpeg "github.com/u2takey/ffmpeg-go"
)

func main() {
	err := ffmpeg.Input("natureofacrime_00_conrad_64kb.mp3").
		Output("natureofacrime_00_conrad_96kb.mp3", ffmpeg.KwArgs{"b:a": "96k"}).
		OverWriteOutput().ErrorToStdOut().Run()

	if err != nil {
		log.Fatal(err)
	}
}

If ffmpeg not available

➜  go-ffmpeg-test git:(main) ✗ go run .
2023/03/08 09:59:49 compiled command: ffmpeg -i natureofacrime_00_conrad_64kb.mp3 -b:a 96k natureofacrime_00_conrad_96kb.mp3 -y
2023/03/08 09:59:49 exec: "ffmpeg": executable file not found in $PATH
exit status 1

else

go run .
2023/03/08 09:58:59 compiled command: ffmpeg -i natureofacrime_00_conrad_64kb.mp3 -b:a 96k natureofacrime_00_conrad_96kb.mp3 -y```
...

[chocolatkey]

Based on a look at the repository you linked, there is no CGO dependency, so this is actually a good solution. If the user doesn't have ffmpeg installed, then they won't be able to run the tasks, but it won't affect the build size. We could have a version of the docker image that bundles ffmpeg so that it's simpler for users to get started

[chocolatkey]

Relared to packaging, I would suggest either as part of this command (if source is an EPUB) or as a separate enhance command, the ability to add a webpub manifest.json to the root of an EPUB file.

This enhances the EPUB ZIP, turning it into a file that is simultaneously an EPUB and a WebPub. The advantage for future consumers of the EPUB is faster parsing, because you can use the manifest.json directly to stream the publication.

I have seen multiple ebook stores in the wild post-processing EPUBs to speed up/avoid having to parse using container.xml -> OPF, so why not offer this functionality?

Described here as well: https://readium.org/webpub-manifest/packaging.html#6-hybrid-epub-3--rpf-packages

[mickael-menu]

Yeah I agree this could be a useful feature. With my proposal above it is already supported with this:

$ rwp package new books/mobydick.epub --title "Moby-Dick"