Merge pull request #3 from ItalyPaleAle/v0.4-wip

This includes features that will be added to prvt 0.4.

- [X] Upload files directly from the UI
- [x] Themes and dark mode
- [X] Storing file date and type in index
- [X] Index file is now encoded with protocol buffers. This requires repos version 3; however, prvt can still read repos v2.
- [X] Web UI shows file date and icon
- [X] Action item in the file's dropdown menu to force a file to be downloaded
- [X] `--store flag` can be passed with the `PRVT_STORE` env var

Encryption.md

@@ -2,7 +2,7 @@
 
 prvt stores files using strong end-to-end encryption.
 
-The files are encrypted on the local machine before being sent to the cloud or to the target directory. To view the files, one would need the encryption key (ie.. the passphrase or GPG private key), as well as the encrypted files and their index.
+The files are encrypted on the local machine before being sent to the cloud or to the target directory. To view the files, one would need the encryption key (ie. the passphrase or GPG private key), as well as the encrypted files and their index.
 
 When viewing files using the web-based interface, files are downloaded and then decrypted locally by prvt's local server, before being sent to the browser in cleartext.
 
@@ -129,30 +129,16 @@ This is done to protect your privacy, by hiding the original name of the file an
 
 To map files back to the original paths, prvt uses an encrypted index file. This is the `_index` file in the repository, and it's encrypted using the same pipeline as the data files, and as such it contains the same headers too.
 
-Decrypted, the `_index` file is a JSON document that contains two keys:
+Decrypted, the `_index` file is a structured document encoded with Protocol Buffer that contains a dictionary with two main keys:
 
-- The version (`v`) of the index file. Currently, this is always `1`.
-- A list of elements (`e`) present in the repository. This is an array of objects, each containing two keys:
-    - The original path (`p`) of the file within the repo's tree (for example, `/folder/sub/file.jpeg`).
-    - The name (`n`) of the encrypted file in the repository (a UUID, stored as a string).
+- The version of the index file. The latest version, used from prvt version 0.4, is `2`.
+- A list of elements present in the repository. This is an array of objects, each containing up to four keys:
+    - The original path of the file within the repository's tree (for example, `/folder/sub/file.jpeg`).
+    - The UUID of the encrypted file in the repository, stored as binary data.
+    - The date when the file was added to the repository, stored as UNIX timestamp.
+    - The mime type of the file, as determined from its extension.
 
-For example (the document below has been pretty-printed for clarity for this example only):
-
-```json
-{
-  "v": 1,
-  "e": [
-    {
-      "p": "/photos/IMG0342.jpeg",
-      "n": "6d0acb54-195c-483c-9b25-7511af9a433f"
-    },
-    {
-      "p": "/documents/something.pdf",
-      "n": "e2175556-4e58-4116-a264-376d69ea4437"
-    }
-  ]
-}
-```
+The `_index` file uses Protocol Buffers for encoding, so it's a binary file and not human-readable. The proto file defining the data structure is saved at [`index/index.proto`](/index/index.proto).
 
 Thanks to this index, prvt can show a tree of all directories and files, and knows what encrypted document to request for each file.
 
@@ -163,7 +149,7 @@ The `_info.json` file is the only file in the repository that is not encrypted.
 This file is a JSON document containing four keys:
 
 - The name of the app (`app`) that created it. This is always `prvt`.
-- The version (`ver`) of the info file. The latest value, prvt version 0.3, is `2`.
+- The version (`ver`) of the info file. The latest value, from prvt version 0.4, is `3`.
 - The data path (`dp`), which is the name of the sub-folder where the encrypted data is stored. The default value is `data`. (This value can't be set using the prvt CLI, but it's defined here to enable backwards compatibility with repositories created by previous versions of prvt.)
 - The list of passphrases and keys (`k`).
 

README.md

@@ -203,6 +203,23 @@ prvt repo key ls --store <string> --key <string>
 
 Note: after running this command, the identifier of the passphrases in the repository might change, since they're always in a sequence.
 
+## Using environmental variables
+
+You can set pre-defined values with environmental variables to reduce repetitions.
+
+### `PRVT_STORE`
+
+Use the `PRVT_STORE` environmental variable to set a default value for the `--store <string>` flag for all commands that require it.
+
+For example:
+
+```sh
+export PRVT_STORE="local:repo"
+prvt repo init
+```
+
+The value defined with environmental variables acts as a fallback, and you can override it by explicitly set the `--store <string>` flag.
+
 # FAQ
 
 ### How does prvt encrypt my files?

cmd/add.go

@@ -85,6 +85,11 @@ You must specify a destination, which is a folder inside the repository where yo
 				return
 			}
 
+			// Require info files version 3 or higher before any operation that changes the store (which would update the index to the protobuf-based format)
+			if !requireInfoFileVersion(info, 3, flagStoreConnectionString) {
+				return
+			}
+
 			// Derive the master key
 			masterKey, _, errMessage, err := GetMasterKey(info)
 			if err != nil {
@@ -145,8 +150,7 @@ You must specify a destination, which is a folder inside the repository where yo
 	}
 
 	// Flags
-	c.Flags().StringVarP(&flagStoreConnectionString, "store", "s", "", "connection string for the store")
-	c.MarkFlagRequired("store")
+	addStoreFlag(c, &flagStoreConnectionString)
 	c.Flags().StringVarP(&flagDestination, "destination", "d", "", "destination folder")
 	c.MarkFlagRequired("destination")
 

cmd/repo-init.go

@@ -85,8 +85,7 @@ In order to use GPG keys, you need to have GPG version 2 installed separately. Y
 	}
 
 	// Flags
-	c.Flags().StringVarP(&flagStoreConnectionString, "store", "s", "", "connection string for the store")
-	c.MarkFlagRequired("store")
+	addStoreFlag(c, &flagStoreConnectionString)
 	c.Flags().StringVarP(&flagGPGKey, "gpg", "g", "", "protect the master key with the gpg key with this address (optional)")
 
 	// Add the command

cmd/repo-key-add.go

@@ -18,7 +18,6 @@ along with this program. If not, see <http://www.gnu.org/licenses/>.
 package cmd
 
 import (
-	"errors"
 	"fmt"
 
 	"github.com/ItalyPaleAle/prvt/fs"
@@ -64,8 +63,7 @@ In order to use GPG keys, you need to have GPG version 2 installed separately. Y
 			}
 
 			// Require info files version 2 or higher
-			if info.Version < 2 {
-				utils.ExitWithError(utils.ErrorUser, "Repository needs to be upgraded", errors.New(`Please run "prvt repo upgrade --store <string>" to upgrade this repository to the latest format`))
+			if !requireInfoFileVersion(info, 2, flagStoreConnectionString) {
 				return
 			}
 
@@ -110,8 +108,7 @@ In order to use GPG keys, you need to have GPG version 2 installed separately. Y
 	}
 
 	// Flags
-	c.Flags().StringVarP(&flagStoreConnectionString, "store", "s", "", "connection string for the store")
-	c.MarkFlagRequired("store")
+	addStoreFlag(c, &flagStoreConnectionString)
 	c.Flags().StringVarP(&flagGPGKey, "gpg", "g", "", "protect the master key with the gpg key with this address (optional)")
 
 	// Add the command

cmd/repo-key-ls.go

@@ -18,7 +18,6 @@ along with this program. If not, see <http://www.gnu.org/licenses/>.
 package cmd
 
 import (
-	"errors"
 	"fmt"
 	"strconv"
 
@@ -62,8 +61,7 @@ Usage: "prvt repo key ls --store <string>"
 			}
 
 			// Require info files version 2 or higher
-			if info.Version < 2 {
-				utils.ExitWithError(utils.ErrorUser, "Repository needs to be upgraded", errors.New(`Please run "prvt repo upgrade --store <string>" to upgrade this repository to the latest format`))
+			if !requireInfoFileVersion(info, 2, flagStoreConnectionString) {
 				return
 			}
 
@@ -90,8 +88,7 @@ Usage: "prvt repo key ls --store <string>"
 	}
 
 	// Flags
-	c.Flags().StringVarP(&flagStoreConnectionString, "store", "s", "", "connection string for the store")
-	c.MarkFlagRequired("store")
+	addStoreFlag(c, &flagStoreConnectionString)
 
 	// Add the command
 	repoKeyCmd.AddCommand(c)

cmd/repo-key-rm.go

@@ -66,8 +66,7 @@ To identify a passphrase or a GPG key among those authorized, you can use the "p
 			}
 
 			// Require info files version 2 or higher
-			if info.Version < 2 {
-				utils.ExitWithError(utils.ErrorUser, "Repository needs to be upgraded", errors.New(`Please run "prvt repo upgrade --store <string>" to upgrade this repository to the latest format`))
+			if !requireInfoFileVersion(info, 2, flagStoreConnectionString) {
 				return
 			}
 
@@ -111,8 +110,7 @@ To identify a passphrase or a GPG key among those authorized, you can use the "p
 	}
 
 	// Flags
-	c.Flags().StringVarP(&flagStoreConnectionString, "store", "s", "", "connection string for the store")
-	c.MarkFlagRequired("store")
+	addStoreFlag(c, &flagStoreConnectionString)
 	c.Flags().StringVarP(&flagKeyId, "key", "k", "", "ID of the key to remove")
 	c.MarkFlagRequired("key")
 

cmd/repo-key-test.go

@@ -18,7 +18,6 @@ along with this program. If not, see <http://www.gnu.org/licenses/>.
 package cmd
 
 import (
-	"errors"
 	"fmt"
 
 	"github.com/ItalyPaleAle/prvt/fs"
@@ -63,8 +62,7 @@ This command is particularly useful to determine the ID of a key that you want t
 			}
 
 			// Require info files version 2 or higher
-			if info.Version < 2 {
-				utils.ExitWithError(utils.ErrorUser, "Repository needs to be upgraded", errors.New(`Please run "prvt repo upgrade --store <string>" to upgrade this repository to the latest format`))
+			if !requireInfoFileVersion(info, 2, flagStoreConnectionString) {
 				return
 			}
 
@@ -81,8 +79,7 @@ This command is particularly useful to determine the ID of a key that you want t
 	}
 
 	// Flags
-	c.Flags().StringVarP(&flagStoreConnectionString, "store", "s", "", "connection string for the store")
-	c.MarkFlagRequired("store")
+	addStoreFlag(c, &flagStoreConnectionString)
 
 	// Add the command
 	repoKeyCmd.AddCommand(c)

cmd/repo-upgrade.go

@@ -77,8 +77,7 @@ Usage: "prvt repo upgrade --store <string>"
 	}
 
 	// Flags
-	c.Flags().StringVarP(&flagStoreConnectionString, "store", "s", "", "connection string for the store")
-	c.MarkFlagRequired("store")
+	addStoreFlag(c, &flagStoreConnectionString)
 
 	// Add the command
 	repoCmd.AddCommand(c)

cmd/rm.go

@@ -69,6 +69,11 @@ To remove a file, specify its exact path. To remove a folder recursively, specif
 				return
 			}
 
+			// Require info files version 3 or higher before any operation that changes the store (which would update the index to the protobuf-based format)
+			if !requireInfoFileVersion(info, 3, flagStoreConnectionString) {
+				return
+			}
+
 			// Derive the master key
 			masterKey, _, errMessage, err := GetMasterKey(info)
 			if err != nil {
@@ -112,8 +117,7 @@ To remove a file, specify its exact path. To remove a folder recursively, specif
 	}
 
 	// Flags
-	c.Flags().StringVarP(&flagStoreConnectionString, "store", "s", "", "connection string for the store")
-	c.MarkFlagRequired("store")
+	addStoreFlag(c, &flagStoreConnectionString)
 
 	// Add the command
 	rootCmd.AddCommand(c)

cmd/serve.go

@@ -82,9 +82,10 @@ You can use the optional "--address" and "--port" flags to control what address
 
 			// Start the server
 			srv := server.Server{
-				Store:   store,
-				Verbose: flagVerbose,
-				Repo:    repo,
+				Store:    store,
+				Verbose:  flagVerbose,
+				Repo:     repo,
+				Infofile: info,
 			}
 			err = srv.Start(flagBindAddress, flagBindPort)
 			if err != nil {
@@ -95,8 +96,7 @@ You can use the optional "--address" and "--port" flags to control what address
 	}
 
 	// Flags
-	c.Flags().StringVarP(&flagStoreConnectionString, "store", "s", "", "connection string for the store")
-	c.MarkFlagRequired("store")
+	addStoreFlag(c, &flagStoreConnectionString)
 	c.Flags().StringVarP(&flagBindAddress, "address", "a", "127.0.0.1", "address to bind to")
 	c.Flags().StringVarP(&flagBindPort, "port", "p", "3129", "port to bind to")
 	c.Flags().BoolVarP(&flagVerbose, "verbose", "v", false, "show request log")

cmd/zz-functions.go

@@ -0,0 +1,52 @@
+/*
+Copyright © 2020 Alessandro Segala (@ItalyPaleAle)
+
+This program is free software: you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation, either version 3 of the License, or
+(at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program. If not, see <http://www.gnu.org/licenses/>.
+*/
+
+package cmd
+
+import (
+	"errors"
+	"os"
+
+	"github.com/ItalyPaleAle/prvt/infofile"
+	"github.com/ItalyPaleAle/prvt/utils"
+
+	"github.com/spf13/cobra"
+)
+
+// Requires a minimum version of the info file to continue
+func requireInfoFileVersion(info *infofile.InfoFile, version uint16, connectionString string) bool {
+	if connectionString == "" {
+		connectionString = "<string>"
+	}
+
+	if info.Version < version {
+		utils.ExitWithError(utils.ErrorUser, "Repository needs to be upgraded", errors.New(`Please run "prvt repo upgrade --store `+connectionString+`" to upgrade this repository to the latest format`))
+		return false
+	}
+
+	return true
+}
+
+// Adds the --store flag, with a default value read from the environment
+func addStoreFlag(c *cobra.Command, flag *string) {
+	// Check if we have a value in the PRVT_STORE env var
+	env := os.Getenv("PRVT_STORE")
+	c.Flags().StringVarP(flag, "store", "s", env, "connection string for the store")
+	if env == "" {
+		c.MarkFlagRequired("store")
+	}
+}

cmd/zz-keys.go

@@ -73,13 +73,35 @@ func NewInfoFile(gpgKey string) (info *infofile.InfoFile, errMessage string, err
 	return info, "", nil
 }
 
-// UpgradeInfoFile upgrades an info file from version 1 to 2
+// UpgradeInfoFile upgrades an info file to the latest version
 func UpgradeInfoFile(info *infofile.InfoFile) (errMessage string, err error) {
-	// Can only upgrade info files version 1
-	if info.Version != 1 {
+	// Can only upgrade info files version 1 and 2
+	if info.Version != 1 && info.Version != 2 {
 		return "Unsupported repository version", errors.New("This repository has already been upgraded or is using an unsupported version")
 	}
 
+	// Upgrade 1 -> 2
+	if info.Version < 2 {
+		errMessage, err = upgradeInfoFileV1(info)
+		if err != nil {
+			return errMessage, err
+		}
+	}
+
+	// Upgrade 2 -> 3
+	// Nothing to do here, as the change is just in the index file
+	// However, we still want to update the info file so older versions of prvt won't try to open a protobuf-encoded index file
+	/*if info.Version < 3 {
+	}*/
+
+	// Update the version
+	info.Version = 3
+
+	return "", nil
+}
+
+// Upgrade an info file from version 1 to 2
+func upgradeInfoFileV1(info *infofile.InfoFile) (errMessage string, err error) {
 	// GPG keys are already migrated into the Keys slice
 	// But passphrases need to be migrated
 	if len(info.Salt) > 0 && len(info.ConfirmationHash) > 0 {
@@ -124,9 +146,6 @@ func UpgradeInfoFile(info *infofile.InfoFile) (errMessage string, err error) {
 		info.ConfirmationHash = nil
 	}
 
-	// Update the version
-	info.Version = 2
-
 	return "", nil
 }