Skip to content

Commit

Permalink
Merge branch 'master' into update-docker-deps
Browse files Browse the repository at this point in the history
  • Loading branch information
fmartingr authored Dec 30, 2023
2 parents 0c01a70 + 7c13626 commit 2525d47
Show file tree
Hide file tree
Showing 6 changed files with 235 additions and 45 deletions.
2 changes: 1 addition & 1 deletion Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,7 @@ serve:
## Runs server for local development
.PHONY: run-server
run-server:
GIN_MODE=$(GIN_MODE) SHIORI_DEVELOPMENT=$(SHIORI_DEVELOPMENT) SHIORI_DIR=$(SHIORI_DIR) SHIORI_HTTP_SECRET_KEY=shiori go run main.go server
GIN_MODE=$(GIN_MODE) SHIORI_DEVELOPMENT=$(SHIORI_DEVELOPMENT) SHIORI_DIR=$(SHIORI_DIR) SHIORI_HTTP_SECRET_KEY=shiori go run main.go server --log-level debug

## Generate swagger docs
.PHONY: swagger
Expand Down
93 changes: 70 additions & 23 deletions docs/Configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,16 +2,56 @@

<!-- TOC -->

- [Data Directory](#data-directory)
- [Webroot](#webroot)
- [Overall Configuration](#overall-configuration)
- [Global configuration](#global-configuration)
- [HTTP configuration variables](#http-configuration-variables)
- [Storage Configuration](#storage-configuration)
- [The data Directory](#the-data-directory)
- [Database Configuration](#database-configuration)
- [MySQL](#mysql)
- [PostgreSQL](#postgresql)
- [Reverse proxies and the webroot path](#reverse-proxies-and-the-webroot-path)
- [Nginx](#nginx)
- [Database](#database)
- [MySQL](#mysql)
- [PostgreSQL](#postgresql)

<!-- /TOC -->

## Data Directory
## Overall Configuration

Most configuration can be set directly using environment variables or flags. The available flags can be found by running `shiori --help`. The available environment variables are listed below.

### Global configuration

| Environment variable | Default | Required | Description |
| -------------------- | ------- | -------- | -------------------------------------- |
| `SHIORI_DEVELOPMENT` | `False` | No | Specifies if the server is in dev mode |

### HTTP configuration variables

| Environment variable | Default | Required | Description |
| ------------------------------------------ | ------- | -------- | ----------------------------------------------------- |
| `SHIORI_HTTP_ENABLED` | True | No | Enable HTTP service |
| `SHIORI_HTTP_PORT` | 8080 | No | Port number for the HTTP service |
| `SHIORI_HTTP_ADDRESS` | : | No | Address for the HTTP service |
| `SHIORI_HTTP_ROOT_PATH` | / | No | Root path for the HTTP service |
| `SHIORI_HTTP_ACCESS_LOG` | True | No | Logging accessibility for HTTP requests |
| `SHIORI_HTTP_SERVE_WEB_UI` | True | No | Serving Web UI via HTTP. Disable serves only the API. |
| `SHIORI_HTTP_SECRET_KEY` | | **Yes** | Secret key for HTTP sessions. |
| `SHIORI_HTTP_BODY_LIMIT` | 1024 | No | Limit for request body size |
| `SHIORI_HTTP_READ_TIMEOUT` | 10s | No | Maximum duration for reading the entire request |
| `SHIORI_HTTP_WRITE_TIMEOUT` | 10s | No | Maximum duration before timing out writes |
| `SHIORI_HTTP_IDLE_TIMEOUT` | 10s | No | Maximum amount of time to wait for the next request |
| `SHIORI_HTTP_DISABLE_KEEP_ALIVE` | true | No | Disable HTTP keep-alive connections |
| `SHIORI_HTTP_DISABLE_PARSE_MULTIPART_FORM` | true | No | Disable pre-parsing of multipart form |

### Storage Configuration

The `StorageConfig` struct contains settings related to storage.

| Environment variable | Default | Required | Description |
| -------------------- | ------------- | -------- | --------------------------------------- |
| `SHIORI_DIR` | (current dir) | No | Directory where Shiori stores its data. |

#### The data Directory

Shiori is designed to work out of the box, but you can change where it stores your bookmarks if you need to.

Expand All @@ -27,7 +67,30 @@ If you pass the flag `--portable` to Shiori, your data will be stored in the `s

To specify a custom path, set the `SHIORI_DIR` environment variable.

## Webroot
### Database Configuration

| Environment variable | Default | Required | Description |
| -------------------------- | ------- | -------- | ----------------------------------------------- |
| `SHIORI_DBMS` (deprecated) | `DBMS` | No | Deprecated (Use environment variables for DBMS) |
| `SHIORI_DATABASE_URL` | `URL` | No | URL for the database (required) |

> `SHIORI_DBMS` is deprecated and will be removed in a future release. Please use `SHIORI_DATABASE_URL` instead.
Shiori uses an SQLite3 database stored in the above [data directory by default](#storage-configuration). If you prefer, you can also use MySQL or PostgreSQL database by setting the `SHIORI_DATABASE_URL` environment variable.

#### MySQL

MySQL example: `SHIORI_DATABASE_URL="mysql://username:password@(hostname:port)/database?charset=utf8mb4"`

You can find additional details in [go mysql sql driver documentation](https://github.com/go-sql-driver/mysql#dsn-data-source-name).

#### PostgreSQL

PostgreSQL example: `SHIORI_DATABASE_URL="postgres://pqgotest:password@hostname/database?sslmode=verify-full"`

You can find additional details in [go postgres sql driver documentation](https://pkg.go.dev/github.com/lib/pq).

## Reverse proxies and the webroot path

If you want to serve Shiori behind a reverse proxy, you can set the `SHIORI_WEBROOT` environment variable to the path where Shiori is served, e.g. `/shiori`.

Expand All @@ -46,19 +109,3 @@ location /shiori {
proxy_set_header X-Real-IP $remote_addr;
}
```

## Database

Shiori uses an SQLite3 database stored in the above data directory by default. If you prefer, you can also use MySQL or PostgreSQL database by setting it in environment variables.

### MySQL

MySQL example: `SHIORI_DATABASE_URL="mysql://username:password@(hostname:port)/database?charset=utf8mb4"`

You can find additional details in [go mysql sql driver documentation](https://github.com/go-sql-driver/mysql#dsn-data-source-name).

### PostgreSQL

PostgreSQL example: `SHIORI_DATABASE_URL="postgres://pqgotest:password@hostname/database?sslmode=verify-full"`

You can find additional details in [go postgres sql driver documentation](https://pkg.go.dev/github.com/lib/pq).
3 changes: 3 additions & 0 deletions internal/cmd/root.go
Original file line number Diff line number Diff line change
Expand Up @@ -71,6 +71,7 @@ func initShiori(ctx context.Context, cmd *cobra.Command) (*config.Config, *depen
}

cfg := config.ParseServerConfiguration(ctx, logger)
cfg.LogLevel = logger.Level.String()

if storageDirectory != "" && cfg.Storage.DataDir != "" {
logger.Warn("--storage-directory is set, overriding SHIORI_DIR.")
Expand Down Expand Up @@ -125,6 +126,8 @@ func initShiori(ctx context.Context, cmd *cobra.Command) (*config.Config, *depen
}
}

cfg.DebugConfiguration(logger)

return cfg, dependencies
}

Expand Down
2 changes: 0 additions & 2 deletions internal/cmd/server.go
Original file line number Diff line number Diff line change
Expand Up @@ -41,8 +41,6 @@ func newServerCommandHandler() func(cmd *cobra.Command, args []string) {

cfg, dependencies := initShiori(ctx, cmd)

cfg.Http.SetDefaults(dependencies.Log)

// Validate root path
if rootPath == "" {
rootPath = "/"
Expand Down
67 changes: 48 additions & 19 deletions internal/config/config.go
Original file line number Diff line number Diff line change
Expand Up @@ -16,14 +16,14 @@ import (

// readDotEnv reads the configuration from variables in a .env file (only for contributing)
func readDotEnv(logger *logrus.Logger) map[string]string {
result := make(map[string]string)

file, err := os.Open(".env")
if err != nil {
return nil
return result
}
defer file.Close()

result := make(map[string]string)

scanner := bufio.NewScanner(file)

for scanner.Scan() {
Expand All @@ -33,6 +33,11 @@ func readDotEnv(logger *logrus.Logger) map[string]string {
}

keyval := strings.SplitN(line, "=", 2)
if len(keyval) != 2 {
logger.WithField("line", line).Warn("invalid line in .env file")
continue
}

result[keyval[0]] = keyval[1]
}

Expand Down Expand Up @@ -60,6 +65,19 @@ type HttpConfig struct {
DisablePreParseMultipartForm bool `env:"HTTP_DISABLE_PARSE_MULTIPART_FORM,default=true"`
}

// SetDefaults sets the default values for the configuration
func (c *HttpConfig) SetDefaults(logger *logrus.Logger) {
// Set a random secret key if not set
if len(c.SecretKey) == 0 {
logger.Warn("SHIORI_HTTP_SECRET_KEY is not set, using random value. This means that all sessions will be invalidated on server restart.")
randomUUID, err := uuid.NewV4()
if err != nil {
logger.WithError(err).Fatal("couldn't generate a random UUID")
}
c.SecretKey = []byte(randomUUID.String())
}
}

type DatabaseConfig struct {
DBMS string `env:"DBMS"` // Deprecated
// DBMS requires more environment variables. Check the database package for more information.
Expand All @@ -73,23 +91,10 @@ type StorageConfig struct {
type Config struct {
Hostname string `env:"HOSTNAME,required"`
Development bool `env:"DEVELOPMENT,default=False"`
LogLevel string // Set only from the CLI flag
Database *DatabaseConfig
Storage *StorageConfig
// LogLevel string `env:"LOG_LEVEL,default=info"`
Http *HttpConfig
}

// SetDefaults sets the default values for the configuration
func (c *HttpConfig) SetDefaults(logger *logrus.Logger) {
// Set a random secret key if not set
if len(c.SecretKey) == 0 {
logger.Warn("SHIORI_HTTP_SECRET_KEY is not set, using random value. This means that all sessions will be invalidated on server restart.")
randomUUID, err := uuid.NewV4()
if err != nil {
logger.WithError(err).Fatal("couldn't generate a random UUID")
}
c.SecretKey = []byte(randomUUID.String())
}
Http *HttpConfig
}

// SetDefaults sets the default values for the configuration
Expand All @@ -108,16 +113,40 @@ func (c Config) SetDefaults(logger *logrus.Logger, portableMode bool) {
if c.Database.DBMS == "" && c.Database.URL == "" {
c.Database.URL = fmt.Sprintf("sqlite:///%s", filepath.Join(c.Storage.DataDir, "shiori.db"))
}

c.Http.SetDefaults(logger)
}

func (c *Config) DebugConfiguration(logger *logrus.Logger) {
logger.Debug("Configuration:")
logger.Debugf(" SHIORI_HOSTNAME: %s", c.Hostname)
logger.Debugf(" SHIORI_DEVELOPMENT: %t", c.Development)
logger.Debugf(" SHIORI_DATABASE_URL: %s", c.Database.URL)
logger.Debugf(" SHIORI_DBMS: %s", c.Database.DBMS)
logger.Debugf(" SHIORI_DIR: %s", c.Storage.DataDir)
logger.Debugf(" SHIORI_HTTP_ENABLED: %t", c.Http.Enabled)
logger.Debugf(" SHIORI_HTTP_PORT: %d", c.Http.Port)
logger.Debugf(" SHIORI_HTTP_ADDRESS: %s", c.Http.Address)
logger.Debugf(" SHIORI_HTTP_ROOT_PATH: %s", c.Http.RootPath)
logger.Debugf(" SHIORI_HTTP_ACCESS_LOG: %t", c.Http.AccessLog)
logger.Debugf(" SHIORI_HTTP_SERVE_WEB_UI: %t", c.Http.ServeWebUI)
logger.Debugf(" SHIORI_HTTP_SECRET_KEY: %d characters", len(c.Http.SecretKey))
logger.Debugf(" SHIORI_HTTP_BODY_LIMIT: %d", c.Http.BodyLimit)
logger.Debugf(" SHIORI_HTTP_READ_TIMEOUT: %s", c.Http.ReadTimeout)
logger.Debugf(" SHIORI_HTTP_WRITE_TIMEOUT: %s", c.Http.WriteTimeout)
logger.Debugf(" SHIORI_HTTP_IDLE_TIMEOUT: %s", c.Http.IDLETimeout)
logger.Debugf(" SHIORI_HTTP_DISABLE_KEEP_ALIVE: %t", c.Http.DisableKeepAlive)
logger.Debugf(" SHIORI_HTTP_DISABLE_PARSE_MULTIPART_FORM: %t", c.Http.DisablePreParseMultipartForm)
}

// ParseServerConfiguration parses the configuration from the enabled lookupers
func ParseServerConfiguration(ctx context.Context, logger *logrus.Logger) *Config {
var cfg Config

lookuper := envconfig.MultiLookuper(
envconfig.MapLookuper(map[string]string{"HOSTNAME": os.Getenv("HOSTNAME")}),
envconfig.MapLookuper(readDotEnv(logger)),
envconfig.PrefixLookuper("SHIORI_", envconfig.OsLookuper()),
envconfig.OsLookuper(),
)
if err := envconfig.ProcessWith(ctx, &cfg, lookuper); err != nil {
logger.WithError(err).Fatal("Error parsing configuration")
Expand Down
113 changes: 113 additions & 0 deletions internal/config/config_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,113 @@
package config

import (
"context"
"os"
"testing"

"github.com/sirupsen/logrus"
"github.com/stretchr/testify/require"
)

func TestHostnameVariable(t *testing.T) {
os.Setenv("HOSTNAME", "test_hostname")
defer os.Unsetenv("HOSTNAME")

log := logrus.New()
cfg := ParseServerConfiguration(context.TODO(), log)

require.Equal(t, "test_hostname", cfg.Hostname)
}

// TestBackwardsCompatibility tests that the old environment variables changed from 1.5.5 onwards
// are still supported and working with the new configuration system.
func TestBackwardsCompatibility(t *testing.T) {
for _, env := range []struct {
env string
want string
eval func(t *testing.T, cfg *Config)
}{
{"HOSTNAME", "test_hostname", func(t *testing.T, cfg *Config) {
require.Equal(t, "test_hostname", cfg.Hostname)
}},
{"SHIORI_DIR", "test", func(t *testing.T, cfg *Config) {
require.Equal(t, "test", cfg.Storage.DataDir)
}},
{"SHIORI_DBMS", "test", func(t *testing.T, cfg *Config) {
require.Equal(t, "test", cfg.Database.DBMS)
}},
} {
t.Run(env.env, func(t *testing.T) {
os.Setenv(env.env, env.want)
t.Cleanup(func() {
os.Unsetenv(env.env)
})

log := logrus.New()
cfg := ParseServerConfiguration(context.Background(), log)
env.eval(t, cfg)
})
}
}

func TestReadDotEnv(t *testing.T) {
log := logrus.New()

for _, testCase := range []struct {
name string
line string
env map[string]string
}{
{"empty", "", map[string]string{}},
{"comment", "# comment", map[string]string{}},
{"ignore invalid lines", "invalid line", map[string]string{}},
{"single variable", "SHIORI_HTTP_PORT=9999", map[string]string{"SHIORI_HTTP_PORT": "9999"}},
{"multiple variable", "SHIORI_HTTP_PORT=9999\nSHIORI_HTTP_SECRET_KEY=123123", map[string]string{"SHIORI_HTTP_PORT": "9999", "SHIORI_HTTP_SECRET_KEY": "123123"}},
} {
t.Run(testCase.name, func(t *testing.T) {
tmpDir, err := os.MkdirTemp("", "")
require.NoError(t, err)

os.Chdir(tmpDir)

t.Cleanup(func() {
require.NoError(t, os.RemoveAll(tmpDir))
})

// Write the .env file in the temporary directory
handler, err := os.OpenFile(".env", os.O_CREATE|os.O_WRONLY, 0655)
require.NoError(t, err)
handler.Write([]byte(testCase.line + "\n"))
handler.Close()

e := readDotEnv(log)

require.Equal(t, testCase.env, e)
})
}

t.Run("no file", func(t *testing.T) {
tmpDir, err := os.MkdirTemp("", "")
require.NoError(t, err)

os.Chdir(tmpDir)

t.Cleanup(func() {
require.NoError(t, os.RemoveAll(tmpDir))
})

e := readDotEnv(log)

require.Equal(t, map[string]string{}, e)
})
}

func TestConfigSetDefaults(t *testing.T) {
log := logrus.New()
cfg := ParseServerConfiguration(context.TODO(), log)
cfg.SetDefaults(log, false)

require.NotEmpty(t, cfg.Http.SecretKey)
require.NotEmpty(t, cfg.Storage.DataDir)
require.NotEmpty(t, cfg.Database.URL)
}

0 comments on commit 2525d47

Please sign in to comment.