Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Move autopilots into autopilot_state #1657

Open
wants to merge 25 commits into
base: dev
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 19 commits
Commits
Show all changes
25 commits
Select commit Hold shift + click to select a range
09a54c3
stores: add migration_00025_autopilot_config
peterjan Nov 5, 2024
7c3f167
stores: remove autopilots
peterjan Nov 6, 2024
3826cde
all: rename autopilot_config to autopilot_state
peterjan Nov 6, 2024
62e584b
stores: temporarily re-add contracts.set
peterjan Nov 6, 2024
fb00944
all: cleanup PR
peterjan Nov 6, 2024
edaf2d7
all: cleanup PR
peterjan Nov 6, 2024
1b86a31
stores: remove debug
peterjan Nov 6, 2024
56baf32
Merge branch 'dev' into pj/autopilot-state
peterjan Nov 6, 2024
1fa83f5
testing: fix TestNewTestCluster NDF
peterjan Nov 6, 2024
884611b
Merge branch 'pj/autopilot-state' of github.com:SiaFoundation/renterd…
peterjan Nov 6, 2024
4889e65
testing: fix TestNewTestCluster NDF
peterjan Nov 6, 2024
95b5651
stores: ensure default autopilot state
peterjan Nov 6, 2024
0d75f55
Merge branch 'dev' of github.com:SiaFoundation/renterd into pj/autopi…
peterjan Nov 7, 2024
668bae8
stores: change autopilot_state to autopilot
peterjan Nov 7, 2024
74f41be
bus: update autopilot routes
peterjan Nov 8, 2024
c741532
Merge branch 'dev' of github.com:SiaFoundation/renterd into pj/autopi…
peterjan Nov 8, 2024
8564b84
autopilot: fix merge
peterjan Nov 8, 2024
c16e960
Merge branch 'dev' of github.com:SiaFoundation/renterd into pj/autopi…
peterjan Nov 8, 2024
75de934
stores: update autopilot routes
peterjan Nov 8, 2024
ce2033f
bus: cleanup client
peterjan Nov 14, 2024
750effb
contractor: sorty by set
peterjan Nov 14, 2024
eaea057
stores: update table name
peterjan Nov 14, 2024
fb7ed16
stores: update migration
peterjan Nov 14, 2024
296d400
testing: remove TestScoredHostsRandSelectByScore
peterjan Nov 14, 2024
da3580d
Merge branch 'dev' of github.com:SiaFoundation/renterd into pj/autopi…
peterjan Nov 14, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
36 changes: 0 additions & 36 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -517,42 +517,6 @@ your node's consensus state using the following endpoint:

- `GET /api/bus/consensus/state`

### Config

The configuration can be updated through the UI or by using the following
endpoints:

- `GET /api/autopilot/config`
- `PUT /api/autopilot/config`

The autopilot will not perform any tasks until it is configured. An example
configuration can be found below. Especially the `contracts` section is
important, make sure the `amount` is set to the amount of hosts with which you
want to form a contract. The `allowance` is the amount of money the autopilot
can spend per period, make sure it is not set to zero or contracts won't get
formed.

```json
{
"hosts": {
"allowRedundantIPs": false,
"maxDowntimeHours": 1440,
"maxConsecutiveScanFailures": 20,
"scoreOverrides": {}
},
"contracts": {
"set": "autopilot",
"amount": 50,
"allowance": "10000000000000000000000000000",
"period": 6048,
"renewWindow": 2016,
"download": 1099511627776, // 1TiB
"upload": 1099511627776, // 1TiB
"storage": 1099511627776 // 1TiB
}
}
```

### Blocklist

Unfortunately the Sia blockchain is subject to hosts that announced themselves
Expand Down
81 changes: 52 additions & 29 deletions api/autopilot.go
Original file line number Diff line number Diff line change
Expand Up @@ -4,38 +4,35 @@ import (
"errors"
"fmt"

"go.sia.tech/core/types"
"go.sia.tech/renterd/internal/utils"
)

const (
// BlocksPerDay defines the amount of blocks that are mined in a day (one
// block every 10 minutes roughly)
BlocksPerDay = 144

// DefaultAutopilotID is the id of the autopilot.
DefaultAutopilotID = "autopilot"
)

var (
// ErrAutopilotNotFound is returned when an autopilot can't be found.
ErrAutopilotNotFound = errors.New("couldn't find autopilot")

// ErrMaxDowntimeHoursTooHigh is returned if the autopilot config is updated
// ErrMaxDowntimeHoursTooHigh is returned if the contracts config is updated
// with a value that exceeds the maximum of 99 years.
ErrMaxDowntimeHoursTooHigh = errors.New("MaxDowntimeHours is too high, exceeds max value of 99 years")

// ErrInvalidReleaseVersion is returned if the version is an invalid release
// string.
ErrInvalidReleaseVersion = errors.New("invalid release version")
)

type (
// Autopilot contains the autopilot's config and current period.
// Autopilot contains its configuration as well as the current period.
Autopilot struct {
ID string `json:"id"`
Config AutopilotConfig `json:"config"`
CurrentPeriod uint64 `json:"currentPeriod"`
CurrentPeriod uint64 `json:"currentPeriod"`
peterjan marked this conversation as resolved.
Show resolved Hide resolved
AutopilotConfig
}

// AutopilotConfig contains all autopilot configuration.
// AutopilotConfig contains configuration settings for the autopilot.
AutopilotConfig struct {
Enabled bool `json:"enabled"`
n8maninger marked this conversation as resolved.
Show resolved Hide resolved
Contracts ContractsConfig `json:"contracts"`
Hosts HostsConfig `json:"hosts"`
}
Expand All @@ -54,19 +51,33 @@ type (

// HostsConfig contains all hosts settings used in the autopilot.
HostsConfig struct {
AllowRedundantIPs bool `json:"allowRedundantIPs"`
MaxDowntimeHours uint64 `json:"maxDowntimeHours"`
MinProtocolVersion string `json:"minProtocolVersion"`
MaxConsecutiveScanFailures uint64 `json:"maxConsecutiveScanFailures"`
ScoreOverrides map[types.PublicKey]float64 `json:"scoreOverrides"`
AllowRedundantIPs bool `json:"allowRedundantIPs"`
peterjan marked this conversation as resolved.
Show resolved Hide resolved
MaxConsecutiveScanFailures uint64 `json:"maxConsecutiveScanFailures"`
MaxDowntimeHours uint64 `json:"maxDowntimeHours"`
MinProtocolVersion string `json:"minProtocolVersion"`
}
)

// EndHeight of a contract formed using the AutopilotConfig given the current
// period.
func (ap *Autopilot) EndHeight() uint64 {
return ap.CurrentPeriod + ap.Config.Contracts.Period + ap.Config.Contracts.RenewWindow
}
var (
DefaultAutopilotConfig = AutopilotConfig{
Enabled: false,
Contracts: ContractsConfig{
Amount: 50,
peterjan marked this conversation as resolved.
Show resolved Hide resolved
Period: 144 * 7 * 6,
RenewWindow: 144 * 7 * 2,
Download: 1e12, // 1 TB
Upload: 1e12, // 1 TB
Storage: 4e12, // 4 TB
Copy link
Member

@n8maninger n8maninger Nov 8, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Similar conversation: since we now have initial contract funding instead of a global allowance and refreshing a contract in v2 will be less painful, are these even necessary? Attempting to calculate this ahead of time adds unnecessary setup complexity and overhead. It may end up in more contracts overall, but storage utilization could be more efficient if the scoring didn't need to check for storage space available based on expected storage * redundancy / target contracts.

If a host runs out of storage space, but is otherwise in working condition, form a contract with a new host instead of churning it out of the set.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We could get rid of the remaining storage score and simplify the collateral score and instead just check whether a host is already full / out of collateral before forming a contract without giving them a score.

Prune: false,
},
Hosts: HostsConfig{
AllowRedundantIPs: false,
MaxConsecutiveScanFailures: 10,
MaxDowntimeHours: 24 * 7 * 2,
MinProtocolVersion: "1.6.0",
},
}
)

type (
// AutopilotTriggerRequest is the request object used by the /trigger
Expand All @@ -84,8 +95,7 @@ type (
// AutopilotStateResponse is the response type for the /autopilot/state
// endpoint.
AutopilotStateResponse struct {
ID string `json:"id"`
Configured bool `json:"configured"`
Enabled bool `json:"enabled"`
Migrating bool `json:"migrating"`
MigratingLastStart TimeRFC3339 `json:"migratingLastStart"`
Pruning bool `json:"pruning"`
Expand Down Expand Up @@ -128,11 +138,24 @@ type (
}
)

func (c AutopilotConfig) Validate() error {
if c.Hosts.MaxDowntimeHours > 99*365*24 {
func (ap Autopilot) EndHeight() uint64 {
return ap.CurrentPeriod + ap.Contracts.Period + ap.Contracts.RenewWindow
}

func (cc ContractsConfig) Validate() error {
if cc.Period == 0 {
return errors.New("period must be greater than 0")
} else if cc.RenewWindow == 0 {
return errors.New("renewWindow must be greater than 0")
}
return nil
}

func (hc HostsConfig) Validate() error {
if hc.MaxDowntimeHours > 99*365*24 {
return ErrMaxDowntimeHoursTooHigh
} else if c.Hosts.MinProtocolVersion != "" && !utils.IsVersion(c.Hosts.MinProtocolVersion) {
return fmt.Errorf("invalid min protocol version '%s'", c.Hosts.MinProtocolVersion)
} else if hc.MinProtocolVersion != "" && !utils.IsVersion(hc.MinProtocolVersion) {
return fmt.Errorf("%w: '%s'", ErrInvalidReleaseVersion, hc.MinProtocolVersion)
}
return nil
}
8 changes: 8 additions & 0 deletions api/bus.go
Original file line number Diff line number Diff line change
Expand Up @@ -97,4 +97,12 @@ type (
Settings rhpv2.HostSettings `json:"settings,omitempty"`
PriceTable rhpv3.HostPriceTable `json:"priceTable,omitempty"`
}

// UpdateAutopilotRequest is the request type for the /autopilot endpoint.
UpdateAutopilotRequest struct {
Enabled *bool `json:"enabled"`
Contracts *ContractsConfig `json:"contracts"`
CurrentPeriod *uint64 `json:"currentPeriod"`
Hosts *HostsConfig `json:"hosts"`
}
)
34 changes: 16 additions & 18 deletions api/host.go
Original file line number Diff line number Diff line change
Expand Up @@ -60,7 +60,6 @@ type (
HostsRequest struct {
Offset int `json:"offset"`
Limit int `json:"limit"`
AutopilotID string `json:"autopilotID"`
FilterMode string `json:"filterMode"`
UsabilityMode string `json:"usabilityMode"`
AddressContains string `json:"addressContains"`
Expand Down Expand Up @@ -88,7 +87,6 @@ type (
// Option types.
type (
HostOptions struct {
AutopilotID string
AddressContains string
FilterMode string
UsabilityMode string
Expand All @@ -101,19 +99,19 @@ type (

type (
Host struct {
KnownSince time.Time `json:"knownSince"`
LastAnnouncement time.Time `json:"lastAnnouncement"`
PublicKey types.PublicKey `json:"publicKey"`
NetAddress string `json:"netAddress"`
PriceTable HostPriceTable `json:"priceTable"`
Settings rhpv2.HostSettings `json:"settings"`
Interactions HostInteractions `json:"interactions"`
Scanned bool `json:"scanned"`
Blocked bool `json:"blocked"`
Checks map[string]HostCheck `json:"checks"`
StoredData uint64 `json:"storedData"`
ResolvedAddresses []string `json:"resolvedAddresses"`
Subnets []string `json:"subnets"`
KnownSince time.Time `json:"knownSince"`
LastAnnouncement time.Time `json:"lastAnnouncement"`
PublicKey types.PublicKey `json:"publicKey"`
NetAddress string `json:"netAddress"`
PriceTable HostPriceTable `json:"priceTable"`
Settings rhpv2.HostSettings `json:"settings"`
Interactions HostInteractions `json:"interactions"`
Scanned bool `json:"scanned"`
Blocked bool `json:"blocked"`
Checks HostChecks `json:"checks,omitempty"`
StoredData uint64 `json:"storedData"`
ResolvedAddresses []string `json:"resolvedAddresses"`
Subnets []string `json:"subnets"`
}

HostInfo struct {
Expand Down Expand Up @@ -156,7 +154,7 @@ type (
PriceTable HostPriceTable `json:"priceTable"`
}

HostCheck struct {
HostChecks struct {
GougingBreakdown HostGougingBreakdown `json:"gougingBreakdown"`
ScoreBreakdown HostScoreBreakdown `json:"scoreBreakdown"`
UsabilityBreakdown HostUsabilityBreakdown `json:"usabilityBreakdown"`
Expand Down Expand Up @@ -192,8 +190,8 @@ type (
}
)

func (hc HostCheck) MarshalJSON() ([]byte, error) {
type check HostCheck
func (hc HostChecks) MarshalJSON() ([]byte, error) {
type check HostChecks
return json.Marshal(struct {
check
Score float64 `json:"score"`
Expand Down
2 changes: 1 addition & 1 deletion api/host_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ import (
)

func TestMarshalHostScoreBreakdownJSON(t *testing.T) {
hc := HostCheck{
hc := HostChecks{
ScoreBreakdown: HostScoreBreakdown{
Age: 1.1,
Collateral: 1.1,
Expand Down
4 changes: 2 additions & 2 deletions api/prometheus.go
Original file line number Diff line number Diff line change
Expand Up @@ -50,9 +50,9 @@ func (asr AutopilotStateResponse) PrometheusMetric() (metrics []prometheus.Metri
Value: float64(asr.UptimeMS),
},
{
Name: "renterd_autopilot_state_configured",
Name: "renterd_autopilot_state_enabled",
Labels: labels,
Value: boolToFloat(asr.Configured),
Value: boolToFloat(asr.Enabled),
},
{
Name: "renterd_autopilot_state_migrating",
Expand Down
Loading
Loading