Releases: spotify/spydra
v0.3.18
Use deterministic names for pooling clusters
This PR changes the way new clusters are created and jobs are assigned to them in the poolable clusters in order to fix a concurrency problem that made it create more clusters than the defined limit. The new implementation uses deterministic names based of the current time to decide the names of the clusters that should be used at a given moment in time. Existing clusters than don't belong to them will be deleted automatically after the max-idle time has passed, as no more jobs will be assigned to them.
v0.3.16
v0.3.15
This release moves Spydra from using custom heartbeats & self-destruct script to using a new dataproc feature (introduced in current beta) called cluster scheduled deletion. More info on the feature here: https://cloud.google.com/dataproc/docs/concepts/configuring-clusters/scheduled-deletion
Note: this transition changes the name and position of configuration parameter responsible for timeout before deletion and format of the value. The parameter responsible for the timeout is now called max-idle and is a part of cluster.options. The format for the value as described in the documentation:
Provide the duration in IntegerUnit format, where the unit can be "s, m, h, d" (seconds, minutes, hours, days, respectively). Examples: "30m" or "1d" (30 minutes or 1 day from when the cluster becomes idle).
The duration should be from the range 10m..14d
The default configuration sets the timeout to 30m which is the same as in previous versions.
