Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WIP Tiot/openriak 3.4/tictacaae and nextgenrepl cli commands #55

Open
wants to merge 9 commits into
base: openriak-3.4
Choose a base branch
from

Conversation

hmmr
Copy link

@hmmr hmmr commented Dec 19, 2024

This PR implements "Tictac runtime management and reporting" section of the TicTac AAE and NextGenRepl CLI commands proposal.

The riak_kv_console:titacaae_cmd/1 functions will be exposed as riak admin tictacaae CMD.

The commands from rebuildwait to aaefoldworkers are simple get/show commands, reading and setting the corresponding app env vars in the riak_kv app, on local node by default or on other node if specified with -n option.

rebuild-soon will schedule a tree rebuild, and rebuild-now, will additionally send a poke to controller(s) associated with partitions or nodes as specified with options -n and -p.

Depends on this PR in kv_index_tictactree.

@martinsumner
Copy link

martinsumner commented Dec 19, 2024

For meck issue - OpenRiak/riak#22

@martinsumner
Copy link

As an aside on this, there is a dependency in Riak on clique - which is intended as a framework for building CLI commands.

Is there anything left within Riak that is active/working and uses clique?

There are some clique things registered within riak_core - https://github.com/OpenRiak/riak_core/blob/openriak-3.2/src/riak_core_cluster_cli.erl and https://github.com/OpenRiak/riak_core/blob/openriak-3.2/src/riak_core_handoff_cli.erl. I think these have now been replaced with commands run via the riak-admin script and riak_core_console.

Should we be removing clique, or is there any value in migrating existing CLI stuff to clique?

@martinsumner
Copy link

I misunderstood what was happening with clique in riak_core. Some of the riak_core CLI is using clique using the command/1 function in riak_core_console:

e.g.
https://github.com/OpenRiak/riak/blob/openriak-3.2/rel/files/riak-admin#L318-L331
https://github.com/OpenRiak/riak/blob/openriak-3.2/rel/files/riak-admin#L984-L994

I've added a comment to the riak_core issue on long-term plans.

@martinsumner
Copy link

One of the features of clique is:

Set, show and describe [cuttlefish](https://github.com/basho/cuttlefish) configuration across one or all nodes: i.e. riak-admin set anti-entropy=on --all

Rather than have specific functions for setting certain config parameters - might there be a better general solution?

There have been issues with confusion when setting run-time configuration using application:set_env/2 - where the operator scripted the cuttlefish on/off value rather than the underlying boolean() - in one case bringing a cluster down, by causing a process to continually crash on an unexpected environment variable type.

@hmmr
Copy link
Author

hmmr commented Dec 20, 2024

Rather than have specific functions for setting certain config parameters - might there be a better general solution?

These are simple set_env calls; importantly, they are non-persistent, instead intended to let operators tweak and experiment with these parameters without having to edit riak.conf and restart. This should be regarded as complementary to the right and proper way of setting these parameters, i.e., via riak.conf.

On the matter of whither to go with clique, I have no strong opinions. Apparently, the push behind clique while it was developed at Basho was to migrate all riak admin callbacks to it -- only to be abandoned midway. To me, still, clique is overengineering. As others have pointed out, argparse and getopt facilities already provide adequate set of functions authors of clique originally intended to implement. I would vote to continue to use it where it is already being used (and not here in riak_kv_console).

@hmmr hmmr changed the title Tiot/openriak 3.4/tictacaae and nextgenrepl cli commands WIP Tiot/openriak 3.4/tictacaae and nextgenrepl cli commands Dec 28, 2024
@@ -31,7 +31,7 @@
]}.

{profiles, [
{test, [{deps, [meck]}]},
{test, [{deps, [{meck, {git, "https://github.com/OpenRiak/meck.git", {branch, "openriak-3.2"}}}]}]},

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cloning your own version of meck seems to me the wrong way to go

@aef- aef- self-requested a review January 8, 2025 14:37
* Tictac runtime management and reporting:
  riak admin tictacaae rebuild_schedule [-n NODE] [-p PARTITION] [RW RD]
  riak admin tictacaae storeheads [-n NODE] [-p PARTITION] [VALUE]
  riak admin tictacaae tokenbucket [-n NODE] [-p PARTITION] [VALUE]
  riak admin tictacaae rebuildtick|exchangetick|maxresults|rangeboost [-n NODE] [VAL]
  riak admin tictacaae rebuild-soon [-n NODE] [-p PARTITION] DELAY
  riak admin tictacaae rebuild-now [-n NODE] [-p PARTITION] DELAY
  riak admin tictacaae treestatus [--format table|json] [--show STATES]

* AAE fold operations, dumping results in JSON format:
  riak tictacaae fold list-buckets NVAL
  riak tictacaae fold find-keys BUCKET KEY_RANGE MODIFIED_RANGE sibling_count=COUNT|object_size=BYTES
  riak tictacaae fold find-keys BUCKET KEY_RANGE MODIFIED_RANGE sibling_count=COUNT|object_size=BYTES
  riak tictacaae fold find|count-tombstones KEY_RANGE SEGMENTS MODIFIED_RANGE
  riak tictacaae fold reap-tombstones KEY_RANGE SEGMENTS MODIFIED_RANGE CHANGE_METHOD
  riak tictacaae fold object-stats BUCKET KEY_RANGE MODIFIED_RANGE
  riak tictacaae fold erase-keys BUCKET KEY_RANGE SEGMENTS MODIFIED_RANGE CHANGE_METHOD
  riak tictacaae fold repair-keys BUCKET KEY_RANGE MODIFIED_RANGE
@hmmr
Copy link
Author

hmmr commented Jan 8, 2025

Reworked much of it, now threading parameters rebuild_schedule, storeheads (with special treatment), tokenbucket to aae_controller and riak_kv_vnode states where they will take effect properly. Some others which can be directly set as riak_kv app env vars, are rebuildtick, exchangetick, maxresults, rangeboost.

AAE fold operations (repair_keys_range, find_keys, object_stats, find_tombs, reap_tombs, erase_keys, list_buckets) are now made to background and write to a file.

Comments are welcome but not really requested as it is still WIP.

I will complement it with proper test in riak_test, too.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants