Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Initial addition of a server "logger" API. #55

Closed
wants to merge 36 commits into from

Conversation

edwbuck
Copy link
Contributor

@edwbuck edwbuck commented Jan 30, 2024

New Data Types

  • Logger : a representation of a logger
    • current_level : the current log level of the logger
    • default_level : the log level of the logger configured in the application's config file

Logger Operations:

  • GetLogger : Fetches the Logger data type

    • Output: a Logger record
  • SetLogLevel: Changes the log level of the logger

    • Input: log_level, the new log level, (DEFAULT to use the default_level)
    • Output: none

closes #54

loveyana and others added 24 commits March 17, 2022 14:28
* proto/agent: add JWT SVID privileged API

Signed-off-by: Yuhan Li <[email protected]>

* Fix comment lint problem

Signed-off-by: Yuhan Li <[email protected]>
This change updates links to github repositories to incorporate renames
to their default branches.

Signed-off-by: Andrew Harding <[email protected]>
update bool name

Co-authored-by: Andrew Harding <[email protected]>
Signed-off-by: Faisal Memon <[email protected]>
Update permission denied details

Co-authored-by: Andrew Harding <[email protected]>
Signed-off-by: Faisal Memon <[email protected]>
Signed-off-by: Faisal Memon <[email protected]>
Also update go.mod to use latest conventions with indirect dependencies.

Signed-off-by: Ryan Turner <[email protected]>
Introduce new SVID count fields for each SVID cache in SPIRE Agent.
These new fields clarify what exactly is represented in the counts and
provide integration tests a way to assert on the state of the various
Agent in-memory caches.

Signed-off-by: Ryan Turner <[email protected]>
This change is in reference to a new SPIRE feature discussed in spiffe/spire#2700

Signed-off-by: Dennis Gove <[email protected]>
Per comment spiffe/spire#3445 (comment) it was decided to
1. Rename the TTL field to X509SvidTtl
2. Add the new JwtSvidTtl field

This change augments commit spiffe@3c5e450

Signed-off-by: Dennis Gove <[email protected]>
Signed-off-by: Guilherme Carvalho <[email protected]>
* Add missing CanReAttest agent listing filter and output mask

Signed-off-by: Guilherme Carvalho <[email protected]>

---------

Signed-off-by: Guilherme Carvalho <[email protected]>
* Introduce an API for key management

Introduces an API that allows callers to manage key
preparation/activation/rotation/revocation in SPIRE. 
Update bundle API to propagate tainted keys.
Add an endpoint to Agent API to post current agent status.

---------

Signed-off-by: Evan Gilman <[email protected]>
Signed-off-by: Marcos Yacob <[email protected]>
Co-authored-by: Evan Gilman <[email protected]>
* Update get local authorities to return authorities by state instead of a list
* Remove Status from responses

Signed-off-by: Marcos Yacob <[email protected]>
* New SyncAuthorizedEntries RPC

Signed-off-by: Andrew Harding <[email protected]>

* Fix up comments

Signed-off-by: Andrew Harding <[email protected]>

* more comments

Signed-off-by: Andrew Harding <[email protected]>

---------

Signed-off-by: Andrew Harding <[email protected]>
@edwbuck
Copy link
Contributor Author

edwbuck commented Feb 13, 2024

@evan2645 This is ready for review.

Copy link
Member

@azdagron azdagron left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, @edwbuck! This is pared down nicely. Mostly just some nitpicks around naming and other conventions.

proto/spire/api/agent/logger/v1/logger.proto Outdated Show resolved Hide resolved
proto/spire/api/types/logger.proto Outdated Show resolved Hide resolved

message SetLogLevelRequest {

enum SetValue {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd suggest the LogLevel enum be lifted from the Logger message in the types definition and reused here instead of duplicating this for each API. Maybe the LOG_LEVEL_UNSPECIFIED value could be interpreted as "set it to the default"?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The main problem with reusing the types.LOG_LEVEL it is that there's no good way (easy to understand, easy to implement, easy to maintain) to add the additional value to an existing enum. This is why I created a SetValue enum instead of reusing the LogLevel with an additional "reset to config value" field. (The combination of the two would then require use to ignore one or the other when they are both set).

I don't think we should use LOG_LEVEL_UNSPECIFIED. That's a default within logrus that carries the meaning of "the log level was not set yet". It is used to handle detection of un-configured loggers in scenarios where file handlers and other items might be configured prior to the logger having a configured log level. In addition, UNSPECIFIED has the effective value of INFO, which means we would have to be very careful not to forward it to logrus, or risk getting the wrong log level.

Also this is identical to the cli options, so it basically keeps the CLI small and accurate to the options, making the parity between the CLI and the API 100%. If we ever expand this to include REST calls, such parity will be appreciated in the need to keep the CLI and API documentation nearly identical.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IMO, the API definition is a high-level representation of a concept, in this case, facilities to observe and adjust the log level of SPIRE core. The API definition of a log level should represent the log levels we want to support at a high level, not necessarily the log levels supported by the underlying logger implementation. Trying to align the definitions of the two should be a non-goal. We currently use logrus but one day we may not (logrus is no longer maintained). I very much anticipate a function as part of the API implementation that converts between the API definition of a log level to that of the (current) logger implementation in SPIRE.

From that perspective, I wouldn't worry too much about having to combine two enums or try to add a value.

Using "LOG_LEVEL_UNSPECIFIED" on SetLogLevel as "go back to the default" has some elegance in my opinion, but there are other ways we can represent "go back to the default" that don't rely on duplicating the log levels for the sake of adding a "DEFAULT" value just for the sake of the SetLogLevel call.

For example, you could two fields on SetLogLevel:

LogLevel log_level = 1;
bool unset = 2;

If unset is true, log_level is ignored and the value is unset. You could fail the API call with InvalidArgument if log_level was specified and unset was true.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd like to follow (as much as it makes sense) a policy of "no disinformation" in the API.

  • UNSPECIFIED means we don't detail what log level to set, and that's not what we are doing, we are detailing the log level to be set to the configured-at-launch level.

  • UNSET means that we are clearing out the log level setting, and no logging system that I'm aware of permits a nil log level, again we are setting it to the configured-at-launch level setting.

  • Changing the word DEFAULT to CONFIGURED_AT_LAUNCH, LAUNCH_VALUE, or creating a new GRPC message to handle the action of setting to the configured-at-launch value are all approaches that are far much more difficult to misinterpret, even if documentation isn't available.

Trying to align the definitions of the two should be a non-goal.

I fully agree, and it's a good justification to not use UNSPECIFIED. It's also a good justification to not use Logrus Log Levels, except that the overriding purpose of the API is to set Logrus Log Levels, so at least some parity between the two is likely to reduce confusion.

I could set two fields, but then everyone will have to maintain the documentation on how to interpret the two fields. There's basically three ways out of this:

  1. Create two enums (one for setting, one for reply)
  2. Create two messages (one for setting to a specified log level, one for setting to the configured-at-launch log level)
  3. Create two fields in one shared message (documenting on how the two interact, programming the code to match the documentation, and creating load on the future clients to generate a message containing both settings with the matching logic)

As you can see, option 3 puts more long lasting load on the maintenance, because while it will be nigh impossible to mess up the logic for options 1 or 2, it will be easier to mess up the logic for option 3, and it requires matching logic in the cli and grpc service. In the event that we expose this API to JSON style REST requests, it will then also require the clients to have matching "set both according to these rules" logic.

And yes, I wrote the code using your approach first, and after a few days realized that it was more elegant (doing more with less) after I merged the two fields into one enum.

Please reconsider your focus on option 3. I would happily implement options 1 or 2 to address your concerns.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Per offline conversation with Andrew:

  • Add an UNSPECIFIED enum value for the log level enum's zero index, so we can tell if the value wasn't set.

After some back-and-forth, we both decided than an additional ResetLogLevel message (for the server and the agent APIs) would be preferable to having a combination of fields in a single SetLogLevel message that then required additional logic / rules to resolve if the log level was both set to a value and reset simultaneously.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@azdagron I would like to close this conversation as resolved, unless there are action items within it you still need to see in the PR.

proto/spire/api/types/logger.proto Outdated Show resolved Hide resolved
proto/spire/api/agent/logger/v1/logger.proto Outdated Show resolved Hide resolved
proto/spire/api/server/logger/v1/logger.proto Outdated Show resolved Hide resolved
message GetLoggerRequest {
}

// The requested log level the Logger should assume.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: this comment seems out of place

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changed to // Set Log Level Request message.


}

// Empty Get Logger Request :essage for future extension
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

typo

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gotta love vim, changed to message

message SetLogLevelRequest {

// The new level the logger should assume
spire.api.types.LogLevel set_level = 1;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
spire.api.types.LogLevel set_level = 1;
spire.api.types.LogLevel log_level = 1;

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd rather keep it as "set_level" as "log_level" can confuse input / output orientation of the value.

If set_level seems wrong, anything indicating a request would be fine by me. I just don't want the stammering of saying it's a log level type and then having nothing else to say except it's a log level (which is captured by the type).

message SetLogLevelRequest {

// The new level the logger should assume
spire.api.types.LogLevel set_level = 1;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
spire.api.types.LogLevel set_level = 1;
spire.api.types.LogLevel log_level = 1;

Copy link
Contributor Author

@edwbuck edwbuck Feb 22, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd rather keep it as "set_level" as "log_level" can confuse input / output orientation of the value.

If set_level seems wrong, anything indicating a request would be fine by me. I just don't want the stammering of saying it's a log level type and then having nothing else to say except it's a log level (which is captured by the type).

some options that are more expressive than log_level

  • set_level
  • new_level
  • requested_level

I know "set" is often overused, but it might fit the bill here. Let me know what you think.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Resolved in an offline chat: new_level will the the name.


// Gets the logger level.
//
// The caller must be local or present an admin X509-SVID.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The agent doesn't have the same authorization mechanisms as the server. This API will only be available over UDS. Probably appropriate to just remove this sentence (for all 3 RPCs) since anybody with access to the private admin socket has permissions to execute any function.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll alter all of the agent commentary to say something about it being serviced on the admin socket.


}

// Empty Get Logger Request :essage for future extension
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

typo

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed. Thank you.

proto/spire/api/types/logger.proto Show resolved Hide resolved
azdagron
azdagron previously approved these changes Feb 22, 2024
@azdagron azdagron changed the base branch from main to next February 22, 2024 18:17
@azdagron azdagron dismissed their stale review February 22, 2024 18:17

The base branch was changed.

@azdagron azdagron changed the base branch from next to main February 22, 2024 18:18
@azdagron
Copy link
Member

One last hiccup here @edwbuck, that I didn't notice until now. All PRs to these SDKs should be done against the next branch, as mentioned here.

Can you rebase this against next and then change to base of this PR to target next?

New Data Types

- Logger : a representation of a logger
  - name : the logger's name, empty for the root logger
  - current_level : the current log level of the logger
  - default_level : the log level of the logger configured
                    in the application's config file

- LoggerMask : which Logger fields must be populated on a
               query response
  - current_level : populate the current log level
  - default_level : populate the default log level

Single Logger Operations:

- GetLogger    : Fetches the log level of the logger
  - Input: name, the logger name
    (optional, unset is root logger)
  - Input: output_mask, the desired logger output fields
    (optional, default is all fields)
  - Output: a Logger record

- AdjustLogger : Changes the log level of the logger
  - Input: name, the logger name
    (optional, unset is the root logger)
  - Input: log_level, the logger's new LogLevel
    (required, unset is error)
  - Output: none

Plural Logger Operations:

- CountLoggers : Returns the number of loggers
  - Input: none
  - Output: the number of loggers

- ListLoggers : Show Logger details for a number of loggers
  - Input: filter, the control over which loggers are returned.
    (optional, unset is the root logger with all sub-loggers)
    - filter.by_name : the name of the logger forming the root
                       of the returned loggers
    - filter.with_subloggers : a flag indicating if sub-loggers
                               are to be included in the response
  - Input: output_mask, the fields to be returned in the output
    - output_mask.current_level : show the current log level of
                                  the logger
    - output_mask.default_level : show the log level of the logger
                                  configured in the process config
                                  file

Signed-off-by: Edwin Buck <[email protected]>
Remove the LoggerMask type and support for it in GetLogger and
ListLogger RPCs.  The Logger type is only a string and two enums (ints)
so making the enums optional doesn't add up to significant savings
compared to the message complexity it would add.

Remove the CountLoggers RPC.  Users of the API can use the ListLoggers
RPC and then count the returned entries.  This simplifies the interface
and removes any count mismatches between a call to CountLoggers and
ListLoggers.

Remove the ListLoggerRequest sub-message Filter, promoting the name and
with_subloggers options into the ListLoggerRequest.  This simplifies the
request by not burdening it with a sub-message.

Fixed all the comments, they previously had references to the server's
agent API, the inspiration of the original API structure.

Signed-off-by: Edwin Buck <[email protected]>
As the context of the loggers is now reduced to only the log level
of the single logrus.Logger, it no longer makes sense to overload
the API namespace with context about how it's being adjusted.

Signed-off-by: Edwin Buck <[email protected]>
By returning the post-set logger state, we can avoid the need to
get the logger afterwards to verify the log level change.

Signed-off-by: Edwin Buck <[email protected]>
These adjustments improve the JSON output and simplify the
adjustment value settings.

Signed-off-by: Edwin Buck <[email protected]>
Change the log_level field in the SetLogLevel message to
set_level to better express its intent.

Signed-off-by: Edwin Buck <[email protected]>
To handle the removal of the SetLevel enums, a new message was
added 'ResetLogLevel' to the server and agent logger v1 apis.

Additionally, added the UNSPECIFIED zero-value for the enum
in log level enum.  This aligns with GRPC best practices.

Signed-off-by: Edwin Buck <[email protected]>
Fixed misspelling of message.
Altered out-of-place commentary for the Agent RPC.
Added comment to Server SetLogLevelRequest
Altered comment on Agent SetLogLevelRequest

Signed-off-by: Edwin Buck <[email protected]>
The name new_level better expresses the intent of the field, set_level
is not quite as expressive.

Signed-off-by: Edwin Buck <[email protected]>
@edwbuck
Copy link
Contributor Author

edwbuck commented Feb 22, 2024

work resubmitted against desired next branch with #57

@edwbuck edwbuck closed this Feb 22, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add UDS API for post-launch log level tuning.
8 participants