Skip to content

Commit

Permalink
Merge pull request #759 from allegro/develop
Browse files Browse the repository at this point in the history
Release 0.11.3
  • Loading branch information
adamdubiel authored Apr 14, 2017
2 parents ba55b43 + 812a305 commit cbad73b
Show file tree
Hide file tree
Showing 51 changed files with 784 additions and 114 deletions.
54 changes: 54 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,57 @@
## 0.11.3 (13.04.2017)

### Features

#### ([753](https://github.com/allegro/hermes/issues/753)) Filtering by headers

Added new filter type: `header` that allows on filtering messages by HTTP headers. Example of filter definition:

```
{"type": "header", "header": "My-Propagated-Header", "matcher": "^abc.*"}
```

Mind that by default no headers are propagated from Frontend to Consumers. To enable headers propagation, define and register
own `HeadersPropagator` via `HermesFrontend.Builder#withHeadersPropagator`.

#### ([749](https://github.com/allegro/hermes/pull/749)) Handling `avro/json` content type

When converting messages from JSON to Avro Hermes uses [json-avro-converter](https://github.com/allegro/json-avro-converter)
to provide smooth experience, that does not require changing already produced JSONs (for instance to to support optional
fields).

However in some rare cases it might be desired to send JSON messages that are compatible with
[standard Avro JSON encoding](https://avro.apache.org/docs/1.8.1/spec.html#json_encoding). To use vanilla JSON -> Avro converter and bypass `json-avro-converter`, send requests with `avro/json` content type.

### Enhancements

#### ([748](https://github.com/allegro/hermes/pull/748)) Topic authorization controls and status in Console

Console now has support for toggling auth on topics. This is an opt-in feature, enable by specifying:

```
{
"topic": {
"authEnabled": true,
}
}
```

In Console `config.json`.

#### ([710](https://github.com/allegro/hermes/issues/710)) Limit size of messages in preview

#### ([751](https://github.com/allegro/hermes/pull/751)) Move Zookeeper cache update logs to DEBUG level

#### ([750](https://github.com/allegro/hermes/pull/750)) Move Schema Registry cache refresh logs to DEBUG level

### Bugfixes

#### ([737](https://github.com/allegro/hermes/issues/737)) Updating subscription in hermes-console resets OAuth password

#### ([734](https://github.com/allegro/hermes/issues/734)) Prevent manual setting of subscription state to PENDING

#### ([743](https://github.com/allegro/hermes/pull/743)) Better defaults for max-rate algorithm

## 0.11.2 (15.03.2017)

### Enhancements
Expand Down
2 changes: 1 addition & 1 deletion build.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,7 @@ allprojects {
guava : '20.0',
jackson : '2.8.3',
jersey : '2.23.2',
jetty : '9.3.6.v20151106',
jetty : '9.3.18.v20170406',
curator : '2.11.1',
wiremock : '1.58',
fongo : '1.6.1',
Expand Down
113 changes: 113 additions & 0 deletions docs/docs/configuration/publishing-security.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,113 @@
# Publishing security

Hermes has a feature to restrict publishing on specific topics to particular services: [publishing permissions](/user/permissions).
How service name is extracted from request is entirely configurable and depends on security requirements within company's ecosystem.

Since Hermes uses Undertow as http server authentication implementation follows Undertow
internal security model which is described here: [Undertow Security](http://undertow.io/undertow-docs/undertow-docs-1.3.0/#security).

Authentication has to be enabled using following configuration:

Option | Description | Options | Default value
------------------------------- | --------------------------------------------------- | ----------------------------- | -------------
frontend.authentication.enabled | enable authentication handler | true, false | false
frontend.authentication.mode | in which circumstances perform authentication | constraint_driven, pro_active | constraint_driven

More about authentication mode can be read here: [AuthenticationMode](http://undertow.io/javadoc/1.3.x/io/undertow/security/api/AuthenticationMode.html)

## Implementing IdentityManager

Authentication is handled by `IdentityManager` exposed via `AuthenticationConfiguration` from `HermesFrontend.Builder`.
Hermes by default do not come with any concrete implementation so it has to be coded.

Frontend is extracting service name from `HttpServerExchange` using following method:
```java
String serviceName = exchange.getSecurityContext().getAuthenticatedAccount().getPrincipal().getName();
```

Worth mentioning is that authenticated account has to contain role `Roles.PUBLISH` otherwise frontend will return 403
without evaluating topic specific authorisation configuration. This behavior can be used for instance to reject requests from
development environment on production.

Below you can see naive implementation of authentication via `Authorization` header used in our integration tests.

```java
Predicate<HttpServerExchange> isAuthenticationRequiredPredicate = exchange -> true;
AuthenticationMechanism basicAuth = new BasicAuthenticationMechanism("basicAuthRealm");
builder.withAuthenticationConfiguration(new AuthenticationConfiguration(
isAuthenticationRequiredPredicate,
Arrays.asList(basicAuth),
new SingleUserAwareIdentityManager("John", "12345")
));
```

```java
public class SingleUserAwareIdentityManager implements IdentityManager {
private final String username;
private final String password;

SingleUserAwareIdentityManager(String username, String password) {
this.username = username;
this.password = password;
}

@Override
public Account verify(Account account) {
return null;
}

@Override
public Account verify(String username, Credential credential) {
String password = new String(((PasswordCredential) credential).getPassword());
if (this.username.equals(username) && this.password.equals(password)) {
return new SomeUserAccount(username);
}
return null;
}

@Override
public Account verify(Credential credential) {
return null;
}

private final class SomeUserAccount implements Account {
private final Principal principal;

private SomeUserAccount(String username) {
this.principal = new SomeUserPrincipal(username);
}

@Override
public Principal getPrincipal() {
return principal;
}

@Override
public Set<String> getRoles() {
return Collections.singleton(Roles.PUBLISHER);
}
}

private final class SomeUserPrincipal implements Principal {
private final String username;

private SomeUserPrincipal(String username) {
this.username = username;
}

@Override
public String getName() {
return username;
}
}

static Map<String, String> getHeadersWithAuthentication(String username, String password) {
String credentials = username + ":" + password;
String token = "Basic " + Base64.encodeBase64String(credentials.getBytes(StandardCharsets.UTF_8));

Map<String, String> headers = new HashMap<>();
headers.put("Authorization", token);
return headers;
}
}
```
4 changes: 3 additions & 1 deletion docs/docs/configuration/rate-limiting.md
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,8 @@ and we always preserve a *MIN_MAX_RATE* for a consumer.
- Recalculation is done every *BALANCE_INTERVAL* seconds.
- Consumer updates it's delivery attempt rate history every *UPDATE_INTERVAL*
if the change from previous recorded value is greater than *MIN_SIGNIFICANT_CHANGE_PERCENT*.
- *MIN_SIGNIFICANT_CHANGE_PERCENT* / 100 must be lower than *BUSY_TOLERANCE*, as otherwise
consumers would (in some cases) not enter busy state
- At the moment *RATE_HISTORY_SIZE* is ignored, defaulting to 1,
and might be used in future versions of the algorithm

Expand All @@ -71,4 +73,4 @@ RATE_HISTORY_SIZE | consumer.maxrate.history.size
BUSY_TOLERANCE | consumer.maxrate.busy.tolerance | 0.1
MIN_MAX_RATE | consumer.maxrate.min.value | 1.0
MIN_CHANGE_PERCENT | consumer.maxrate.min.allowed.change.percent | 1.0
MIN_SIGNIFICANT_CHANGE_PERCENT | consumer.maxrate.min.significant.update.percent | 10.0
MIN_SIGNIFICANT_CHANGE_PERCENT | consumer.maxrate.min.significant.update.percent | 9.0
37 changes: 36 additions & 1 deletion docs/docs/user/permissions.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,11 +26,46 @@ Add new topic | any logged in user
Remove existing topic | **topic owner**
Modify topic | **topic owner**

### Publishing permission

You can configure which services can publish on which topic configuring **topic.auth** section.
How publisher name is evaluated is deployment specific. It can be extracted from ssl certificate or read from supplied header for instance.
Worth noting is that for authorization features to work those have to be enabled on your hermes cluster by your administrator.


Option | Description | Options | Default value
---------------------------- | --------------------------------------------------- | ----------- | -------------
enabled | enable topic authorization | true, false | false
unauthenticatedAccessEnabled | allow publishing for services without credentials | true, false | false
publishers | array of service names that are allowed to publish | - | []

Example:

```json
{
"name": "my-group.my-topic",
"description": "This is my topic",
"contentType": "JSON",
"retentionTime": {
"duration": 1
},
"owner": {
"source": "Plaintext",
"id": "My Team"
},
"auth": {
"enabled": true,
"unauthenticatedAccessEnabled": false,
"publishers": ["my-publisher-1", "my-publisher-2"]
}
}
```

## Subscriptions

Operation | Permissions
---------------------------- | -----------
Add new subscription | any logged in user
Remove existing subscription | **subscription owner** or **topic owner**
Modify subscription | **subscription owner** or **topic owner**
Retransmit messages | **subscription owner** or **topic owner**
Retransmit messages | **subscription owner** or **topic owner**
Original file line number Diff line number Diff line change
Expand Up @@ -3,4 +3,6 @@
public class AvroMediaType {

public final static String AVRO_BINARY = "avro/binary";

public final static String AVRO_JSON = "avro/json";
}
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,10 @@ public String getMatcher() {
return getStringValue("matcher");
}

public String getHeader() {
return getStringValue("header");
}

@SuppressWarnings("unchecked")
public <T> T getFieldValue(String key) {
return (T) spec.get(key);
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
package pl.allegro.tech.hermes.api;

public class MessageTextPreview {

private final String content;

private final boolean truncated;

public MessageTextPreview(String content, boolean truncated) {
this.content = content;
this.truncated = truncated;
}

public String getContent() {
return content;
}

public boolean isTruncated() {
return truncated;
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -11,24 +11,24 @@ public class PublishingAuth {

private final List<String> publishers;
private final boolean enabled;
private final boolean unauthorisedAccessEnabled;
private final boolean unauthenticatedAccessEnabled;

@JsonCreator
public PublishingAuth(@JsonProperty("publishers") List<String> publishers,
@JsonProperty("enabled") boolean enabled,
@JsonProperty("unauthorisedAccessEnabled") boolean unauthorisedAccessEnabled) {
@JsonProperty("unauthenticatedAccessEnabled") boolean unauthenticatedAccessEnabled) {

this.publishers = publishers;
this.enabled = enabled;
this.unauthorisedAccessEnabled = unauthorisedAccessEnabled;
this.unauthenticatedAccessEnabled = unauthenticatedAccessEnabled;
}

public boolean isEnabled() {
return enabled;
}

public boolean isUnauthorisedAccessEnabled() {
return unauthorisedAccessEnabled;
public boolean isUnauthenticatedAccessEnabled() {
return unauthenticatedAccessEnabled;
}

public boolean hasPermission(String publisher) {
Expand All @@ -53,12 +53,12 @@ public boolean equals(Object o) {
}
PublishingAuth that = (PublishingAuth) o;
return enabled == that.enabled
&& unauthorisedAccessEnabled == that.unauthorisedAccessEnabled
&& unauthenticatedAccessEnabled == that.unauthenticatedAccessEnabled
&& Objects.equals(publishers, that.publishers);
}

@Override
public int hashCode() {
return Objects.hash(publishers, enabled, unauthorisedAccessEnabled);
return Objects.hash(publishers, enabled, unauthenticatedAccessEnabled);
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -192,8 +192,8 @@ public boolean isAuthEnabled() {
}

@JsonIgnore
public boolean isUnauthorisedAccessEnabled() {
return publishingAuth.isUnauthorisedAccessEnabled();
public boolean isUnauthenticatedAccessEnabled() {
return publishingAuth.isUnauthenticatedAccessEnabled();
}

public boolean hasPermission(String publisher) {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -126,6 +126,7 @@ public enum Configs {
FRONTEND_AUTHENTICATION_MODE("frontend.authentication.mode", "constraint_driven"),

FRONTEND_MESSAGE_PREVIEW_ENABLED("frontend.message.preview.enabled", false),
FRONTEND_MESSAGE_PREVIEW_MAX_SIZE_KB("frontend.message.preview.max.size.kb", 10),
FRONTEND_MESSAGE_PREVIEW_SIZE("frontend.message.preview.size", 3),
FRONTEND_MESSAGE_PREVIEW_LOG_PERSIST_PERIOD("frontend.message.preview.log.persist.period.seconds", 30),

Expand Down Expand Up @@ -179,7 +180,7 @@ public enum Configs {
CONSUMER_MAXRATE_BUSY_TOLERANCE("consumer.maxrate.busy.tolerance", 0.1),
CONSUMER_MAXRATE_MIN_MAX_RATE("consumer.maxrate.min.value", 1.0),
CONSUMER_MAXRATE_MIN_ALLOWED_CHANGE_PERCENT("consumer.maxrate.min.allowed.change.percent", 1.0),
CONSUMER_MAXRATE_MIN_SIGNIFICANT_UPDATE_PERCENT("consumer.maxrate.min.significant.update.percent", 10.0),
CONSUMER_MAXRATE_MIN_SIGNIFICANT_UPDATE_PERCENT("consumer.maxrate.min.significant.update.percent", 9.0),

CONSUMER_HEALTH_CHECK_PORT("consumer.status.health.port", 8000),
CONSUMER_WORKLOAD_ALGORITHM("consumer.workload.algorithm", "selective"),
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
package pl.allegro.tech.hermes.domain.topic.preview;

import com.fasterxml.jackson.annotation.JsonCreator;
import com.fasterxml.jackson.annotation.JsonProperty;

public class MessagePreview {

private final byte[] content;

private final boolean truncated;

@JsonCreator
public MessagePreview(@JsonProperty("content") byte[] content, @JsonProperty("truncated") boolean truncated) {
this.content = content;
this.truncated = truncated;
}

public MessagePreview(byte[] content) {
this(content, false);
}

public byte[] getContent() {
return content;
}

public boolean isTruncated() {
return truncated;
}
}
Loading

0 comments on commit cbad73b

Please sign in to comment.