Http2 plugin not always switch to h2 protocol

[syampol13]

Jmeter: 5.6.2
http plugin: 2.0.5

I have an internally available service under test. Service located behind the F5 and configured to support both http1/1 & http2.
Tried access it via Jmeter using bzm http2 plugin and faced with an issue where chosen during negotiation protocol appear http1/1 while http2 expected.

Debug logs shows:

2024-02-21 15:05:36,141 DEBUG o.e.j.c.d.HttpClientTransportDynamic: ALPN negotiated http/1.1 among [http/1.1, h2c, h2]

Test plan & jmeter.log attached. Endpoint is not available externally so the TP is just to see the request being made.

Test plan also contains publicly available endpoint for different but still http2 service - bzm - HTTP2 Sampler galerabet.
This one just for comparison as this sample does work as expected. It communicates over http2 and negotiation completed with:

... ALPN negotiated h2 among [http/1.1, h2c, h2]
And the response starts with
HTTP/2.0 200 OK

Now regarding the original internal service (problematic one) - I've tried to execute exactly the same request (same endpoint, type, headers, body) via K6 and it was done via http2. Protocol check shows that used protocol was http2 and the response starts with 'HTTP/2.0 200 OK'. Tested service also captured incoming request as a http2 one. This all to make sure tested service does support http2 and does work with it

[diego-ferrand]

Hey @syampol13 i can't reproduce this issue from my end. Do you know of any public server that presents the issue that i can use to debug this?

You could enable java network logs to get more info on why it is going to http1. To enable this you can run jmeter with
-Djavax.net.debug=all.

Let me know if you find anything or can provide more details into the issue.

[syampol13]

Hi @diego-ferrand
No, unfortunately I don't have a public one.

Tried to reproduce this with javax.net.debug but this didn't help me much. Logs are attached. Please take a look, maybe you will see smth interesting. http2_dbg.zip

From my pov it sounds weird that there is a application_layer_protocol_negotiation in a ClientHello and then I can see Ignore unsupported extension: application_layer_protocol_negotiation. (there are also other unsupported extension, however I'm not familiar with such details so can't tell whether this is expected or not)
I'd suppose server side might not support h2 but if I check the HTTP1 Upgrade checkbox in the HTTP2 plugin UI or add following headers explicitly:

HTTP2-Settings:
Upgrade: h2c
Connection: Upgrade, HTTP2-Settings

Same request works via HTTP2.
Unfortunately, this is not workable workaround as this need to be done on every consequent request and this will create a new TCP connection per each such request

[3dgiordano]

Hi @syampol13

Looking at the jmeter log we can see that in the ALPN negotiation, when it is detected that it does not have ALPN, it decides to go for HTTP/1.1

2024-06-09 20:10:17,760 DEBUG o.e.j.c.d.HttpClientTransportDynamic: No ALPN protocol, using HTTP11@28c91ad9[http/1.1]

If the server does not have TLS/ALPN then it is not following the current RFC standard for HTTP2.
In your case ServerHello does not mention support for the ALPN extension (application_layer_protocol_negotiation)

There are tools that provide information about the level of HTTP2 support on a server.
https://blog.cloudflare.com/tools-for-debugging-testing-and-using-http-2/
https://tools.keycdn.com/http2-test

There are tools that allow you to directly use the protocol without involving negotiation. Probably K6 setting with prior knowledge the http2 and not use ALPN (also I remember LoadRunner do the same change ALPN for a specific protocol).
In the past our plugin did that, it directly forced the protocol to HTTP2. But that generate problems with servers that required ALPN negotiation (and is why we align the plugin with the standard). K6 don't follow the standard by default, and if you need to use more realistic simulation you need to use xk6 extensions to comply with a more complete test.

We plan to incorporate in the future that it is possible to force protocol without using negotiation (like in the past for legacy servers).
This would be for those servers that are not aligned to the current standard (for example some older servers with SPDY or older gRPC servers).

You can try version v2.0.2 or v2.0.1 of the plugin, those versions did not yet have standard-aligned support with TLS/ALPN incorporated, forcing HTTP2 or HTTP1 Upgrade without using ALPN.

Tell us if that version works with you problematic server
Just to confirm that the server does not support ALPN and it is one of the main problems with the current plugin version.

[syampol13]

Hi @3dgiordano
Tried 2.0 and indeed it does work.

[3dgiordano]

@syampol13 Great, perfect.
Great, perfect.
We still have to incorporate in the future that it may be indicated that ALPN negotiations do not need to be carried out.
We will incorporate the suggestion for improvement.

Thanks for the information provided.

[syampol13]

@3dgiordano one more note - 2.0 works but on Jmeter 5.5
When tried on jmeter 5.6.2 - any of 2.0, 2.0.1, 2.0.2 and even 2.0.3 results in no HTTP2 sample in jmeter UI. Only 2.0.4 and higher allows me to add HTTP2 element into the TP. (according to 2.0.4 description this is probably expected)

Also, for 2.0.4 & 2.0.5 - on those versions not all requests are working via http1.1. Odds are http1.1 and even are http2.0

[3dgiordano]

Hi @syampol13
Yeah that's right. JMeter 5.6.0 modified something internally and we had to fix our plugin to be compatible with JMeter 5.6.x

You cannot use any version of JMeter 5.6 in the version prior to 2.0.4 where we added support for JMeter 5.6.x,
Unfortunately we incorporated ALPN in 2.0.3, so you will have to use JMeter 5.5 or earlier until we incorporate allowing not to use ALPN..

We will try to raise the priority of this need to be able to specify that ALPN not be used in our latest version, in order to resolve this difficulty that some are having.

[syampol13]

Hi @3dgiordano
We were trying to configure our balancer (F5) to support ALPN.
According to curl & openssl - tested target server do support ALPN and the response is HTTP2. But via http2 plugin it still works via HTTP1.1

curl:

• ALPN, offering h2
• ALPN, offering http/1.1
• ..
• SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
• ALPN, server accepted to use h2

openssl:

New, TLSv1.3, Cipher is TLS_AES_128_GCM_SHA256
Server public key is 2048 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
ALPN protocol: h2

http2 plugin:

"application_layer_protocol_negotiation (16)": {
  [http/1.1, h2c, h2]
},

...
"supported_versions (43)": {
"versions": [TLSv1.3, TLSv1.2]
},
...
"ServerHello": {
"server version" : "TLSv1.2",
"random" : "0A C4 3A E4 0D 57 C7 6E 3E 41 59 2F 27 F2 C0 E5 D6 79 2F F2 B4 9F E0 18 32 EF 41 68 39 62 6B 57",
"session id" : "0F 63 BF 64 4A F7 54 5A A3 44 CA EB EC 87 8A 4F B5 12 65 4B DC 9B 19 25 11 DD E9 03 0A 42 E3 5E",
"cipher suite" : "TLS_AES_128_GCM_SHA256(0x1301)",
"compression methods" : "00",
"extensions" : [
"key_share (51)": {
"server_share": {
"named group": x25519
"key_exchange": {
0000: 43 49 42 B6 BC 37 A9 DC 17 9E 9C 4A C8 8F 98 22 CIB..7.....J..."
0010: 93 DD 80 D7 50 62 43 C8 3E 7D 21 AC 76 CA 29 31 ....PbC.>.!.v.)1
}
},
},
"supported_versions (43)": {
"selected version": [TLSv1.3]
}
]
}
)
...
javax.net.ssl|DEBUG|36|http2[https://pop-trtpopdev-mvc1.extdev.eu:8088:48]-54|2024-06-12 21:25:51.747 EEST|ServerHello.java:963|Negotiated protocol version: TLSv1.3
...
javax.net.ssl|DEBUG|36|http2[https://pop-trtpopdev-mvc1.extdev.eu:8088:48]-54|2024-06-12 21:25:51.755 EEST|EncryptedExtensions.java:171|Consuming EncryptedExtensions handshake message (
"EncryptedExtensions": [
"application_layer_protocol_negotiation (16)": {
[http/1.1]
}
]
)

Attaching curl, openssl and java.net logs - http2_debugging.zip

[3dgiordano]

Hi @syampol13

Try to see from the server logs side if there is any evidence why in the negotiation, the server decides to go for http 1.1

The implementation of SSL and its negotiation in Java may be different from that implemented in cURL and some different step in the negotiation may lead to different results.

When the client does not directly indicate the negotiation, it directly uses h2 and the server accepts it directly.
The part to investigate is why the server, when ALPN negotiation is enabled, tells the client to use http1.1.
There is probably more information on the server side and the decision it makes than on the client side.

[syampol13]

Hi @3dgiordano
Thanks, will try to investigate.
Meanwhile we came up with the workable for us solution with either removing http/1.1 from the suggested by client protocols or switch order of the protocols to try on ALPN fail in HttpClientTransportDynamic.
Do you know if any of those may cause some unpredictable results? (considering we are not expecting to test http1 via bzm-http2 plugin)