guestagent over grpc broken with WSL2

[pendo324]

Description

I previously fixed this issue and confirmed it was working (#2118), but I was just testing off of main and found there's a new issue. Seems like the change to use gRPC somehow broke guest agent communication with WSL2 over VSOCK.

I confirmed that communication is indeed working over VSOCK using a simple socat echo server (inside the VM socat VSOCK-LISTEN:1234,fork EXEC:cat) and a simple Go program on my host:

Simple program

Comment out the test code, and then uncomment the grpc code for a simple repro.

package main

import (
	"context"
	"fmt"
	"net"

	"github.com/Microsoft/go-winio"
	"github.com/Microsoft/go-winio/pkg/guid"
	"google.golang.org/protobuf/runtime/protoimpl"
)

type IPPort struct {
	state         protoimpl.MessageState
	sizeCache     protoimpl.SizeCache
	unknownFields protoimpl.UnknownFields

	Ip   string `protobuf:"bytes,1,opt,name=ip,proto3" json:"ip,omitempty"`
	Port int32  `protobuf:"varint,2,opt,name=port,proto3" json:"port,omitempty"`
}

type Info struct {
	state         protoimpl.MessageState
	sizeCache     protoimpl.SizeCache
	unknownFields protoimpl.UnknownFields

	LocalPorts []*IPPort `protobuf:"bytes,1,rep,name=local_ports,json=localPorts,proto3" json:"local_ports,omitempty"`
}

func getCon(ctx context.Context) (net.Conn, error) {
	// This was my VM's VM ID. You can get your's by following the logic in this file:
	// https://github.com/lima-vm/lima/blob/master/pkg/windows/wsl_util_windows.go#L13
	// OR running these commands (basically the same thing)
	// Get-CimInstance Win32_Process -Filter "name = 'wslhost.exe'" | Select CommandLine | ConvertTo-Json
	// --distro-id will correlate to the registry keys under
	// Computer\HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\Lxss
	// Most likely, one is for your default distro, and one for your Lima distro.
	VMIDStr := "a2fae325-ef5c-4ecc-a78e-043ab330a3e6"
	// The port of the currently running lima-guestagent. This can be retrieved from logging
	// into the VM and running systemctl status lima-guestagent
	VSockPort := 123456
	VMIDGUID, err := guid.FromString(VMIDStr)
	if err != nil {
		fmt.Printf("Error converting to GUID %v\n", err)
	}
	sockAddr := &winio.HvsockAddr{
		VMID:      VMIDGUID,
		ServiceID: winio.VsockServiceID(uint32(VSockPort)),
	}
	fmt.Printf("Dialing vsock %s with port %d\n", VMIDStr, VSockPort)
	return winio.Dial(ctx, sockAddr)
}

func main() {
	conn, err := getCon(context.Background())
	fmt.Printf("err: %v\n", err)
	fmt.Printf("remoteAddr: %v\n", conn.RemoteAddr())
	buf := []byte("Test!")
	bytesWritten, err := conn.Write(buf)
	if err != nil {
		fmt.Printf("Error reading from conn: %v\n", err)
	}
	fmt.Printf("bytesWritten: %d, buf: %s\n", bytesWritten, buf)
	read := make([]byte, bytesWritten)
	bytesRead, err := conn.Read(read)
	if err != nil {
		fmt.Printf("Error reading from conn: %v\n", err)
	}
	fmt.Printf("bytesRead: %d, read: %s\n", bytesRead, read)

	// opts := []grpc.DialOption{
	// 	grpc.WithContextDialer(func(ctx context.Context, target string) (net.Conn, error) {
	// 		return getCon(ctx)
	// 	}),
	// 	grpc.WithTransportCredentials(insecure.NewCredentials()),
	// }

	// clientConn, err := grpc.Dial("", opts...)
	// if err != nil {
	// 	fmt.Printf("Error converting to GUID %v", err)
	// }
	// out := new(Info)
	// err = clientConn.Invoke(context.Background(), "/GuestService/GetInfo", emptypb.Empty{}, out)
	// fmt.Printf("err: %v\n", err)
	// fmt.Printf("out: %v\n", out)
}

You can monitor established vsock connections by using something like:

watch -n1 "ss | grep 123456"

If you remove the Read from the test program so it only ever writes, then the connection will linger as established, making it easier to see in the output of ss. This works even with the lima-guestagent server, which leads me to believe the issue is somehow with the grpc communication (maybe the client?) not vsock itself.

So I'm not yet sure why running Lima and trying to use the grpc connection gives the following error:

rpc error: code = Unavailable desc = connection error: desc = "error reading server preface: http2: frame too large"

[pendo324]

Weirdly, the grpc-go Hello World example works just fine over vsock (modified slightly from the upstream example).

I don't yet see how this is any different than what is being done in Lima, but I'm still investigating.

Client

/*
 *
 * Copyright 2015 gRPC authors.
 *
 * Licensed under the Apache License, Version 2.0 (the "License");
 * you may not use this file except in compliance with the License.
 * You may obtain a copy of the License at
 *
 *     http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 *
 */

// Package main implements a client for Greeter service.
package main

import (
	"context"
	"flag"
	"fmt"
	"log"
	"net"
	"time"

	"github.com/Microsoft/go-winio"
	"github.com/Microsoft/go-winio/pkg/guid"
	"google.golang.org/grpc"
	"google.golang.org/grpc/credentials/insecure"
	pb "google.golang.org/grpc/examples/helloworld/helloworld"
)

const (
	defaultName = "world"
)

var (
	name = flag.String("name", defaultName, "Name to greet")
)

func getCon(ctx context.Context) (net.Conn, error) {
	// This was my VM's VM ID. You can get your's by following the logic in this file:
	// https://github.com/lima-vm/lima/blob/master/pkg/windows/wsl_util_windows.go#L13
	// OR running these commands (basically the same thing)
	// Get-CimInstance Win32_Process -Filter "name = 'wslhost.exe'" | Select CommandLine | ConvertTo-Json
	// --distro-id will correlate to the registry keys under
	// Computer\HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\Lxss
	// Most likely, one is for your default distro, and one for your Lima distro.
	// Get-ItemProperty -Path HKCU:\Software\Microsoft\Windows\CurrentVersion\Lxss
	VMIDStr := "0acf4aab-5fa4-497d-b78e-80748d431e3c"
	// The port of the currently running lima-guestagent. This can be retrieved from logging
	// into the VM and running systemctl status lima-guestagent
	VSockPort := 123456
	VMIDGUID, err := guid.FromString(VMIDStr)
	if err != nil {
		fmt.Printf("Error converting to GUID %v\n", err)
	}
	sockAddr := &winio.HvsockAddr{
		VMID:      VMIDGUID,
		ServiceID: winio.VsockServiceID(uint32(VSockPort)),
	}
	fmt.Printf("Dialing vsock %s with port %d\n", VMIDStr, VSockPort)
	return winio.Dial(ctx, sockAddr)
}

func main() {
	flag.Parse()
	// Set up a connection to the server.
	opts := []grpc.DialOption{
		grpc.WithContextDialer(func(ctx context.Context, target string) (net.Conn, error) {
			return getCon(ctx)
		}),
		grpc.WithTransportCredentials(insecure.NewCredentials()),
	}

	conn, err := grpc.Dial("", opts...)
	if err != nil {
		log.Fatalf("did not connect: %v", err)
	}
	defer conn.Close()
	c := pb.NewGreeterClient(conn)

	// Contact the server and print out its response.
	ctx, cancel := context.WithTimeout(context.Background(), time.Second)
	defer cancel()
	r, err := c.SayHello(ctx, &pb.HelloRequest{Name: *name})
	if err != nil {
		log.Fatalf("could not greet: %v", err)
	}
	log.Printf("Greeting: %s", r.GetMessage())
}

Server

/*
 *
 * Copyright 2015 gRPC authors.
 *
 * Licensed under the Apache License, Version 2.0 (the "License");
 * you may not use this file except in compliance with the License.
 * You may obtain a copy of the License at
 *
 *     http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 *
 */

// Package main implements a server for Greeter service.
package main

import (
	"context"
	"flag"
	"log"

	"github.com/mdlayher/vsock"
	"google.golang.org/grpc"
	pb "google.golang.org/grpc/examples/helloworld/helloworld"
)

var (
	port = flag.Int("port", 50051, "The server port")
)

// server is used to implement helloworld.GreeterServer.
type server struct {
	pb.UnimplementedGreeterServer
}

// SayHello implements helloworld.GreeterServer
func (s *server) SayHello(ctx context.Context, in *pb.HelloRequest) (*pb.HelloReply, error) {
	log.Printf("Received: %v", in.GetName())
	return &pb.HelloReply{Message: "Hello " + in.GetName()}, nil
}

func main() {
	flag.Parse()
	lis, err := vsock.Listen(uint32(123456), nil)
	if err != nil {
		log.Fatalf("failed to listen: %v", err)
	}
	s := grpc.NewServer()
	pb.RegisterGreeterServer(s, &server{})
	log.Printf("server listening at %v", lis.Addr())
	if err := s.Serve(lis); err != nil {
		log.Fatalf("failed to serve: %v", err)
	}
}

Output

Client output:

PS> go run .\main.go
Dialing vsock 0acf4aab-5fa4-497d-b78e-80748d431e3c with port 123456
2024/02/16 17:30:53 Greeting: Hello world

Server output:

# ./greeter_server
2024/02/16 17:29:59 server listening at vm(4294967295):123456
2024/02/16 17:30:53 Received: world

[pendo324]

I had two different lima-guestagent.Linux-x86_64's in Lima's search path, and my instance was using the old HTTP server when I was trying to test with a newer limactl, which explains why it (rightly) gave that response.

Regarding grpc, everything works as expected on Windows using vsocks 👍. I found another couple of minor issues regarding port forwarding, but I'll just open a PR directly for that since they are minor.

[AkihiroSuda]

Is this issue closable now?

[pendo324]

Is this issue closable now?

Yeah, should be good to close now. Thanks for following up.