[Feature Request] Implement CrossQ

[danielpalen]

🚀 Feature

I would like to implement CrossQ (https://openreview.net/pdf?id=PczQtTsTIX) in SB3, as also suggested by @araffin (araffin/sbx#36 (comment)),

Motivation

CrossQ is one of the current state-of-the-art deep reinforcement learning methods in terms of sample efficiency and substantionally more computationally efficefficient than the previous state-of-the-art (e.g. DroQ or REDQ), as it uses a low update-to-data ratio of 1. It is the first successful application of batch normalization within deep reinforcement learning, which is at the heart of it's efficiency. I think a PyTorch based reference implementation in SB3 would be very valuable for the research community.

Pitch

As one of the first authors on the paper, I want to contribute a PyTorch based reference implementation of CrossQ to SB3, since the paper's implementation is in JAX.

Alternatives

No response

Additional context

No response

Checklist

• I have checked that there is no similar issue in the repo
• If I'm requesting a new feature, I have proposed alternatives