Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Turn off game specific behaviors based on time #13

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

BartekCupial
Copy link

It turns out that the game is somehow different based on local time https://nethackwiki.com/wiki/Time. For reproducibility and evaluation reasons we shouldn't allow this changes. For example my agent score dropped from 5k to 3k just because it was full moon.

I turned off all four such cases. moon phases, friday 13th, midnight, and night.

@BartekCupial BartekCupial changed the title turn off game specific behaviors based on time Turn off game specific behaviors based on time May 24, 2024
@heiner
Copy link
Owner

heiner commented May 24, 2024

Not a fan of just disabling it -- If people feel they want to have this part reproducible but setting the time to something static isn't possible in their setup (understandable on both counts) we'd need to add a hook to optionally override getlt().

But also if the goal is to beat the real NetHack we should perhaps not dumb it down too much.

@Miffyli
Copy link

Miffyli commented Jun 22, 2024

Randomly pitching in (saw that NLE is getting some love again :) ).

Agree with both here. The reliance on time is indeed very painful for running any experiments, and it is easy to miss unless you go deeper into Nethack functionality. This was a painful lesson during the NLE competition 😅

But also completely replacing it won't cut it, as it can be unexpected to people who know the logic.

I would prefer informing users about this logic more in docs. Additionally, if "completing Nethack" is baked into Gym/Gymnsium-like environments, the environment should specify the same game-time with a hook heiner suggested.

@StephenOman StephenOman added the enhancement New feature or request label Jun 22, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants