Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix automatic evaluation with run_trial.bash #62

Merged
merged 7 commits into from
Jun 26, 2023

Conversation

M1chaelM
Copy link
Collaborator

@M1chaelM M1chaelM commented Jun 22, 2023

This PR updates run_trial.bash and related helper scripts to get the VRX testing tutorial working again using @j-herman 's example competitor image to test.

To test, follow the updated VRX testing tutorial but use example_team as your team name so the script will use our ghost ship image as the competitor system.

Everything should work except the replay script, which will be addressed in a separate PR. The current script includes a cheap workaround for osrf/vrx#680 (see below for details).

@M1chaelM M1chaelM marked this pull request as ready for review June 22, 2023 22:58
@M1chaelM M1chaelM requested a review from caguero June 22, 2023 22:58
@M1chaelM
Copy link
Collaborator Author

It turns out that the reason the ghost ship isn't moving has to do with a bug in the core code, rather than the ghost ship or this repository. See osrf/vrx#680

A workaround is to use the practice worlds already installed in VRX. I've pushed a modification to the run_trial.bash script that does this for now. This won't work in the long run, but it should be good enough to get PR #60 tested and merged, as well as to verify that this scoring script is working.

@caguero Could you review this when you get a chance?

Copy link
Contributor

@caguero caguero left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A few minor detail from the tutorial:

In Step1->(3):

source ~/vrx_ws/devel/setup.bash

This should be:

source ~/vrx_ws/install/setup.bash

I already updated the tutorial.

The simulation starts and stops a few minutes later but the WAM-V doesn't move for me.

@M1chaelM
Copy link
Collaborator Author

Now that osrf/vrx#682 is merged I've restored the original behavior. Testing again, but it looks like fuel is broken again, so we have to wait for it to come back up.

@M1chaelM
Copy link
Collaborator Author

@caguero Looks like fuel is back up. I have added a layer to our docker image to download models from fuel so they will be cached in the image.

I also reran the trial. It works for me using both the workaround and the restored behavior. Note that the expected movement of the WAM-V in @j-herman 's systems is subtle. It waits until the task goes into "running" and then moves about a meter and stops. If you exit early or leave it running without looking at it, you won't see it move.

Could you please rebuild your image and try again? Also, please make sure you aren't using a cached image, since the VRX repo has changed. One way to do this is to uncomment the no-cache option on line 27 of vrx_server/build_image.bash.

DOCKER_ARGS="$DOCKER_ARGS --no-cache"

then rerun vrx_server/build_image.bash to rebuild.

After you run the trial for TASK=stationkeeping TRIAL=0 please check the score. It should be around 3.424.

@caguero
Copy link
Contributor

caguero commented Jun 26, 2023

Thanks for the clarifications. After looking carefully at the propellers I can confirm that it works as expected then.

@M1chaelM M1chaelM merged commit e339134 into jessica/mwes Jun 26, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants