Skip to content

Latest commit

 

History

History
94 lines (86 loc) · 5.1 KB

TODO.md

File metadata and controls

94 lines (86 loc) · 5.1 KB
Raw

Main Branch

  • ✓ (Remote environment like OpenAI Universe): Support interaction with environment hosted on a remote server. To enable training RL agents on GPU servers without KVM enabled.
  • ✓ (Partial) (Easier Deployment): Maybe we can construct a docker to simplify the deployment of Mobile-Env. Even I feel that it is too complicated to set up an Android environment before install Mobile-Env. However, a docker won't help KVM configuration, either. Under consideration.
  • ✓ (VH Node Specification): Using CSS selector to locate a VH node should be easier than the current way
  • ✓ (Repeatability control for event slots): I find it useful to control repeatability of an event slot. Will implement it when I have the time.
  • ✓ (About new event source RHU): Update the annotation tool to support the new type of event source.
  • ✓ (JPEG compression for transfer): Applied JEPG compression before transferring the screen to RemoteSimulator
  • ✓ (Fuzzy Match): Maybe fuzzy match in text event sources will help. Maybe through rapidfuzz library.
  • ✓ (Beyond dm_env interfaces): Will follow new gymnasium to distinguish successful finishing and error termination. Will give a DmEnvWrapper instead for back-compatibility.
  • ✓ (Migration to Gymnasium): Migrate GymInterfaceWrapper from deprecated gym to gymnasium.
  • ✓ (Temporary Storage for Event Values): I think this will be cool if we can temporarily store the submitted value of a virtual event and recall it in later steps. This will enable interactions between heterochronic events. For example, the task manager can note a piece of key information during execution and compare it with a later response. This is cool.
  • ✓ (Non-ASCII character input through clipboard): I found that clipboard synching can copy UTF-8 strings. This may be an idea to enable non-ASCII character input. The emulator gPRC has interfaces to operate clipboard.
  • ✗ (Non-ASCII character input through emulator gPRC): sendKey interface of the emulator may be explored to implement input of UTF-8 strings.
  • (Non-ASCII character input through a dedicated input method): Follow https://blog.csdn.net/qq_37148270/article/details/109514727 to make a solution.
  • (FileSystem Events): Events arising from changes in Android file system. Just a conceptual idea.
  • (Replace pexpect with a self-composing module): As pexpect doesn't work on Windows, maybe replace it with a custom module.
  • (Other problems on Windows): The main problem seems to lie on uiautomator, or the adb shell. Maybe change it to adb exec-out will solve.
  • (Tutorial of how to manage events): I need to supplement a tutorial of how to manage events to the current document which simply lists all the fields out.
  • (Icon Model): The embedded icon model. Maybe I can just use IconNet. But, I didn't find a ready-to-use IconNet implementation. Still need to train an own one. Found some candidates for icon detection, vision-ui and UIED. However, no icon match model is found. I know that this may be a minor demand. So maybe a runnable workaround will be implemented first by wrapping these out-of-box tools. And will train our own model when I have enough time. Maybe 1 year later, when I'm ready for graduation. (> <)
  • (Unit Tests): Have a thorough check to the original unit test codes.
  • (Task Debugger): Maybe a debugger for task definition composing? I think a visualizer to the event trees is needed.
  • ✓ (Long Click): Long click action for VhIoWrapper.
  • (Accessibility Forwarding): Maybe have a try on the accessibility forwarder to ameliorate the efficiency of VH obtaining following AndroidWorld
  • (Investigate the snapshot function of Android Emulator): I want to conduct an investigation on the snapshot funciton of Android Emulator. This may help to set different initial state for tasks more easily.
  • (Simulator based on real mobile devices): Replace the implementation based on the gRPC interfaces of Android Emulator in current EmulatorSimulator to pure ADB implementations so as to form a simulator that can connect to real mobile devices.

Annotation Tool

  • (Migration to 4.0): Update the annotation-tool to better work with new episode end mechanism of 4.0.
  • (Keyboard Input for Annotation Tool): Supports free keyboard input for the web interface of annoatation tool.

Template Toolkit

  • (Method to merge into new command): Add a mode to allow to keep only the first command in the final result rather than concatenating all the intructions.

Task Sets

  • (OpenMoneyBox): I want to define a group of tasks based on OpenMoneyBox.