Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix augmentations and matplotlib #1179

Merged
merged 4 commits into from
May 25, 2024
Merged

Fix augmentations and matplotlib #1179

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
079ecaf
* skip augmentations when list is empty
DocGarbanzo May 21, 2024
7ec7211
* fix layout height for ui widgets in macos (i.e. platform darwin)
DocGarbanzo May 21, 2024
1d29961
* pin pytest to 8.1 for non-backwards compatible change
DocGarbanzo May 21, 2024
488fdc1
Fix test selection to correctly identify RPi hardware as 'aarch64' an…
DocGarbanzo May 23, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 1 addition & 2 deletions donkeycar/config.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -66,8 +66,7 @@ def to_pyfile(self, path):
def load_config(config_path=None, myconfig="myconfig.py"):

if config_path is None:
import __main__ as main
main_path = os.path.dirname(os.path.realpath(main.__file__))
main_path = os.getcwd()
config_path = os.path.join(main_path, 'config.py')
if not os.path.exists(config_path):
local_config = os.path.join(os.path.curdir, 'config.py')
Expand Down
5 changes: 3 additions & 2 deletions donkeycar/management/ui/common.kv
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
#:import platform sys.platform

#:set common_height 30
#:set reduced_height 40 if platform == 'darwin' else 20
#:set common_height 60 if platform == 'darwin' else 30
#:set spacing 10
#:set layout_height common_height + spacing

Expand Down Expand Up @@ -226,7 +227,7 @@
current_field: data_spinner.text
BoxLayout:
size_hint_y: None
height: 20
height: reduced_height
spacing: 2
MyLabel:
id: label
Expand Down
6 changes: 3 additions & 3 deletions donkeycar/management/ui/pilot_screen.kv
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -104,7 +104,7 @@
pilot_loader: pilot_loader
DataPanel:
size_hint_y: None
height: 20 + (self.minimum_height - 20) * 15
height: reduced_height + (self.minimum_height - reduced_height) * 15
id: data_panel
is_linked: True
font_color: [0.2, 0.2, 1, 1]
Expand All @@ -126,7 +126,7 @@
BackgroundBoxLayout:
orientation: 'vertical'
size_hint_y: None
height: self.minimum_height +60
height: self.minimum_height + 2 * common_height
Header:
title: 'Multi Pilot Loader'
description:
Expand Down Expand Up @@ -217,7 +217,7 @@
BackgroundBoxLayout:
orientation: 'vertical'
size_hint_y: None
height: self.minimum_height + 90
height: self.minimum_height + 3 * common_height
Slider:
size_hint_y: None
height: common_height
Expand Down
2 changes: 1 addition & 1 deletion donkeycar/management/ui/train_screen.kv
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -123,7 +123,7 @@
orientation: 'vertical'
spacing: spacing
size_hint_y: None
height: self.minimum_height + 80
height: self.minimum_height + 4 * reduced_height
Header:
size_hint_y: 1.5
id: cfg_header
Expand Down
2 changes: 1 addition & 1 deletion donkeycar/management/ui/ui.kv
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ BoxLayout:
ActionView:
id: av
size_hint_y: None
height: 20
height: reduced_height

ActionPrevious:
with_previous: False
Expand Down
2 changes: 2 additions & 0 deletions donkeycar/pipeline/augmentations.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,8 @@ def create(cls, aug_type: str, config: Config, prob, always) -> \

# Parts interface
def run(self, img_arr):
if len(self.augmentations) == 0:
return img_arr
aug_img_arr = self.augmentations(image=img_arr)["image"]
return aug_img_arr

2 changes: 1 addition & 1 deletion donkeycar/pipeline/training.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -153,7 +153,7 @@ def train(cfg: Config, tub_paths: str, model: str = None,
assert val_size > 0, "Not enough validation data, decrease the batch " \
"size or add more data."
logger.info(f'Train with image caching: '
f'{getattr(cfg, "CACHE_IMAGES", True)}')
f'{getattr(cfg, "CACHE_IMAGES", "ARRAY")}')
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@DocGarbanzo I don't know that this means. Does this limit the size of the cache?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the previous PR, we have upgraded the CACHE_IMAGES config parameter to allow for 3 values, i.e NOCACHE, ARRAY and BINARY, please see here: https://github.com/autorope/donkeycar/pull/1173/files#diff-8a130617f883f549bf089e3ab0c46cabd4c3e4a69b50568c3b27ef73414e7c28. ARRAY represents the original setting where the whole image is cached as a np.uint8 array (consuming about 58k of mem ~ 120 * 160 * 3). With BINARY we can now cache the image as binary jpeg encoded which uses only 4k of memory. This caching still provides significantly faster access to the image than re-reading it from disk, even with a fast M2 SSD. With that much smaller memory footprint we can now train >> 100k images with caching, speeding up the training on large datasets by a lot. In this line here, I just fixed a left-over from the previous PR and here, it only affects logging the caching status into the shell, because the default changed from the binary True to the enum/string 'ARRAY'.

history = kl.train(model_path=model_path,
train_data=dataset_train,
train_steps=train_size,
Expand Down
2 changes: 1 addition & 1 deletion donkeycar/tests/setup.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@


def on_pi():
if 'arm' in platform.machine():
if 'aarch64' in platform.machine():
return True
return False

Expand Down
10 changes: 5 additions & 5 deletions setup.cfg
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,7 @@ pi =
flatbuffers==24.3.*
tensorflow-aarch64==2.15.*
opencv-contrib-python
matplotlib
matplotlib==3.8.*
kivy
kivy-garden.matplotlib
pandas
Expand All @@ -73,24 +73,24 @@ nano =

pc =
tensorflow==2.15.*
matplotlib
matplotlib==3.8.*
kivy
kivy-garden.matplotlib
pandas
plotly
albumentations

macos =
tensorflow-macos==2.15.*
matplotlib
tensorflow==2.15.*
matplotlib==3.8.*
kivy
kivy-garden.matplotlib
pandas
plotly
albumentations

dev =
pytest
pytest==8.1.*
pytest-cov
responses
mypy
Expand Down
Loading