Image Annotation Moderation and FLUX Model Integration #65
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…age dataset annotations.
…General cleanup for PR.
benliang99
requested review from
aliang322,
dylanuys and
kenobijon
and removed request for
kenobijon
aliang322
reviewed
Sep 23, 2024
aliang322
reviewed
Sep 23, 2024
aliang322
reviewed
Sep 23, 2024
aliang322
reviewed
Sep 23, 2024
bitmind/synthetic_image_generation/notebooks/gs1_nis100_1024_1024.ipynb
Outdated
Show resolved
Hide resolved
aliang322
reviewed
Sep 23, 2024
aliang322
reviewed
Sep 23, 2024
benliang99
commented
Sep 24, 2024
| @@ -39,19 +41,36 @@ | |||
| { | |||
| "path": "stabilityai/stable-diffusion-xl-base-1.0", | |||
| "use_safetensors": True, | |||
| "torch_dtype": torch.float16, | |||
There was a problem hiding this comment.
torch_dtype is no longer hardcoded, so it must be specified in the pipeline / DIFFUSER_ARGS.
| "path": 'black-forest-labs/FLUX.1-dev', | ||
| "use_safetensors": True, | ||
| "torch_dtype": torch.bfloat16, | ||
| "generate_args": { |
There was a problem hiding this comment.
Arguments decided based on experimental generation latencies.
| } for m in VALIDATOR_MODEL_META['diffusers'] | ||
| } | ||
|
|
||
| GENERATE_ARGS = { |
There was a problem hiding this comment.
New field for generation arguments, which FLUX-1.dev can utilize.
| @@ -80,11 +140,17 @@ def generate_description(self, | |||
|
|
|||
| if not verbose: | |||
| transformers_logging.set_verbosity_info() | |||
|
|
|||
|
|
|||
| if description.startswith(prompts[0]): | |||
There was a problem hiding this comment.
Check for the moderation model repeating the prompt in its response.
| @@ -122,9 +136,13 @@ def clear_gpu(self): | |||
| torch.cuda.empty_cache() | |||
| self.diffuser = None | |||
|
|
|||
| def load_diffuser(self, diffuser_name) -> None: | |||
| def load_diffuser(self, diffuser_name, gpu_id=None) -> None: | |||
There was a problem hiding this comment.
Added GPU parameterization for image generation parallelization. Defaults to cuda if not specified (previous behavior).
bitmind/synthetic_image_generation/notebooks/gs1_nis100_1024_1024.ipynb
Outdated
Show resolved
Hide resolved
… logging cleanup.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR upgrades our synthetic image pipeline with a new state-of-the-art (SOTA) model and various quality of life enhancements. It introduces Black Forest Labs' advanced FLUX.1-dev model, upgrades our BLIP-2 annotation model from 2.7B to 6.7B parameters, and incorporates Unsloth's Meta-Llama-3.1-8B-Instruct-bnb-4bit for image moderation. Further refinements include optimized parameterization for both CPU and GPU operations, streamlined .env setup for automatic login credentials management, and verification of model loading before a validation starts.
Key Changes
Model Upgrades and Enhancements:
Infrastructure and Configuration:
min_compute.ymlto detail new GPU requirements like tflops.setup_validator_env.shandstart_validator.shto automate Weights * Biases API key and Hugging Face token user login.autoupdate_validator_steps.shsequence and executed prior tostart_validator.sh. This modification eliminates the initial download delay when models are loaded for the first time.Code Quality and Maintenance: