Possible enhancements & additions #53
Comments
|
Also loving the script! I do love lots of output and progress bars so on my fork I added progress updates using rich to see how fast each thread is going and how many thumbnails were produced (based on the ffmpeg stdout output). Changes here: e13ac9a If there's interest I'm happy to open a PR to add this. I tried having 16 progress bars (= number of GPU threads) but I couldn't get it to work, so I just ended up logging progress every 1%. |
Oh wow that's really awesome that you did that. Can you share what the output looks like so we can get an idea? Thank you! |
|
@ali-ramadhan def interested.. please open a PR. :) |
|
Hi @dezerving re your requests Enhancement: Choose which libraries to run
Enhancement: Choose order of libraries
Enhancement: Compress existing thumbnails
Add: Extend support to other scanner activities
Intro/credits detection, voice activity
Add: Output log including: Thumbnails generated, thumbnails skipped due to existing, missing thumbnails, size of thumbnail folder
|
I think for this I would expect it to be done through a cli argument (
Cool.
With This would also allow a more standardized approach to thumbnails. The ones generated from your script would be small (especially if I set the quality to 6) compared to Plex's. Would lead to a mismatch in thumbnail quality between different media depending on how the thumbs were generated.
The obvious use case would be to offload the intro/credits detection to a more powerful computer. Mainly would be useful for large libraries who had corrupt dbs that couldn't be repaired who have to start over. Yeah not sure exactly how this works. Would have to look deeper into seeing if it's even possible through PlexAPI. There are a couple projects that insert credit markers but that's not really what we need as those custom insertions would be deleted if the library was re-analyzed. Not sure if possible but just want to be able to do the task that the server does but through a different computer.
Personally I think the way the script currently runs (directly on Windows) has the progress bar and everything which is nice. Docker doesn't show that btw. The additional logging should be shown interactively as is right now - just extended with the enhancements above and be outputted as a log file at the end of the run as well. Log file creation can be determined by Thanks for the awesome script! |
|
I’ve just discovered this after reading a Reddit post :) I’d like to regenerate all of my chapter thumbnails, all the HDR & DoVi ones aren’t tone-mapped, or have wrong color space. You should look at FFMPEG libplacebo; it has superior HDR tone mapping. Here’s an example of one of my typical FFMPEG video filters for extracting screenshots (here full resolution, just as an example) :) In practice it may be faster to not offload to GPU for a single frame (my use case chapter previews). -init_hw_device vulkan \
-vf "hwupload,libplacebo=tonemapping=bt.2446a:colorspace=bt709:color_primaries=bt709:color_trc=bt709:range=limited,hwdownload,format=yuv420p10" \ |
Hey @ali-ramadhan! Just wondering if you are planning on PRing the logging from your fork: https://github.com/ali-ramadhan/plex_generate_vid_previews |
|
@ali-ramadhan following up on the above. You able to open a PR? Thanks :) |
thanks for the feedback but I'm unsure how to handle this.. It would need logic to determine the file type.. if HDR & DoVi then apply those args? If you want to open a PR i can merge it in? |
Sure. I'm working on a version myself to see how it goes. This is very experimental: USE_LIB_PLACEBO = os.getenv("USE_LIB_PLACEBO", 'False').lower() in ('true', '1', 't') # Check if we have a HDR Format. Note: Sometimes it can be returned as "None" (string) hence the check for None type or "None" (String)
if media_info.video_tracks:
if media_info.video_tracks[0].hdr_format != "None" and media_info.video_tracks[0].hdr_format is not None:
logger.debug('HDR format reported by Plex')
if USE_LIB_PLACEBO:
vf_parameters = f"hwupload,libplacebo=fps=1/{PLEX_BIF_FRAME_INTERVAL}:frame_mixer=none:tonemapping=bt.2446a:colorspace=bt709:color_primaries=bt709:color_trc=bt709:range=tv:w=320:h=240:force_original_aspect_ratio=decrease:format=yuv420p10le,hwdownload,format=yuv420p10le"
else:
vf_parameters = f"fps=fps={round(1 / PLEX_BIF_FRAME_INTERVAL, 6)}:round=up,zscale=t=linear:npl=100,format=gbrpf32le,zscale=p=bt709,tonemap=tonemap=hable:desat=0,zscale=t=bt709:m=bt709:r=tv,format=yuv420p,scale=w=320:h=240:force_original_aspect_ratio=decrease" if len(gpu_ffmpeg) < GPU_THREADS or CPU_THREADS == 0:
hw = True
if USE_LIB_PLACEBO:
args.insert(5, "-init_hw_device")
args.insert(6, "vulkan")
else:
args.insert(5, "-hwaccel")
args.insert(6, "cuda")if __name__ == '__main__':
logger.info('GPU Detection (with AMD Support) was recently added to this script.')
logger.info('Please log issues here https://github.com/stevezau/plex_generate_vid_previews/issues')
logger.info(f"USE_LIB_PLACEBO={USE_LIB_PLACEBO}")
if USE_LIB_PLACEBO:
logger.info('Using libplacebo for HDR tone-mapping.') |
|
@stevezau I've also changed the library section value returned by PlexAPI to logger.info(f"Generated Video Preview for {video_file} HW={hw} TIME={seconds}seconds SPEED={speed}x ")My example below of implementing some of these feature requests: @dezerving This is my script that I run. Can select which Libraries to process, and can easily be extended to select Movies and/or Shows In this example you can see to generate previews for a single movie: :) I've also added an option to run the process at a lower-priority :) .env: FORCE_REGENERATION_OF_BIF_FILES=True
PLEX_MEDIA_TYPES_TO_PROCESS='["movies", "shows"]'
PLEX_LIBRARIES_TO_PROCESS='["Films", "Movies", "Documentaries", "Shows"]'
USE_LIB_PLACEBO=False
RUN_PROCESS_AT_LOW_PRIORITY=Trueplex_generate_vid_previews.py: USE_LIB_PLACEBO = os.getenv("USE_LIB_PLACEBO", 'False').lower() in ('true', '1', 't')
FORCE_REGENERATION_OF_BIF_FILES = os.getenv("FORCE_REGENERATION_OF_BIF_FILES", 'False').lower() in ('true', '1', 't')
PLEX_MEDIA_TYPES_TO_PROCESS = os.getenv("PLEX_MEDIA_TYPES_TO_PROCESS", '').lower()
PLEX_LIBRARIES_TO_PROCESS = os.getenv("PLEX_LIBRARIES_TO_PROCESS", '')
RUN_PROCESS_AT_LOW_PRIORITY = os.getenv("RUN_PROCESS_AT_LOW_PRIORITY", "False").lower() in ('true', '1', 't')def process_item(item_key, gpu):
...
bundle_path = sanitize_path(os.path.join(PLEX_LOCAL_MEDIA_PATH, 'localhost', bundle_file))
indexes_path = sanitize_path(os.path.join(bundle_path, 'Contents', 'Indexes'))
index_bif = sanitize_path(os.path.join(indexes_path, 'index-sd.bif'))
tmp_path = sanitize_path(os.path.join(TMP_FOLDER, bundle_hash))
if not os.path.isfile(index_bif) or FORCE_REGENERATION_OF_BIF_FILES:
logger.debug(f"Generating bundle_file for {media_file} at {index_bif}")
...This below is essential, albeit not elegant. On a very large Plex library, and with a number of sub-processes started with This sends SIGKILL to the main parent process (=9, as A slightly more elegant approach requires managing the sub-processes manually and sending a signal to each running sub-process, which would have a signal handler and stop-processing, clean-up, and exit. There is no convenient way in Python. def signal_handler(arg1, arg2):
"""
Signal interrupt handler.
"""
print(f"⚠️Received SIGTERM or SIGINT {arg1} {arg2}⚠️")
print("Sending SIGKILL to PPID...")
os.kill(os.getpid(), 9)
# Register SIGERM signal handler
signal.signal(signal.SIGTERM, signal_handler)
# Register SIGINT (CTRL-C) signal handler
signal.signal(signal.SIGINT, signal_handler)def low_priority():
""" Set the priority of the process to below-normal."""
try:
sys.getwindowsversion()
except AttributeError:
isWindows = False
else:
isWindows = True
if isWindows:
import win32api, win32process
# priorityclasses:
# win32process.IDLE_PRIORITY_CLASS,
# win32process.BELOW_NORMAL_PRIORITY_CLASS,
# win32process.NORMAL_PRIORITY_CLASS,
# win32process.ABOVE_NORMAL_PRIORITY_CLASS,
# win32process.HIGH_PRIORITY_CLASS,
# win32process.REALTIME_PRIORITY_CLASS
win32process.SetPriorityClass(win32api.GetCurrentProcess(), win32process.BELOW_NORMAL_PRIORITY_CLASS)
else:
os.nice(10)def run(gpu):
# Ignore SSL Errors
sess = requests.Session()
sess.verify = False
plex = PlexServer(PLEX_URL, PLEX_TOKEN, session=sess)
for section in plex.library.sections():
if section.title not in PLEX_LIBRARIES_TO_PROCESS:
logger.info(f"Skipping library {section.title} as not in list of libraries to process {PLEX_LIBRARIES_TO_PROCESS}")
continue
if section.type not in PLEX_MEDIA_TYPES_TO_PROCESS:
logger.info(f"Skipping library {section.title} as not in list of media types to process {PLEX_MEDIA_TYPES_TO_PROCESS}")
continue
logger.info(f"Getting the media files from library \'{section.title}\'")
# ['movie', 'show', 'artist', 'photo']
if section.type == 'show':
media = [m.key for m in section.search(libtype='episode')]
elif section.type == 'movie':
# media = [m.key for m in section.search()] # default search returns all titles
# media = [m.key for m in section.search(title="Anora")] # just search for a single title
media = [m.key for m in section.search(title="Twilight")] # returns all the movies with Twilight in the title
else:
logger.info(f"Skipping library {section.title} as \'{section.type}\' is unsupported")
continue
logger.info(f"Got {len(media)} media files for library {section.title}")
if len(media) == 0:
continue
with Progress(SpinnerColumn(), *Progress.get_default_columns(), MofNCompleteColumn(), console=console) as progress:
with ProcessPoolExecutor(max_workers=CPU_THREADS + GPU_THREADS) as process_pool:
futures = [process_pool.submit(process_item, key, gpu) for key in media]
for future in progress.track(futures):
future.result()main: if __name__ == '__main__':
logger.info('GPU Detection (with AMD Support) was recently added to this script.')
logger.info('Please log issues here https://github.com/stevezau/plex_generate_vid_previews/issues')
logger.info(f"PLEX_LIBRARIES_TO_PROCESS={PLEX_LIBRARIES_TO_PROCESS}")
logger.info(f"PLEX_MEDIA_TYPES_TO_PROCESS={PLEX_MEDIA_TYPES_TO_PROCESS}")
logger.info(f"FORCE_REGENERATION_OF_BIF_FILES={FORCE_REGENERATION_OF_BIF_FILES}")
logger.info(f"USE_LIB_PLACEBO={USE_LIB_PLACEBO}")
logger.info(f"RUN_PROCESS_AT_LOW_PRIORITY={RUN_PROCESS_AT_LOW_PRIORITY}")
if USE_LIB_PLACEBO:
logger.info('Using libplacebo for HDR tone-mapping.')
if FORCE_REGENERATION_OF_BIF_FILES:
logger.warning('⚠️Force regeneration of BIF files is enabled, this will regenerate *all* files!⚠️')
if RUN_PROCESS_AT_LOW_PRIORITY:
low_priority()
logger.info('Running processes at lower-priority') |
|
Great, will you open a PR after your testing? :) |
I sure will! Give me some time though, as work is super busy right now.
I've also been experimenting with full offload to GPU, however it's slower than keeping some processing on CPU. I may not have optimal settings for libplacebo - I've run out of time today for investigating. |
|
How did you go @planetrocky ? |
|
Hey all, just throwing my 2c in I'd love to be able to set the user and group for the processes to run as/create new files and directories as, when running as a docker container - as it stands I have to run I've tried adding (Adding here as this appears to be a combined wish-list, so let me know if I should open my own issue) Edit: I have found a docker native method to achieve what I needed, which others can use easily. I'll create a PR shortly to update the readme to include this as I am sure others will find it helpful. |
|
thanks @jeremysherriff i merged the PR and updated the doc. |
Still working on it. I discovered loguru on Windows requires extra effort to make multiprocessing logging work. That’s all done and fixed! I’ve also added command line arguments that can be used as well as ENV variables. I’ve added a dictionary mapping for Plex libraries and local disk location. I have many libraries and some are split across multiple volumes. The dictionary mapping is specified with an ENV variable containing JSON, as shown below. It then uses the appropriately mapped local path. As a Plex Library can contain many directories, this is all supported: # these paths must have a trailing path separator (TODO: add code to check/add)
PLEX_LOCAL_VIDEOS_PATH_MAPPINGS_JSON='{
"/data/films/" : "\\\\fs2\\video2\\Films\\",
"/data/tv/" : "\\\\fs2\\video\\TV\\",
"/data/tv2/" : "\\\\fs2\\video2\\TV\\",
"/data/tv3/" : "\\\\fs2\\video3\\TV\\",
"/data/documentaries/" : "\\\\fs2\\video\\Documentaries\\",
"/data/music_videos/" : "\\\\fs2\\video\\Music Videos\\",
"/data/misc_videos/" : "\\\\fs2\\video\\Misc Videos\\"
}'
LOG_LEVEL=info # info, debug
USE_LIB_PLACEBO=False
PLEX_MEDIA_TYPES_TO_PROCESS='["movie", "show"]'
PLEX_LIBRARIES_TO_PROCESS='["Films", "TV Shows", "Music Videos", "Documentaries"]'I’ve added the ability to search for a specific title; and to only process a list of libs and types specified; and a force override to regenerate BIF files. So you can search for a title in a library and force regenerate just that one. $ python plex_generate_previews.py --thumbnail_quality 2 --bif_interval=4 --search "Title Name Here"
2025/01/03 06:10:21 | ℹ️ - GPU Detection (with AMD Support) was recently added to this script.
2025/01/03 06:10:21 | ℹ️ - Please log issues here https://github.com/stevezau/plex_generate_vid_previews/issues
2025/01/03 06:10:21 | ℹ️ - PLEX_LIBRARIES_TO_PROCESS=["TV Shows"]
2025/01/03 06:10:21 | ℹ️ - PLEX_MEDIA_TYPES_TO_PROCESS=["movie", "show"]
2025/01/03 06:10:21 | ℹ️ - PLEX_BIF_FRAME_INTERVAL=4
2025/01/03 06:10:21 | ℹ️ - THUMBNAIL_QUALITY=2
2025/01/03 06:10:21 | ℹ️ - FORCE_REGENERATION_OF_BIF_FILES=False
2025/01/03 06:10:21 | ℹ️ - USE_LIB_PLACEBO=False
2025/01/03 06:10:21 | ℹ️ - RUN_PROCESS_AT_LOW_PRIORITY=True
2025/01/03 06:10:21 | ℹ️ - LOG_LEVEL=INFO
2025/01/03 06:10:21 | ℹ️ - Running processes at lower-priority
2025/01/03 06:10:21 | ℹ️ - 🔸Searching for media titles matching Title Name Here🔸
2025/01/03 06:10:21 | ℹ️ - Found NVIDIA GPU
2025/01/03 06:10:21 | ℹ️ - Skipping library Films as not in list of libraries to process ["TV Shows"]
2025/01/03 06:10:21 | ℹ️ - Skipping library Documentaries as not in list of libraries to process ["TV Shows"]
2025/01/03 06:10:21 | ℹ️ - Getting the media files from library 'TV Shows'
2025/01/03 06:10:21 | ℹ️ - Got 0 media files for library TV Shows
2025/01/03 06:10:21 | ℹ️ - Skipping library Radio as not in list of libraries to process ["TV Shows"]
2025/01/03 06:10:21 | ℹ️ - Skipping library Misc Videos as not in list of libraries to process ["TV Shows"]
2025/01/03 06:10:21 | ℹ️ - Skipping library Music Videos as not in list of libraries to process ["TV Show"]ENV file are default, CLI args (if specified) override the ENV - very useful to quickly run on a single movie/show. I've only added about half the CLI args so far - very easy to do - I added enough to get more complex things in a working-state. $ python plex_generate_previews.py -h
usage: plex_generate_vid_previews [-h] [-s SEARCH] [-f] [-q [2-6]] [-i [1-30]]
What the program does
options:
-h, --help show this help message and exit
-s, --search SEARCH Search Plex for title (default: None)
-f, --force Force regeneration of BIF (default: False)
-q, --thumbnail_quality [2-6]
Preview image quality (2-6) (default: 3):
--thumbnail_quality=3 good balance between quality and file-size (default and recommend setting)
--thumbnail_quality=2 the highest quality and largest file size
--thumbnail_quality=6 the lowest quality and smallest file size
-i, --bif_interval [1-30]
Interval between preview images in seconds (1-30) (default: 2):
--bif_interval=2 generate a preview thumbnail every 2 seconds (default and recommend setting)
--bif_interval=1 generate a preview thumbnail every second (largest file size, longest processing time, best resolution for trick-play)
--bif_interval=30 generate a preview thumbnail every 30 seconds (smaller file size, shorter processing time, worst resolutionm for trick-play)
Text at the bottom of helplibplacebo and DoVi processing is in, but needs more work. I’ve had to add a lot of changes to support the features described above. You’ll have to be patient - January is a busy month work-wise; so it’ll be a minute before you should check again :) These extra features I find extremely useful; and are exactly what I wanted, and others probably! |
|
https://docs.docker.com/reference/compose-file/services/#user Most things I have in Docker Compose and if I need to will use
Hard-code IDs like 100:1000 or use some ENV magic :) |
|
INFO level only prints newly generated video preview thumbnails. I reformatted it to put the filename at the end, so that it stays tabulated: time, icon log level, BIF generated file size, HW, SDR|HDR, time, speed, location. DEBUG level prints a lot more; including all the files that were skipped etc. It would be easy to add an option to list each file skipped - though it does flood the screen and log. 2025/01/03 05:32:28 | ℹ️ - Generated Video Preview SIZE= 5.0MiB HW=True SDR TIME= 16.6seconds SPEED= 238x for \\fs2\video\TV\The Show Name (2015)\Season 5\The Show Name.S05E09.The Episode Title.mkv
|
|
@planetrocky great, when do you think you can open a PR even if it is draft? I plan on updating the logging to use a similar config as Delgan/loguru#444 (comment) which i think will fix the issue where the progress bar freezes. I don't want to make any changes unless yours is merged in otherwise it will be painful to merge both our PRs. |
|
@stevezau loguru setup can't be global, as in Windows Python executes the entire file each time a new process is launched; so you get multiple logs being setup; it's a mess. This is how I set it up; and it's been running for a week or so flawlessly now :) I also added a log to disk, which can/will be wrapped in options. def set_logger(logger_):
global logger
logger = logger_
def setup_logging():
# Logging setup
LOG_FORMAT = '<green>{time:YYYY/MM/DD HH:mm:ss}</green> | {level.icon} - <level>{message}</level>'
global console
console = Console()
logger.remove()
logger.add(
lambda _: console.print(_, end=''),
level = LOG_LEVEL,
format = LOG_FORMAT,
enqueue = True
)
logger.add(
os.path.join("logs", "plex_generate_previews_{time}.log"),
retention="14 days",
level = LOG_LEVEL,
format = LOG_FORMAT,
colorize = False,
enqueue = True
)
if __name__ == '__main__':
setup_logging()
logger.info('GPU Detection (with AMD Support) was recently added to this script.')
logger.info('Please log issues here https://github.com/stevezau/plex_generate_vid_previews/issues')
logger.info(f"PLEX_LIBRARIES_TO_PROCESS={PLEX_LIBRARIES_TO_PROCESS}")
logger.info(f"PLEX_MEDIA_TYPES_TO_PROCESS={PLEX_MEDIA_TYPES_TO_PROCESS}")
logger.info(f"PLEX_BIF_FRAME_INTERVAL={PLEX_BIF_FRAME_INTERVAL}")
logger.info(f"THUMBNAIL_QUALITY={THUMBNAIL_QUALITY}")
logger.info(f"FORCE_REGENERATION_OF_BIF_FILES={FORCE_REGENERATION_OF_BIF_FILES}")
logger.info(f"USE_LIB_PLACEBO={USE_LIB_PLACEBO}")
logger.info(f"RUN_PROCESS_AT_LOW_PRIORITY={RUN_PROCESS_AT_LOW_PRIORITY}")
logger.info(f"LOG_LEVEL={LOG_LEVEL}")
if USE_LIB_PLACEBO:
logger.info('Using libplacebo for HDR tone-mapping.')
if FORCE_REGENERATION_OF_BIF_FILES:
logger.warning('⚠️Force regeneration of BIF files is enabled, this will regenerate *all* files!⚠️')
if RUN_PROCESS_AT_LOW_PRIORITY:
low_priority()
logger.info('Running processes at lower-priority')
if cli_args.search:
logger.info(f"🔸Searching for media titles matching {cli_args.search}🔸") |
If you are seeing the progress bar problems on Windows; it will be the way Since I added the change above all the logging has been smooth, correct, not overlapping each other, and no other bad things happening. |
|
@planetrocky i'm not a windows users.. i'm mac and linux and the freezing still happens, it only happens in the docker container tho. However, ill wait for your PR, review, merge then take a look if it does not fix it. |
|
@stevezau Go ahead and make your changes, please don't wait on me - I can do the painful merge later on! One thing I did do, was to change everything to use f-strings. logger.info(f"Generated Video Preview SIZE={sizeof_fmt(bif_filesize):>9} HW={results_gen_imgs['hw']!r:<5} {'HDR' if results_gen_imgs['hdr'] else 'SDR'} TIME={results_gen_imgs['seconds']:>6}seconds SPEED={(str(results_gen_imgs['speed']) + 'x'):>6} for {media_file}")
else:
logger.debug(f"Not generating bundle_file for {media_file} at {index_bif} as it already exists!")I also added a keyboard interrupt handler, as it was impossible to I develop on all platforms! Almost everything I have is running on a Linux server, in a Docker Containers However my Windows desktop has an RTX 4080 Super FE, so it's good for testing the hardware video processing. I have a server with an RTX 3080 Ti FE, but I've not finished it's rebuild. The problem is I'm working on about 100x projects at the same time! |
|
@planetrocky, maybe but it will be better to wait and will cause less pain for me as i have to review all PR's. Can you give me an idea when you think you will open the PR? Perhaps you can open it in draft format for now? |
|
@planetrocky any update on the PR? Would love to see it even in draft? |
|
@stevezau if no one works on the requested PRs in this thread over the next few weeks I may look at combining some of the functionality that they've added (improved logging and modular approach) and open a PR myself. Note No guarantees as I'm pretty busy until March but I will definitely look at it as this script has become part of my workflow since last year. |
|
It’s slowly evolving. One feature I added that I like the most is the search! You can specify added since; so it will only pull media added in the last 1h, 2h, 1d, 7d, 1y etc. NB this is Plex’s added, when the media was added to the library, not when a new version or different edition was added |
|
Thanks for all the hard work, @planetrocky! I’m looking forward to testing everything out. When do you think we can merge? The PR is getting quite large, and reviewing it all at once will become tougher the larger it gets. It might be easier to merge and add features in smaller PRs moving forward. |
Love the script by the way (finally got it working with my overly convoluted setup).
Was wondering if these enhancements & additions would be possible or if they're out of the scope of this project/not possible:
With these improvements this script could go from just being run once to rebuild to something that is used to maintain our libraries (until Plex natively offers the ability to offload thumbnail generation and compress thumbnails).
I will probably turn off the scheduled task of adding thumbnails and just trigger this script nightly to add thumbnails for newly added shows and movies.
The text was updated successfully, but these errors were encountered: