Support Milkdrop user sprites

[kblaschke]

Milkdrop supports drawing user-defined "sprites", which are simple rectangular images with an optional color key (for transparency in image formats which don't have an alpha channel). These sprites can optionally be animated via expression code, using a few input values like preset timers and the beat detection values.

projectM should also support rendering these sprites, as they're very useful for streamers and making videos with watermarks or additional animated logos on top to save post-processing work and sync the image movement exactly the the visuals.

Milkdrop Sprite Definition File Format

The sprite def file is very similar to the .milk preset format, as it also uses INI syntax. Each sprite is defined in its own section named [imgXX], where XX is a two-digit number. Each section can contain the following lines:

• img = <filename>: The sprite texture image to load.
• colorkey = 0xAARRGGBB: Transparency color key. Defaults to black, alpha is always set to 0xFF in Milkdrop. Might not work using the hexadecimal representation as Milkdrop uses GetPrivateProfileInt(), which doesn't support parsing 0x... values.
• init_N = <code>: Initialization expression, run once after loading the sprite. N is the line number, starting with 1. Numbering must be continuous. Each line of <code> is appended and executed as one program.
• code_N = <code>: Per-frame expression, run once every time the sprite is drawn. N is the line number, starting with 1. Numbering must be continuous. Each line of <code> is appended and executed as one program.

The following expression variables are available in the code_N expressions, updated with the current frame data:

• time: Time passed since program start. Also available in init code.
• frame: Total frames rendered so far. Also available in init code.
• fps: Current (or, if not available, target) frames per second value.
• progress: Preset blending progress (only if blending).
• bass: Bass frequency loudness, median of 1.0, range of ~0.7 to ~1.3 in most cases.
• mid: Middle frequency loudness, median of 1.0, range of ~0.7 to ~1.3 in most cases.
• treb: Treble frequency loudness, median of 1.0, range of ~0.7 to ~1.3 in most cases.
• bass_att: More attenuated/smoothed value of bass.
• mid_att: More attenuated/smoothed value of mid.
• treb_att: More attenuated/smoothed value of treb.

The following output variables are used to draw the sprite:

• done: If this becomes non-zero, the sprite is deleted. Default: 0.0
• burn: If non-zero, the sprite will be "burned" into currently rendered presets when done is also true, effectively "dissolving" the sprite in the preset. Default: 1.0
• x, y: Sprite x/y position (position of the image center). Range from -1000 to 1000. Default: 0.5
• sx, sy: Sprite x/y scaling factor. Range from -1000 to 1000. Default: 1.0
• rot: Sprite rotation in radians (2*PI equals one full rotation). Default: 0.0
• flipx, flipy: If a flag is non-zero, the sprite is flipped in this axis. Default: 0.0
• repeatx, repeaty: Repeat count of the image on the sprite quad. Fractional values allowed. Range from 0.01 to 100.0. Default: 1.0
• blendmode: Image blending mode. 0 = Alpha blending (default), 1 = Decal mode (no transparency), 2 = Additive blending, 3 = Source color blending, 4 = Color key blending. Default: 0
• r, g, b, a: Modulation color used in some blending modes. Default: 1.0

Default values are used if the expressions don't explicitly set a value. No q or t variables are available to sprite expressions. The regXX vars, gmegabuf and megabuf are individual per-sprite contents, not shared between sprites or with any running presets.

The blending modes are using this "effect" matrix:

// blendmodes                                      src alpha:        dest alpha:
// 0   blend      r,g,b=modulate     a=opacity     SRCALPHA          INVSRCALPHA
// 1   decal      r,g,b=modulate     a=modulate    D3DBLEND_ONE      D3DBLEND_ZERO
// 2   additive   r,g,b=modulate     a=modulate    D3DBLEND_ONE      D3DBLEND_ONE
// 3   srccolor   r,g,b=no effect    a=no effect   SRCCOLOR          INVSRCCOLOR
// 4   colorkey   r,g,b=modulate     a=no effect   

Implementation Notes

Sprite positioning and aspect-correction calculations could be done in a sprite-specific vertex shader, as well as implementing the blending modes in the fragment shader. This should make the C++ code a bit cleaner.

When implementing user sprite support in projectM, these sprites must be drawn after the whole preset rendering process and any transition blending is done, effectively being the last step before returning the render_frame method. This makes sure that sprites are only drawn once and continue working even after introducing new preset formats besides Milkdrop.

Burning-in stuff into presets is currently not implemented. Could be implemented easily by adding a method to the Preset interface class, returning a reference to the framebuffer holding the current texture which is passed to the next frame, with this texture being the only color attachment. Instead of drawing into the final output framebuffer, the sprite can be drawn here. Note that during blending, this must be done with both running presets!

Put all sprite-related API functions into a separate header, e.g. sprites.h and include the header in projectm.h.

Adding sprites should be done via a new API function, which takes the sprite definition as text (const char*) and also could have an additional "type" parameter so we can later add additional sprite types which differ from the Milkdrop format. Sprite data should be formatted in the same way Milkdrop accepts them (see below).

In Milkdrop, sprites are "launched" or killed by the user entering a two-digit number. The sprites exists either until the sprite sets the done flag or the user actively kills the sprite. Since libprojectM doesn't handle any keyboard input, we need API methods to start or kill specific sprites.

There should also be a method to clear all loaded sprites, and optionally additional methods to enumerate and replace/unload specific sprites. Do not confuse this with the launch/kill methods above, which control whether a sprite is drawn or not.

libprojectM must not search for sprite definition files on its own - adding sprites is solely application-controlled.

Milkdrop Source References

Here's a list of relevant code parts in the Milkdrop sources related to sprite rendering:

• Launching and killing sprites via hotkeys
• Load sprite if triggered
• Sprite drawing code
• Most sprite-related management code is in texmgr.cpp