EOF proposed project format.txt

EOF proposed file format (revision 'H'):

	FILE HEADER:
8 bytes:	'E','O','F','S','O','N','H','\0'
8 bytes:	(padding)

	CHART PROPERTIES:
4 bytes:	Project revision number
*1 byte:	Timing format (0=Milliseconds,1=Deltas)
*4 bytes:	Time division (only to be used if the timing format is "deltas", will probably be informational only unless EOF changes the time division used)

	SONG PROPERTIES:
2 bytes:	Number of INI strings
INI STRING CHUNK, for each INI string:
{
*	1 byte:		INI string type (0=Custom,1=Album,2=Artist,3=Title,4=Frettist,5=(unused),6=Year,7=Loading Text,8=Album,9=Genre,10=Track number.  *Other potential future ones can be: #=Icon ID,#=Unlock ID,#=Unlock ID required,#=Unlock text string,...)
!	The following Icon IDs are supported natively in FoFiX: rb1,rb2,rbdlc,rbtpk,gh1,gh2,gh2dlc,gh3,gh3dlc,gh80s,gha,ghm,ph1,ph2,ph3,ph4,phm.  Custom icon strings can be used in FoFiX.
!	Unlock text string is the text that FoFiX will display if this chart is not unlocked
*	2 bytes:	INI string length
*	[varies:]	INI string
}

2 bytes:	Number of INI booleans
INI BOOLEAN CHUNK, for each INI boolean:
{
*	1 byte:		INI boolean type (0=Reserved,128-255=DISALLOWED,1=Lyrics present,2=Eighth note HOPO,3=Guitar fret hand pos of 0,4=Bass fret hand pos of 0,5=Tempo map locked,6=Expert+ drums disabled,7=Click/drag disabled,8=Export RS chord techniques,9=Unshare drum phrasing,10=Highlight non grid snapped notes,11=Accurate T/S,12=Highlight notes in arpeggios,13=Suppress DD warnings,14=FoFLC export w/o pitch shifts)
!	The MSB is the boolean status of the INI setting (1 is True), allowing for an explicit "value = False" setting to be stored if necessary
!	The low 7 bits represents the boolean INI setting in question, numbered from 0 to 127 (number 0 reserved for future use)
!	Tutorial songs are hidden during quickplay in FoFiX
}

2 bytes:	Number of INI numbers
INI NUMBER CHUNK, for each INI number:
{
*	1 byte:		INI number type (0=Reserved,1=Cassette color,2=Band difficulty level,3=HOPO frequency,4=User specified chart version #,...)
*	4 bytes:	INI number value
!	Cassette color is an 8 bit intensity each for Red, Green and Blue
!	HOPO frequency is 0-5, and is used if the player's "Song HOPO Freq" FoFiX setting is set to "Auto"
}

	CHART DATA:
2 bytes:	Number of OGG profiles
OGG PROFILE CHUNK, for each OGG profile:
{
*	2 bytes:	OGG file name string length
	[varies:]	OGG file name string
*	2 bytes:	Original audio file name string length
*	[varies]:	Original audio file name
*	2 bytes:	OGG profile description string length
*	[varies:]	OGG profile description string
	4 bytes:	MIDI delay for this profile (milliseconds)
*	4 bytes:	OGG profile flags (such as whether the OGG is loaded as an additional stem, is muted, etc)
}

4 bytes:	Number of beats in the chart
BEAT CHUNK, for each beat:
{
	4 bytes:	PPQN
	4 bytes:	Position (in milliseconds or delta ticks)
	4 bytes:	Beat flags (highest byte stores a custom TS numerator, second highest byte stores a custom TS denominator)
*	1 byte:		Key signature (If negative, the value defines the number of flats present, ie. -2 is Bb.  If positive, the value defines the number of sharps present, ie. 4 is E)
}
!For next revision of the format, take the custom time signature out of the flags and make it a conditional set of bytes that are written if a TS beat flag is set
!Also make the key signature byte conditional on a beat flag being set

4 bytes:	Number of text events
TEXT EVENT CHUNK, for each text event:
{
*	2 bytes:	Text event string length
*	[varies:]	Text event string
	4 bytes:	Associated beat number or millisecond position (depending on whether the floating position flga is set)
*	2 bytes:	Associated track number (0 if it applies to the entire chart, such as generic section markers)
	2 bytes:	Text event flags (1 = Rocksmith phrase, 2 = Rocksmith section, 4 = Rocksmith event, 8 = Rocksmith solo phrase, 16 = Floating position)
}

*4 bytes:	Number of custom data blocks
CUSTOM DATA BLOCK CHUNK, for each custom data block
{
*	4 bytes:	The number of bytes this custom data block consumes (including the block ID and the custom data itself)
*	4 bytes:	Custom data block ID (1 = Raw MIDI data, 2 = Floating point beat timings, 3 = Start/end points, 0xFFFFFFFF = Empty debugging block)
*	[varies:]	(Custom data, can store anything)
}
!This will allow the storage of any data not specifically accounted for in this format, helping to future proof the format
=========================================
Custom data block ID 1:  Raw MIDI data:
	2 bytes:  Number of MIDI tracks
	1 byte:  (Flags:  Bit 0= Delta timings allowed)
	1 byte:  (Unused)
	For each track:
	{
		2 bytes:  Length of MIDI track name
		[varies]:  MIDI track name (no NULL terminator)
		2 bytes:  Length of track description string
		[varies]:  Track description string
		4 bytes:  Number of events in the track
		[1 byte]:  Track includes delta timings (If the delta timings allowed flag is set, this byte is present.  If this byte is nonzero, each event entry includes a 4 byte delta time.)
		[4 bytes]:  Time division (If the delta timings present flag above is nonzero, the original time division is required to convert to real time so it will be stored here)
		For each event:
		{
			2 bytes:  The size of the realtime position string
			[varies]:  Realtime position of event (as written by fprintf("%f"...) with no NULL terminator, this is for cross platform compatibility)
			[4 bytes]:  If the track includes delta timings (as per the byte above), this field contains the absolute delta time of the event
			2 bytes:  The size of the event
			[varies]:  Event data
		}
	}
Custom data block ID 2:  Floating point beat timings:
	For each beat in the project:
	{
		2 bytes:  Length of floating point time string
		[varies]:  String representation of the beat's position
	}
Custom data block ID 3:  Start/end points
	4 bytes:  Start point
	4 bytes:  End point


	TRACK DATA:
*4 bytes:	Number of tracks (including track 0, which stores non track-specific sections like bookmarks)
TRACK CHUNK, for each track:
{
*	2 bytes:	Track name string length
*	[varies]:	Track name string
*	1 byte:		Track format (0=Global,1=Legacy,2=Vocal,3=Pro Keys,4=Pro Guitar/Bass,5=Variable Lane Legacy,...)
*	1 byte:		Track behavior (1=Guitar/Bass,2=Drum/Pro Drum,3=Vocal,4=Keys,5=Pro Guitar/Bass,6=Pro Keys,7=Dance...)
*	1 byte:		Track type (1=Guitar,2=Bass,3=Lead Guitar,4=Rhythm Guitar,5=Drum/Pro Drum,6=Vocal,7=Keys,8=17 Fret Pro Bass,9=17 Fret Pro Guitar,10=Pro Keys,11=22 Fret Pro Guitar,12=22 Fret Pro Bass,13=Phase Shift drum track,14=Rocksmith bonus arrangement,...)
!	Rock Band 3 charts have separate tracks for the full 22 fret chart (PART REAL_GUITAR_22) usable on the Squier guiter, and a version transposed to be accurate on 17 frets (PART REAL_GUITAR)
!	Both EOF and Rock Band allow for the existence of one of each of the usual tracks, so the first of each will be assigned a unique number as the track type.  Any secondary tracks (ie. a second PART KEYS) will be marked as a custom track
!	and will be identified by its format and behavior fields.  This will allow the ability to maintain some compatibility with EOF's current method of comparing tracks to defined macros
!	The track format and behavior abstract the track to the point where any number of tracks can be handled, allowing an easy way to verify a track's ability to use HOPOs, etc. instead of relying on the track's number
!	Until track number comparison is completely removed for this purpose, the track type can serve as a replacement
*	1 byte:		Track difficulty level (0-6), 0xFF represents an undefined track difficulty
*	4 bytes:	Track flags (ie. which difficulties are locked for editing)
	[varies]:	Extended note flags (if the MSB of the note flags field is set, another 4 byte flag field follows, and if its MSB is set, another 4 byte flag field, etc)
*	2 bytes:	Track compliance flags (ie. if it is valid as per constraints of Rock Band, etc.  Currently unused)
	[2 bytes]:	//If the track flags (flags | 2) indicated an alternate name is given for the track, the alternate track name string length
	[varies]:	//If the track flags (flags | 2) indicated an alternate name is given for the track, the alternate track name string

*	[varies:]	Track data (depending on the format type as defined below)

*	2 bytes:	Number of section types stored for this track
*	SECTION TYPE CHUNK, for each section type
*	{
*		2 bytes:	Type of section (1=solo,2=star power,3=bookmark,4=fret catalog entry,5=lyric phrase,6=yellow cymbal,7=blue cymbal,8=green cymbal,9=trill,10=arpeggio,11=trainer,12=custom MIDI note,13=preview section,14=tremolo section,15=slider section,16=fret hand positions,17=tone change...)

*		4 bytes:	Number of the specified sections stored
*		SECTION CHUNK, for each of the specified type of sections
*		{
*			2 bytes:	Section name string length
*			[varies:]	Section name string
*			1 byte:		The type/difficulty that this phrase applies to (0xFF if it applies to all difficulties instead of just one, such as with a solo phrase)
*			4 bytes:	Start position of section (in milliseconds or delta ticks)
*			4 bytes:	End position of section (in milliseconds or delta ticks, this is used to store other data for sections that have no end position, such as for fret hand positions and bookmarks)
*			4 bytes:	Section flags
*		}
*	}
!	The first track maintained by EOF will be the global track (track 0), which stores information that applies to the whole chart
!	(ie. song preview sections, bookmarks, catalog entries) or information that otherwise applies to more than one track (ie. unison SP sections)
!		For bookmarks, the end position indicates which bookmark number it is
!		For fret catalog entries, the flags field stores the track number the catalog entry is from
!		For lyric phrases, the difficulty field indicates which lyric set number (ie. PART VOCALS) the phrase applies to

*	4 bytes:	Number of custom data blocks
*	TRACK CUSTOM DATA BLOCK CHUNK, for each custom data block
*	{
*		4 bytes:	The number of bytes this custom data block consumes  (including the block ID and the custom data itself)
*		4 bytes:	Custom data block ID (Defined below.  0xFFFFFFFF = Empty debugging block)
*		[varies:]	(Custom data block, can store anything)
*	}
=========================================
Custom track data block definitions
=========================================
	Custom data block ID 1:  Extra note data (not current used):
		1 byte:		Number of additional data bytes per note
		[varies]:	Number of data bytes * number of notes (defined above in the track data) bytes
	Custom data block ID 2:  Pro guitar finger arrays, for each note in the pro guitar track:
		FINGER CHUNK, for each set bit in the note's bitflag (defined in the prior track's note data)
		{
			1 byte:	Finger number used for this string (0 = unused/undefined, 1 = index, 2 = middle, 3= ring, 4 = pinky, 5 = thumb)
		}
	Custom data block ID 3:  Pro guitar track arrangement type
		1 byte:		Arrangement type (1 = Combo, 2 = Rhythm, 3 = Lead, 4 = Bass)
	Custom data block ID 4:  Pro guitar track tuning not honored
		1 byte:		If nonzero, indicates that the chord name detection reflects the DEFAULT tuning for the arrangement, and not the track's actual specified tuning.
	Custom data block ID 5:  Pro guitar vibrato speeds (this data block is deprecated but is reserved for backwards compatibility with 1.8RC9 era release candidates)
		VIBRATO CHUNK, for each note in the track that is marked with vibrato (note flag value 16777216)
		{
			1 byte: Vibrato speed used for this note (0 = undefined, 40 = slow, 80 = medium, 120 = fast, the default value is 80)
		}
	Custom data block ID 6:  Pro guitar capo position
		1 byte:		The fret number at which a capo is placed for this track
	Custom data block ID 7:  Pro guitar tech notes
		4 bytes:	Number of tech notes
		PRO GUITAR NOTE CHUNK, for each tech note
		{
			(As defined in the "PRO GUITAR/PRO BASS TRACK FORMAT" section)
		}
	Custom data block ID 8:  Accent note bitmask
		ACCENT CHUNK, for each note in the legacy track
		{
			1 byte: The accent note bitmask for the legacy note
		}
	Custom data block ID 9:  Track difficulty count (currently only written for pro guitar track with a difficulty count higher than 5)
		1 byte:		Number of difficulties in the track (either with or without notes)
	Custom data block ID 10:  SP deploy bitmask
		SP DEPLOY CHUNK, for each note in the legacy track
		{
			1 byte: The SP deploy bitmask for the legacy note
		}
	Custom data block ID 11:  Ghost note bitmask
		GHOST CHUNK, for each note in the legacy track
		{
			1 byte:  The ghost note bitmask for the legacy note
		}

=========================================		
}
(End of file)


	TRACK FORMAT DEFINITIONS
["LEGACY" (GUITAR/BASS/DRUM/PRO DRUM/KEYS/DANCE) TRACK FORMAT]
*	1 byte:		The number of lanes, keys, etc. used in this track (5 for a standard legacy track, 6 for legacy track with open strum, etc.)
	4 bytes:	Number of notes
	NOTE CHUNK, for each note
	{
*		2 bytes:	Chord name string length
*		[varies:]	Chord name string
		1 byte:		Note type (difficulty)
		1 byte:		Note bitflag (on/off statuses)
		4 bytes:	Note position (in milliseconds or delta ticks)
		4 bytes:	Note length (in milliseconds or delta ticks)
*		4 bytes:	Note flags
		[4 bytes:]	Extended note flags (if the MSB of the note flags field is set, another 4 byte flag field follows, and if its MSB is set, another 4 byte flag field, etc)
	}
!This track format could be referred to as "legacy" format, because it is Harmonix's old format used prior to RB3
!The MSB of the PART DRUMS' flags variable stores the additional "pro drums" difficulty (the high nibble stored the "PS Real Drums" difficulty in late beta versions of EOF 1.8, the low nibble stores the Pro Drums difficulty)
!The high nibble of the PART DRUMS flag is set to 0xF for backwards compatibility with the beta versions
!The low nibble of the most significant byte of PART VOCALS' flags variable stores the harmony difficulty
!For the next variation of the format, store additional difficulties outside the track flags (ie. as INI numbers)

[VOCAL TRACK FORMAT]
*	1 byte:		Tone set number assigned to this track (ie. 0=Grand Piano, 1=MIDI device, ...)
	4 bytes:	Number of lyrics
	LYRIC CHUNK, for each lyric:
	{
*		2 bytes:	Lyric text string length
*		[varies:]	Lyric text string
*		1 byte:		Lyric set number (0=PART VOCALS, 1=HARM1, 2=HARM2...)
		1 byte:		Note pitch
		4 bytes:	Lyric position (in milliseconds or delta ticks)
		4 bytes:	Lyric length (in milliseconds or delta ticks)
*		4 bytes:	Lyric flags
		[4 bytes:]	Extended lyric flags (if the MSB of the note flags field is set, another 4 byte flag field follows, and if its MSB is set, another 4 byte flag field, etc)
	}

[PRO GUITAR/PRO BASS TRACK FORMAT]
*	1 byte:		The highest fret number allowed in this track (17 for Mustang, 22 for Squier)
*	1 byte:		The number of strings on this track's guitar (ie. 4 for standard bass guitar, 6 for standard guitar, for now, the max number supported is 8)
*	PRO GUITAR TUNING CHUNK, for each string (starting from the string for lane 1):
*	{
*		1 byte:		(Signed value) The number of half steps above or below (negative value) the standard tuning for this string
*	}
!		The first string defined refers to the lowest guage string (low E on a standard guitar)
!		The tuning information will be written to song.ini from lowest to highest guage, ie. "real_guitar_tuning (-2 0 0 0 0 0)" would indicate drop D tuning

*	4 bytes:	Number of notes
*	PRO GUITAR NOTE CHUNK, for each note
*	{
*		2 bytes:	Chord name string length
*		[varies:]	Chord name string
*		1 byte:		Chord number
*		1 byte:		Note type (difficulty)
*		1 byte:		Note bitflag (string use statuses: set=played, reset=not played)
!				Bit 0 refers to lane 1 (ie. string 6, low E), bit 5 refers to lane 6 (ie. string 1, high E), consistent with guitar terminology
!		1 byte:		Ghost bitflag (specifies which lanes in the note are treated as "ghost" notes, ie. for Arpeggio phrases)
*		FRET CHUNK, for each set bit in the guitar note bitflag
*		{
*			1 byte:		Fret # tab (0=Open strum, #=Fret # pressed, 0xFF=Muted (no fret specified), if MSB is set, the string is considered muted)
*		}
*		1 byte:		Legacy bitflags (the 5 least significant bits represent the lanes that would be set when pasting this note into a legacy track)
!			If the MSB is set, this bitmask was user-defined, otherwise it will be dynamically defined during the paste operation (basically just keep lanes 1-5 set/clear based on the pro guitar note's lanes)
*		4 bytes:	Note position (in milliseconds or delta ticks)
*		4 bytes:	Note length (in milliseconds or delta ticks)
*		4 bytes:	Note flags (allow 32 flags for an unforsee-able number of techniques such as tapping, bending, etc)
		[1 byte:]	Slide ending fret (if the Rocksmith notation flag, decimal value 33554432, is set and this note is flagged as having an up/down slide, value 4096 or 8192)
		[1 byte:]	Bend strength in half/quarter steps (if the Rocksmith notation flag, decimal value 33554432, is set and this note is flagged as a bend, value 2097152)
				(If the MSB of this value is set, the other 7 bits define the number of quarter steps the bend is, otherwise the value is in half steps)
		[1 byte:]	Unpitched slide ending fret (if this note is flagged as having an unpitched slide, value 256)
		[4 bytes:]	Extended note flags (if the MSB of the note flags field is set, another 4 byte flag field follows, and if its MSB is set, another 4 byte flag field, etc)
*	}

[PRO KEYS TRACK FORMAT]
*	1 byte:		Tone set number assigned to this track (ie. 0=Grand Piano, 1=MIDI device, ...)
*	1 byte:		The lowest note used in this track (Rock Band standard is 48)
*	1 byte:		The highest note used in this track (Rock Band standard is 72)

*	4 bytes:	Number of chords
!		For the sake of this format, "chord" will refer to all notes defined at a given position in a difficulty in a keys track,
!		even if there are not multiple notes, and notes will refer to individual notes in the chord
*	KEY CHORD CHUNK, for each note
*	{
*		2 bytes:	Chord name string length
*		[varies:]	Chord name string
*		1 byte:		Chord type (difficulty)
*		1 byte:		Chord hand (1=right hand- ie. melody, 2=left hand- ie. chords)
!			Rock Band 3 won't support two hand keys tracks, so this would be a feature for other games such as Synthesia/Doremi Mania)
!			Two chords should be allowed to be defined at any timestamp, as each hand should be allowed to play one
*		1 byte:		Number of keys played in this note/chord
*		KEY NOTE CHUNK, for each key in the chord
*		{
*			1 byte:		Note number
*			1 byte:		Note velocity
*			1 byte:		Note flags (for storing accidentals)
*		}
*		1 byte:		Legacy bitflags (how this chord is pasted into a legacy track)

*		4 bytes:	Note position (in milliseconds or delta ticks)
*		4 bytes:	Note length (in milliseconds or delta ticks)
*		4 bytes:	Chord flags
*		[4 bytes:]	Extended note flags (if the MSB of the note flags field is set, another 4 byte flag field follows, and if its MSB is set, another 4 byte flag field, etc)
*	}

[VARIABLE LANE "LEGACY" FORMAT]
*	1 byte:		The number of lanes included in this track

*	4 bytes:	Number of notes
*	EXTRA LANE CHORD CHUNK, for each note
*	{
*		1 byte:		Note type (difficulty)
*		1 byte:		Number of lanes played at this position
*		EXTRA LANE NOTE CHUNK, for each lane played at this position
*		{
*			1 byte:		Lane number containing a gem
			1 byte:		Number of flag bytes for this gem
			EXTRA LANE FLAG CHUNK, for each flag byte specified above
			{
				[varies]:	The flags for this gem
			}
*		}
*		4 bytes:	Note position (in milliseconds or delta ticks)
*		4 bytes:	Note length (in milliseconds or delta ticks)
*		4 bytes:	Note flags (among other specialized flags, DTXMania supposedly tracks whether hi-hat is open/closed)
*		[4 bytes:]	Extended note flags (if the MSB of the note flags field is set, another 4 byte flag field follows, and if its MSB is set, another 4 byte flag field, etc)
*	}
!For the next variation of the format, have a 4 byte common flags variable that all track formats use, and then one or more 4 byte flags variables with statuses that are specific to each format
!Also don't store dynamically updated flags (ie. EOF_NOTE_FLAG_HOPO and EOF_NOTE_FLAG_SP) in the note's limited flags, they can be kept in a temporary flags variable that doesn't save to file


!NOTES:

!Rock Band 3 has a new type of phrase people are referring to as trill phrases
!This is sort of like a section with a freestyle strumming for a single button (fret 1-5), perhaps more
!Supposedly, the player can strum as slowly or quickly as desired provided that the specific notes defined
!in the trill phrase are hit.

!Rock Band 2 and Rock Band 3 (Pro drums) use MIDI markers to define which yellow, blue and green notes are to be played
!as toms (pads) instead of cymbals (which are separate from the 4 pads on a compatible drum set controller).
!	Note:  Pro drums are currently being implemented as note statuses, requiring extra logic to simulate phrasing

!For keys and vocal track charting in grand staff mode, track accidentals for each note in each chord (4 bits):
!The flat/sharp status of each line in the staff is refreshed at the beginning of each measure
!If a note extends across a measure, its flat/sharp status is retained for the duration of that measure
	00 : No accidental (any previous flat/sharp is still in effect)
	01 : Flat (lower note by one semitone)
	10 : Sharp (raise note by one semitone)
	11 : Natural (cancel any flat/sharp in effect for this line in the staff, until the next measure/accidental/key change)

!Difficulty levels 0 through 5 are a regular graduated scale of difficulty, 6 represents "5 devil skulls" which is extra hard
!FoFiX doesn't reportedly support a vocal difficulty value yet, but planning ahead doesn't hurt