Worked on Spotlight store format support

Author: joachimmetz

documentation/Apple Spotlight store database file format.asciidoc

@@ -1,4 +1,4 @@
-= Apple Spotlight store database file format
+= Apple Spotlight store file formats
 
 :toc:
 :toclevels: 4
@@ -7,29 +7,29 @@
 [abstract]
 == Summary
 
-The Apple Spotlight store database file format is used by MacOS and iOS to
-store the Spotlight desktop search information. This specification is based
-on the source code and documentation.
+The Apple Spotlight store file formats are used by MacOS and iOS to store
+the Spotlight desktop search information. This specification is based
+on available source code and documentation.
 
 This document is intended as a working document for the Apple Spotlight
-store database file format specification.
+store file formats specification.
 
 [preface]
 == Document information
 
 [cols="1,5"]
 |===
 | Author(s): | Joachim Metz <[email protected]>
-| Abstract: | Apple Spotlight store database file format
+| Abstract: | Apple Spotlight store file formats
 | Classification: | Public
-| Keywords: | Apple Spotlight store database, store.db
+| Keywords: | Apple Spotlight store, store.db
 |===
 
 [preface]
 == License
 
 ....
-Copyright (C) 2020, Joachim Metz <[email protected]>.
+Copyright (C) 2020-2023, Joachim Metz <[email protected]>.
 Permission is granted to copy, distribute and/or modify this document under the
 terms of the GNU Free Documentation License, Version 1.3 or any later version
 published by the Free Software Foundation; with no Invariant Sections, no
@@ -44,28 +44,69 @@ in the section entitled "GNU Free Documentation License".
 |===
 | Version | Author | Date | Comments
 | 0.0.1 | J.B. Metz | June 2020 | Initial version based on earlier notes, with thanks to Everest Munro-Zeisberger
+| 0.0.2 | J.B. Metz | June 2023 | Additional changes based on format analysis
 |===
 
 :numbered:
 == Overview
 
-The Apple Spotlight store database file format is used by MacOS and iOS to
-store the Spotlight desktop search information. This specification is based
-on the source code and documentation.
+The Apple Spotlight store file formats are used by MacOS and iOS to store
+the Spotlight desktop search information. This specification is based on
+available source code and documentation.
+
+The Spotlight store is typically stored under:
+
+....
+/.Spotlight-V100/Store-V2/${UUID}/
+....
+
+But can also be stored in other locations such as:
 
 ....
-/.Spotlight-V100/Store-V2/${STORE_UUID}/store.db
-/.Spotlight-V100/Store-V2/${STORE_UUID}/.store.db
+/Users/${USERNAME}/Library/Caches/com.apple.helpd/index.spotlightV3/
+/Users/${USERNAME}/Library/Metadata/CoreSpotlight/index.spotlightV3/
+/Users/${USERNAME}/Library/Developer/Xcode/DocumentationCache/v18/9.0.1/DeveloperDocumentation.index/
 ....
 
+The Spotlight store consists of the following files:
+
+* .store.db
+* store.db
+
+In macOS 10.15 (Catalina) the following <<database_streams_map_files,database streams map files>>
+were added:
+
+* dbStr-#.map.buckets
+* dbStr-#.map.data
+* dbStr-#.map.header
+* dbStr-#.map.offsets
+
 [cols="1,5",options="header"]
 |===
 | Characteristics | Description
 | Byte order | little-endian
-| Date and time values |
+| Date and time values | Cocoa and POSIX date and time values
 | Character strings | UTF-8 formatted strings
 |===
 
+=== Test versions
+
+The following version of programs were used to test the information within this
+document:
+
+* Mac OS X 10.7 (Lion)
+* Mac OS X 10.8 (Mountain Lion)
+* Mac OS X 10.9 (Mavericks)
+* Mac OS X 10.10 (Yosemite)
+* Mac OS X 10.11 (El Capitan)
+* macOS 10.12 (Sierra)
+* macOS 10.13 (High Sierra)
+* macOS 10.14 (Mojave)
+* macOS 10.15 (Catalina)
+* macOS 11 (Big Sur)
+* macOS 12 (Monterey)
+* macOS 13 (Ventura)
+
 == Store database file
 
 A store database (store.db) file consists of:
@@ -85,7 +126,8 @@ The file header is at least 720 bytes in size and consists of:
 |===
 | Offset | Size | Value | Description
 | 0 | 4 | "8tsd" | Signature
-| 4 | 4 | | Flags
+| 4 | 4 | | Flags +
+See section: <<file_header_flags,File header flags>>
 | 8 | 4 | | [yellow-background]*Unknown (0-byte values)*
 | 12 | 4 | | [yellow-background]*Unknown* +
 Seen: 0x0c
@@ -113,6 +155,24 @@ The file header is stored in the first 4096 bytes
 The signature is "dst8" in little-endian, which cloud represention something
 in line of "data store".
 
+==== [[file_header_flags]]File header flags
+
+[cols="1,1,5",options="header"]
+|===
+| Value | Identifier | Description
+| 0x00000001 | | Seen in .store.db and store.db
+| | |
+| 0x00000004 | | Seen in .store.db and store.db
+| 0x00000008 | | Seen in .store.db
+| | |
+| 0x00000100 | | Seen in .store.db and store.db
+| | |
+| 0x00000400 | | Seen in .store.db
+| 0x00000800 | | Seen in .store.db and store.db
+| | |
+| 0x00010000 | | [yellow-background]*Unknown (Has database streams map files)?*
+|===
+
 === Pages
 
 ....
@@ -161,7 +221,7 @@ The map page value is 16 bytes in size and consists of:
 00001050  00 40 00 00 0e d2 01 00  00 00 00 00 51 00 00 00  |[email protected]...|
 ....
 
-TODO what about "1mbd"
+[yellow-background]*TODO what about "1mbd"*
 
 === Property table
 
@@ -201,8 +261,9 @@ Page contains a property table header
 Page contains a property table header
 | 0x00000081 | | Metadata lists or localized strings +
 Page contains a property table header
-| 0x00001009 | | data records +
-Page contains LZ4 compressed data
+3+| _Flags_
+| 0x00001000 | | Data is LZ4 compressed
+| 0x00004000 | | [yellow-background]*Unknown*
 |===
 
 ==== Compressed data
@@ -211,22 +272,35 @@ Page contains LZ4 compressed data
 |===
 | Value | Identifier | Description
 | "\x78" | | start of zlib+DEFLATE compressed data
-| "bv41" | | start of LZ4 compressed block
+| "bv41" | | LZ4 compressed block marker +
+See section: <<lz4_compressed_block,LZ4 compressed block>>
+| "bv4-" | | LZ4 uncompressed block marker +
+See section: <<lz4_uncompressed_block,LZ4 uncompressed block>>
+| "bv4$" | | end of LZ4 compressed stream marker
 |===
 
-===== [[lz4_compressed_block]]LZ4 compressed block
+==== [[lz4_compressed_block]]LZ4 compressed block
 
 [cols="1,1,1,5",options="header"]
 |===
 | Offset | Size | Value | Description
 4+| _LZ4 compressed block header_
-| 0 | 4 | "bv41" | Signature
+| 0 | 4 | "bv41" | LZ4 compressed block marker
 | 4 | 4 | | Uncompressed data size (in bytes)
-| 8 | 4 | | Block size (in bytes)
+| 8 | 4 | | LZ4 compressed data size (in bytes)
 4+| _LZ4 compressed block data_
 | 12 | ... | | LZ4 compressed data
-4+| _LZ4 compressed block footer_
-| ... | 4 | "bv4$" | | LZ4 end of compressed data marker
+|===
+
+==== [[lz4_uncompressed_block]]LZ4 uncompressed block
+
+[cols="1,1,1,5",options="header"]
+|===
+| Offset | Size | Value | Description
+| 0 | 4 | "bv4-" | LZ4 uncompressed block marker
+| 4 | 4 | | Uncompressed data size (in bytes)
+4+| _LZ4 uncompressed block data_
+| 8 | ... | | Uncompressed data
 |===
 
 ==== Property table header
@@ -252,7 +326,8 @@ The data record (type 0x09) is variable of size and consists of:
 4+| _Record data_
 | 4 | ... | | Identifier +
 A variable size integer that contains the file system identifier, e.g. CNID on HFS, of the corresponding file (system) entry
-| ... | 1 | | Flags
+| ... | 1 | | Data record flags +
+See section: <<data_record_flags,Data record flags>>
 | ... | ... | | Item identifier +
 Contains a variable size integer
 | ... | ... | | Parent identifier +
@@ -263,22 +338,21 @@ Contains a variable size integer that contains the number of microseconds since
 | ... | ... | | Properties array
 |===
 
-TODO: describe flags
+==== [[data_record_flags]]Data record flags
 
 [cols="1,1,5",options="header"]
 |===
 | Value | Identifier | Description
 | 0x01 | | [yellow-background]*Unknown (Is metadata?)* +
-Seen in record with identifier 0 *
-Does this influence the behavior of the value type of kMDStoreAccumulatedSizes ?
+Seen in record with identifier 0
 | 0x02 | |
 3+|
 | 0x10 | |
 | 0x20 | |
 | 0x40 | |
 |===
 
-TODO: describe property
+[yellow-background]*TODO: describe property*
 
 [cols="1,1,1,5",options="header"]
 |===
@@ -677,6 +751,109 @@ Data:
 | kMDItemWhereFroms | | |
 |===
 
+== [[database_streams_map_files]]Database streams map (dbStr-#.map) files
+
+The # in the filename corresponds to the nature of the strings in the map.
+
+[cols="1,1,5",options="header"]
+|===
+| Value | Identifier | Description
+| 1 | | Metadata types streams map
+| 2 | | Metadata values streams map
+| 3 | | Unknown values 0x41 streams map
+| 4 | | Metadata lists streams map
+| 5 | | Metadata localized strings streams map
+|===
+
+=== Database streams map header file (dbStr-#.map.header)
+
+The database streams map header file (dbStr-#.map.header) file is 56 bytes in
+size and consists of:
+
+[cols="1,1,1,5",options="header"]
+|===
+| Offset | Size | Value | Description
+| 0 | 8 | "\x00PataD\x00\x00" | Signature
+| 8 | 4 | | [yellow-background]*Unknown (Seen: 13)*
+| 12 | 4 | | [yellow-background]*Unknown (Seen: 0, 2)*
+| 16 | 4 | | [yellow-background]*Unknown (Seen: 1)*
+| 20 | 4 | | [yellow-background]*Unknown (size of the corresponding dbStr-#.map.data file?)*
+| 24 | 4 | | [yellow-background]*Unknown (page/block size or flags?)*
+| 28 | 4 | | [yellow-background]*Unknown (number of entries in the corresponding dbStr-#.map.ofsets file?)*
+| 32 | 4 | | [yellow-background]*Unknown (similar to value at offset 20)*
+| 36 | 4 | | [yellow-background]*Unknown (similar to value at offset 24)*
+| 40 | 4 | | [yellow-background]*Unknown (similar to value at offset 28)*
+| 44 | 4 | 0 | [yellow-background]*Unknown (empty)*
+| 48 | 4 | 0 | [yellow-background]*Unknown (empty)*
+| 52 | 4 | 0 | [yellow-background]*Unknown (empty)*
+|===
+
+=== Database streams map offsets file (dbStr-#.map.offsets)
+
+The database streams map offsets file (dbStr-#.map.offsets) file is 4096 bytes
+in size and consists of:
+
+[cols="1,1,1,5",options="header"]
+|===
+| Offset | Size | Value | Description
+| 0 | 4 x number of entries | | Array of 32-bit offsets +
+The offset of the value in the corresponding dbStr-#.map.data file
+| ... | ... | 0 | [yellow-background]*Unknown (empty)*
+|===
+
+=== Database streams map data file (dbStr-#.map.data)
+
+The database streams map data file (dbStr-#.map.data) is variable of size and
+consists of:
+
+* One or more stream values
+
+[NOTE]
+Note that the first stream value always appears to be a single 0-byte value.
+
+A stream value is variable of size and consists of:
+
+[cols="1,1,1,5",options="header"]
+|===
+| Offset | Size | Value | Description
+| 0 | ... | | Stream value size
+Contains a variable size integer +
+See section: <<variable_size_integer,Variable size integer>> +
+| ... | Stream value size | | Stream value
+|===
+
+==== Metadata attribute types
+
+The dbStr-1.map.data file contains metadata attribute types that consist of:
+
+[cols="1,1,1,5",options="header"]
+|===
+| Offset | Size | Value | Description
+| 0 | 1 | | Value type +
+See section: <<metadata_attribute_value_types,Metadata attribute value types>>
+| 1 | 1 | | Property type
+| 2 | ... | | Key name +
+Contains an UTF-8 encoded string with an end-of-string character
+|===
+
+==== Metadata attribute values
+
+The dbStr-2.map.data file contains metadata attribute values that consist of:
+
+[cols="1,1,1,5",options="header"]
+|===
+| Offset | Size | Value | Description
+| 0 | ... | | Metadata attribute value name +
+Contains an UTF-8 encoded string with an end-of-string character
+|===
+
+=== Database streams map buckets file (dbStr-#.map.buckets)
+
+The database streams map buckets file (dbStr-#.map.buckets) file is 4096 bytes
+in size and consists of:
+
+[yellow-background]*TODO: describe*
+
 == Notes
 
 ....