Full Text Search

There's an extension for that.

The full text search extension for YapDatabase is built atop the FTS module for SQLite. The FTS module was contributed by Google. As you might imagine, it's pretty fast. Especially when you compare it to Core Data:

I wrote a simple sample app in order to see what kind of performance gain I might see from using a separate full text index in SQLite. The app loads 1682 text files from textfiles.com (where else?), that’s about 42mb of plain text, into both Core Data model objects and an SQLite database. I then timed how long it took to find a single word using both a full text query and a core data fetch request. For one query running on the main thread on slowest device I have (a 4th gen. iPod touch) the Core Data fetch took 9.34 seconds while the SQLite query only took 1.48 seconds. (link)

With the full text search extension, you can have search up and running in minutes.

The primary header files can be found here:

• YapDatabaseFullTextSearch
• YapDatabaseFullTextSearchTransaction

 

Creating the FTS extension

The first step is deciding what you want to search. That is, deciding what you want the FTS module to index. For example, if you're storing tweets you may want to index the tweet content and the author name.

let propertiesToIndex = ["tweet", "author"]

The next step is to create a block which will extract the properties you want to index from an object in the database. It will look something like this:

let handler = YapDatabaseFullTextSearchHandler.withObjectBlock {
  (transaction, dict, collection, key, object: Any) in

  if let tweet = object as? Tweet {
    
    dict["author"] = tweet.author
    dict["tweet"] = tweet.tweet
  }
  else {
    // Don't need to index this item.
    // So we simply don't add anything to the dict.
  }
}

Just add the proper fields to the given dict parameter. The other parameters correspond to a row in the database. So all you have to do is inspect the row, decide if you want to index anything, and if so then extract the proper information and add it to the dictionary.

And lastly, we create the extension and register it with the database.

// The 'versionTag' is a string that tells the extension if you've
// made changes to the handler since last app launch.
// If the versionTag hasn't changed, the extension doesn't need to do anything.
// But if the versionTag HAS changed, the extension will automatically
// re-populate itself. That is, it will:
// - flush its internal tables
// - enumerate the database, and invoke your handler to see
//   if the row should be indexed
// 
// In other words, if you make changes to your handler code,
// then just change your versionTag, and the extension will do the right thing.
// 
let versionTag = "2019-05-20" // <= change me if you modify handler

let fts =
  YapDatabaseFullTextSearch(columnNames: propertiesToIndex,
                                handler: handler,
                             versionTag: versionTag)

let extName = "tweets_fts" // <= Name we will use to access this extension
database.asyncRegister(view, withName: extName) {(ready) in
			
  if !ready {
    print("Error registering \(extName) !!!")
  }
}

 

Performing searches

Once you've created your extension, and registered it, you're ready to start searching. Here's the basics:

dbConnection.read {(transaction) in

  // Find matches for: "board meeting"
  if let ftsTransaction = transaction.ext("tweets_fts") as? YapDatabaseFullTextSearchTransaction {
    
    ftsTransaction.enumerateKeys(matching: "board meeting") {
      (collection, key, stop) in
      // ...
    }
  }
}

You'll notice you access the extension within a transaction. This is in keeping with the rest of the database architecture. Which also means you get "atomic searches", and other benefits of transactions such as LongLivedReadTransactions (to simplify database access on the main thread).

In terms of search options, the extension supports everything that the SQLite FTS extension supports. Which is a LOT... token queries, token prefixes, phrases, NEAR, AND, OR, NOT ...

Rather than try to document all the possibilities here, I'll point you to the excellent examples and documentation here: SQLite FTS3 and FTS4 Extensions

Here's a few really simple examples:

dbConnection.read {(transaction) in

  guard let ftsTransaction = transaction.ext("tweets_fts") as? YapDatabaseFullTextSearchTransaction else {
    return
  }

  // tweet.author matches: john
  // and row matches: board meeting

  var query = "author:john board meeting"
  ftsTransaction.enumerateKeys(matching: query) {
    (collection, key, stop) in
    // ...
  }

  // tweet.author matches: john
  // tweet.tweet contains phrase: "board meeting"

  query = "author:john tweet:\"board meeting\""
  ftsTransaction.enumerateKeysAndObjects(matching: query) {
    (collection, key, obj, stop) in
    // ...
  }

  // find any tweets with the words "meeting" or "conference"

  query = "meeting OR conference"
  ftsTransaction.enumerateRows(matching: query) {
    (collection, key, obj, metadata, stop) in
    // ...
  }
}

Also, snippets are supported! So if you're searching large documents, you can use the snippets feature to get fragments of surrounding text that matched the search query. Very helpful to display to the user so they can easily identify the context of each search result, and quickly find exactly what they're looking for.

// Query matching + Snippets

func enumerateKeys(matching query: String, with options: YapDatabaseFullTextSearchSnippetOptions?, using block: (String, String, String, UnsafeMutablePointer<ObjCBool>) -> Void)