Assemble list of quirks with our full text search engine (esp. punctuation)

[joewiz]

• As discovered by @mcdanielhn, a search for Banquo will not yield documents containing Banquo's or Banquo’s. (Lucene's Standard Analyzer doesn't treat an apostrophe (straight or curly) as a word marking boundary, following the Unicode word boundary specification rules.)

  • https://history.state.gov/search?q=Banquo
  • https://history.state.gov/search?q=Banquo%E2%80%99s

• A search for f-16 (the airplane) is effectively identical to a search for f 16 - and will yield documents containing f or 16. Making this a phrase search "f-16" helps, but returns results without the hyphen like f 16. Same with the proximity search, "f 16"~0.

  • https://history.state.gov/search?q=f-16
  • https://history.state.gov/search?q=%22f-16%22
  • https://history.state.gov/search?q=%22f+16%22
  • https://history.state.gov/search?q=%22f+16%22~0

• A search for an acronym with a slash returns a Lucene parsing error. For example, s/s (Office of the Secretariat Staff, Department of State) error:

  Syntax error in Lucene query string: Cannot parse 's/s': Lexical error at line 1, column 4. Encountered: after : "/s"

  • https://history.state.gov/search?q=s%2Fs

  A workaround is to use phrase or proximity search, but again we would get undesired results, e.g., U.S.S.R. and U.S.S. are matches for "s/s".

  • https://history.state.gov/search?q=%22s%2Fs%22

@HistoryAtState/editors: Please post any more examples you know of, and we'll work with @HistoryAtState/existsolutions to try out different Lucene analyzers and/or add advanced search form controls to see what combination can produce our expected results.

[joewiz]

See HistoryAtState/frus@fc3f8b3 for an experiment with stopwords.

[joewiz]

For more on stopwords, see:

• Lucene's default list of stopwords: https://stackoverflow.com/questions/17527741/what-is-the-default-list-of-stopwords-used-in-lucenes-stopfilter
• Pros/cons of stopwords: https://www.elastic.co/guide/en/elasticsearch/guide/current/stopwords.html (linked from the stackoverflow page above)
• Example of how our use of default stopwords leads to unexpected/unwanted search hits, see https://twitter.com/awmarrs/status/893213357251203077

[joewiz]

Regarding the banquo's example above, I did some experiments on this some months ago:

• https://gist.github.com/joewiz/99ce7ea1082bc4ad14f013da9ba2def5

Related issues:

• Lucene KeywordAnalyzer cannot be found eXist-db/exist#794
• Unexpected results using ft:get-field() with whitespace analyzer index eXist-db/exist#795
• exist:match missing from Lucene proximity searches eXist-db/exist#833

[joewiz]

Search scope, index configuration, and plumbing issues

• administrative-timeline: needs to be added to $search:SECTIONS
• frus-history/articles: need index on tei:div, needs to be added to $search:SECTIONS
• frus-history/documents: needs index on tei:body, needs to be added to $search:SECTIONS
• other-publications/serial-set: needs to be added to $search:SECTIONS
• other-publications/vietnam-guide: needs to be added to $search:SECTIONS
• wwdai: needs index on tei:div

Search omissions

• carousel and frus-history/events: have never been included in search, could be useful
• frus: volume titles not searched, only document titles
• pocom: searched only on persName, not on position title, org name, or country
• hsg-shell/pages: contains substantive text in some HTML
• tags: not searched or leveraged

[joewiz]

From Michael McCoyer:

Sending you another search “quirk” I just encountered (I hope when you encouraged me to send these along that you really meant it!) –

So, our common terminology in FRUS headers for the National Security Advisor is “President’s Assistant for National Security Affairs.” When I searched that syntax as a phrase in Kristin’s Human Rights volume (https://history.state.gov/historicaldocuments/frus1977-80v02), however, I received no hits. The phrase does, however, appear in a number of headers in the volume – e.g., Docs 4 and 16.

Is it possible that the search queries don’t search on headers, or that they search them differently? I will leave this puzzle in your capable hands…

The exact URL for his search that returned zero hits is:

https://history.state.gov/search?q=%22president%27s+assistant+for+national+security+affairs%22&volume-id=frus1977-80v02

However, I found that if I changed the apostrophe in "president's" from straight (') to curly (’), the expected hits suddenly appeared:

https://history.state.gov/search?q=%22president%E2%80%99s+assistant+for+national+security+affairs%22&volume-id=frus1977-80v02&within=documents&sort-by=relevance

Thus, it appears that our search engine treats the curly quote as a literal character - like a letter in a word - rather than as punctuation that should be dropped. We need to get the search engine to treat curly quotes as straight quotes.

[joewiz]

From @joshbotts via the mailbox:

Wanted to flag this mailbox inquiry as an instance where a case-sensitive search capability would come in handy. Searching for "goa" returns several hundred hits, but most of them are for abbreviations for "Government of ..." rather than the geographical entity in South Asia.

--> User story: I want to search for "Goa" and exclude hits that are upper case ("GOA")

[plutonik-a]

This comment #255 (comment) has been already issued here -> #289

[plutonik-a]

This issue has been spliced into different existing and new issues (including backlinks to this one):

• Implement Exact Phrase, Case & Punctuation Sensitivity search options #289
• [Search] Search scope, index configuration, and plumbing issues #349
• [Search] Enhance Lucene word boundaries / stopwords handling #350

Therefore closing this parent issue.

[marmoure]

@joewiz for searching for the term s/s we can escape the forward slash with a backward slash
s\/s and this will pass without the nasty lucene error