☂️ Handle remaining unimplemented content parsing features

[PIG208]

This is a follow-up to #190 and #917, to serve as an umbrella issue for unimplemented features to be handled.

Current features

• <table>: Handle <table> elements #360
• <span class="topic-mention">, <span class="topic-mention silent">: Parse @-topic mentions #892
• <div class="codehilite">: Handle codehilite elements with multiple classes #933
• <span class="tex-error">: Handle "tex-error" spans #1003
• <span class="timestamp-error">: Handle "timestamp-error" <span> elements #922
• <div class="message_inline_ref">: Handle legacy message_inline_ref Dropbox previews #962

Legacy features/behaviors in old messages

• (to mid-2023) <div class="inline-preview-twitter">: Handle Twitter previews #660
• (to 2022-03) several old forms of KaTeX output: <span class="katex-display">, <span class="katex">, <p>: Handle old forms of HTML for math/TeX #1004
• (to 2020) <img class="message_body_gravatar">: Handle legacy !gravatar images (img.message_body_gravatar) #963
• (to 2019) a legacy form of <div class="message_inline_image">: Handle legacy form of message_inline_image #964

Short-lived experiments

In the public chat.zulip.org message history, each of these appears for only a brief period. It's not clear any of them went to Zulip Cloud or into a published Zulip Server release.

• Issue for the whole of this section: Handle old experimental content features, or get them cleaned up #965

• <span class="timestamp">: Added -> Removed (example)
• <span class="inline-subscribe">. Added -> Removed (example)
• <img>: It was this experiment for markdown: Add support for ![alt text](url) syntax for images. zulip#29300.
• <div class="message_embed"> (example)

---

Full Output

Found unimplemented features in 33742 out of 1295577 public messages:

• <span class="topic-mention">
  Oldest message: 1609467; newest message: 1925998 (15/33742)

• <table>
  Oldest message: 33947; newest message: 1931762 (348/33742)

• <div class="codehilite">
  Oldest message: 3444; newest message: 1845456 (10175/33742)

• <img>
  Oldest message: 1792633; newest message: 1792683 (3/33742)

• <span class="tex-error">
  Oldest message: 176408; newest message: 1768950 (25/33742)

• <span class="topic-mention silent">
  Oldest message: 1609468; newest message: 1704065 (2/33742)

• <div class="inline-preview-twitter">
  Oldest message: 29260; newest message: 1574645 (22176/33742)

• <span class="katex-display">
  Oldest message: 202662; newest message: 1355972 (19/33742)

• <span class="timestamp-error">
  Oldest message: 925908; newest message: 1267388 (10/33742)

• <div class="message_inline_ref">
  Oldest message: 61290; newest message: 945000 (26/33742)

• <img class="message_body_gravatar">
  Oldest message: 15312; newest message: 927237 (55/33742)

• <span class="timestamp">
  Oldest message: 882554; newest message: 908075 (38/33742)

• <p>
  Oldest message: 176412; newest message: 908053 (65/33742)

• <div class="message_inline_image">
  Oldest message: 4324; newest message: 751747 (709/33742)

• <span class="katex">
  Oldest message: 308073; newest message: 426840 (3/33742)

• <div class="message_embed">
  Oldest message: 192764; newest message: 193181 (28/33742)

• <span class="inline-subscribe">
  Oldest message: 4297; newest message: 97826 (45/33742)

---

These features are categorized to help us determine the priorities.

• The current features should be supported before launch. There can also be potential bugs in the content parser affecting current features.
• Most of the legacy features (such as inline-subscribe) are irrelevant enough that we probably just need to acknowledge them and render them as plain text.
• The one-off experiments can likely be addressed by just removing the messages from CZO, without us handling them.

We might add more unimplemented features here as we find more of them later. We tested on all public messages from CZO (1295577 messages).

Also related:

• Revise "unimplemented content" UX for general release #194

[gnprice]

Thanks @PIG208 for doing this survey, and for compiling all this data!

I've gone and filed issues for the remaining items that didn't already have them:

• Handle legacy message_inline_ref Dropbox previews #962
• Handle legacy !gravatar images (img.message_body_gravatar) #963
• Handle legacy form of message_inline_image #964
• Handle old experimental content features, or get them cleaned up #965

and added the link for one that did:

• Handle Twitter previews #660

All of those are post-launch issues, as is #922.

It's encouraging to see that almost all the features this found were fairly obscure. In the list above, the issues that I think we'll want to handle before launching this app to replace the legacy app are three features we knew about already:

• Handle <table> elements #360
• TeX in the message list #46
• Parse @-topic mentions #892

and one issue we didn't:

• Handle codehilite elements with multiple classes #933

Now that I've filed those other sub-issues (in addition to #922 and #933 which you filed), I'll close this issue in favor of the specific sub-issues. This doesn't belong in the "Launch" milestone since we won't be completing it before launch… but if it's in "Post-launch" then we won't be seeing it on our dashboards, so we'll be relying on the sub-issues for tracking anyway.

[gnprice]

After looking through the #917 script's output myself, I realized that three classes that we had attributed to the known issue that we don't yet support TeX:

• TeX in the message list #46

are really a pair of other, low-priority issues which we hadn't known about (and I just filed):

• Handle "tex-error" spans #1003
• Handle old forms of HTML for math/TeX #1004

I've edited the description to reflect that.

In fact #46 itself doesn't relate to this search for unimplemented content, because TeX nodes — apart from the cases covered by #1003 and #1004 — are already implemented by our parser, as part of #359. Our display of them isn't as rich as we'd like it to be (that's #46), but this survey doesn't have a way to know about that.