Document the tree-sitter grammars used on GitHub (github-linguist#5605)

* Document the tree-sitter grammars used on GitHub

* Move to vendor/README.md

* Words

* Add and update missing link

* Use tree-sitter urls

* Put back orig wording

* Use UTF-8

docs/releasing.md

@@ -28,11 +28,13 @@ This is the procedure for making a new release of Linguist. The entire process n
 12. Tag and push: `git tag vx.xx.xx; git push --tags`
 13. Create a GitHub release with the pushed tag (https://github.com/github/linguist/releases/new) and populate it with a list of the commits from `git log --pretty=format:"- %s" --reverse refs/tags/[OLD TAG]...refs/tags/[NEW TAG]` [like this](https://github.com/github/linguist/releases/tag/v7.2.0)
 14. Build a grammars tarball (`./script/build-grammars-tarball`) and attach it to the GitHub release
-15. Push to rubygems.pkg.github.com -- `gem push --key github --host https://rubygems.pkg.github.com/github github-linguist-3.0.0.gem`. See [Configuring RubyGems for use with GitHub Package Registry][gpr] for more details.
+15. Push to rubygems.pkg.github.com -- `gem push --key github --host https://rubygems.pkg.github.com/github github-linguist-3.0.0.gem`. See [Working with the RubyGems registry][gpr] for more details.
 16. Push to rubygems.org -- `gem push github-linguist-3.0.0.gem`
 17. Update and deploy the following repositories to use the new gem in production:
     - `github/github` - label for backporting to the latest version of GitHub Enterprise Server only.
     - `github/treelights` - this only needs the Linguist version updated to pull the compiled grammars from the Linguist release. Label for backporting to the latest version of GitHub Enterprise Server only.
     - `github/lightshow`
 
     Note: syntax highlighting changes won't take effect until the updated `github/treelights` repo has been deployed.
+
+[gpr]: https://docs.github.com/en/packages/working-with-a-github-packages-registry/working-with-the-rubygems-registry

script/list-grammars

@@ -5,6 +5,19 @@ require "linguist"
 require "json"
 require "yaml"
 
+TREE_SITTER_GRAMMARS = {
+  "CodeQL" => "https://github.com/tree-sitter/tree-sitter-ql",
+  "CSS" => "https://github.com/tree-sitter/tree-sitter-css",
+  "EJS" => "https://github.com/tree-sitter/tree-sitter-embedded-template",
+  "ERB" => "https://github.com/tree-sitter/tree-sitter-embedded-template",
+  "Go" => "https://github.com/tree-sitter/tree-sitter-go",
+  "HTML" => "https://github.com/tree-sitter/tree-sitter-html",
+  "JavaScript" => "https://github.com/tree-sitter/tree-sitter-javascript",
+  "Python" => "https://github.com/tree-sitter/tree-sitter-python",
+  "Ruby" => "https://github.com/tree-sitter/tree-sitter-ruby",
+  "TypeScript" => "https://github.com/tree-sitter/tree-sitter-typescript"
+}
+
 class GrammarList
 
   ROOT = File.expand_path "../../", __FILE__
@@ -62,6 +75,7 @@ class GrammarList
     markdown = ""
     @language_names.each do |item|
       lang  = Linguist::Language["#{item}"]
+      marker = ""
       scope = lang.tm_scope
       next if scope == "none"
       path  = @sources[scope] || scope
@@ -80,9 +94,14 @@ class GrammarList
         next unless submodule
         short_url = submodule[:short]
         long_url  = submodule[:url]
+        if TREE_SITTER_GRAMMARS.keys.include?(item)
+          short_url = shorten(TREE_SITTER_GRAMMARS[item])
+          long_url  = TREE_SITTER_GRAMMARS[item]
+          marker = " 🐌"
+        end
       end
       item = item.gsub("*", "\\*")
-      markdown += "- **#{item}:** [#{short_url}](#{long_url})\n"
+      markdown += "- **#{item}:** [#{short_url}](#{long_url})#{marker}\n"
     end
 
     markdown
@@ -91,7 +110,7 @@ class GrammarList
   # Update the file displaying the reader-friendly list of grammar repos
   def update_readme
     readme = "#{ROOT}/vendor/README.md"
-    preamble = File.read(readme).match(/\A.+?<!--.+?-->\n/ms)
+    preamble = File.read(readme).match(/\A.+?<!--.+?-->\n/mu)
     list = self.to_markdown
     File.write(readme, preamble.to_s + list)
   end

test/test_grammars.rb

@@ -37,7 +37,7 @@ def test_submodules_are_in_sync
   end
 
   def test_readme_file_is_in_sync
-    current_data = File.read("#{ROOT}/vendor/README.md").to_s.sub(/\A.+?<!--.+?-->\n/ms, "")
+    current_data = File.read("#{ROOT}/vendor/README.md").to_s.sub(/\A.+?<!--.+?-->\n/mu, "")
     updated_data = `script/list-grammars --print`
     assert_equal current_data, updated_data, "Grammar list is out-of-date. Run `script/list-grammars`"
   end

vendor/README.md

@@ -3,6 +3,8 @@ Grammar index
 
 This is a list of grammars that Linguist selects to provide syntax highlighting on GitHub. If you've encountered an error with highlighting, please find the grammar in the list below and report it to the appropriate repository.
 
+**Note:** grammars marked with 🐌 are not updated when Linguist is so upstream fixes may take longer to appear on GitHub.
+
 <!-- Everything below this line is auto-generated by script/list-grammars. Manual edits will be lost -->
 - **1C Enterprise:** [xDrivenDevelopment/atom-language-1c-bsl](https://github.com/xDrivenDevelopment/atom-language-1c-bsl)
 - **4D:** [ayoubserti/lang-4d](https://github.com/ayoubserti/lang-4d)
@@ -71,7 +73,7 @@ This is a list of grammars that Linguist selects to provide syntax highlighting
 - **CODEOWNERS:** [Alhadis/language-etc](https://github.com/Alhadis/language-etc)
 - **COLLADA:** [textmate/xml.tmbundle](https://github.com/textmate/xml.tmbundle)
 - **CSON:** [atom/language-coffee-script](https://github.com/atom/language-coffee-script)
-- **CSS:** [atom/language-css](https://github.com/atom/language-css)
+- **CSS:** [tree-sitter/tree-sitter-css](https://github.com/tree-sitter/tree-sitter-css) 🐌
 - **CUE:** [cue-sh/vscode-cue](https://github.com/cue-sh/vscode-cue)
 - **Cabal Config:** [atom-haskell/language-haskell](https://github.com/atom-haskell/language-haskell)
 - **Cap'n Proto:** [textmate/capnproto.tmbundle](https://github.com/textmate/capnproto.tmbundle)
@@ -88,7 +90,7 @@ This is a list of grammars that Linguist selects to provide syntax highlighting
 - **Closure Templates:** [mthadley/language-closure-templates](https://github.com/mthadley/language-closure-templates)
 - **Cloud Firestore Security Rules:** [jaysquared/atom-firestore-grammar](https://github.com/jaysquared/atom-firestore-grammar)
 - **CoNLL-U:** [odanoburu/conllu-linguist-grammar](https://github.com/odanoburu/conllu-linguist-grammar)
-- **CodeQL:** [github/vscode-codeql](https://github.com/github/vscode-codeql)
+- **CodeQL:** [tree-sitter/tree-sitter-ql](https://github.com/tree-sitter/tree-sitter-ql) 🐌
 - **CoffeeScript:** [atom/language-coffee-script](https://github.com/atom/language-coffee-script)
 - **ColdFusion:** [SublimeText/ColdFusion](https://github.com/SublimeText/ColdFusion)
 - **ColdFusion CFC:** [SublimeText/ColdFusion](https://github.com/SublimeText/ColdFusion)
@@ -123,7 +125,7 @@ This is a list of grammars that Linguist selects to provide syntax highlighting
 - **EBNF:** [Alhadis/language-grammars](https://github.com/Alhadis/language-grammars)
 - **ECL:** [hpcc-systems/ecl-tmLanguage](https://github.com/hpcc-systems/ecl-tmLanguage)
 - **ECLiPSe:** [alnkpa/sublimeprolog](https://github.com/alnkpa/sublimeprolog)
-- **EJS:** [gregory-m/ejs-tmbundle](https://github.com/gregory-m/ejs-tmbundle)
+- **EJS:** [tree-sitter/tree-sitter-embedded-template](https://github.com/tree-sitter/tree-sitter-embedded-template) 🐌
 - **EQ:** [dotnet/csharp-tmLanguage](https://github.com/dotnet/csharp-tmLanguage)
 - **Eagle:** [textmate/xml.tmbundle](https://github.com/textmate/xml.tmbundle)
 - **Easybuild:** [MagicStack/MagicPython](https://github.com/MagicStack/MagicPython)
@@ -172,7 +174,7 @@ This is a list of grammars that Linguist selects to provide syntax highlighting
 - **Glyph:** [textmate/tcl.tmbundle](https://github.com/textmate/tcl.tmbundle)
 - **Glyph Bitmap Distribution Format:** [Alhadis/language-fontforge](https://github.com/Alhadis/language-fontforge)
 - **Gnuplot:** [mattfoster/gnuplot-tmbundle](https://github.com/mattfoster/gnuplot-tmbundle)
-- **Go:** [AlanQuatermain/go-tmbundle](https://github.com/AlanQuatermain/go-tmbundle)
+- **Go:** [tree-sitter/tree-sitter-go](https://github.com/tree-sitter/tree-sitter-go) 🐌
 - **Go Checksums:** [golang/vscode-go](https://github.com/golang/vscode-go)
 - **Go Module:** [golang/vscode-go](https://github.com/golang/vscode-go)
 - **Golo:** [TypeUnsafe/sublime-golo](https://github.com/TypeUnsafe/sublime-golo)
@@ -187,7 +189,7 @@ This is a list of grammars that Linguist selects to provide syntax highlighting
 - **HAProxy:** [abulimov/atom-language-haproxy](https://github.com/abulimov/atom-language-haproxy)
 - **HCL:** [alexlouden/Terraform.tmLanguage](https://github.com/alexlouden/Terraform.tmLanguage)
 - **HLSL:** [tgjones/shaders-tmLanguage](https://github.com/tgjones/shaders-tmLanguage)
-- **HTML:** [atom/language-html](https://github.com/atom/language-html)
+- **HTML:** [tree-sitter/tree-sitter-html](https://github.com/tree-sitter/tree-sitter-html) 🐌
 - **HTML+ECR:** [atom-crystal/language-crystal](https://github.com/atom-crystal/language-crystal)
 - **HTML+EEX:** [elixir-lang/elixir-tmbundle](https://github.com/elixir-lang/elixir-tmbundle)
 - **HTML+ERB:** [atom/language-ruby](https://github.com/atom/language-ruby)
@@ -227,7 +229,7 @@ This is a list of grammars that Linguist selects to provide syntax highlighting
 - **Java:** [textmate/java.tmbundle](https://github.com/textmate/java.tmbundle)
 - **Java Properties:** [textmate/java.tmbundle](https://github.com/textmate/java.tmbundle)
 - **Java Server Pages:** [textmate/java.tmbundle](https://github.com/textmate/java.tmbundle)
-- **JavaScript:** [atom/language-javascript](https://github.com/atom/language-javascript)
+- **JavaScript:** [tree-sitter/tree-sitter-javascript](https://github.com/tree-sitter/tree-sitter-javascript) 🐌
 - **JavaScript+ERB:** [atom/language-javascript](https://github.com/atom/language-javascript)
 - **Jest Snapshot:** [jest-community/vscode-jest](https://github.com/jest-community/vscode-jest)
 - **Jinja:** [textmate/python-django.tmbundle](https://github.com/textmate/python-django.tmbundle)
@@ -367,7 +369,7 @@ This is a list of grammars that Linguist selects to provide syntax highlighting
 - **Pug:** [davidrios/jade-tmbundle](https://github.com/davidrios/jade-tmbundle)
 - **Puppet:** [russCloak/SublimePuppet](https://github.com/russCloak/SublimePuppet)
 - **PureScript:** [purescript-contrib/atom-language-purescript](https://github.com/purescript-contrib/atom-language-purescript)
-- **Python:** [MagicStack/MagicPython](https://github.com/MagicStack/MagicPython)
+- **Python:** [tree-sitter/tree-sitter-python](https://github.com/tree-sitter/tree-sitter-python) 🐌
 - **Python console:** [MagicStack/MagicPython](https://github.com/MagicStack/MagicPython)
 - **Python traceback:** [MagicStack/MagicPython](https://github.com/MagicStack/MagicPython)
 - **Q#:** [microsoft/qsharp-compiler](https://github.com/microsoft/qsharp-compiler)
@@ -403,7 +405,7 @@ This is a list of grammars that Linguist selects to provide syntax highlighting
 - **Roff:** [Alhadis/language-roff](https://github.com/Alhadis/language-roff)
 - **Roff Manpage:** [Alhadis/language-roff](https://github.com/Alhadis/language-roff)
 - **Rouge:** [atom/language-clojure](https://github.com/atom/language-clojure)
-- **Ruby:** [atom/language-ruby](https://github.com/atom/language-ruby)
+- **Ruby:** [tree-sitter/tree-sitter-ruby](https://github.com/tree-sitter/tree-sitter-ruby) 🐌
 - **Rust:** [zargony/atom-language-rust](https://github.com/zargony/atom-language-rust)
 - **SAS:** [rpardee/sas.tmbundle](https://github.com/rpardee/sas.tmbundle)
 - **SCSS:** [atom/language-sass](https://github.com/atom/language-sass)
@@ -473,7 +475,7 @@ This is a list of grammars that Linguist selects to provide syntax highlighting
 - **Turtle:** [peta/turtle.tmbundle](https://github.com/peta/turtle.tmbundle)
 - **Twig:** [Anomareh/PHP-Twig.tmbundle](https://github.com/Anomareh/PHP-Twig.tmbundle)
 - **Type Language:** [goodmind/language-typelanguage](https://github.com/goodmind/language-typelanguage)
-- **TypeScript:** [Microsoft/TypeScript-TmLanguage](https://github.com/Microsoft/TypeScript-TmLanguage)
+- **TypeScript:** [tree-sitter/tree-sitter-typescript](https://github.com/tree-sitter/tree-sitter-typescript) 🐌
 - **Unified Parallel C:** [textmate/c.tmbundle](https://github.com/textmate/c.tmbundle)
 - **Unity3D Asset:** [atom/language-yaml](https://github.com/atom/language-yaml)
 - **Unix Assembly:** [calculuswhiz/Assembly-Syntax-Definition](https://github.com/calculuswhiz/Assembly-Syntax-Definition)