Handle Markdown and Tutorial files in textDocument/doccDocumentation

[matthewbastien]

Adds a handler to DocumentationLanguageService for the textDocument/doccDocumentation LSP extension. The request is handled much in the same way as for SwiftLanguageService where the SwiftDocC convert request is assembled and then sent to SwiftDocC for processing. All of the SwiftDocC specific code has been moved into a new DocCDocumentation module to try and keep SourceKit-LSP clean.

SwiftDocC supports four different kinds of Markdown/Tutorial files:

• Symbol Extensions - Written in markdown with a symbol link surrounded by double backticks as the header. The symbol link must be found in the index and matched to a symbol within a Swift file.
• Articles - Written in markdown as a standalone page. Can contain links to other Markdown/Tutorial content.
• Tutorials - Uses the *.tutorial file extension and represents a self-contained tutorial.
• Tutorial Overviews - Also uses the *.tutorial file extension, but represents a group of tutorials.

In addition to finding the Swift symbol for a symbol extension page, we must do the reverse when generating documentation for a symbol within a Swift file. This involves looking for *.docc documentation catalogs within the target and asking SwiftDocC to build an index for us. We can then retrieve all of the markdown extension pages and their associated symbols to do the lookup.

[matthewbastien]

@swift-ci please test

[matthewbastien]

@swift-ci please test

[matthewbastien]

@swift-ci please test windows platform

[matthewbastien]

@swift-ci please test

[matthewbastien]

@swift-ci please test windows platform

[ahoppen]

Thanks for working on this @matthewbastien 🙏🏽

I left some comments inline. On a layering level, I think we should not build the DocCDocumentation module at all if we are building SourceKit-LSP without docc support. That way DocCDocumentation doesn’t need any #if statements and we should be able to eliminate all the protocols. Instead the code in the SourceKitLSP module would need to be guarded by #if statements.

[ahoppen]

How do we resolve overloads here? Eg. if you have foo(x: Int) and foo(y: Int) in your module, how do we know which one to pick?

[matthewbastien]

Using LinkCompletionTools gets us 80% of the way there for disambiguations, but currently this PR does not handle function overloads. We'll need a way to get the function parameter names from the index. I'll open up a separate PR for this after this one gets merged.

[matthewbastien]

@swift-ci please test

[matthewbastien]

@swift-ci please test

[matthewbastien]

@swift-ci please test

[matthewbastien]

@swift-ci please test

[matthewbastien]

@swift-ci please test windows platform

[matthewbastien]

@swift-ci please test

[matthewbastien]

@swift-ci please test windows platform

[ahoppen]

Architecturally this looks good to me now, with one concern about the cursor info request on the helper document, the rest should be smaller comments. Also, just for my own sake, the following comments are still open from the last review:

• Handle Markdown and Tutorial files in textDocument/doccDocumentation #1959 (comment)
• Handle Markdown and Tutorial files in textDocument/doccDocumentation #1959 (comment)
• Handle Markdown and Tutorial files in textDocument/doccDocumentation #1959 (comment)
• Handle Markdown and Tutorial files in textDocument/doccDocumentation #1959 (comment)

[ahoppen]

I might have missed it but I didn’t find a use of the patching method taking a String. Do we need it? Also, I didn’t find any use of patching in your code, so I think this can continue to be internal.

[matthewbastien]

This was a remnant I forgot to clean up from an experiment I was doing locally. It's being used by cursorInfoFromDisk() now.

[ahoppen]

Also, I just noticed that DoccDocumentationRequest is not covered in MessageHandlingDependencyTracker.swift. We definitely need to fix that.

This approach is racy. Consider the following two scenarios:

1. We get a documentation request for a tutorial document. The .swift file for the symbol is open so we pass this check. Now, before we execute the cursor info request the clients sends us a textDocument/didClose notification for the symbol’s .swift file. Unless textDocument/doccDocumentation is a workspace request, we can handle the didClose for the .swift file concurrently to the documentation request for the tutorial file because they don’t affect the same document. Now, the document will be closed in sourcekitd and the cursor info will fail.
2. We get a documentation request for a tutorial document. The .swift file for the symbol is not open so we open it here. Before we finish this request, we get a textDocument/didOpen for the symbol’s document, which triggers another open of the document. Then we close the document from here, which will close it in sourcekitd and now the user can’t get any functionality in the .swift document they opened in the editor unless they close and re-open it.

For other requests that use cursor info like hover, this is not an issue because they are .documentRequest, which cannot be handled simultaneously to .documentUpdate requests like didOpen or didClose.

I think there are two solutions here:

1. Implement some kind of reference-counting-like logic by which we can say that we are interested in this document to prevent it from getting closed in sourcekitd. This will definitely be non-trivial to get right because you also need to consider cases like us opening a helper document with on-disk contents here and then a user sending a textDocument/didOpen request with contents that don’t match the on-disk contents.
2. We open a helper document that has a prefix like DocumentSymbols: here. If we can ensure that this document name is not getting used by anything else and that there will never be overlapping requests that open the same document, this will be safe because we have exclusive ownership of that helper document. One thing to consider here is that we could have multiple concurrent textDocument/doccDocumentation requests that could try and open the same helper document. One advantage of this approach is that we could also always open the on-disk contents, which means that in-memory modifications of the file can’t shift the positions of the symbol.

All of this would need to be handled by a separate cursorInfo function and it can’t be implicit in the current cursorInfo function, I think.

[matthewbastien]

Added a new cursorInfoFromDisk() function that will always load the DocumentSnapshot from disk for situations where a request needs to access a document that it isn't directly tied to. It uses a UUID prefix to avoid name collisions.

[matthewbastien]

Also, DoccDocumentationRequest is a TextDocumentRequest which is handled already by MessageHandlingDependencyTracker.

[matthewbastien]

@swift-ci please test

[matthewbastien]

@swift-ci please test windows platform

[matthewbastien]

@swift-ci please test

[ahoppen]

@swift-ci Please test macOS

[ahoppen]

@swift-ci Please test Windows

[matthewbastien]

@swift-ci please test

[matthewbastien]

@swift-ci please test Windows

[matthewbastien]

@swift-ci please test Linux