Enabling more IDE features that manipulate user code

[cartermp]

The F# tools offer plenty of features today, and although plenty more could be added with the current system we have in place, it has gaps that realistically prevent several kinds of features to be written.

I identified in #3750 (comment) that we have 3 fundamental gaps in F# tooling for VS today:

1. The F# tools in VS do not have a mapping from a document to a syntax tree and vice/versa. There is no "get syntax tree" operation on a document nor is there an ability to say, "here is the modified tree, now write it to the document". It is possible to work around this, which we do in all tooling, but it takes a lot of code and ends up with a lot of helpers being written.
2. The F# syntax tree is (1) lossy and (2) lacks key abilities like "find the first ancestor to this node" and "insert a new node before or after this other node". It's just not something the F# compiler service supports as concepts. This makes determining key information difficult (lots of code) and also makes writing new data into a user's document very, very challenging. What would normally be pretty simple - find the syntax node we're adjacent to and insert a new node of a particular kind - now requires manually wiring up the functionality to communicate between our lossy tree's data and an underlying document's text representation. It's error-prone and complicated code that is simply working at the wrong abstraction level.
3. We don't support the Roslyn syntax tree API, which is used for these refactorings, critically, to mark specific nodes as needing to be renamed. Roslyn's infrastructure will take care of starting a new rename session once the changes to the syntax tree are written with those annotations. So there's no way to trigger the rename after the text changes even if points (1) and (2) were worked around well enough.

What this means, concretely

These really hold for all tooling though, not just F# tools in VS. The following kinds of features are blocked by this issue:

• Code scaffolding features like:
  • A sane implement of "Implement Interface"
  • Implement abstract members in abstract class
  • Record field boilerplate generation when constructing a new record
  • Union case generation in pattern match expressions
  • etc.
• Code addition refactorings and quick fixes like:
  • Extract value from expression
  • Extract to function / local function
  • Inline value
  • Make reference cell a mutable value
  • Make type a struct
  • Make class abstract
  • Add missing [<Extension>] attribute
  • Add ignore to expression
  • Add catch-all pattern match case to enum pattern match
  • etc.
• Code removal refactorings, perhaps backed by a custom code analyzer
  • Remove unused local binding
  • Remove redundant pattern match expression
  • Remove redundant new keyword in object construction
  • Remove [<Struct>] to make a struct a reference type
  • etc.
• Refactorings that do it all
  • Simplify pipeline (e.g., replacing a filter --> map with a choose call)
  • etc.

In theory, every one of these features could be built in the F# tools today. In fact, the implement interface code fix does this today. However, the complexity of doing so would be incredibly high. Implementing these would be horribly error-prone and unmaintainable because we're working at the wrong abstraction level. Using the interface implementation code fix as an example, it is several hundred lines of code operating at a document level to identify where to insert new strings. This is simply the wrong layer of abstraction.

Fundamentally, these constructs are about manipulating a syntax tree. We need to be able to add or remove nodes at particular positions, then pass the updated tree to a routine or set of routines that understands how to write it into a document form. Code that directly manipulates a document, like we have today, is horribly complicated.

Things get even more complicated when you consider that formatting matters here. Let's say that you want to scaffold out a new record with the record scaffolding refactoring. This record has 200 fields (which we've seen a customer have). Do you really want that all to be on one line? Of course not? Unfortunately, it now gets complicated because we need to place fields on new lines and make sure all the columns line up correctly. Now repeat this for every single feature that implies some amount of code formatting that adheres to the rest of the user's code. The complexity is just too high to handle this at the document level.

What we'd like to do

@TIHan and I spoke at length about this, since he had prototyped a few things that can help us build features like I previously mentioned.

We're planning on picking one of the more complicated features - union case generation - and using it to help drive the infrastructure improvements that have to be added to the F# compiler service to enable the features to be written in a maintainable way. The feature would include:

• A better way to traverse a syntax tree to identify what we need to scaffold out
• Perhaps some additions to the syntax tree if we're missing information
• An ability to add nodes to a syntax tree
• An ability to format nodes the nodes that are added to a tree such that the formatting is aligned with the user's code / is correct

If done correctly, this will result in several capabilities added to the F# compiler service that can be used to implement more of the previously-mentioned features. That would then allow all F# editing environments to surface these features and wire them up to their own editor bindings (i.e., we would not push Roslyn APIs into this).

Additionally, this would allow us to rewrite several code fixes and refactorings we have already to be less code and no longer directly manipulate a document.

Prior art

The rider tools for F# have their own lexer/parser that constructs a ReSharper tree that backs the user's source code. It contains APIs to add/remove nodes, identify if things like a newline needs to be added, etc.

Roslyn refactorings and code fixes are all based on syntax trees. There is no direct document manipulation. Any time a refactoring adds or changes code, it changes the tree and passes the changed tree to an engine that knows how to serialize it to disk. Note that Roslyn has a smart diffing engine that doesn't redundantly re-create trees every time they operate on them.

The Visual F# Power Tools enabled a subset of these features for VS 2015. However, since the same limitations existed then, it was a lot of code (600+ lines per feature) of logic to traverse trees and manipuate strings, in addition to requiring their own smaller set of abstractions to make that work a bit easier. We would be unlikely to use their approach.

[cartermp]

An additional part of this, to address problem (3) I identified, is to write our own mapping to a Roslyn language-agnostic syntax tree in the VS F# tools layer. This would let us leverage things like annotating a ndoe to be renambed and light up specific refactorings.

There may also be a different path that implies a change on the Roslyn end to incorporate some F#-isms.

But either way, we don't have the tools to enable certain features.

[heronbpv]

What would be the feasibility of going full-on Roslyn for F#? Certainly not now, but as longer term prospect?
I'm asking mostly because it seems like the majority of our woes, tooling wise, comes from this difference in compiler services providers, so to speak.

[cartermp]

We would not just go full-on Roslyn, no. It would require a complete rewrite of the compiler.