Experimental inline completer

[krassowski]

References

Relates to #14267

Reference implementation: krassowski/jupyterlab-transformers-completer

Code changes

• Adds a new IInlineCompletionProvider for extensions to provide inline completions (for example from various LLMs)
• Extends public ICompletionManager API with new methods:
  • .registerInlineProvider(provider: IInlineCompletionProvider)
  • inline actions namespace:
    • .inline.invoke(id: string)
    • .inline.cycle(id: string, direction: 'next' | 'previous')
    • .inline.accept(id: string)
    • .inline.configure(settings)
• Implements an example provider of inline completions using kernel history request

Provider API overview

/**
 * The interface extensions should implement to provide inline completions.
 */
export interface IInlineCompletionProvider {
  /**
   * Name of the provider to be displayed in the user interface.
   */
  readonly name: string;

  /**
   * Unique identifier, cannot change on runtime.
   *
   * The identifier is also added on data attribute of ghost text widget,
   * allowing different providers to style the ghost text differently.
   */
  readonly identifier: string;

  /**
   * The icon representing the provider in the user interface.
   */
  readonly icon?: LabIcon.ILabIcon;

  /**
   * Settings schema contributed by provider for user customization.
   */
  readonly schema?: ISettingRegistry.IProperty;

  /**
   * The method called when user requests inline completions.
   *
   * The implicit request (on typing) vs explicit invocation are distinguished
   * by the value of `triggerKind` in the provided `context`.
   */
  fetch(
    request: CompletionHandler.IRequest,
    context: IInlineCompletionContext
  ): Promise<IInlineCompletionList>;

  /**
   * Optional method called when user changes settings.
   *
   * This is only called if `schema` for settings is present.
   */
  configure?(settings: { [property: string]: JSONValue }): void;

  /**
   * Optional method to stream remainder of the `insertText`.
   */
  stream?(token: string): AsyncGenerator<{ response: IInlineCompletionItem }>;
}

IInlineCompletionItem

interface IInlineCompletionItem {
  /**
   * The text to replace the range with. Must be set.
   * Is used both for the preview and the accept operation.
   */
  insertText: string;

  /**
   * A text that is used to decide if this inline completion should be
   * shown. When `falsy` the insertText is used.
   *
   * An inline completion is shown if the text to replace is a prefix of the
   * filter text.
   */
  filterText?: string;

 /**
  * Token passed to identify the completion when streaming updates
  */
  token?: string;

  /**
   * Whether generation of `insertText` is still ongoing. If your provider supports streaming,
   * you can set this to true, which will result in the provider's `stream()` method being called
   * with `token` which has to be set for incomplete completions.
   */
  isIncomplete?: boolean;
}

InlineCompletionTriggerKind

export enum InlineCompletionTriggerKind {
  /**
   * Completion was triggered explicitly by a user gesture.
   * Return multiple completion items to enable cycling through them.
   */
  Invoke = 0,
  /**
   * Completion was triggered automatically while editing.
   * It is sufficient to return a single completion item in this case.
   */
  Automatic = 1
}

IInlineCompletionContext

/**
 * The context which will be passed to the inline completion provider.
 */
export interface IInlineCompletionContext {
  /**
   * The widget (notebook, console, code editor) which invoked
   * the inline completer
   */
  widget: Widget;

  /**
   * Describes how an inline completion provider was triggered.
   */
  triggerKind: InlineCompletionTriggerKind;

  /**
   * The session extracted from widget for convenience.
   */
  session?: Session.ISessionConnection | null;
}

CompletionHandler.IRequest

Note: this differs from context in that the request can change within editor depending on editor state (content, cursor position), while context (except for triggerKind) is largely stable. mimeType can change when transclusions are present, for example JavaScript within <script></script> tag. For markdown cells a markdown mime type is returned.

/**
 * The details of a completion request.
 */
export interface IRequest {
  /**
   * The cursor offset position within the text being completed.
   */
  offset: number;
  /**
   * The text being completed.
   */
  text: string;

  /**
   * The MIME type under the cursor.
   */
  mimeType: string;
}

Advantages:

• the IInlineCompletionItem and InlineCompletionTriggerKind is a super-set of proposed Language Server Protocol v3.18 API for inline completions which will enable connecting LSP servers as a source of inline completions in the future
• streaming is supported; this means that provider can return a number of empty completion candidates and only populate them later
• different cell types and fragments of cells can be easily distinguished by mimeType of request
• allows providers to put settings in the Inline Completer tab of settings rather than having the settings scattered around (as it is currently a problem and confusion for LSP users because the Tab Completer does not offer such API)

Limitations:

• as-proposed the provider needs to know how many completion candidates it will yield; it would be relatively easy to remove this limitation if we want to; possible approaches would be:
  • mirroring the LSP approach for tab completions by adding add isIncomplete: bool on IInlineCompletionList and a new trigger kind TriggerForIncompleteCompletions on InlineCompletionTriggerKind; then when provider's fetch() method would be called for the second time with this trigger kind if the list was incomplete and provider would return additional items.
  • adding a new token on IInlineCompletionList result and a new method streamList(token) method on IInlineCompletionProvider (but by virtue of sticking with LSP land I think the above is better; we could even propose it upstream)

User-facing changes

Shortcuts on (default)	Shortcuts off

The last icon represents provider (source) of the completions (here history) and is cusomizable by providers.

Note: the intention is to allow accepting the suggestion with Tab but this will likely follow in a separate pull request once #14115 is merged.

New settings section

(plus Code Completer got an icon ✨)

Backwards-incompatible changes

To check.

To be done

• inline completion widget (hover box)
  • reduce space taken by the widget (UI/UX input welcome)
  • ensure it is placed above the line so that it never obstructs neither the existing nor proposed code
  • add a setting to enable/disable it
  • decide if it should be opt-in or opt-out
• settings
  • document how to disable inline completions globally
  • setting to enable/disable specific providers
    • make the history provider off by default (so no providers are there and effectively the feature is off)
• tests:
  • unit
  • integration
• document the new API
• rethink what shortcuts to use to allow implementing accepting partial suggestions in the future (it seems we should go for tab and then use arrows with a modifier for partial completions)
• implement streaming support
• improve history provider
• nice to have:
  • make the provider icon a menu with provider options
  • maybe put the counter inside prev/next buttons and hug them
  • check for multi-cursor behaviour
  • allow to only show the widget on hover
  • ensure clicking on the widget does not hide the suggestion (happens due to blur if we click on non-button)
  • loading indicator when more than one provider and when streaming
  • improve rendering of the shortcuts (use kbd)
  • experiment with partial transparency and blur behind the widget to improve user awareness
  • "stop" signal to notify provider that model was reset and generation can be aborted (important if tokens cost)

[jupyterlab-probot]

Thanks for making a pull request to jupyterlab!
To try out this branch on binder, follow this link:

[fcollonval]

better default icon/icon for the history provider

What about https://pictogrammers.com/library/mdi/icon/history/

[fcollonval]

Thanks @krassowski

Here are my questions/suggestions on the API.

[fcollonval]

should we use @Alpha or @experimental to align with the new API ts doc?

[krassowski]

I wanted to ask about this. Do you have a link explaining differences? From the docs it seems that @experimental == @beta. I see that in injectExtension we have both:

jupyterlab/packages/codemirror/src/editor.ts

Lines 226 to 227 in e00fdc4

	* @alpha
	* @experimental

I don't know which one we should prefer.

[fcollonval]

If I recall correctly, this is exactly what is mentioned in tsdoc:

Same semantics as @\beta, but used by tools that don't support an @\alpha release stage.

Some tools have experimental and not alpha/beta, other the other way around (like API extractor). I think experimental was even the only one supported by tsdoc in the past.

[krassowski]

Would you prefer me to change the existing tag comments to:

    * @alpha 
    * @experimental 

or are they good as they are?

[fcollonval]

Let's keep it like this as alpha / beta is clearer than experimental in term of API stability expectation.

[krassowski]

Thank you for taking a look @fcollonval, hugely appreciated!

[krassowski]

I wanted to ask about this. Do you have a link explaining differences? From the docs it seems that @experimental == @beta. I see that in injectExtension we have both:

jupyterlab/packages/codemirror/src/editor.ts

Lines 226 to 227 in e00fdc4

	* @alpha
	* @experimental

I don't know which one we should prefer.

[krassowski]

Good point, there is a place in the IModel where it seems more relevant (but then it is further away from the extension point). I do not see it for cycleInline specifically, can you elaborate? The future possibilities for extension that I see could be direction: 'first' | 'last'.

[krassowski]

bot please update galata snapshots

[github-actions]

Galata snapshots updated.

[dongs0104]

Hello @krassowski In this case, I don't get additional information like type of documentation from the inline completer like I do from the completer widget, is this intended behavior?
The current LLM model is more generative when you give it a lot of context, so it would be nice to get more information.
Can i Parse using context on it?

[krassowski]

Hello @dongs0104 👋

I don't get additional information like type of documentation from the inline completer like I do from the completer widget, is this intended behavior?

Yes, this is documented, quoting from completer.rst file:

Inline completer

JupyterLab 4.1+ includes an experimental inline completer, showing the suggestions
as greyed out "ghost" text. Compared to the completer widget, the inline completer:

• can present multi-line completions
• is automatically invoked as you type
• does not offer additional information such as type of documentation for the suggestions
• can provide completions in both code and markdown cells (the default history provider only suggests in code cells)

The inline completer is disabled by default and can be enabled in the Settings Editor by enabling the History Provider.

We could add additional information about the inline completion in the future (e.g. allow models to generate both text and a plain-text comment) but "type" really does not make sense when returning multiple lines of code, for example if type impo and the model completes it to:

import pandas
x = pandas.DataFrame()
y = 1
print(x, y)

There is no single type here. Similarly for documentation.

The current LLM model is more generative when you give it a lot of context, so it would be nice to get more information.
Can i Parse using context on it?

Yes, you can get the content of editor/notebook from context. Extracting a single line is presented the History provider, while the reference implementation in https://github.com/krassowski/jupyterlab-transformers-completer demonstrates using more lines (with a cap at the context length because this has an impact on generation speed).

[krassowski]

If there are no further comments, it would be helpful to merge and cut a pre-release to allow wider testing (setting up a binder with krassowski/jupyterlab-transformers-completer is proving to be difficult due to Binder runners restrictions).

[fcollonval]

Hey @krassowski

I only have one comment about the icon for the settings. It seems they are not using jp-icon classes and therefore may not be styled properly depending on the theme. For example in dark mode:

[fcollonval]

I was not seeing a specific case for cycleInline. It was more to be sure to think a second times about that; as you did 😉

[fcollonval]

Let's keep it like this as alpha / beta is clearer than experimental in term of API stability expectation.

[fcollonval]

Thanks @krassowski