Implement ClientIO interaction support on Agent-Server by GeorgeNgMsft · Pull Request #2178 · microsoft/TypeAgent

GeorgeNgMsft · 2026-04-09T22:49:23Z

Async ClientIO Callbacks: Non-Blocking Deferred-Promise Pattern
Three ClientIO methods — askYesNo, proposeAction, and popupQuestion — were previously implemented as blocking RPC invoke calls in the agent server. This meant the server-side command processing was tied to a specific client connection: a disconnect would permanently hang the operation, reconnecting clients couldn't recover pending questions, and popupQuestion was entirely disabled in server mode.

This PR converts all three to a non-blocking deferred-promise pattern:

The server generates a unique interactionId, logs the pending interaction to the DisplayLog, stores a deferred Promise in a new PendingInteractionManager, and sends a fire-and-forget requestInteraction notification to the client(s). Command processing suspends at await as before, but is no longer coupled to the connection.

The client responds via a new respondToInteraction dispatcher method, which resolves the deferred promise and resumes processing exactly where it left off.
On client disconnect, pending interactions owned by that connection are cancelled (rejecting their promises), and all callers treat this as a cancellation of the operation.
popupQuestion is now fully supported in server mode via broadcast to all connected clients, with first-responder resolution.

Pending interactions are logged to DisplayLog so they can be replayed to reconnecting or newly joining clients.
Existing callers (askYesNoWithContext, confirmTranslation, sessionContext.popupQuestion) required no changes — the ClientIO interface signatures are unchanged.

Proposes converting the blocking askYesNo, proposeAction, and popupQuestion callbacks to a non-blocking deferred-promise pattern modeled on the existing requestChoice/respondToChoice mechanism. Covers protocol changes, DisplayLog schema extensions, pending interaction management, replay/reconnection strategy, and a phased implementation plan. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

…cks (Phases 1-8) Convert askYesNo, proposeAction, and popupQuestion from blocking RPC callbacks to a non-blocking deferred-promise pattern with correlation IDs, enabling multi-client broadcast and resilient disconnect handling. - Define PendingInteractionRequest/Response types and discriminated union for askYesNo, proposeAction, popupQuestion interaction types - Add PendingInteractionManager with create/resolve/cancel lifecycle, per-connection cleanup, and timeout support - Extend DisplayLog with logPendingInteraction and logInteractionResolved for interaction persistence and replay - Add requestInteraction/interactionResolved fire-and-forget methods to ClientIO interface and wire through RPC client/server layers - Add respondToInteraction to Dispatcher interface with RPC support - Rewrite SharedDispatcher to use deferred promises instead of blocking callbacks, with broadcast to all clients and first-responder resolution - Add requestInteraction/interactionResolved stubs to all ClientIO implementations (shell, CLI, browser extension, console, MCP, tests) - Add @typeagent/dispatcher-types dependency to agent-server and protocol Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

…b.com/microsoft/TypeAgent into dev/georgeng/async-clientio-callbacks

Copilot

Pull request overview

This PR introduces an async “deferred interaction” mechanism for ClientIO prompts in agent-server mode, so server-side command execution is no longer coupled to a single blocking RPC invoke call and can survive disconnect/reconnect scenarios.

Changes:

Adds new dispatcher-types for pending interaction requests/responses and extends ClientIO/Dispatcher APIs to support requestInteraction + respondToInteraction.
Implements server-side tracking/logging of pending interactions via PendingInteractionManager and new DisplayLog entry types.
Wires new RPC surface area and propagates pending interactions to clients on session join.

Reviewed changes

Copilot reviewed 29 out of 30 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
ts/pnpm-lock.yaml	Adds workspace dependency links for `@typeagent/dispatcher-types`.
ts/packages/shell/src/renderer/src/main.ts	Adds `ClientIO` stubs for deferred interaction events (currently no-op).
ts/packages/dispatcher/types/src/pendingInteraction.ts	New shared types for pending interaction request/response payloads.
ts/packages/dispatcher/types/src/index.ts	Re-exports pending interaction types.
ts/packages/dispatcher/types/src/displayLogEntry.ts	Adds `pending-interaction` and `interaction-resolved` log entry types.
ts/packages/dispatcher/types/src/dispatcher.ts	Adds `respondToInteraction` to the Dispatcher interface.
ts/packages/dispatcher/types/src/clientIO.ts	Adds `requestInteraction` and `interactionResolved` to `ClientIO`.
ts/packages/dispatcher/rpc/src/dispatcherTypes.ts	Exposes `respondToInteraction` through dispatcher RPC invoke functions.
ts/packages/dispatcher/rpc/src/dispatcherServer.ts	Implements RPC server handler for `respondToInteraction`.
ts/packages/dispatcher/rpc/src/dispatcherClient.ts	Implements RPC client call for `respondToInteraction`.
ts/packages/dispatcher/rpc/src/clientIOTypes.ts	Adds new ClientIO call-function typings for deferred interactions.
ts/packages/dispatcher/rpc/src/clientIOServer.ts	Wires new ClientIO call handlers on the RPC server side.
ts/packages/dispatcher/rpc/src/clientIOClient.ts	Wires new ClientIO call senders on the RPC client side.
ts/packages/dispatcher/nodeProviders/test/provider.spec.ts	Updates test ClientIO stub to satisfy the extended interface.
ts/packages/dispatcher/dispatcher/src/internal.ts	Exposes `PendingInteractionManager` via internal exports.
ts/packages/dispatcher/dispatcher/src/helpers/console.ts	Adds no-op stubs for deferred interactions in console ClientIO.
ts/packages/dispatcher/dispatcher/src/displayLog.ts	Adds logging helpers for pending interactions and their resolution.
ts/packages/dispatcher/dispatcher/src/dispatcher.ts	Adds base `respondToInteraction` method (throws by default).
ts/packages/dispatcher/dispatcher/src/context/pendingInteractionManager.ts	New manager to store deferred promises and handle cancel/resolve.
ts/packages/dispatcher/dispatcher/src/context/interactiveIO.ts	Updates `nullClientIO` to include the new methods.
ts/packages/commandExecutor/src/commandServer.ts	Updates MCP ClientIO stub to include the new methods.
ts/packages/cli/src/enhancedConsole.ts	Adds no-op stubs for deferred interactions in CLI ClientIO.
ts/packages/agentServer/server/src/sharedDispatcher.ts	Implements deferred interaction flow in server shared dispatcher.
ts/packages/agentServer/server/src/sessionManager.ts	Returns pending interactions on session join.
ts/packages/agentServer/server/src/server.ts	Includes pending interactions in join-session response payload.
ts/packages/agentServer/server/package.json	Adds dependency on `@typeagent/dispatcher-types`.
ts/packages/agentServer/protocol/src/protocol.ts	Extends JoinSessionResult with optional `pendingInteractions`.
ts/packages/agentServer/protocol/package.json	Adds dependency on `@typeagent/dispatcher-types`.
ts/packages/agentServer/docs/async-clientio-design.md	Adds design documentation for the deferred interaction pattern.
ts/packages/agents/browser/src/extension/serviceWorker/dispatcherConnection.ts	Adds no-op stubs for deferred interactions in extension ClientIO.

Files not reviewed (1)

ts/pnpm-lock.yaml: Language not supported

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

…cancellation notification

…b.com/microsoft/TypeAgent into dev/georgeng/async-clientio-callbacks

Copilot

Pull request overview

Copilot reviewed 31 out of 32 changed files in this pull request and generated 10 comments.

Files not reviewed (1)

ts/pnpm-lock.yaml: Language not supported

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

curtisman · 2026-04-10T15:02:48Z

                }
            });
        },
+        async respondToInteraction(


I am wondering if this needs to be exposed in the API, can it all be hidden by the protocol?

Technically it could, if we moved respondToInteraction to use rpc.invoke (await) instead of rpc.send (fire and forget). However, the reason I chose this is because with the PendingInteractionManager and use of the FOF requestInteraction, we can actually extend functionality to allow "stashing" of interactions in the future.

As per our offline discussion, if the server asks the user a question, the next response can be interpreted by the LLM to determine if the user is responding to the question (and making a call to respondToInteraction) versus implicitly ignoring the question for now (sending a new command or following a different path of thought). These are things that happen naturally in a conversation such as coming back to a question later.

Having this respondToInteraction() helps partially enable the scenario above, though it would take significantly more work to design and implement that interaction described above. I think it's a great future work item though

curtisman · 2026-04-10T15:04:40Z

                            connectionId: result.connectionId,
                            sessionId,
                            name: result.name,
+                            pendingInteractions:


Doesn't the display log replay sufficient?

Technically, you could retrieve the last question asked from the previous message in display log. However, that's adding additional (potentially fragile) extraction logic on an ordered stream of events. In addition, potential future capabilities like interruptions or moving operations into the background could make extracting from display log much more difficult.

I think it's cleaner to just store pending interactions in this array (which is also already filtered for the specific connection). No need to run any parsing logic.

curtisman · 2026-04-10T15:06:08Z

+            if (notified === 0) {
+                // No clients to handle it — resolve immediately without logging
+                // a pending entry (which would have no corresponding resolution).
+                return defaultValue ?? false;


That would cause intermittent disconnection to miss some interactions? Probably want to keep it pending until either user dismiss or the dispatcher session is closing down?

That is a great callout, this scenario will come into play if the user sends a command, disconnects, and the server sends a question back. The user isn't currently able to reconnect and respond.

In a scenario where the user was connected when the server sends the questions, but then disconnects, they will preserve the ability to respond until the timeout.

I will match the behavior in both scenarios so that the user can always respond until the timeout. I think setting a timeout is good policy for now, and something we can change later if we want to allow a question to hang indefinitely (Claude, I believe operates on a timeout as well).

curtisman · 2026-04-10T15:09:14Z

+            context.displayLog.logPendingInteraction(request);
+            context.displayLog.saveQueued();
+
+            return pendingInteractions.create<unknown>(


Instead of a timeout, at the server level, I think letting both the user or the requester (i.e. agent) able to explicit cancel will give them more control over the experience.

The user can always cancel by simply responding, I think that part is fine. Though exposing an api to allow the agent to explicitly cancel (in other potential scenarios) is definitely useful.

I prefer keeping a configurable timeout because it is good to guarantee completion of an interaction and resource cleanup.

Both aren't mutually exclusive capabilities and we can tune them as we go.

curtisman · 2026-04-10T15:22:30Z

+            }
+
+            // Log only after we know at least one client was notified
+            context.displayLog.logPendingInteraction(request);


we don't log the message if it is not notified? Wouldn't that be missed on replay?

This has been revaluated in the follow up PR along with not broadcasting on 0 users.

curtisman · 2026-04-10T15:23:03Z

+        // respond via respondToInteraction resolves the promise.
+
+        askYesNo: async (requestId, message, defaultValue?) => {
+            const interactionId = randomUUID();


if we log in up front, can we just use the sequence number?

I think this goes back to the conversation about having PendingInteractionRequests versus parsing the logs. I think maintaining a queue of PendingInteractionRequests is better structure that allows for more extensibility than log extraction (especially for the long run).

Resolved conflicts in favour of explicit cancelInteraction approach over the auto-cancel-on-disconnect (cancelByConnection) approach from #2178. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

This PR addresses comments in #2178 **Changes :** Explicit client-initiated cancellation (cancelInteraction) instead of auto-cancelling interactions on client disconnect. PR #2178 (already in main) implemented the async deferred-promise pattern with cancelByConnection — when a client disconnects, all its pending interactions are immediately cancelled. This branch replaces that with: - Interactions survive disconnect — they remain pending until timeout (10 min) or an explicit call, allowing a reconnecting client to still respond - New cancelInteraction(interactionId) method on Dispatcher — clients cancel explicitly (e.g. user dismisses UI), not implicitly on disconnect - Always-log invariant — logPendingInteraction is called unconditionally, even when no clients are connected at broadcast time, so the interaction appears in DisplayLog for session replay --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>

GeorgeNgMsft and others added 3 commits April 8, 2026 13:33

Add initial config persistence design to agentServerSessions.md

5316bbd

GeorgeNgMsft had a problem deploying to development-fork April 9, 2026 22:49 — with GitHub Actions Error

Merge branch 'main' into dev/georgeng/async-clientio-callbacks

dfff4c7

GeorgeNgMsft had a problem deploying to development-fork April 9, 2026 22:49 — with GitHub Actions Error

GeorgeNgMsft changed the title ~~Implement ClientIO interactions on Agent-Server~~ Explore ClientIO interactions on Agent-Server Apr 9, 2026

Merge branch 'main' into dev/georgeng/async-clientio-callbacks

680bcb6

GeorgeNgMsft had a problem deploying to development-fork April 9, 2026 22:57 — with GitHub Actions Error

GeorgeNgMsft added 2 commits April 9, 2026 16:29

Update documentation

8316fd5

Merge branch 'dev/georgeng/async-clientio-callbacks' of https://githu…

3c04335

…b.com/microsoft/TypeAgent into dev/georgeng/async-clientio-callbacks

GeorgeNgMsft had a problem deploying to development-fork April 9, 2026 23:31 — with GitHub Actions Error

GeorgeNgMsft changed the title ~~Explore ClientIO interactions on Agent-Server~~ Implement ClientIO interaction support on Agent-Server Apr 9, 2026

GeorgeNgMsft requested a review from Copilot April 9, 2026 23:38

Copilot started reviewing on behalf of GeorgeNgMsft April 9, 2026 23:38 View session

Copilot AI reviewed Apr 9, 2026

View reviewed changes

Update documentation and tests

43c7091

GeorgeNgMsft had a problem deploying to development-fork April 10, 2026 00:06 — with GitHub Actions Error

GeorgeNgMsft added 2 commits April 9, 2026 22:09

Add clientIO timeout in edgecase where no valid clients/no clients + …

b818baf

…cancellation notification

Address comments for cancellation clarity and logging

fe299c9

GeorgeNgMsft had a problem deploying to development-fork April 10, 2026 06:59 — with GitHub Actions Error

Merge branch 'main' into dev/georgeng/async-clientio-callbacks

b28f861

GeorgeNgMsft marked this pull request as ready for review April 10, 2026 06:59

GeorgeNgMsft had a problem deploying to development-fork April 10, 2026 06:59 — with GitHub Actions Error

GeorgeNgMsft added 2 commits April 10, 2026 00:05

Fix build

94b4766

Merge branch 'dev/georgeng/async-clientio-callbacks' of https://githu…

cc4ac42

…b.com/microsoft/TypeAgent into dev/georgeng/async-clientio-callbacks

GeorgeNgMsft had a problem deploying to development-fork April 10, 2026 07:06 — with GitHub Actions Error

GeorgeNgMsft requested a review from Copilot April 10, 2026 07:07

Copilot started reviewing on behalf of GeorgeNgMsft April 10, 2026 07:07 View session

Copilot AI reviewed Apr 10, 2026

View reviewed changes

Address clients and update document

1a908ad

GeorgeNgMsft had a problem deploying to development-fork April 10, 2026 08:10 — with GitHub Actions Failure

GeorgeNgMsft temporarily deployed to development-fork April 10, 2026 08:10 — with GitHub Actions Inactive

GeorgeNgMsft added this pull request to the merge queue Apr 10, 2026

Merged via the queue into main with commit 54f1519 Apr 10, 2026
20 of 21 checks passed

curtisman reviewed Apr 10, 2026

View reviewed changes

This was referenced Apr 10, 2026

Update Dispatcher ClientIO interactions #2183

Merged

Agent-Server: ClientIO should not have blocking callback. #2147

Closed

Conversation

GeorgeNgMsft commented Apr 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

GeorgeNgMsft Apr 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

curtisman Apr 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

GeorgeNgMsft commented Apr 9, 2026 •

edited

Loading

GeorgeNgMsft Apr 10, 2026 •

edited

Loading

curtisman Apr 10, 2026 •

edited

Loading