Skip to content

[WIP] Create Excalidraw Agent#2168

Draft
GeorgeNgMsft wants to merge 15 commits intomainfrom
dev/georgeng/excalidraw_agent
Draft

[WIP] Create Excalidraw Agent#2168
GeorgeNgMsft wants to merge 15 commits intomainfrom
dev/georgeng/excalidraw_agent

Conversation

@GeorgeNgMsft
Copy link
Copy Markdown
Contributor

@GeorgeNgMsft GeorgeNgMsft commented Apr 8, 2026 •
edited
Loading

Add Excalidraw Agent for AI-Powered Diagram Generation

Summary
Adds a new excalidraw TypeAgent that converts documentation, markdown, Visio XML exports, Mermaid
diagrams, and plain-text architecture descriptions into https://excalidraw.com/ diagram files using
AI. The agent integrates with the existing TypeAgent dispatcher and agent provider infrastructure.

What's Changed
New package: ts/packages/agents/excalidraw/

  • excalidrawActionSchema.ts — Defines two actions:
    • createDiagram — accepts sourceContent, sourceType (markdown | text | visio-xml | mermaid |
      architecture), optional diagramTitle, and optional outputPath
    • exportDiagram — writes provided Excalidraw JSON to a specified path
  • excalidrawActionHandler.ts — Calls the OpenAI chat model with a structured system prompt that
    instructs the LLM to output valid Excalidraw JSON (type, version, elements[], appState). Writes the
    resulting .excalidraw file to the user-specified path, or defaults to ~/Documents/ with a timestamped
    filename.
  • excalidrawManifest.json — Agent manifest for TypeAgent discovery

Agent registration: ts/packages/defaultAgentProvider/

  • Registered in config.json, config.all.json, and config.agent.json with execMode: "dispatcher"
  • Added excalidraw-agent as a workspace dependency

Usage Examples

▎ "Create an Excalidraw diagram from this architecture document"
▎ "Convert this Mermaid diagram to an Excalidraw file and save it to C:\diagrams\arch.excalidraw"
▎ "Turn this Visio XML into an Excalidraw diagram"

Testing

  • pnpm run build excalidraw-agent passes (114 tasks, tsc + asc compilation)
  • Schema compiles to excalidrawSchema.pas.json in dist/

Notes

  • Output defaults to ~/Documents/<title>_.excalidraw when no path is provided
    (cross-platform: works on Windows and Unix)
  • The LLM prompt includes explicit Excalidraw JSON structure guidance (elements, text binding,
    arrows, colors, layout) to maximize diagram quality

GeorgeNgMsft and others added 2 commits April 8, 2026 13:33
@GeorgeNgMsft
Add initial config persistence design to agentServerSessions.md
5316bbd
@GeorgeNgMsft @claude
Add excalidraw agent for converting documents to Excalidraw diagrams
7691a5d 
Implements a new TypeAgent agent (excalidraw-agent) that uses an LLM to
convert source materials (markdown, plain text, Visio XML, Mermaid, and
architecture descriptions) into valid Excalidraw JSON diagrams. Includes
createDiagram and exportDiagram actions, with output defaulting to the
user's Documents folder.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
GeorgeNgMsft and others added 3 commits April 10, 2026 18:10
@GeorgeNgMsft @claude
Redesign excalidraw agent with iterative correction pipeline
7c5b62d 
Replace the lossy 2-phase Mermaid intermediate approach with a typed
DiagramPlan IR and an iterative LLM correction loop (up to 3 passes)
backed by deterministic programmatic validation. Adds explicit support
for nested/grouped objects and plan-to-element traceability via stable
ID conventions.

- diagramPlan.ts: typed IR with parentGroupId/childNodeIds for nesting
- prompts.ts: plan extraction, generation, and correction prompt builders
- diagramValidator.ts: 8-category deterministic validator (no LLM)
- excalidrawActionHandler.ts: rewritten iterative pipeline
- test/: 24-test suite covering validation and prompt correctness
- excalidraw_agent_design.md: architecture design document

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@GeorgeNgMsft @claude
Fix max_tokens error and improve JSON parse robustness in excalidraw …
e5f58d4 
…agent

Remove hardcoded max_tokens overrides (8000/16000) that exceeded the
configured model's 4096 token limit, causing "max_tokens is too large"
errors. The API now uses the model's default maximum.

Also add stripMarkdownFences() to handle models that emit markdown
code fences even in json_object mode, and improve parse error messages
to include response context for debugging.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@GeorgeNgMsft @claude
Update sprint-status.md with bug fix details
bbc5816 
Document the max_tokens fix, JSON parse robustness improvements,
and better error diagnostics added in the previous commit.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@GeorgeNgMsft GeorgeNgMsft marked this pull request as ready for review April 11, 2026 03:08
@GeorgeNgMsft GeorgeNgMsft marked this pull request as draft April 11, 2026 03:09
GeorgeNgMsft and others added 6 commits April 11, 2026 00:03
@GeorgeNgMsft @claude
Fix crash on undefined childNodeIds/childGroupIds in DiagramPlan groups
91c4e2d 
LLM sometimes omits childNodeIds or childGroupIds arrays from PlanGroup
objects. Added normalization in the plan post-parse step and defensive
null-coalescing in the validator to prevent .includes() on undefined.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@GeorgeNgMsft @claude
Fix JSON truncation for large diagrams with compact output and chunke…
736b742 
…d generation

Large diagrams (16+ nodes) caused LLM output to exceed token limits, producing
truncated JSON (e.g., "Unterminated string in JSON at position 12564"). This adds
a 3-layer defense: (1) compact output via default field omission with programmatic
injection post-parse, reducing output ~40%; (2) chunked generation splitting Phase 2
into 3 sequential LLM calls (groups, nodes, edges) for diagrams with >12 plan items;
(3) truncation recovery as a last-ditch salvage mechanism. Adds 30 new tests (54 total).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@GeorgeNgMsft @claude
Fix persistent JSON truncation by switching to MinimalDiagram schema …
fdab30e 
…with deterministic expansion

LLM now generates a compact MinimalDiagram (~7 fields/element) instead of verbose Excalidraw JSON
(~20+ fields/element). New expandToExcalidraw() function deterministically adds all required
Excalidraw fields in TypeScript, eliminating reliance on LLM to omit default fields. A 31-element
diagram now produces well under 4000 chars of LLM output vs ~12,761 chars previously. Correction
loop also updated to work in minimal format. 81 tests pass (27 new).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@GeorgeNgMsft @claude
Improve excalidraw layout engine with depth-based sizing and axis-awa…
e5540b6 
…re flow

- Add normalizeLayout() pre-processing pass in expandToExcalidraw()
- Group frames sized by 2^N multiplier where N = max child nesting depth (cap 4)
- Frames re-flow children along dominant axis (LR if wider, TD if taller)
- Frames grow by 50% when children don't fit, propagating up ancestor chain
- resolveOverlaps() pushes siblings along parent frame's dominant axis
- Snap all coordinates to 20px grid for clean alignment
- Increase prompt spacing: 100px row gap, 60px within-group, 80px group top padding
- Fix group label text: small strip at top-left instead of spanning full frame
- Debug error message includes raw response preview for parse failures

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@GeorgeNgMsft
Merge main
0894a4f
@GeorgeNgMsft
Merge branch 'main' into dev/georgeng/excalidraw_agent
18ab2da
@GeorgeNgMsft GeorgeNgMsft changed the title Create Excalidraw Agent [WIP] Create Excalidraw Agent Apr 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@GeorgeNgMsft