[WIP] Implement model management for Gemma, Whisper, and Tesseract#39
[WIP] Implement model management for Gemma, Whisper, and Tesseract#39
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
Important Review skippedBot user detected. To trigger a single review, invoke the You can disable this status message by setting the Note Other AI code review bot(s) detectedCodeRabbit has detected other AI code review bot(s) in this pull request and will avoid duplicating their findings in the review comments. This may lead to a less comprehensive review. Comment |
…dels, services, and UI Co-authored-by: Shangjin-Xiao <84136399+Shangjin-Xiao@users.noreply.github.com>
…t, test flexibility Co-authored-by: Shangjin-Xiao <84136399+Shangjin-Xiao@users.noreply.github.com>
Co-authored-by: Shangjin-Xiao <84136399+Shangjin-Xiao@users.noreply.github.com>
|
@copilot 你进行flutter analyze了吗？修复所有编译错误啊！编译报错了:
class RecordLinux extends RecordPlatform { INFO: Installing Rust target: aarch64-linux-android FAILURE: Build failed with an exception.
|
Co-authored-by: Shangjin-Xiao <84136399+Shangjin-Xiao@users.noreply.github.com>
Fixed in 1d27824. The |
|
@copilot 时间: 2025-12-28 15:11:33.233489 |
Co-authored-by: Shangjin-Xiao <84136399+Shangjin-Xiao@users.noreply.github.com>
Fixed in 217c1cf. The model download functionality has been updated:
Model download status:
To fully enable Gemma models, the project SDK version needs to be updated to >=3.6.0, then |
…st' for result instances
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
| try { | ||
| final extractedPath = await ModelExtractor.extract( | ||
| task.savePath, | ||
| path.dirname(task.savePath), | ||
| onProgress: (progress) { | ||
| // 解压进度占 80%-100% | ||
| task.onProgress?.call(0.8 + progress * 0.2); | ||
| }, | ||
| ); | ||
|
|
||
| // 记录解压后的路径 | ||
| _extractedModelPaths[task.modelId] = extractedPath; | ||
| logInfo('模型解压完成: $extractedPath', source: 'ModelManager'); |
There was a problem hiding this comment.
🔴 ModelExtractor._findRootExtractPath returns parent models directory instead of extracted model subdirectory
When a model archive (e.g., sherpa-onnx-whisper-tiny.tar.bz2) is extracted, _findRootExtractPath receives the shared models directory (path.dirname(task.savePath)) as extractDir. It then checks if this directory has exactly one visible entity. But the models directory already contains the archive file itself (and possibly other model files), so visibleEntities.length == 1 is almost always false, causing it to return the parent models directory path instead of the specific extracted model subdirectory.
Root Cause
In model_manager.dart:398, the extraction is called as:
final extractedPath = await ModelExtractor.extract(
task.savePath, // e.g. .../local_ai_models/sherpa-onnx-whisper-tiny.tar.bz2
path.dirname(task.savePath), // e.g. .../local_ai_models/
...
);_findRootExtractPath at model_extractor.dart:65-80 lists all visible entities in the shared models directory. Even for the very first extraction, the .tar.bz2 archive file is still present alongside the extracted directory, so visibleEntities.length is ≥ 2 and the method returns the parent directory.
At model_manager.dart:406, _extractedModelPaths[task.modelId] is then set to the parent models directory. Later, getModelPath() at line 714-716 returns this wrong path. When validateWhisperModel() receives the entire models directory, it searches recursively and may pick up files from other models.
Note: On app restart, _checkExtractedPath independently computes the correct path, so this bug only affects the session during which the model was downloaded.
Impact: Model preparation after download may use the wrong directory path, potentially causing model validation to fail or use incorrect model files.
Prompt for agents
In lib/services/local_ai/model_manager.dart, in the _executeDownload method around line 395-407, instead of relying on ModelExtractor._findRootExtractPath (which examines the shared parent directory), compute the expected extraction directory name directly from the model filename, similar to how _checkExtractedPath does it. For example:
String extractDirName = path.basenameWithoutExtension(task.savePath);
if (task.savePath.endsWith('.tar.bz2')) {
extractDirName = path.basename(task.savePath).replaceAll('.tar.bz2', '');
} else if (task.savePath.endsWith('.tar.gz')) {
extractDirName = path.basename(task.savePath).replaceAll('.tar.gz', '');
}
final expectedExtractedPath = path.join(path.dirname(task.savePath), extractDirName);
Then after ModelExtractor.extract completes, check if expectedExtractedPath exists and use it as _extractedModelPaths[task.modelId] instead of the return value from _findRootExtractPath.
Was this helpful? React with 👍 or 👎 to provide feedback.
| try { | ||
| await (session as dynamic).addQueryChunk( | ||
| Message.text(text: prompt, isUser: true), | ||
| ); | ||
| final dynamic resp = await (session as dynamic).getResponse(); | ||
| if (resp is String) return resp; | ||
| } catch (_) { | ||
| // ignore | ||
| } | ||
|
|
||
| // 1) 常见：getResponse(prompt: ...) | ||
| try { | ||
| final dynamic resp = await (session as dynamic).getResponse(prompt: prompt); | ||
| if (resp is String) return resp; | ||
| } catch (_) { | ||
| // ignore and try other patterns | ||
| } | ||
|
|
||
| // 2) 常见：setPrompt / prompt 属性 + getResponse() | ||
| try { | ||
| await (session as dynamic).setPrompt(prompt); | ||
| final dynamic resp = await (session as dynamic).getResponse(); | ||
| if (resp is String) return resp; | ||
| } catch (_) { | ||
| // ignore | ||
| } | ||
|
|
||
| // 3) 常见：addQuery + getResponse() | ||
| try { | ||
| await (session as dynamic).addQuery(prompt); | ||
| final dynamic resp = await (session as dynamic).getResponse(); | ||
| if (resp is String) return resp; | ||
| } catch (_) { | ||
| // ignore | ||
| } |
There was a problem hiding this comment.
🔴 _tryGemmaPrompt sequentially tries multiple API patterns on the same session, corrupting session state
The method _tryGemmaPrompt at lib/services/local_ai/text_processing_service.dart:671 tries 4 different API patterns sequentially on the same _gemmaSession object. If an earlier pattern partially succeeds (e.g., addQueryChunk at line 684 adds a message but getResponse returns non-String), the session's conversation history is polluted. Subsequent patterns then add the same prompt again, compounding the corruption.
Root Cause and Impact
Pattern 0 (line 683-691) calls addQueryChunk which modifies the session state by adding a user message. If getResponse() returns a non-String value, the code falls through to pattern 1. Pattern 1 (line 694-698) tries getResponse(prompt: prompt) — now the session has two copies of the prompt in its history. Patterns 2 and 3 further add the prompt via setPrompt and addQuery.
Since _gemmaSession is reused across calls (it's an instance field), this session corruption persists for all future invocations of _tryGemmaPrompt. Every subsequent call (e.g., text correction, source recognition, tag suggestion) operates on a corrupted session with leftover conversation fragments.
Impact: After the first partially-failed API probe, the LLM session may produce nonsensical or degraded results for all subsequent text processing operations until the model is reloaded.
Prompt for agents
In lib/services/local_ai/text_processing_service.dart, the _tryGemmaPrompt method (lines 671-724) tries multiple API patterns on the same session object. Each failed pattern may leave the session in a corrupted state. Fix this by: (1) determining the correct API pattern once during initialization/loadModel and caching which pattern works, then only using that pattern in _tryGemmaPrompt; or (2) creating a fresh session for each _tryGemmaPrompt call to avoid state pollution; or (3) at minimum, only trying the first pattern that does not throw and not falling through to subsequent patterns that modify session state.
Was this helpful? React with 👍 or 👎 to provide feedback.
…el-management-features
| // 统一抛出 FormatException，保持异常处理的一致性 | ||
| if (e is ArgumentError) { | ||
| throw FormatException(e.message.toString()); | ||
| } |
There was a problem hiding this comment.
🟡 Dead code: if (e is ArgumentError) is unreachable because ArgumentError is already caught and rethrown
The new code adds an if (e is ArgumentError) check inside the catch (e) block at line 256. However, this condition can never be true because the on ArgumentError { rethrow; } clause at lines 252-253 already catches and rethrows all ArgumentError exceptions before the general catch (e) is reached.
Root Cause
Dart's exception handling evaluates on Type clauses in order. The on ArgumentError at line 252 matches first and immediately rethrows, so no ArgumentError ever reaches the catch (e) block. The dead code at lines 256-258 was clearly intended to convert ArgumentError to FormatException, but it will never execute.
Actual behavior: The if (e is ArgumentError) block is dead code — never reached.
Expected behavior: If the intent was to convert ArgumentError to FormatException, the on ArgumentError { rethrow; } clause should be removed, OR this dead code should be removed to avoid confusion.
| // 统一抛出 FormatException，保持异常处理的一致性 | |
| if (e is ArgumentError) { | |
| throw FormatException(e.message.toString()); | |
| } | |
| // 非预期解析错误：包装为 FormatException 并附带原始 JSON 便于排查。 | |
Was this helpful? React with 👍 or 👎 to provide feedback.
…ownload ergonomics Co-authored-by: Shangjin-Xiao <84136399+Shangjin-Xiao@users.noreply.github.com>
…e-add-note Restore add-note long-press ASR flow and improve local model import/download ergonomics
… reduce recording lag, auto-populate model URLs Co-authored-by: Shangjin-Xiao <84136399+Shangjin-Xiao@users.noreply.github.com>
…seOverlayIfOpen helper Co-authored-by: Shangjin-Xiao <84136399+Shangjin-Xiao@users.noreply.github.com>
…ionality Fix local AI: ASR stop button, FlutterGemma init, recording lag, model downloads
| } catch (e) { | ||
| logError('FlutterGemma 全局初始化失败: $e', source: 'ModelManager'); | ||
| // 重置，允许后续重试 | ||
| _flutterGemmaInitCompleter = null; | ||
| } |
There was a problem hiding this comment.
🔴 Completer never completed on error causes concurrent callers to hang forever
When FlutterGemma.initialize() throws, the Completer is set to null without being completed with an error. Any concurrent caller that already obtained the future via _flutterGemmaInitCompleter!.future (line 485) will await indefinitely on a Completer that will never complete.
Root Cause and Impact
The race condition occurs as follows:
- Caller A enters
_ensureFlutterGemmaInitialized(), sees_flutterGemmaInitCompleter == null, creates a new Completer (line 487), and beginsawait FlutterGemma.initialize()(line 489). - While A is awaiting, Caller B enters the same method, sees
_flutterGemmaInitCompleter != null(line 484), and returns_flutterGemmaInitCompleter!.future(line 485) — now B is awaiting this future. FlutterGemma.initialize()fails in Caller A. The catch block sets_flutterGemmaInitCompleter = null(line 495) but never calls_flutterGemmaInitCompleter!.completeError(e).- Caller B is now stuck forever on a future that will never resolve.
Impact: On initialization failure, any concurrent model download or prepare operation will hang indefinitely, freezing the UI if awaited on the main isolate. The identical pattern exists in text_processing_service.dart:119-135.
| } catch (e) { | |
| logError('FlutterGemma 全局初始化失败: $e', source: 'ModelManager'); | |
| // 重置，允许后续重试 | |
| _flutterGemmaInitCompleter = null; | |
| } | |
| _flutterGemmaInitCompleter!.complete(); | |
| } catch (e) { | |
| logError('FlutterGemma 全局初始化失败: $e', source: 'ModelManager'); | |
| // Complete with error so concurrent waiters don't hang, then reset for retry. | |
| _flutterGemmaInitCompleter!.completeError(e); | |
| _flutterGemmaInitCompleter = null; | |
| } |
Was this helpful? React with 👍 or 👎 to provide feedback.
…el-management-features
…nt-features' into copilot/implement-model-management-features
- voice_input_overlay: add missing onStopRecording parameter + stop button UI - voice_input_overlay: show stop button when recording, fill button when done - l10n: add voiceStopRecording key to zh/en ARB files - text_processing_service: complete Completer with error before reset to prevent hanging callers - text_processing_service: close Gemma session in unloadModel() to prevent resource leak - speech_recognition_service: cancel timers/stream/asrStream in startRecording error handler - speech_recognition_service: remove dead _audioBuffer field (never written, only cleared) - model_manager: clear errorMessage when retrying download - local_ai_model: fix copyWith sentinel pattern to allow setting errorMessage to null - home_page: remove dead _onFABLongPress/_showVoiceInputOverlay (LocalAIFab handles this internally) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Local AI Implementation (flutter_gemma + sherpa_onnx + Tesseract)
Implementation of on-device AI features for ThoughtEcho using:
Implementation Checklist
1. Dependencies - INTEGRATED
2. Create models for local AI
3. Create services - INTEGRATED WITH PACKAGES
model_manager.dart- Real HTTP download with error codesspeech_recognition_service.dart- Integrated with sherpa_onnxocr_service.dart- Integrated with flutter_tesseract_ocrembedding_service.dart- Embedding generationvector_store.dart- Fixed async error handling and disposetext_processing_service.dart- Integrated with flutter_gemmalocal_ai_service.dart- Unified integration service4. Add localization strings - COMPLETE
5. Create UI components
ModelManagementPage- Model download/import UILocalAISettingsPagewith model management entryAIActionButtonsto use LocalAIService6. Testing
7. Code Review Fixes
Original prompt
💬 We'd love your input! Share your thoughts on Copilot coding agent in our 2 minute survey.