Skip to content

Python: Introducing Realtime Clients for OpenAI and Azure OpenAI#10127

Merged
moonbox3 merged 51 commits intomicrosoft:mainfrom
eavanvalkenburg:realtime
Mar 4, 2025
Merged

Python: Introducing Realtime Clients for OpenAI and Azure OpenAI#10127
moonbox3 merged 51 commits intomicrosoft:mainfrom
eavanvalkenburg:realtime

Conversation

@eavanvalkenburg
Copy link
Copy Markdown
Member

@eavanvalkenburg eavanvalkenburg commented Jan 8, 2025 •
edited
Loading

Motivation and Context

Implements the OpenAI Realtime API with Semantic Kernel

Description

Adds a RealtimeClientBase
Adds RealtimeEvents
Adds OpenAI and Azure Realtime clients

Closes: #10108

Contribution Checklist

@eavanvalkenburg eavanvalkenburg requested a review from a team as a code owner January 8, 2025 16:04
@eavanvalkenburg eavanvalkenburg marked this pull request as draft January 8, 2025 16:04
@markwallace-microsoft markwallace-microsoft added the python Pull requests for the Python Semantic Kernel label Jan 8, 2025
TaoChenOSU
TaoChenOSU reviewed Jan 8, 2025
View reviewed changes
Comment thread python/samples/concepts/audio/audio_player.py Outdated
Comment thread python/pyproject.toml Outdated
Comment thread python/samples/concepts/audio/audio_recorder_stream.py Outdated
Comment thread ...ernel/connectors/ai/open_ai/prompt_execution_settings/open_ai_realtime_execution_settings.py
Comment thread python/semantic_kernel/connectors/ai/open_ai/services/open_ai_realtime_base.py Outdated
Comment thread python/semantic_kernel/connectors/ai/realtime_client_base.py
Comment thread python/semantic_kernel/connectors/ai/open_ai/services/open_ai_realtime_base.py Outdated
Comment thread python/semantic_kernel/connectors/ai/open_ai/services/open_ai_realtime_utils.py Outdated
Comment thread python/semantic_kernel/connectors/ai/realtime_client_base.py Outdated
Comment thread python/tests/unit/contents/test_audio_content.py Outdated
@markwallace-microsoft
Copy link
Copy Markdown
Contributor

markwallace-microsoft commented Jan 9, 2025 •
edited
Loading

Python Test Coverage

Python Test Coverage Report •
FileStmtsMissCoverMissing
semantic_kernel
   kernel.py2024777%143, 154, 158, 308–311, 428, 442–485
semantic_kernel/connectors/ai
   chat_completion_client_base.py124298%398, 408
   function_calling_utils.py48296%163, 168
   realtime_client_base.py33197%114
semantic_kernel/connectors/ai/anthropic/services
   anthropic_chat_completion.py162895%159, 165, 178, 184, 188, 245–247, 380
semantic_kernel/connectors/ai/azure_ai_inference/services
   azure_ai_inference_chat_completion.py110695%120–123, 132, 156, 182
semantic_kernel/connectors/ai/bedrock/services
   bedrock_chat_completion.py1351490%115, 137, 162, 166–169, 227, 245–264, 323
semantic_kernel/connectors/ai/google/google_ai/services
   google_ai_chat_completion.py122695%126, 131, 155, 160, 183, 185
semantic_kernel/connectors/ai/google/vertex_ai/services
   vertex_ai_chat_completion.py117497%123, 150, 176, 178
semantic_kernel/connectors/ai/ollama/services
   ollama_chat_completion.py1373475%116, 141, 145–146, 156, 169, 186, 206–207, 211, 224–234, 245–247, 258–267, 279, 289–290, 312, 323–324, 350, 359–367
semantic_kernel/connectors/ai/open_ai/services
   azure_config_base.py56296%95, 97
   azure_realtime.py29872%83–98, 116
   open_ai_realtime.py44113071%342–344, 507–508, 511–512, 515–519, 522, 530–533, 540, 542, 545–546, 596–599, 608–611, 623–624, 663, 669–677, 687–732, 738–739, 742–743, 747–777, 785–794, 798–817, 874–875, 877, 911, 914, 917–927, 930–936, 949, 1010–1011, 1013
semantic_kernel/connectors/ai/open_ai/settings
   azure_open_ai_settings.py26485%119–122
semantic_kernel/contents
   audio_content.py21195%88
   binary_content.py1171389%99, 120, 128, 134, 147, 152–153, 194–199
   function_call_content.py107397%194, 222–223
TOTAL19586255187% 

Python Unit Test Overview

Tests Skipped Failures Errors Time
3207 5 💤 0 ❌ 0 🔥 1m 42s ⏱️

@eavanvalkenburg eavanvalkenburg force-pushed the realtime branch 4 times, most recently from 20f5270 to 2afa19f Compare January 23, 2025 10:04
moonbox3
moonbox3 reviewed Jan 27, 2025
View reviewed changes
Comment thread python/semantic_kernel/connectors/ai/open_ai/services/realtime/open_ai_realtime_base.py Outdated
Comment thread python/semantic_kernel/connectors/ai/open_ai/services/realtime/open_ai_realtime_websocket.py Outdated
Comment thread python/semantic_kernel/connectors/ai/open_ai/services/realtime/open_ai_realtime_base.py Outdated
moonbox3
moonbox3 reviewed Jan 27, 2025
View reviewed changes
Comment thread docs/decisions/00XX-realtime-api-clients.md Outdated
Comment thread docs/decisions/00XX-realtime-api-clients.md Outdated
Comment thread docs/decisions/00XX-realtime-api-clients.md Outdated
Comment thread docs/decisions/00XX-realtime-api-clients.md Outdated
Comment thread docs/decisions/00XX-realtime-api-clients.md Outdated
@eavanvalkenburg eavanvalkenburg mentioned this pull request Jan 31, 2025
Closed
5 tasks
@eavanvalkenburg eavanvalkenburg marked this pull request as ready for review February 17, 2025 16:01
@eavanvalkenburg eavanvalkenburg changed the title Python: draft initial implementation of Realtime API Python: Introducing Realtime Clients for OpenAI and Azure OpenAI Feb 17, 2025
TaoChenOSU
TaoChenOSU reviewed Feb 18, 2025
View reviewed changes
Comment thread python/samples/concepts/realtime/01a-chat_with_realtime_websocket.py
Comment thread python/samples/concepts/realtime/01a-chat_with_realtime_websocket.py
Comment thread python/samples/concepts/realtime/02b-chat_with_function_calling_webrtc.py Outdated
Comment thread python/samples/demos/call_automation/main.py Outdated
Comment thread python/samples/demos/call_automation/main.py
Comment thread python/samples/demos/call_automation/main.py Outdated
Comment thread python/semantic_kernel/connectors/ai/function_calling_utils.py
moonbox3
moonbox3 reviewed Feb 20, 2025
View reviewed changes
Comment thread python/samples/concepts/realtime/01a-chat_with_realtime_websocket.py
Comment thread python/samples/concepts/realtime/01a-chat_with_realtime_websocket.py Outdated
Comment thread python/samples/concepts/realtime/01a-chat_with_realtime_websocket.py Outdated
Comment thread python/samples/concepts/realtime/01a-chat_with_realtime_websocket.py Outdated
Comment thread python/samples/concepts/realtime/01b-chat_with_realtime_webrtc.py
Comment thread python/samples/concepts/realtime/01b-chat_with_realtime_webrtc.py
Comment thread ...ernel/connectors/ai/open_ai/prompt_execution_settings/open_ai_realtime_execution_settings.py
Comment thread python/semantic_kernel/connectors/ai/open_ai/services/realtime/utils.py Outdated
Comment thread python/samples/concepts/realtime/utils.py Outdated
@eavanvalkenburg eavanvalkenburg force-pushed the realtime branch 2 times, most recently from 4142f8e to bbb89d5 Compare February 21, 2025 10:17
Copilot AI reviewed Feb 21, 2025
View reviewed changes
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot reviewed 37 out of 37 changed files in this pull request and generated 1 comment.

Comment thread python/samples/concepts/realtime/01b-chat_with_realtime_webrtc.py Outdated
@eavanvalkenburg eavanvalkenburg force-pushed the realtime branch 3 times, most recently from f6620ae to fe2204b Compare February 28, 2025 08:13
TaoChenOSU
TaoChenOSU approved these changes Mar 3, 2025
View reviewed changes
Copy link
Copy Markdown
Contributor

@TaoChenOSU TaoChenOSU left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it possible to add some integration tests?

@moonbox3 moonbox3 added this pull request to the merge queue Mar 4, 2025
@github-merge-queue github-merge-queue bot removed this pull request from the merge queue due to failed status checks Mar 4, 2025
@moonbox3 moonbox3 added this pull request to the merge queue Mar 4, 2025
Merged via the queue into microsoft:main with commit de54193 Mar 4, 2025
28 checks passed
rracanicci pushed a commit to rracanicci/semantic-kernel that referenced this pull request Mar 5, 2025
@eavanvalkenburg @moonbox3 @rracanicci
Python: Introducing Realtime Clients for OpenAI and Azure OpenAI (mic…
90ef1d7 
…rosoft#10127)

### Motivation and Context

<!-- Thank you for your contribution to the semantic-kernel repo!
Please help reviewers and future users, providing the following
information:
  1. Why is this change required?
  2. What problem does it solve?
  3. What scenario does it contribute to?
  4. If it fixes an open issue, please link to the issue here.
-->
Implements the OpenAI Realtime API with Semantic Kernel

### Description

<!-- Describe your changes, the overall approach, the underlying design.
These notes will help understanding how your code works. Thanks! -->

Adds a RealtimeClientBase
Adds RealtimeEvents
Adds OpenAI and Azure Realtime clients

Closes: microsoft#10108

### Contribution Checklist

<!-- Before submitting this PR, please make sure: -->

- [x] The code builds clean without any errors or warnings
- [x] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [x] All unit tests pass, and I have added new tests where possible
- [x] I didn't break anyone 😄

---------

Co-authored-by: Evan Mattson <35585003+moonbox3@users.noreply.github.com>
musale pushed a commit to musale/semantic-kernel that referenced this pull request Mar 10, 2025
@eavanvalkenburg @moonbox3 @musale
Python: Introducing Realtime Clients for OpenAI and Azure OpenAI (mic…
2bfd519 
…rosoft#10127)

### Motivation and Context

<!-- Thank you for your contribution to the semantic-kernel repo!
Please help reviewers and future users, providing the following
information:
  1. Why is this change required?
  2. What problem does it solve?
  3. What scenario does it contribute to?
  4. If it fixes an open issue, please link to the issue here.
-->
Implements the OpenAI Realtime API with Semantic Kernel

### Description

<!-- Describe your changes, the overall approach, the underlying design.
These notes will help understanding how your code works. Thanks! -->

Adds a RealtimeClientBase
Adds RealtimeEvents
Adds OpenAI and Azure Realtime clients

Closes: microsoft#10108

### Contribution Checklist

<!-- Before submitting this PR, please make sure: -->

- [x] The code builds clean without any errors or warnings
- [x] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [x] All unit tests pass, and I have added new tests where possible
- [x] I didn't break anyone 😄

---------

Co-authored-by: Evan Mattson <35585003+moonbox3@users.noreply.github.com>
jcruzmot-te pushed a commit to thousandeyes/aia-semantic-kernel that referenced this pull request Sep 15, 2025
@eavanvalkenburg @moonbox3
Python: Introducing Realtime Clients for OpenAI and Azure OpenAI (mic…
461b547 
…rosoft#10127)

### Motivation and Context

<!-- Thank you for your contribution to the semantic-kernel repo!
Please help reviewers and future users, providing the following
information:
  1. Why is this change required?
  2. What problem does it solve?
  3. What scenario does it contribute to?
  4. If it fixes an open issue, please link to the issue here.
-->
Implements the OpenAI Realtime API with Semantic Kernel

### Description

<!-- Describe your changes, the overall approach, the underlying design.
These notes will help understanding how your code works. Thanks! -->

Adds a RealtimeClientBase
Adds RealtimeEvents
Adds OpenAI and Azure Realtime clients

Closes: microsoft#10108

### Contribution Checklist

<!-- Before submitting this PR, please make sure: -->

- [x] The code builds clean without any errors or warnings
- [x] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [x] All unit tests pass, and I have added new tests where possible
- [x] I didn't break anyone 😄

---------

Co-authored-by: Evan Mattson <35585003+moonbox3@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot Copilot left review comments

@TaoChenOSU TaoChenOSU TaoChenOSU approved these changes

@moonbox3 moonbox3 moonbox3 approved these changes

Assignees

No one assigned

Labels

documentation python Pull requests for the Python Semantic Kernel

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Python: add support for OpenAI realtime audio

5 participants

@eavanvalkenburg @markwallace-microsoft @TaoChenOSU @moonbox3