Support retrieved_from: :doc_values for direct leaf fields#1110
Open
Support retrieved_from: :doc_values for direct leaf fields#1110
retrieved_from: :doc_values for direct leaf fields#1110Conversation
jwils
changed the title
retrieved_from: :doc_values for direct leaf fields
jwils
force-pushed
the
joshuaw/docvalue-leaf-fetching
branch
3 times, most recently
from
ad363a5 to
a2b2407
Compare
jwils
force-pushed
the
joshauw/fetchable-false-prototype
branch
from
8e9b12f to
bde522d
Compare
jwils
force-pushed
the
joshuaw/docvalue-leaf-fetching
branch
from
a2b2407 to
fdf2582
Compare
jwils
force-pushed
the
joshauw/fetchable-false-prototype
branch
4 times, most recently
from
caabea3 to
fd1d61a
Compare
jwils
added a commit
that referenced
this pull request
Apr 14, 2026
Add a narrow secondary retrieval path for fields that should stay returnable in GraphQL while being excluded from stored `_source`. When a field is marked `retrieved_from: :doc_values`, ElasticGraph: - keeps the field in GraphQL output types - excludes the field from datastore `_source` via `_source.excludes` - requests the field through datastore `docvalue_fields` - resolves the field from `fields` in datastore hits when `_source` omits it The change is intentionally narrow: only direct, non-list, non-text GraphQL leaf fields on indexed root document types are supported. Query planning only uses `docvalue_fields` when every participating index definition agrees on the retrieval method. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
jwils
force-pushed
the
joshuaw/docvalue-leaf-fetching
branch
from
fdf2582 to
83f0f4b
Compare
jwils
added a commit
that referenced
this pull request
Apr 14, 2026
Add a narrow secondary retrieval path for fields that should stay returnable in GraphQL while being excluded from stored `_source`. When a field is marked `retrieved_from: :doc_values`, ElasticGraph: - keeps the field in GraphQL output types - excludes the field from datastore `_source` via `_source.excludes` - requests the field through datastore `docvalue_fields` - resolves the field from `fields` in datastore hits when `_source` omits it The change is intentionally narrow: only direct, non-list, non-text GraphQL leaf fields on indexed root document types are supported. Query planning only uses `docvalue_fields` when every participating index definition agrees on the retrieval method. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
jwils
force-pushed
the
joshuaw/docvalue-leaf-fetching
branch
from
83f0f4b to
86787f5
Compare
jwils
added a commit
that referenced
this pull request
Apr 14, 2026
Add a narrow secondary retrieval path for fields that should stay returnable in GraphQL while being excluded from stored `_source`. When a field is marked `retrieved_from: :doc_values`, ElasticGraph: - keeps the field in GraphQL output types - excludes the field from datastore `_source` via `_source.excludes` - requests the field through datastore `docvalue_fields` - resolves the field from `fields` in datastore hits when `_source` omits it The change is intentionally narrow: only direct, non-list, non-text GraphQL leaf fields on indexed root document types are supported. Query planning only uses `docvalue_fields` when every participating index definition agrees on the retrieval method. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
jwils
force-pushed
the
joshuaw/docvalue-leaf-fetching
branch
from
86787f5 to
8e29a0b
Compare
jwils
added a commit
that referenced
this pull request
Apr 14, 2026
Add a narrow secondary retrieval path for fields that should stay returnable in GraphQL while being excluded from stored `_source`. When a field is marked `retrieved_from: :doc_values`, ElasticGraph: - keeps the field in GraphQL output types - excludes the field from datastore `_source` via `_source.excludes` - requests the field through datastore `docvalue_fields` - resolves the field from `fields` in datastore hits when `_source` omits it The change is intentionally narrow: only direct, non-list, non-text GraphQL leaf fields on indexed root document types are supported. Query planning only uses `docvalue_fields` when every participating index definition agrees on the retrieval method. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
jwils
force-pushed
the
joshuaw/docvalue-leaf-fetching
branch
from
8e29a0b to
f9afdfc
Compare
jwils
added a commit
that referenced
this pull request
Apr 14, 2026
Add a narrow secondary retrieval path for fields that should stay returnable in GraphQL while being excluded from stored `_source`. When a field is marked `retrieved_from: :doc_values`, ElasticGraph: - keeps the field in GraphQL output types - excludes the field from datastore `_source` via `_source.excludes` - requests the field through datastore `docvalue_fields` - resolves the field from `fields` in datastore hits when `_source` omits it The change is intentionally narrow: only direct, non-list, non-text GraphQL leaf fields on indexed root document types are supported. Query planning only uses `docvalue_fields` when every participating index definition agrees on the retrieval method. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
jwils
force-pushed
the
joshuaw/docvalue-leaf-fetching
branch
from
f9afdfc to
6e0529f
Compare
jwils
added a commit
that referenced
this pull request
Apr 14, 2026
Add a narrow secondary retrieval path for fields that should stay returnable in GraphQL while being excluded from stored `_source`. When a field is marked `retrieved_from: :doc_values`, ElasticGraph: - keeps the field in GraphQL output types - excludes the field from datastore `_source` via `_source.excludes` - requests the field through datastore `docvalue_fields` - resolves the field from `fields` in datastore hits when `_source` omits it The change is intentionally narrow: only direct, non-list, non-text GraphQL leaf fields on indexed root document types are supported. Query planning only uses `docvalue_fields` when every participating index definition agrees on the retrieval method. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
jwils
force-pushed
the
joshuaw/docvalue-leaf-fetching
branch
from
6e0529f to
a006605
Compare
jwils
added a commit
that referenced
this pull request
Apr 14, 2026
Add a narrow secondary retrieval path for fields that should stay returnable in GraphQL while being excluded from stored `_source`. When a field is marked `retrieved_from: :doc_values`, ElasticGraph: - keeps the field in GraphQL output types - excludes the field from datastore `_source` via `_source.excludes` - requests the field through datastore `docvalue_fields` - resolves the field from `fields` in datastore hits when `_source` omits it The change is intentionally narrow: only direct, non-list, non-text GraphQL leaf fields on indexed root document types are supported. Query planning only uses `docvalue_fields` when every participating index definition agrees on the retrieval method. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
jwils
force-pushed
the
joshuaw/docvalue-leaf-fetching
branch
from
a006605 to
4fe1200
Compare
Add a narrow secondary retrieval path for fields that should stay returnable in GraphQL while being excluded from stored `_source`. When a field is marked `retrieved_from: :doc_values`, ElasticGraph: - keeps the field in GraphQL output types - excludes the field from datastore `_source` via `_source.excludes` - requests the field through datastore `docvalue_fields` - resolves the field from `fields` in datastore hits when `_source` omits it The change is intentionally narrow: only direct, non-list, non-text GraphQL leaf fields on indexed root document types are supported. Query planning only uses `docvalue_fields` when every participating index definition agrees on the retrieval method. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
jwils
force-pushed
the
joshuaw/docvalue-leaf-fetching
branch
from
4fe1200 to
a6256d3
Compare
myronmarston
requested changes
Apr 14, 2026
Collaborator
myronmarston
left a comment
myronmarston
left a comment
There was a problem hiding this comment.
retrieved_from: :doc_values exposes a level of detail that, until now, we haven't exposed to ElasticGraph users. ElasticGraph does many things to query the data as efficiently as possible, without asking users to configure those things.
Can we do something similar here? For example, can we determine in which situations it's more efficient to use doc_values and automatically do so without requiring users to configure it at the field level?
Also, if we're going to change how fields are retrieved, I'd like to take a step back and review a full design that takes into account all the options Elasticsearch/OpenSearch offer us here. Besides _source and docvalue_fields there's also fields and stored_fields. I'd like to consider a unified design rather than just tacking this on as a one-off.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR adds a narrow secondary retrieval path for fields that should stay returnable in GraphQL while being excluded from stored
_source.When a field is marked
retrieved_from: :doc_values, ElasticGraph:_sourcevia_source.excludesdocvalue_fieldsfieldsin datastore hits when_sourceomits itImplementation
The change is intentionally narrow:
name_in_indexpaths are rejecteddocvalue_fieldswhen every participating index definition agrees on the retrieval methodThe runtime metadata now records how an index field is retrieved, the GraphQL datastore query can split requested fields between
_sourceanddocvalue_fields, and datastore response helpers can read direct leaf values back from hitfieldswhen needed.Tests
Updated unit specs cover:
retrieved_from: :doc_valuesfields in GraphQL output types_source.excludesand runtime metadata for those fieldsdocvalue_fieldsand falling back to_sourcewhen index definitions disagreefieldsretrieved_fromusagesI was not able to run the RSpec files in this local environment because Bundler is missing required gems/native extensions.
Stack Info: Stacked on #1108