error: Cannot read properties of undefined (reading 'resurrectTimeout')

[pmuellr]

On Kibana main, using @elastic/transport 8.7.0

We've started to see some errors with resurrectTimeout like the following: Cannot read properties of undefined (reading 'resurrectTimeout'). These seem to have started when we changed our access patterns against an index - different operations and different frequency. Haven't yet looked into any other context on them, but figured I'd go ahead and at least post here that we are seeing them.

[pmuellr]

Issue we are tracking within Kibana: elastic/kibana#196379

[pmuellr]

Took a quick peek at the code, seems like it's going to be one of the references after this.connections.find(), which can return undefined. Presumably there's a race condition somewhere removing the connection? I saw two references like this, in that module.
This would align with the changes we made, where we are making ES calls at a faster rate than before (when we didn't see this often). More chances for a race.

elastic-transport-js/src/pool/ClusterConnectionPool.ts

Lines 151 to 156 in 3374510

	const connection = this.connections.find(c => c.id === this.dead[0]) as Connection
	if (opts.now < connection.resurrectTimeout) {
	debug('Nothing to resurrect')
	return
	}

[JoshMock]

@pmuellr Hey Patrick! Started looking at this, finally, and am not certain a race condition would cause this. The code you linked to in the resurrect function is synchronous, so when it checks a couple lines earlier whether the list of dead connections is empty or not, that list would not have have changed its state at all before hitting the lines you referenced.

Instead, I'm now curious if there's a bug somewhere in ClusterConnectionPool that either does not assign an id to each connection, or sets it to some inconsistent value, or is not correctly adding connections to the list of dead connections in all cases where it should.

In order to hone in on the issue, it would be super helpful to get some snippets of code from you, particularly the code where you create your client instance, especially the list of nodes, or anywhere you might be directly accessing client.connectionPool to add, modify or remove connections yourself. Any client configs related to sniffing would also be relevant, as well as any interesting notes about your on-prem (presumably) ES setup that would affect the output of /_nodes/_all/http (which is used by the sniffing transport, if you have it enabled).

That's a lot to ask, but hopefully some of those bits will add the necessary context to why it's not finding a dead connection that should be there.

[pmuellr]

I looked at the logs again - previously I had seen this in serverless and not ECH, now I'm not seeing it in serverless since 3/25, but can see it in ECH across versions 8.3 -> 8.11. It looks like they are all occurring in the saved object service, and looking at some of them, seem to be at start time when initializing the SO indices. For example:

[2025-04-21T09:23:29.552+00:00][ERROR][savedobjects-service] TypeError: Cannot read properties of undefined (reading 'resurrectTimeout')
    at ClusterConnectionPool.resurrect (/usr/share/kibana/node_modules/@elastic/transport/lib/pool/ClusterConnectionPool.js:141:35)
    at ClusterConnectionPool.getConnection (/usr/share/kibana/node_modules/@elastic/transport/lib/pool/ClusterConnectionPool.js:206:14)
    at KibanaTransport.getConnection (/usr/share/kibana/node_modules/@elastic/transport/lib/Transport.js:548:48)
    at KibanaTransport.wrappedGetConnection (/usr/share/kibana/node_modules/elastic-apm-node/lib/instrumentation/modules/@elastic/elasticsearch.js:91:38)
    at KibanaTransport.request (/usr/share/kibana/node_modules/@elastic/transport/lib/Transport.js:403:40)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at KibanaTransport.request (/usr/share/kibana/src/core/server/elasticsearch/client/create_transport.js:58:16)
    at ClientTraced.OpenPointInTimeApi [as openPointInTime] (/usr/share/kibana/node_modules/@elastic/elasticsearch/lib/api/api/open_point_in_time.js:36:12)
    at stateActionMachine (/usr/share/kibana/src/core/server/saved_objects/migrations/state_action_machine.js:67:28)
    at migrationStateActionMachine (/usr/share/kibana/src/core/server/saved_objects/migrations/migrations_state_action_machine.js:92:24)
    at async Promise.all (index 0)

 FATAL  Error: Unable to complete saved object migrations for the [.kibana_task_manager] index: Unable to complete the OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT step after 15 attempts, terminating. The last failure message was: search_phase_execution_exception: 

We'll have to get @elastic/kibana-core involved to point out where the associated code for this is ...