WebAssembly compile-time imports

[syg]

Hot off the presses, WebAssembly now has a proposal for compile-time imports: WebAssembly/design#1479.

This is a very early stage proposal on the WebAssembly side, but it'd be good for the champions to be aware of. Namely, should imports be passable both during compile time (i.e. the action that produces a WebAssembly.Module), how might this proposal be extended to support that? In other words, there should be an "out" here so if compile-time imports become a thing, that ideally does not preclude usage of this proposal.

At the same time, because the WebAssembly proposal is so early stage, we definitely should not design the JS proposal speculatively.

Thoughts?

[guybedford]

This did come up in the Wasm CG meeting when we discussed the topic, but it seemed at the time like there wouldn't really be a conflict.

Reading over the design space again, there definitely will be interactions we need to figure out though. Whether compile-time imports build on top of WebAssembly.Module or on top of something lower-level definitely seems like it affects interaction with the ESM integration. That said, I'm not even clear on what the baseline ESM-integration expectation would be for compile-time imports as well? Would standard host-resolution be sufficient?

Then I suppose JS doesn't have a concept of link-based specialization or optimization at all? So this is simply not a concern for JS? Wondering if partial linking will come up for JS at some point.

I'd be very interested to hear if @eqrion has any thoughts with regards to how this might work as well.

[eqrion]

Okay, had some time to read and think through this a bit.

The current proposal for wasm compile-time imports only touches the Wasm JS-API and avoids touching the core wasm spec. The motivation is that this keeps things simple and less controversial.

The other option would have been to add a new wasm binary section for 'compile-time imports' (sometimes called pre-imports by some people) in addition to the normal 'imports section' which is used at instantiation.

By only modifying the wasm JS-API, we accomplish this by specifying which imports are for 'compile-time' and which for 'instantiation-time' dynamically through which import values are given to the compile JS-API. Anything omitted at compile-time must be provided at instantiation time.

One future with wasm-esm-integration and source-phase-imports is that a wasm module imported as 'source' from JS would be equivalent to compiling with the wasm JS-API and not specifying any compile-time imports. In the future to fill the expressivity gap, we could then add the 'compile-time imports' section to wasm binaries and have that be interpreted by the ESM loader to be resolved at compile-time.

The JS equivalent (if you wanted an earlier phase for imports) would be some form of compile import {..} from "..". I can't speak if that would have any practical use, but that's how it could be harmonized.

[guybedford]

Thanks for putting some thought to this one @eqrion, appreciated for keeping it on your list! Very interesting to hear the motivations and great to see your progress on this topic.

The goal of the source phase is to provide a module abstraction that is both transferrable and generic for virtualization. The source phase is also supposed to represent the earliest loading phase of a given source in the current environment. It can have some environment specific host data associated with it as well though.

To expose compile time imports in the ESM integration, it seems to me like the options might be:

1. Define host-defaults from the browser, so that compile-time imports can be automatically provided to a WebAssembly.Module returned by the source phase.
2. Require passing compile options with something like import attributes. We only have primitives here currently so it would require some kind of syntax extension I think.
3. Define a JS representation for an uncompiled module that could be used for the source phase in a generic way - WebAssembly.Source say.

I'm not sure that a separate compile phase would make sense as it would still require a JS representation of this compilation something like (3), which would then be better served by the source representation since I don't believe it would be possible to justify creating a phase that has no meaning for JS modules. Unless you were pointing to something more like (2) for a parameterized source phase?

Then to open up the discussion a bit more - there might also be a similar concept to compile time imports in the module declarations spec which permits an earlier linking phase between sources. See https://github.com/tc39/proposal-module-declarations#example.

In whatever form, it would be great to figure out a path for supporting compile time imports in the ESM integration both with and without the source phase though. If it would help to have an in-person discussion, it would be great to have you join our weekly modules meeting on Tuesday at 9am pacific if you're free then - I can send you an invite if you are interested.

[guybedford]

@eqrion we discussed this topic further in the modules meeting today, and a suggestion on the topic that came up with support from members including @nicolo-ribaudo and @lucacasonato was what if WebAssembly were to assume a special privileged namespace for imports, something like wasmsys:builtin.

On the JS side at least, we could possibly even look at reserving / ensuring this namespace will never be reachable for JS modules, such that it could be assumed to be reachable only from WebAssembly.

In that case, a core module with an import of "wasmsys:builtin-type" "builtin-func" could be known at compile time to only resolve to its specific delegated host builtin meaning, achieving the same goals of compile time imports I believe, without altering interaction with WebAssembly.Module or phasing.

That does leave open the question of other types of non-host compile time imports, although I'm not sure if there is a use case there being prioritised.

Another benefit of the above would then be first-class ESM integration support for these optimizations.

Would something like the above be a viable option do you think?

[eqrion]

Okay, coming back to this after discussing this with several folks offline.

I didn't fully understand the issue because I thought the issue was with how to specify certain wasm imports as being 'compile-time' when using ESM-integration. But the bigger issue is that even if you have that, you need that compile-time import to get some value from another module's exports and that doesn't fit in the ESM module phases (evaluation comes last) without extending it quite a bit.

Proposed new direction

So here's another option (restated and tweaked from @guybedford and others) that supports the use-case of js-builtins in wasm, without using compile-time imports:

1. Add an option to the WebAssembly JS-API compilation endpoints (module constructor, compile, compileStreaming, etc):

dictionary WebAssemblyCompileOptions {
    optional sequence<USVString> builtinModules;
}

Every string in builtinModules must be a host-recognized 'builtin-module', and the list must not have duplicates. Every 'builtin-module' that's provided is available as an import when compiling the WebAssembly module.

Any module import that refers to something from a builtin-module is checked eagerly when compiling and then is no longer needed to be provided when instantiation happens (same as with compile-time imports). They're 'eagerly' applied and this allows us to specialize codegen to them.

2. Define a 'wasm/string' builtin-module, basically the same as the proposal currently has it in the WebAssembly namespace

3. In a future with ESM-integration, builtin-modules are available to be imported and because they're guaranteed to always resolve to the host we can recognize and specialize codegen to them while we compile the module for instantiation later.

Advantages

1. No longer need any restrictions around serializability/shareability that compile-time imports introduced. Anything that's within a builtinModule is reasonably expected to be serializable and shareable.
2. Coherent story for working with ESM-integration in the future

Disadvantages

1. Unclear if we could make Web features available in the wasm-builtin-modules or only just JS features that are purely computational and don't represent a 'capability'. Even if true, this may not matter as we may only care about computational features?

Open question

Would source importing a wasm module that imports a wasm builtin-module have the same 'eager' behavior as in the JS-API above?

If the answer is yes, then this would seem to make certain import namespaces behave differently, which is not ideal. If the answer is no, then we can't easily specialize codegen to these builtin modules, which is the whole point of this proposal.

Maybe it could be optional using an import attribute of some kind to opt-in to eagerly applying some imports?

[guybedford]

Very interesting, thanks @eqrion for putting this together. This would be really nice to see for ESM integration compatibility.

1. Add an option to the WebAssembly JS-API compilation endpoints (module constructor, compile, compileStreaming, etc):

dictionary WebAssemblyCompileOptions {
  optional sequence<USVString> builtinModules;
}

Does this mean that providing a string that is not a host-recognized builtin module will result in an immediate compilation error? If so, will there be some kind of reflection API for determining which host APIs are supported?

Would source importing a wasm module that imports a wasm builtin-module have the same 'eager' behavior as in the JS-API above?

Would eagerly compiled imports be usefully transferrable as WebAssembly.Module instances to workers? If so, then I wouldn't personally see strong reasons for not wanting that in the source phase imports case by default.

Your suggestion of having control over this through import attributes sounds like it might well be an option if further control is needed. For example, to entirely turn off all compile time imports one might have import mod from './mod.wasm' with { compiledHostImports: false } for example. Obviously to be figured out further, but I think having this config on top of the default case of the compile time imports automatically applying may well be preferable.

[eqrion]

1. Add an option to the WebAssembly JS-API compilation endpoints (module constructor, compile, compileStreaming, etc):

dictionary WebAssemblyCompileOptions {
  optional sequence<USVString> builtinModules;
}

Does this mean that providing a string that is not a host-recognized builtin module will result in an immediate compilation error? If so, will there be some kind of reflection API for determining which host APIs are supported?

Good question, I could see that being useful but I'm not sure the best way to do it. One option could be to duplicate these modules on the WebAssembly namespace (e.g. WebAssembly.String). But that could be a lot of work for just feature detection.

Would source importing a wasm module that imports a wasm builtin-module have the same 'eager' behavior as in the JS-API above?

Would eagerly compiled imports be usefully transferrable as WebAssembly.Module instances to workers? If so, then I wouldn't personally see strong reasons for not wanting that in the source phase imports case by default.

Yeah, any Module with eagerly compiled imports should be transferrable under the current set of builtin modules we're thinking of (no capabilities outside of basic JS primitives).

[bakkot]

Looks like the string builtins are going to ship in Chrome, including the compile-time imports mechanism. I don't immediately see a way to use them with this proposal.

[guybedford]

The initial PR to the Wasm ESM Integration to add support for compile time imports is now up in WebAssembly/esm-integration#95. Further additions might include customization with import attributes, although we might want unordered list or boolean attributes for such a feature.