Extend ImageLoader interface with texture channel mask.

[castano]

This API extends the ITextureImageLoader interface with a new overload of the LoadImage method that also takes an int channelMask argument. This is a bitmask that indicates what channels of the texture are consumed by the material.

The default implementation forwards calls to the original LoadImage method, so existing implementations are unaffected.

Computing the texture channel masks is trivial. It only requires traversing the materials and tagging the textures based on the slots in which the texture is used. This is similar to the way we currently tag sRGB textures.

Motivation:

When loading textures it's often possible to chose the storage format based on how the texture is used. For example, a texture that is only bound as an occlusion map only uses the R channel, but the input file may be decoded in RGBA format and consume 4x more video memory than required.

This contribution is sponsored by Ludicon.

[atteneder]

Thanks for the contribution!

May I ask you to target either branch main, ideally directly on the Unity fork. Warning upfront this openupm branch is merely a copy of the original, which changed its structure to monorepo, so you have to put your changes into subfolder Packages/com.unity.cloud.gltfast.

Do you mind elaborating the concept a bit further to me? I need a better understanding of the subject first.

Expanding on your example, a texture that's used as an occlusion map only could get loaded into R8_UNorm instead of R8G8B8A8_UNorm, right?
How would that work for a texture that used as a roughness-metallness texture only (channels GB). I assume one could load those into R8G8_UNorm, but now the channel mapping changed and I guess the shader needs to adjust channel mapping, right? Either this or choosing R8G8B8_UNorm (with only one channel wasted).

Does that work in conjunction with compressed formats? For example how to transcode a Basis Universal supercompressed texture with a non-RGBA mask? How does it influence picking a transcode format?

Code itself, on a first glimpse, looks good.

May I suggest a more idiomatic alternative to the byte/int type choice for the mask, a flag enum, similar to UnityEngine.Rendering.ColorWriteMask (I'd use it directly if it wasn't for the misleading name/purpose/XML docs).

namespace UnityEngine.Rendering;

/// <summary>
///   <para>Specifies which color components will get written into the target framebuffer.</para>
/// </summary>
[Flags]
public enum ColorWriteMask
{
  /// <summary>
  ///   <para>Write alpha component.</para>
  /// </summary>
  Alpha = 1,
  /// <summary>
  ///   <para>Write blue component.</para>
  /// </summary>
  Blue = 2,
  /// <summary>
  ///   <para>Write green component.</para>
  /// </summary>
  Green = 4,
  /// <summary>
  ///   <para>Write red component.</para>
  /// </summary>
  Red = 8,
  /// <summary>
  ///   <para>Write all components (R, G, B and Alpha).</para>
  /// </summary>
  All = Red | Green | Blue | Alpha, // 0x0000000F
}

Again, thanks for the effort!

[castano]

May I ask you to target either branch main, ideally directly on the Unity fork.

No problem, I'll open a PR on that repo.

Expanding on your example, a texture that's used as an occlusion map only could get loaded into R8_UNorm instead of R8G8B8A8_UNorm, right?

That's right.

How would that work for a texture that used as a roughness-metallness texture only (channels GB). I assume one could load those into R8G8_UNorm, but now the channel mapping changed and I guess the shader needs to adjust channel mapping, right? Either this or choosing R8G8B8_UNorm (with only one channel wasted).

The most optimal solution would be to use R8G8_Unorm and then swizzle the channels in the shader, but that's out of the scope of this proposal. Most APIs also allow creating custom texture views, which allows mapping the RG channels to GB, but this is not exposed through the Unity rendering APIs.

In practice the simplest approach would be to use the R8G8B8_Unorm format and waste the R8 channel.

Does that work in conjunction with compressed formats?

In the context of runtime compression this is extremely useful. Having this information allows much more optimal format selection. For example, R-only textures can be encoded in BC4 or EAC. RGB-only textures can use BC1/ETC which is half the size as ASTC/BC7. Even when targeting the same format, knowing what subset of the channels is used allows the use of more efficient, higher quality encoders.

For example how to transcode a Basis Universal supercompressed texture with a non-RGBA mask? How does it influence picking a transcode format?

What to do with the mask is up to the implementation of the loader interface. The only requirement is that the resulting texture has storage for the channels specified in the mask.

When transcoding KTX files you want to take other factors into consideration. An UASTC file transcodes to ASTC much more efficiently than to EAC_RG, so in some way the preferred target format is decided when the KTX file is compressed, not when it's loaded.

On the other hand, when loading a JPEG or an AVIF file having a mask allows you to specify the most efficient target format, whether it's compressed or not.

May I suggest a more idiomatic alternative to the byte/int type choice for the mask, a flag enum, similar to UnityEngine.Rendering.ColorWriteMask (I'd use it directly if it wasn't for the misleading name/purpose/XML docs).

I chose byte to avoid introducing new types, but I can certainly use an enum.