[FEATURE]: Asset Browser Integration

[Victorlouisdg]

Asset Browser Integration

With the 3.0 release, Blender added a new editor: The Asset Browser.

What's great about the asset browser is that it finally provides a unified way to organize assets.

I'll first explain the basics of the Asset Browser, and then what it could mean for BlenderProc.

1. The Asset Browser

1.1 Creating Assets

Various Blender data-blocks can be assets: objects, materials, worlds, lights, cameras, poses.

To make a data-block available as an asset, you need to do two things:

1. Select the data-block, right click and "Mark as Asset"
2. Put the .blend file containing the asset into an Asset Library.

An Asset Library is simply a folder which contains .blend files (including its subfolders).
The default Asset Library is the User Library, located at ~/Documents/Blender/Assets.

1.2 Catalogs and Tags

There are two ways to organize Assets: Catalogs and Tags.

A catalog is a tree structure in which an asset can appear only once, similar to how files are organized in a file system.

A tag is simply a string that describes an asset, e.g. Wood, Poly Haven, License CC-0. An asset can have multiple tags.

I believe that BlenderProc should focus on using tags.
Tags are simpler, more flexible and in my opinion, less ambiguous.

2. BlenderProc Integration

One of BlenderProc's strengths is the ability to download and load assets from several online asset providers e.g. Poly Haven and CC-0 Textures.

I believe that with the Asset Browser we can make this process even better.

I have two main ideas for this: Make Everything an Asset and Tag-based Asset Loading.

2.1 Make Everything an Asset

BlenderProc should try to make as many downloaded assets into Blender Assets as possible.
This has several advantages:

• Once something is an asset, (I assume) it can be appended/linked to the current scene fast
• To check out the downloaded Assets, users can simply open up Blender and drag and drop them into a scene from the Asset Browser
• Easy to integrate with assets the user created themselves
• Its future proof: asset providers such as Poly Haven will likely allow us to download their assets as Blender Assets directly.
  This would greatly simplify the code to download and load haven textures.

BlenderProc would download all assets to the User Library by default.
Additionally, the User Library could become the default place BlenderProc functions and tests would look for assets (replacing the resources folder).

For other assets e.g CC-0 textures, BlenderProc would automaticallly create assets. This could mean unzipping, setting up the node tree, adding tags etc.

2.2 Tag-based Asset Loading

Each asset can have multiple tags.
These can come with the downloads e.g. from Poly Haven, and users can add additional tags themselves.
Tags would allow users to easily filter which assets they want to load.

Here's an example of what the API for materials could look like:

floor = bproc.object.create_primitive("PLANE")

### Manual loading
wood_material = bproc.asset.load_material("wood_floor_001")
floor.add_material(wood_material)

wood_names = bproc.asset.materials(tags=["Wood"])
wood_material = bproc.asset.load_material(wood_names[0])

### Convenience methods
floor = brpoc.object.create_primitive("PLANE")
floor.add_material(name="wood_floor_001")  # Searches for specific asset
floor.add_material(tags=["Wood", "Stone"]) # Random asset with tag Wood or Stone
floor.add_material(required_tags=["Wood", "Poly Haven"]) # Random asset with tags Wood and Poly Haven

# Random Poly Haven asset with tag Wood or Stone
floor.add_material(tags=["Wood", "Stone"], required_tags=["Poly Haven"]) 

Other assets would have similar APIs:

world_1 = bproc.asset.load_world("snow_field")
world_2 = bproc.asset.load_world(required_tags=["Snow", "Poly Haven"])
bproc.set_world(world_1)

fruit = bproc.asset.load_object(tags=["Apple", "Orange", "Pear"])

camera = bproc.asset.load_camera("Intel Realsense D435")

3. Summary

The User Library becomes the default location to download and load assets, and we add a bproc.asset module with:

• load_material()
• load_object()
• load_light()
• load_camera()
• load_world()

These functions load the assets, and also immediately "apply" them to the current scene.
E.g. bproc.asset.load_material() would load the world, and also set it as the "active" world, because this is usually what users want.

These functions would have arguments:

• name: if name is given, search for asset with the exact name
• tags: search random asset with one of these tags
• required_tags: the random asset must have all of these tags

[themasterlink]

Hey,

that is a great improvement to BlenderProc, we have considered this in the past already, but never that detailed. It requires a major restructure of the library, though. I am currently super deep down in work. Would you be willing to work with us on this starting mid march, so like in a month? I would love to work with you on this, I already liked your last PRs.

Best regards,
Max

[Victorlouisdg]

I'm happy to hear that you like the general idea! I'd gladly help working on this mid march :)

[Victorlouisdg]

Update: I haven't forgotten about this yet, but to properly implement this feature we need Blender to extend its Python API so that it can list the available assets in all the Blend files of the Asset Library. The Blender developers say this is planned, but there's still some work required to make it possible.