Unable to get threejs camera worldMatrix for realtime light

[skywalkershen]

mapbox-gl-js version: 2.10.0

Question

Hi, my goal is to integrate threejs scene into mapbox, I managed to sync their camera frustums with official demo, yet there is a problem: the light in the scene is like baked texture rather than real-time rendered light.

After testing with the official demo and digging into threejs source code, I think I find the issue. The frustum of rendered scene is calculated with camera's projectionMatrix and matrixWorldInverse while the realtime light is calculated based on camera matrixWorld. So, instead of setting the projectionMatrix as transformedMapboxMatrix, my approach is calculating and setting worldMatrix of the camera based on projectionMatrix and transformedMapboxMatrix. Yet there is always issue with my solutions:

Apporach 1:
jsfiddle based on mapbox official demo
Following @indus comment, I set the camera to perspective camera so it gets fixed projectionMatrix(I also tried calculating projectionMatrix on every render, the result is the same), then calculate worldMatrixInverse with projectionMatrixInverse.multiply(transformedMapboxCamMatrix), decompose worldMatrix to get pos, rotation and scale of the camera.
The realtime light works(I use the model from official demo in my jsfiddle, you have to change model to see the effect), yet the model is apparently larger than it should be. And the model rotate and shift(float) on map rotates, especially obvious when pitching to an 2d above view.

Approach 2:
jsfiddle
Instead of transforming the camera, I put everything in the scene into a group and transform the group, so I can use the mapbox frustum matrix directly as threejs camera frustum. As can be seen, the basic position, scale and rotation of the model is correct, yet the details of the rendering, the vertices become a mess. That's for directly using mapbox matrix as three camera projection matrix.
If calculating mapbox matrix into camera position, rotation, scale as in approach 1, the model is no longer in the frustum. After browsing this post, I notice this might be a scale issue related to mapbox mercator coordinate. Yet it still doesn't explain why the model is not rendered when using calculated camera matrix or using inversed transformMatrix on the group.

I think my general direction might be correct, but I couldn't figure out what is off in detailed approach. It would be most appreciated if someone can help me with this issue. Thank you.

Links to related documentation

#7395 (comment)
https://stackoverflow.com/questions/59103698/mangled-rendering-when-transforming-scene-coordinates-instead-of-camera-coordina#comment104489627_59103698
https://docs.mapbox.com/mapbox-gl-js/example/add-3d-model/
https://docs.mapbox.com/mapbox-gl-js/api/properties/#customlayerinterface#render
https://docs.mapbox.com/mapbox-gl-js/api/geography/#mercatorcoordinate

[alexcrist-planted]

@skywalkershen thanks for writing this up. Did you ever find a solution to this? I seem to be facing the same issue

[skywalkershen]

@skywalkershen thanks for writing this up. Did you ever find a solution to this? I seem to be facing the same issue

As mentioned in the issue, you must touch both worldMatrix and projection matrix for the camera to get correct render. And to avoid big coordinate issue, it is also necessary to translate/rotate/scale the whole scene. If you don't want write all the logics yourself, try threebox, it works.