Add screenX, screenY and screenZ

[brendandawes]

Would be great if you could add screenX etc for getting the 2D screen coordinates of a 3D vector. Currently trying to map HTML text to a 3D object and maths is making my head hurt!

Thanks for all the great work.

[AkashGutha]

that sounds great.

i think we need to use projection of vectors on planes.
correct me if i'm wrong.

it would be particularly nice if we could do this.

//example code
var screenVector = threeDVector.getCanvasPoint();

[tcoppex]

Hi @brendandawes ,
As @AkashGutha suggested you need to project a point on a plane, namely the front plane of your camera view frustum.

Fortunately for us it is exactly what OpenGL and the graphics pipeline do when we push vertices to the GPU to be rendered. All this steps are done using transformation 4x4 matrices in different stages changing the space coordinates of our vertices :

• the ModelView matrix set the vertices from object to camera space,
• the Projection matrix set the vertices in Normalize Device Coordinates ranging in [-1, 1]³ (roughly, a normalized 3D cube mapping the screen),
• the NDC coordinates are then transformed in screen-space using our screen viewport informations.

Perspective vs NDC space (credits to songho).

So depending on our needs we have to simulate this steps (world to canvas coordinates) or undo them (canvas to world coordinates), to get our vertices in the right space 😉

Here's what you're looking for :

/* unlicense.org */

/* Multiply a 4x4 homogeneous matrix by a Vector4 considered as point
 * (ie, subject to translation). */
function multMatrixVector(m, v) {
  if (!(m instanceof p5.Matrix) || !(v instanceof p5.Vector)) {
    print('multMatrixVector : Invalid arguments');
    return;
  }

  var _dest = createVector();
  var mat = m.mat4;

  // Multiply in column major order.
  _dest.x = mat[0] * v.x + mat[4] * v.y + mat[8] * v.z + mat[12];
  _dest.y = mat[1] * v.x + mat[5] * v.y + mat[9] * v.z + mat[13];
  _dest.z = mat[2] * v.x + mat[6] * v.y + mat[10] * v.z + mat[14]; 
  var w =   mat[3] * v.x + mat[7] * v.y + mat[11] * v.z + mat[15];

  if (Math.abs(w) > Number.EPSILON) {
    _dest.mult(1.0 / w);
  }

  return _dest;
}

/* Project a vector from Canvas to World coordinates. */
function projectCanvasToWorld(canvas, vCanvas) {
  // Retrieve the ModelView and Projection matrices.
  var mv = canvas.uMVMatrix.copy();
  var p  = canvas.uPMatrix.copy();

  // Compute the ModelViewProjection matrix.
  var mvp = mv.mult(p);

  // Inverts the MVP.
  var invMVP = mvp.invert(mvp);
 
  // Transform the canvas vector to Normalized Device Coordinates (in [-1, 1]³),
  // Here viewport is (0, 0, drawingBufferWidth, drawingBufferHeight).
  var vNDC = createVector();
  vNDC.x = (-1.0 + 2.0 * (vCanvas.x / canvas.GL.drawingBufferWidth));
  vNDC.y = (-1.0 + 2.0 * (vCanvas.y / canvas.GL.drawingBufferHeight));
  vNDC.z = (-1.0 + 2.0 * (vCanvas.z));

  // Transform vector from NDC to world coordinates.
  var vWorld = multMatrixVector(invMVP, vNDC);

  return vWorld;
}

/* Project a vector from World to Canvas coordinates. */
function projectWorldToCanvas(canvas, vWorld) {
  // Calculate the ModelViewProjection Matrix.
  var mvp = (canvas.uMVMatrix.copy()).mult(canvas.uPMatrix);

  // Transform the vector to Normalized Device Coordinate.
  var vNDC = multMatrixVector(mvp, vWorld);

  // Transform vector from NDC to Canvas coordinates.
  var vCanvas = createVector();
  vCanvas.x = 0.5 * (vNDC.x + 1.0) * canvas.GL.drawingBufferWidth;
  vCanvas.y = 0.5 * (vNDC.y + 1.0) * canvas.GL.drawingBufferHeight;
  vCanvas.z = 0.5 * (vNDC.z + 1.0);

  return vCanvas;
}

Note : Matrices operations being costly (especially inversion) you might want to optimize this for bulk transforms.

Due to a lack of 2d / 3d interop I'm not one hundred percent sure of the need to implement this to p5.js right now, but this could still be useful for mouse picking in 3d (especially for interactive applications).

Cheers 👍

[brendandawes]

Wow! @tcoppex Thanks for such a great explanation @AkashGutha

[jaschanarveson]

@AkashGutha - this is cool. I don't fully understand why some kind of conversion function wouldn't be useful to implement in p5js right now, though - it sure seems like it would be handy!

[lmccart]

thanks for the suggestion @brendandawes! I think this is beyond the scope of p5 for now, we're trying to focus on getting core webgl working before expanding the api. but this could make a nice addon library using some of @tcoppex's solution above. see the docs here for how to create an addon: https://github.com/processing/p5.js/wiki/Libraries#creating-a-new-library

[brendandawes]

@tcoppex Thanks to your wonderful piece of code I've started to build the interface I've wanted to do for a while as a way to navigate my work archive. A few work in progress vids here: https://www.instagram.com/p/BeV2tckn0dX/?taken-by=brendandawes The text overlays are all html divs and I'm using the EasyCam library for navigation, together with the sound lib. Amazing what Javascript can do these days. Love p5js! @lmccart

[nevahid]

Hi @brendandawes ,
As @AkashGutha suggested you need to project a point on a plane, namely the front plane of your camera view frustum.

Fortunately for us it is exactly what OpenGL and the graphics pipeline do when we push vertices to the GPU to be rendered. All this steps are done using transformation 4x4 matrices in different stages changing the space coordinates of our vertices :

• the ModelView matrix set the vertices from object to camera space,
• the Projection matrix set the vertices in Normalize Device Coordinates ranging in [-1, 1]³ (roughly, a normalized 3D cube mapping the screen),
• the NDC coordinates are then transformed in screen-space using our screen viewport informations.

Perspective vs NDC space (credits to songho).

So depending on our needs we have to simulate this steps (world to canvas coordinates) or undo them (canvas to world coordinates), to get our vertices in the right space 😉

Here's what you're looking for :

/* unlicense.org */

/* Multiply a 4x4 homogeneous matrix by a Vector4 considered as point
 * (ie, subject to translation). */
function multMatrixVector(m, v) {
  if (!(m instanceof p5.Matrix) || !(v instanceof p5.Vector)) {
    print('multMatrixVector : Invalid arguments');
    return;
  }

  var _dest = createVector();
  var mat = m.mat4;

  // Multiply in column major order.
  _dest.x = mat[0] * v.x + mat[4] * v.y + mat[8] * v.z + mat[12];
  _dest.y = mat[1] * v.x + mat[5] * v.y + mat[9] * v.z + mat[13];
  _dest.z = mat[2] * v.x + mat[6] * v.y + mat[10] * v.z + mat[14]; 
  var w =   mat[3] * v.x + mat[7] * v.y + mat[11] * v.z + mat[15];

  if (Math.abs(w) > Number.EPSILON) {
    _dest.mult(1.0 / w);
  }

  return _dest;
}

/* Project a vector from Canvas to World coordinates. */
function projectCanvasToWorld(canvas, vCanvas) {
  // Retrieve the ModelView and Projection matrices.
  var mv = canvas.uMVMatrix.copy();
  var p  = canvas.uPMatrix.copy();

  // Compute the ModelViewProjection matrix.
  var mvp = mv.mult(p);

  // Inverts the MVP.
  var invMVP = mvp.invert(mvp);
 
  // Transform the canvas vector to Normalized Device Coordinates (in [-1, 1]³),
  // Here viewport is (0, 0, drawingBufferWidth, drawingBufferHeight).
  var vNDC = createVector();
  vNDC.x = (-1.0 + 2.0 * (vCanvas.x / canvas.GL.drawingBufferWidth));
  vNDC.y = (-1.0 + 2.0 * (vCanvas.y / canvas.GL.drawingBufferHeight));
  vNDC.z = (-1.0 + 2.0 * (vCanvas.z));

  // Transform vector from NDC to world coordinates.
  var vWorld = multMatrixVector(invMVP, vNDC);

  return vWorld;
}

/* Project a vector from World to Canvas coordinates. */
function projectWorldToCanvas(canvas, vWorld) {
  // Calculate the ModelViewProjection Matrix.
  var mvp = (canvas.uMVMatrix.copy()).mult(canvas.uPMatrix);

  // Transform the vector to Normalized Device Coordinate.
  var vNDC = multMatrixVector(mvp, vWorld);

  // Transform vector from NDC to Canvas coordinates.
  var vCanvas = createVector();
  vCanvas.x = 0.5 * (vNDC.x + 1.0) * canvas.GL.drawingBufferWidth;
  vCanvas.y = 0.5 * (vNDC.y + 1.0) * canvas.GL.drawingBufferHeight;
  vCanvas.z = 0.5 * (vNDC.z + 1.0);

  return vCanvas;
}

Note : Matrices operations being costly (especially inversion) you might want to optimize this for bulk transforms.

Due to a lack of 2d / 3d interop I'm not one hundred percent sure of the need to implement this to p5.js right now, but this could still be useful for mouse picking in 3d (especially for interactive applications).

Cheers 👍

hi there... i saw this code as i was wandering around to find a way to do this transformation... i just got confused... in your function projectCanvasToWorld(canvas, vCanvas), what is canvas and what is vCanvas? thank you so much again for your help...

[tcoppex]

Hi @nevahid,

It's old code but to my remembrance canvas is the processing canvas object and vCanvas is the vector in canvas-space coordinates as the comments suggested. Consequently vWorld is the vector in world-space coordinates in the second function.

Hope that helps.

[nevahid]

@tcoppex thank you very much. i'm trying to figure it. but it gives me confusing positions, maybe i'm wrong somewhere... what i try to do is to get mouse position in p5.easycam camera... as i had to set the camera canvas to null, i used the p5.renderer as the canvas... maybe that is the point...
i still try to make it work.... thank you so much, and if you have any hints or help, i would really appreciate it...

[ffd8]

Can anyone provide some insight for how the above code can be used? I don't quite understand what params should be fed as canvas and vWorld into the projectWorldToCanvas() function. Here's an example of something I'd be interested to extract screenX/Y/Z coords from. Also wonder if this works both with/without WEBGL mode?

[bohnacker]

@ffd8: Here is my solution. It works with 2D and WEBGL mode. Both have its own difficulties:
In 2D I didn't find a way of getting the actual transformation matrix from the CanvasRenderingContext2D, so I've implemented some functions to track them.
In WEBGL mode the projection math is a bit more difficult. Thanks to @tcoppex it was already almost done. I had to adjust it a bit, maybe because p5js' way of handling WEBGL coordinates changed since 2016.

You'll find the function here:
https://github.com/bohnacker/p5js-screenPosition

And how it's used here:
https://editor.p5js.org/bohnacker/sketches/nUk3bVW7b

[ffd8]

@bohnacker Really nice solution and easy to implement, many thanks!

[jumpjack]

@ffd8: Here is my solution. It works with 2D and WEBGL mode. Both have its own difficulties: In 2D I didn't find a way of getting the actual transformation matrix from the CanvasRenderingContext2D, so I've implemented some functions to track them. In WEBGL mode the projection math is a bit more difficult. Thanks to @tcoppex it was already almost done. I had to adjust it a bit, maybe because p5js' way of handling WEBGL coordinates changed since 2016.

You'll find the function here: https://github.com/bohnacker/p5js-screenPosition

And how it's used here: https://editor.p5js.org/bohnacker/sketches/nUk3bVW7b

I don't understand if your library is suitable for my needs: I want to draw with the mouse over the sides of a cube which has a corner in the origin; so how do I convert MouseX and MouseY to 3d coordinates using your library? Or should I use @nevahid solution instead? Or p5.projection? Or this snippet?

[bohnacker]

Hi @jumpjack,
I'm sure that my solution is not what you need. It just works the other way round. Same with the snippet/discussion you found on the processing forum.
I was thinking a bit if the p5.projection library could work for you, but I don't think so.
And the last one, the solution from @nevahid: It looks a bit more promissing for you, but I'm actually not sure from the quick glance I had.

I found a jsfiddle that is more or less doing what you need. Unfortunately it is using three.js and not p5.js.

[jumpjack]

@nevahid solution is working quite good ... as long as I don't rotate the scene: orbitContol() disrupts everything... :-(
But anyway also without rotating I get a vector only on a plane parallel to the screen, I don't know how to implement support for planes parallel to XZ and YZ.

[jumpjack]

How can I know the rotation angles used by orbitControl(), i.e. the angles of X, Y and Z w.r.t. screen normal?

[jumpjack]

It looks like this topic is actually named "3d picking".

[jumpjack]

How can I know the rotation angles used by orbitControl(), i.e. the angles of X, Y and Z w.r.t. screen normal?

I found the answer:

createCamera()._getLocalAxes() returns {x[], y[], z[]}, which represents orientation of axes after moving them with orbitControl();

asin(cam._getLocalAxes().x[0]) will give you rotation of X axis of the scene w.r.t X axis of the camera, and so on.

Also cameraNear, cameraFar and cameraFOV are interesting, although I dind't yet understand how to use them.

[jumpjack]

I found in p5js libraries the P5XR library and its getRayFromScreen(x, y) ad intersectsPlane([ray]) ... but I don't get how to port the algorithm to "standard" p5JS (not VR).