🐛 Frame orientation is in landscape-right when phone is held in portrait mode #3051

Marius-Adam · 2024-07-05T11:42:11Z

What's happening?

I want to scan text in portrait mode but the camera does not pick up anything only if the text is sideways or the phone is in landscape mode.

When logging out the frame.orientation it comes back as landscape-right when the phone is held in portrait

RPReplay_Final1720178652.1.mp4

Reproduceable Code

import React from "react";
import { StyleSheet } from "react-native";
import { Camera, useCameraDevice, useFrameProcessor } from "react-native-vision-camera";
import { useTextRecognition } from "react-native-vision-camera-text-recognition";

function App() {
  const device = useCameraDevice("back");
  const options = { language: "latin"};
  const { scanText } = useTextRecognition(options);
  const frameProcessor = useFrameProcessor((frame) => {
    "worklet";
    console.log(frame.orientation)
    const data = scanText(frame);
   
  }, []);
  return (
    <>
      {!!device && (
        <Camera
          style={StyleSheet.absoluteFill}
          device={device}
          isActive
          frameProcessor={frameProcessor}
        />
      )}
    </>
  );
}
export default App;

Relevant log output

Running "main" with {"rootTag":1,"initialProps":{"concurrentRoot":false}}
Loading react-native-worklets-core...
Worklets loaded successfully
VisionCameraProxy: Creating Worklet Context...
VisionCameraProxy: Worklet Context Created!
🟢 Creating JS object for module 'ExpoKeepAwake'
Looking up Frame Processor Plugin "scanText"...
Frame Processor Plugin "scanText" found! Initializing...
Initialized TensorFlow Lite runtime.
INFO: Initialized TensorFlow Lite runtime.
13:27:21.231: [info] 📸 VisionCamera.didSetProps(_:): Updating 22 props: [onInitialized, cameraId, position, enableBufferCompression, onOutputOrientationChanged, onStarted, preview, top, onCodeScanned, right, isActive, isMirrored, onViewReady, onError, onStopped, onPreviewOrientationChanged, onPreviewStopped, enableFrameProcessor, onPreviewStarted, left, bottom, onShutter]
13:27:21.236: [info] 📸 VisionCamera.configurePreviewOrientation(_:): Updating Preview rotation: landscapeLeft...
13:27:21.237: [info] 📸 VisionCamera.configureOutputOrientation(_:): Updating Outputs rotation: landscapeLeft...
13:27:21.237: [info] 📸 VisionCamera.configure(_:): configure { ... }: Waiting for lock...
13:27:21.246: [info] 📸 VisionCamera.configure(_:): configure { ... }: Updating CameraSession Configuration... Difference(inputChanged: true, outputsChanged: true, videoStabilizationChanged: true, orientationChanged: true, formatChanged: true, sidePropsChanged: true, torchChanged: true, zoomChanged: true, exposureChanged: true, audioSessionChanged: true, locationChanged: true)
13:27:21.246: [info] 📸 VisionCamera.configureDevice(configuration:): Configuring Input Device...
13:27:21.246: [info] 📸 VisionCamera.configureDevice(configuration:): Configuring Camera com.apple.avfoundation.avcapturedevice.built-in_video:0...
13:27:21.261: [debug] 📸 VisionCamera.sensorOrientation: Sensor Orientation changed from landscapeLeft -> portrait
13:27:21.261: [info] 📸 VisionCamera.configurePreviewOrientation(_:): Updating Preview rotation: portrait...
13:27:21.261: [info] 📸 VisionCamera.configureOutputOrientation(_:): Updating Outputs rotation: portrait...
13:27:21.261: [info] 📸 VisionCamera.configureDevice(configuration:): Successfully configured Input Device!
13:27:21.261: [info] 📸 VisionCamera.configureOutputs(configuration:): Configuring Outputs...
13:27:21.261: [info] 📸 VisionCamera.configureOutputs(configuration:): Adding Video Data output...
13:27:21.263: [info] 📸 VisionCamera.configurePreviewOrientation(_:): Updating Preview rotation: portrait...
13:27:21.263: [info] 📸 VisionCamera.configureOutputOrientation(_:): Updating Outputs rotation: portrait...
13:27:21.263: [info] 📸 VisionCamera.configureOutputs(configuration:): Successfully configured all outputs!
13:27:21.265: [info] 📸 VisionCamera.setTargetOutputOrientation(_:): Setting target output orientation from device to device...
13:27:21.266: [info] 📸 VisionCamera.getPixelFormat(for:): Available Pixel Formats: ["420v", "420f", "BGRA", "&8v0", "&8f0", "&BGA"], finding best match... (pixelFormat="yuv", enableHdr={false}, enableBufferCompression={false})
13:27:21.266: [info] 📸 VisionCamera.getPixelFormat(for:): Using PixelFormat: 420f...
13:27:21.606: [info] 📸 VisionCamera.init(frame:session:): Preview Layer started previewing.
13:27:21.609: [info] 📸 VisionCamera.configure(_:): Beginning AudioSession configuration...
13:27:21.609: [info] 📸 VisionCamera.configureAudioSession(configuration:): Configuring Audio Session...
13:27:21.609: [info] 📸 VisionCamera.configure(_:): Beginning Location Output configuration...
13:27:21.611: [info] 📸 VisionCamera.configure(_:): Committed AudioSession configuration!
13:27:21.624: [info] 📸 VisionCamera.configure(_:): Finished Location Output configuration!

Camera Device

{
  "id": "com.apple.avfoundation.avcapturedevice.built-in_video:0",
  "maxZoom": 120,
  "formats": [],
  "position": "back",
  "hardwareLevel": "full",
  "hasTorch": true,
  "minZoom": 1,
  "name": "Back Camera",
  "hasFlash": true,
  "minExposure": -8,
  "sensorOrientation": "portrait",
  "physicalDevices": [
    "wide-angle-camera"
  ],
  "neutralZoom": 1,
  "supportsRawCapture": false,
  "supportsFocus": true,
  "isMultiCam": false,
  "supportsLowLightBoost": false,
  "maxExposure": 8,
  "minFocusDistance": 15
}

Device

iPhone 12 Pro Max (IOS 17.5.1)

VisionCamera Version

4.4.1

Can you reproduce this issue in the VisionCamera Example app?

Yes, I can reproduce the same issue in the Example app here

Additional information

I am using Expo
I have enabled Frame Processors (react-native-worklets-core)
I have read the Troubleshooting Guide
I agree to follow this project's Code of Conduct
I searched for similar issues in this repository and found none.

maintenance-hans · 2024-07-05T11:42:17Z

Guten Tag, Hans here.

Note

New features, bugfixes, updates and other improvements are all handled mostly by @mrousavy in his free time.
To support @mrousavy, please consider 💖 sponsoring him on GitHub 💖.
Sponsored issues will be prioritized.

christophemenager · 2024-07-05T13:52:00Z

I do have the same issue, only on iOS

Marius-Adam · 2024-07-06T17:05:04Z

I do have the same issue, only on iOS

IOS for me too

emilje · 2024-07-13T23:06:12Z

Same, iPhone 11.

frabanca · 2024-07-15T12:20:26Z

Same issue on iPhone 14 Pro and "react-native-vision-camera": "^4.5.0".

Marius-Adam · 2024-07-15T12:38:36Z

Blocking use of any frame processor with text recognition @mrousavy

mrousavy · 2024-07-15T14:31:22Z

Blocking use of any frame processor with text recognition @mrousavy
Please stop pinging me, I am not free 24h consultancy

The problem is with a Frame Processor Plugin you are using, not with my library.

I explained how Frame.orientation works in #3077, this is now finally the last change to the Frame.orientation prop and should give the plugin developer all they need to make it work.

In the case of react-native-vision-camera-text-recognition, he needs to update this switch statement with the correct values now. For example 0 is .up (as in the documentation) - not .right. With the current code it wouldn't have worked in selfie cameras or iPads.

If a Frame Processor Plugin wants to actually support rotation, they just need to add a parameter to their plugin where you can pass the current outputOrientation of the Camera.

cc @gev2002

frabanca · 2024-07-15T14:43:10Z

Hi @mrousavy , thanks for your reply and your time!

As often happens, your comments are very useful and allow us to understand the situation and in some cases manage it.

For example, your previous comment was very useful for me, because although I use another engine for OCR (this one: https://www.npmjs.com/package/@ismaelmoreiraa/vision-camera-ocr), it It was easy to find the file to edit in the plugin source (this file https://github.com/gev2002/react-native-vision-camera-text-recognition/blob/758041042d6608368fe2a41c38e3368c04f6b9c0/ios/VisionCameraTextRecognition.swift#L146-L151 a line 142) and insert the .right value instead of the .up value.

This small change - which I'm still testing - seems to be enough to solve the problem.

Thanks again for your time.

mrousavy · 2024-07-15T14:44:46Z

Thank you.

Yea the problem again was that Frame.orientation was already adjusted to output orientation before - and if you did that, it was impossible to reverse it for Skia based FPs. So we now just provide the raw value, and users can rotate it if they want to. Otherwise it only works in portrait :)

frabanca · 2024-07-15T14:46:44Z

Seems a good idea to give the plugin the raw value.
Maybe - to prevent incompatibility - you can pass both so you can allow legacy plugin to work like before.

mrousavy · 2024-07-15T16:30:55Z

Well it's both the same name unfortunately.

I could rename it to something like bufferOrientation and keep the old one (orientation)? I'll think about it, but to be fair, no frame processor plugin truly supported orientation before anyways.

frabanca · 2024-07-15T17:19:25Z

Considering how easy is to "fix" the plugin(s) code, maybe it's better to leave the property name "orientation" and let:

Plugin maintainers update their code
Plugin utilizers update/fix the code manually (as I did in less than 3 mins after your anware)

Marius-Adam added the 🐛 bug Something isn't working label Jul 5, 2024

maintenance-hans bot changed the title ~~🐛 Frame orientation is in landscape-right when phone is held in portrait mode~~ Jul 5, 2024

Marius-Adam mentioned this issue Jul 5, 2024

🐛 Orientation have different values on front and back camera #3049

Closed

5 tasks

mrousavy closed this as completed Jul 15, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🐛 Frame orientation is in landscape-right when phone is held in portrait mode #3051

🐛 Frame orientation is in landscape-right when phone is held in portrait mode #3051

Marius-Adam commented Jul 5, 2024 •

edited

Loading

maintenance-hans bot commented Jul 5, 2024 •

edited

Loading

christophemenager commented Jul 5, 2024

Marius-Adam commented Jul 6, 2024

emilje commented Jul 13, 2024

frabanca commented Jul 15, 2024

Marius-Adam commented Jul 15, 2024

mrousavy commented Jul 15, 2024

frabanca commented Jul 15, 2024

mrousavy commented Jul 15, 2024

frabanca commented Jul 15, 2024

mrousavy commented Jul 15, 2024

frabanca commented Jul 15, 2024

🐛 Frame orientation is in landscape-right when phone is held in portrait mode #3051

🐛 Frame orientation is in landscape-right when phone is held in portrait mode #3051

Comments

Marius-Adam commented Jul 5, 2024 • edited Loading

What's happening?

Reproduceable Code

Relevant log output

Camera Device

Device

VisionCamera Version

Can you reproduce this issue in the VisionCamera Example app?

Additional information

maintenance-hans bot commented Jul 5, 2024 • edited Loading

christophemenager commented Jul 5, 2024

Marius-Adam commented Jul 6, 2024

emilje commented Jul 13, 2024

frabanca commented Jul 15, 2024

Marius-Adam commented Jul 15, 2024

mrousavy commented Jul 15, 2024

frabanca commented Jul 15, 2024

mrousavy commented Jul 15, 2024

frabanca commented Jul 15, 2024

mrousavy commented Jul 15, 2024

frabanca commented Jul 15, 2024

Marius-Adam commented Jul 5, 2024 •

edited

Loading

maintenance-hans bot commented Jul 5, 2024 •

edited

Loading