Visual automation not working on macOS with Retina display - run this command

[HKailen]

Hello,

thanks for your contribution and I'm a beginner to RPA.

I tried to automate the process of opening a 'sample.png' file but it's not working. It shows [RPA][ERROR] - cannot find example.png. I have cd to Desktop and the 'example.png' file is also located at Desktop.

I have checked my OpenJDK and Phantomjs and they are both working. (*tested by using tagui > flow > samples > 1_google.tag) I have also tried using Python rpa to automate website and its working.

I have also checked that java and terminal are both ticked in my security > accessibility setting.

Looking forward to your answer.

Thank you in advance.

[HKailen]

BTW I'm using MacOS Catalina

[Conutant]

Same here, did you ever find a solution?

[HKailen]

@Conutant Nope still no solution

[frank3ling]

Same here. :(

[ck81]

@HKailen, @Conutant and @frank3ling,

1. Can you please first try the sample codes in: https://yuchenkuang.com/TagUI-Python-Setup/setup-tagui-python-on-mac-with-visual-automation.php#step4

2. Make sure that the sample code works first. This will allow you to know if Visual Automation is set up correctly on your Mac.

Once (1) and (2) above is ok, then please try this. The reason your sample.png didn't work is because if you are using MacBook Pro, it has a retina display and when you do screen capture, the image is captured at a much higher resolution.

Please give the following a try. Hopefully you have a second monitor that is a standard monitor and not the Apple monitor with Retina Display. Capture your screen in the second monitor. The image will be captured at the standard size. Note: if you are trying to capture an Application Icon on your Desktop, then duplicate the screen instead of extending the screen. In this way you can capture the Application Icons on your Desktop on the second monitor.

Now try to see if this image works withTagUI. You can use r.dclick('sample.png') or r.hover('sample.png')

If it works, you will see the mouse automatically move over to the icon. dlick() will do a double click on the image. hover() will just move the mouse over the image.

It should work now.

If you don't have a second monitor, hopefully you have Photoshop. Try to scale the image from 144 pixels/inch to 72 pixels/inch, which is what Sikuli uses.

You can find out more information about this here: aisingapore/TagUI#240

Please give the above a try and let me know if it works, ok?

[HKailen]

@ck81 @kensoh ,

thanks, @ck81 for the solution. It's working fine for me. Here is how I work:

1. Connect to one additional monitor (NOT from apple monitor)
2. Drag the file which I want to be automatically opened with RPA to the additional monitor.
3. Make a screenshot (CMD + Shift + 4) of the file here at this additional monitor.
4. Rename the screenshot file (works with arbitrary name).
5. Drag both of the files back to the main monitor of my MacBook pro.
6. Run the RPA script.

An alternative way that I found which works fine too without additional monitor.

1. Make a screenshot of the file.
2. Rename the screenshot file (works with arbitrary name).
3. Double Click on the screenshot file and reset the ppi from 144 to 72. (Werkzeuge -> Größenkorrektur -> Auflösung)
   https://www.maclife.de/ratgeber/einfach-passen-bildgroesse-vorschau-app-10094145.html
   *sorry for the german here but the steps of resetting the ppi of png could be found easily on google I guess
4. Run the RPA script.

So it turns out to be the ppi problem.

Once again, thanks for the solution.

[frank3ling]

@ck81 @kensoh ,

thanks, @ck81 for the solution. It's working fine for me. Here is how I work:

1. Connect to one additional monitor (NOT from apple monitor)
2. Drag the file which I want to be automatically opened with RPA to the additional monitor.
3. Make a screenshot (CMD + Shift + 4) of the file here at this additional monitor.
4. Rename the screenshot file (works with arbitrary name).
5. Drag both of the files back to the main monitor of my MacBook pro.
6. Run the RPA script.

An alternative way that I found which works fine too without additional monitor.

1. Make a screenshot of the file.
2. Rename the screenshot file (works with arbitrary name).
3. Double Click on the screenshot file and reset the ppi from 144 to 72. (Werkzeuge -> Größenkorrektur -> Auflösung)
   https://www.maclife.de/ratgeber/einfach-passen-bildgroesse-vorschau-app-10094145.html
   *sorry for the german here but the steps of resetting the ppi of png could be found easily on google I guess
4. Run the RPA script.

So it turns out to be the ppi problem.

Once again, thanks for the solution.

Nice and easy workaround. Thanks.

[kensoh]

Hi RPA for Python users, 🅗🅐🅟🅟🅨 ②⓪②①!

It has been many months since I last worked on this personal side project. Just want to shout out that this project is very much alive, and I hope to look into the 30+ issues raised since July (and bug-fixes if required) soon when I get a moment again.

Below are details if you are interested, why I'm away past few months and why I believe my darkest days are over 🍀🤞🏻🙏🏻

---

Since May last year, I've stopped all work to juggle with my mum's critical illness, taking care of my toddler, and doing housework chores like cleaning and cooking. TBH, my wife and I felt that taking care of a baby and giving her the best nutrition (nursing and home-cooked organic food) and care requires at least 1.5 full-time headcount. There is no life to speak of, other than trying to stay afloat with baby care needs, and keep finding alternative ways to treat my mum, where there're some wins.

Eg importing soy-based tube feeds from Netherlands / Germany / Australia --> after switching away from milk-based tube feeds for 2 weeks she stops requiring dialysis. She started requiring 3x a week dialysis for more than 3 months, after admitting to the hospital. Also, applying herbs like mashed garlic or diced onions to her feet somehow consistently improve her breathing and lung function. Have to keep trying alternative healing methods when her recovery keeps hitting plateau in hospital.

She has been discharged from the hospital a month ago and recovering at home, though very slowly. A longterm medication seems to be one of the contributing causes for her condition. After stopping it, there seems to be improvement to her breathing capacity. She is still bed-bound, but over the course of the year I hope she'll regain strength to move about, and hopefully lungs to recover sufficiently to not require oxygen tank support through a tracheal mask (a hole through her throat).

---

Since November, I've gone back to work at AI Singapore, a government-funded programme to accelerate AI in Singapore. I'm with Product Engineering team, focusing on RPA and TagUI (open-source RPA tool), which RPA for Python is a wrapper for. There are pretty interesting stuffs on TagUI 2021 roadmap. 2020 was the year which my family was going to move to the UK and work, after I receive its tier 1 exceptional tech talent visa. Then everything changed with the virus outbreak, and my mum's illness. The best thing that happened to me in 2020 was being able to go back to work at AI Singapore - I'm grateful.

I've just hired someone to help with housework, cooking and supporting my wife with baby care. I hope this new arrangement will allow me to deliver on my work at AI Singapore, and soon be able to reply countless friends' messages which circumstances leave me with no choice but go missing for months, and then look into the issues raised here!

Eg of the cost of best-in-class nutrition standard researched and set by wife. Baby just felt restless and broke her plate. Even Corelle's break-resistant plate breaks. Why not use plastic plate? Cuz plastic can leech toxins to the food on it. Why not use Silicon plate? Cuz there is no white silicon materials for plate. Why white? Cuz that helps to improve visibility of the food instead of contrasting against some pink colour or some other colours. I would be spending much time clearing up the mess of 70 over pieces of plate fragments, while baby gets nursed to calm down, if not for our newfound helper.

That's why I feel like my darkest days are over and now seeing glimpses of light at the end of the tunnel.

---

My vision for RPA for Python remains - I would like it to be the go-to Python package for RPA.

A best-in-class tool in Python users toolbox which you combine with other cool packages to build amazing stuffs.

[kensoh]

this issue is also encountered by AIDP folks

There is an issue for Mac with Retina displays, the macOS screenshot tool captures at double pixels.

Reply from SikuliX's creator to confirm - https://answers.launchpad.net/sikuli/+question/667091

Solution using the Automator tool in macOS -
https://apple.stackexchange.com/questions/105185/how-can-i-stop-my-retina-display-from-taking-2x-sized-screenshots

Manual solution using image preview -
https://macissues.com/2014/03/29/adjust-retina-macbook-screenshots-to-be-actual-size/

[kensoh]

Adding on, this RPA package is based on a forked version of TagUI open-source RPA tool.

Free feel to join Telegram group chat for TagUI community to post any questions - https://t.me/rpa_chat

[kensoh]

Oh found a good solution from a newer option in SikuliX! For mac computers with Retina display, the images captured are at double resolution. Running below in your Python script will auto-scale your supplied images down to the actual resolution to do visual automation. I don't have Retina display, but have tried and tested successfully with a fake double-sized image.

r.vision('Settings.AlwaysResize = 0.5')