bug: Error creating EyeWitness screenshots with a non-standard URL

[solo10010]

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

The task of creating screenshots stops completely if EyeWitness encounters a strange URL

python3 /usr/src/github/EyeWitness/Python/EyeWitness.py -f /usr/src/scan_results/**********.com_9/endpoints_alive.txt -d /usr/src/scan_results/* *********.com_9/screenshots --no-prompt --timeout 10 --threads 40

Scan logs.

TERM environment variable not set.
################################################## ###############################
#EyeWitness#
################################################## ###############################
# Red Siege Information Security - https://www.redsiege.com #
################################################## ###############################

ERROR: You potentially provided an mal-formed URL!
ERROR: URL is - https://account.servicenow.com/sign-in?client_id=0oapi9ote73XCWSst0x7&redirect_uri=https://account.servicenow.com/auth&response_type=code&state=/profile/home?locale=en-us&scope=openid&source_id= account&locale=en-us

Expected Behavior

I expected that if errors occurred, the collection of screenshots would continue and not be interrupted completely.

Steps To Reproduce

1. Install the latest version of rengine
2. Set up any scan with collection of subdomains and creation of screenshots for the servicenow.com domain
3. Monitor scanning tasks, screenshot creation has been stopped on the account.servicenow.com subdomain

Environment

- reNgine: 2.0.2
- OS: Ubuntu 22.04.1 LTS.
- Python: 3.10.4
- Docker engine: 24.0.7
- Docker Compose: v2.23.0
- Browser: Chrome 120.0.6099.224.

Anything else?

No response

[github-actions]

👋 Hi @solo10010,
Issues is only for reporting a bug/feature request. Please read documentation before raising an issue https://rengine.wiki
For very limited support, questions, and discussions, please join reNgine Discord channel: https://discord.gg/azv6fzhNCE
Please include all the requested and relevant information when opening a bug report. Improper reports will be closed without any response.

[AnonymousWP]

Why didn't you fill in the template regarding versions. We can't help you if you didn't fill it correctly.

[solo10010]

I'm sorry

• rengin: 2.0.2
• OS: Ubuntu 22.04.1 LTS.
• Python: 3.10.4
• Docker engine: 24.0.7
• Docker Compose: v2.23.0
• Browser: Chrome 120.0.6099.224.

So far I have found such a way out that the scanning of subdomains will be completed, run Further Scan Subdomain on all subdomains - screenshot

[yogeshojha]

@solo10010 The URL is malformed.

if you check the get parameters source_id= account&locale=en-us space shouldn't be there in URLs. The problem does not lie in Eyewitness itself.

[yogeshojha]

@solo10010 this will be fixed in #1308

I have added validators before storing them in file. this will prevent storing any junk http urls in a file.