Questions tagged [hadoop]
Hadoop is an Apache open-source project that provides software for reliable and scalable distributed computing. The core consists of a distributed file system (HDFS) and a resource manager (YARN). Various other open-source projects, such as Apache Hive use Apache Hadoop as persistence layer.
hadoop
44,421
questions
0
votes
0
answers
8
views
How to use API and API key on Python
I am trying to implement API keys for alpha vantage, Bloomberg and newsapi to load data into hadoop using spark
ALPHA_VANTAGE_API_URL = "https://www.alphavantage.co/query?function=...
0
votes
0
answers
6
views
How to understand the result of yarn queue myqueue
When I run the following command to see the status of my queue:
$ yarn queue -status my-queue
Queue Information :
Queue Name : my-queue
State : RUNNING
Capacity : 10.0%
...
- 6,184
0
votes
0
answers
10
views
Apache oozie JA008 error - job state changed from SUCCEDED to FAILED
I'm running oozie HA 5.2.1 on EMR and I have an issue with this temporary directory. I have a workflow which has start node -> action node -> end node. The job start running -> runs for 10-15 ...
- 65
0
votes
0
answers
8
views
Apache Ranger Yarn Plugin Installation: Class Loading and Logging Configuration Problems
Class Loading Issue with Commons Logging: I have confirmed the presence of commons-logging*.jar in the directory /usr/bigtop/3.2.0/usr/lib/hadoop-yarn/share/hadoop/hdfs/lib/. Despite this, I am ...
- 1
0
votes
0
answers
16
views
The jmx data obtained from Flume is empty. How to get the correct result？
i set flume-env.sh JAVA_OPTS= -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=5445 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false.
i ...
- 43
-2
votes
0
answers
28
views
How to mount HDFS on ubuntu? [closed]
I am unable to mount HDFS on ubuntu 22.04 what are the ways to do that? I tried these commands
sudo apt-get install sshfs
sudo mkdir -p /mnt/hdfs
sudo sshfs user@localhost:/ /mnt/hdfs
the above is ...
0
votes
0
answers
7
views
AWS EMR - reading multiple "zip" files from S3 bucket returns Your key is too long
In my daily job I use EMR to process large amount of data. This data are stored in CSV files on S3 bucket. The idea I had was to try to process ziped csv files instead of plain csv.
In Hive app I use ...
- 131
-1
votes
0
answers
19
views
Hive sql repeating records with amount differing only by +-5%?
How to delete repeating records with amount differing only by 5%? by amount in the range amout-5% to amount+5%?
Here is an example table on hive
[# a.uk b.uk AMOUNT START_DATE FINISHED_DATE source
1 ...
0
votes
0
answers
15
views
How does Yarn shut down containers?
ApplicationMaster can send a request to NodeManager to close the container. At the same time, ApplicationMaster can send a heartbeat to ResourceManager to report the container to be closed.
...
- 1
0
votes
0
answers
13
views
How Ambari manages Apache hadoop
Ambari is an open source project, but HDP is no longer available for download, so how does Ambari manage components such as Apache Hadoop, Hive, and Spark? Can Ambari directly manage existing Hadoop ...
- 1
-1
votes
1
answer
43
views
Getting java.lang.UnsatisfiedLinkError when trying to run my Code
i am trying to follw this tutorial https://dzone.com/articles/databricks-delta-lake-using-java
So i want to use spark with delta lake, but for my storage i use minio and not hdfs or s3 (But it is ...
- 51
0
votes
0
answers
23
views
How to setup HDFS with Geoserver?
I want to test the performance of HDFS with geospatial data. So I setup HDFS but unable to fetch the data from HDFS and give it to Geoserver. Is there any way to fetch the data from HDFS and give it ...
0
votes
0
answers
9
views
HDFS: how to touch recursively
We can use -R to list files recursively. HDFS: How do you list files recursively?
However, there is no -R option in hadoop fs -touch command.
When using the pipe hadoop fs -ls -R <path> | xargs ...
- 1,003
0
votes
0
answers
15
views
Alternative to storage OCR Information Mongo, Hadoop [closed]
I need to develop a web software to receive mainly PDF documents and extract information using OCR. I need to save the original document and the OCR extraction for later searches. I have already ...
- 1
0
votes
0
answers
17
views
What is the solution - Hadoop yarn is not working [closed]
I installed Hadoop to my laptop on linux but when I opened localhost:8088, yarn wasn't worked. I don't understand about this problem and I couldn't looking for anything information about this problem.
- 1