All Questions
Tagged with apache-zookeeper apache-spark
67
questions
0
votes
0
answers
54
views
The Hbase docker container is not accessible under version 2.4.4 but works with same setup under 2.2.6
There is a testing environment including a dockerised HBase unit which was developed previously and works fine on a Linux machine but doesn't work on Mac locally.
In particular, after running the ...
1
vote
0
answers
58
views
Spark job stuck at running process
I use docker-compose to make a cluster of Spark and Kafka. When I tried to consume a topic by submitting it to this cluster, the job was always stuck and not executed.
Here is my docker-compose file:
...
0
votes
1
answer
96
views
Is it a good idea to have masters and workers running on the same node in a small Spark cluster for HA?
I need to set up an Apache Spark cluster in Standalone mode for a production environment. For this task, I only have three available nodes. Each node has:
128GB RAM
24 CPU cores
There is no example ...
0
votes
0
answers
59
views
Configure ACL on spark with zookeeper
I have configured spark to use zookeeper as RECOVERYMODE for high availability. We need to enable ACLs on zookeeper with authentication. I have also configured spark to pass zookeeper client ...
- 1,182
1
vote
0
answers
332
views
Zookeeper Leader Auto Failover
We are using spark standalone cluster 3 zookeepers in HA mode.I am seeing this issue in zookeeper.log.
Exception causing close of session 0x0 due to java.io.IOException: Len error 1195725856
Closed ...
- 11
0
votes
0
answers
124
views
Spark not receiving streams from Kafka?
Iam trying to stream logs/topic from the kafka to the pyspark script.
I have used Docker Spark cluster to submit this job
I have done following configuration so far
# Used 2182 for both zookeeper and ...
- 15
1
vote
3
answers
9k
views
Topic doesn't exist as expected in kafka
After running this command :
bin/kafka-topics.bat --list --bootstrap-server localhost:6666, localhost:6667, localhost:6668, localhost:6669
Output - Test
After running this command :
bin/kafka-topics....
- 371
0
votes
0
answers
2k
views
ConnectionLoss for /hbase/meta-region-server
We encountered an error when run spark-submit command on AWS EMR cluster.
Our spark job previously was running on HDP cluster without any additional configuration in a scala code. When we start the ...
6
votes
0
answers
2k
views
HDP3.1.4 - Spark2 with Hive Warehouse Connector error using spark-submit and pyspark shell: KeeperErrorCode = ConnectionLoss
Environment:
HDP 3.1.4 - configured and tested Hive server 2 - tested and working
Hive server 2 LLAP - tested and working Spark configured as per documentation to use Hive Warehouse Connector (HWC)
...
- 61
0
votes
1
answer
5k
views
failed for get of /hbase/hbaseid, code = CONNECTIONLOSS, retries = 6
I am trying to connect spark application with hbase. Below is the configuration I am giving
val conf = HBaseConfiguration.create()
conf.set("hbase.master", "localhost:16010")
conf.setInt("timeout", ...
- 433
0
votes
1
answer
96
views
Spark submit failed due to exception in ZooKeeperPersistenceEngine
Spark submit fails due to an exception caused in ZooKeeperPersistenceEngine.
Tested the jar deployment in my local environment however when I try to push it to the cluster it fails.
Spark: 2.4.0
...
- 500
0
votes
1
answer
42
views
Prevent spark executors from accessing Mesos or ZooKeeper apis
Currently Spark on Mesos is run in cluster mode, Mesos/ZooKeeper doesn't have any access control, and executors are started on host network.
What would be the best practice for preventing spark ...
- 623
0
votes
1
answer
215
views
Does Spark Kafka streaming 0.8 indirect stream KafkaUtils.createStream updateWaterMark save offsets to Zookeeper?
I am bound to use
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-streaming-kafka-0-8_2.11</artifactId>
<version>${...
- 3,713
1
vote
1
answer
305
views
Why does Apache Spark Master shutdown when Zookeeper expires the session
As I understand, Apache Spark Master can be run in high availability mode using Zookeeper. That is, multiple Spark masters can run in Leader/Follower mode and these modes are registered with ZooKeeper....
- 369