All Questions
Tagged with snowflake-cloud-data-platform apache-spark
123
questions
0
votes
3
answers
44
views
How to connect Snowflake with PySpark with Google Colab?
I am trying to connect to Snowflake with Pyspark on Google Colab.
Spark version 3.4
Scala version 2.12.17
from pyspark.sql import SparkSession
from pyspark.sql.functions import *
from pyspark import ...
- 23
0
votes
0
answers
33
views
Snowflake setting of sql variables using pyspark
I am trying to read a view in snowflake using spark.read.format("snowflake").options(**options).option("query","Set date='03-02-2018';Select * from View123").load()
This ...
- 1
0
votes
1
answer
226
views
How to overwrite a single partition in Snowflake when using Spark connector
Is there a way for Spark to read a single date partition from a Snowflake table, update it and then to overwrite this single date partition. Concurrent writes should be supported. Currently Spark has ...
- 65
0
votes
0
answers
19
views
Spark connectors from Azure Databricks to Snowflake using AzureAD login [duplicate]
I am trying to connect below Spark connectors from Databricks to Snowflake. On the Snowflake side, I can access the account using "Login via AzureAD" which are my Azure account credentials.
...
- 195
0
votes
0
answers
73
views
Not able to run COPY INTO commands during spark write in Snowflake
GET @spark_connector_load_stage_jtVp1DVrvm/ file:///tmp/dummy_location_spark_connector_tmp/
PUT file:///tmp/dummy_location_spark_connector_tmp/ @spark_connector_load_stage_jtVp1DVrvm
copy into <...
- 46
0
votes
0
answers
184
views
Upserting data into Snowflake?
I have a use case where I need programmatically copy data from one or more JSON files into a Snowflake table. However, when I copy the files, I would also need to do an upsert of records. For example, ...
- 404
1
vote
1
answer
92
views
junk(Null) char appending to Actual snowflake table data
I am performing some couple of transformations before writing pyspark dataframe to snowflakes table. I have written Snowflake write in below way.
df.spark.format("snoflake").opton(**...
- 109
0
votes
0
answers
191
views
Py4JJavaError while trying to get data from Snowflake into Databricks
I am trying to get data from a Snowflake table into a Databricks spark dataframe. Here is my current code.
from pyspark.sql import SparkSession
from pyspark.sql.functions import *
from pyspark.sql....
- 118
1
vote
1
answer
934
views
Read Data from Snowflake through Snowpark and then insertion into folder on local machine with .csv format using Python
Here is code I have to query from snowflake but I am not able to write the pulled data into csv on local machine folder. I am getting
snowflake.snowpark.dataframe has no attribute csv.
snowspark = ...
- 79
0
votes
0
answers
253
views
Capture error or rejected records while loading data from spark df to snowflake table
I have written a spark code which writes the dataframe into snowflake table.
Code Snippet:
val data = Seq(("1", "product12345678", "20000"), ("2", "...
- 84
1
vote
0
answers
162
views
What's the easiest way to insert a Spark dataframe into Snowflake without duplicate rows?
I need to create a Spark dataframe and add the values to a Snowflake table. It works, the only problem is I occasionally get duplicate entries.
How can I avoid duplicates?
# create DF that will be ...
- 1,241
0
votes
0
answers
263
views
Spark query pushdown is not working as expected for join
I am writing a Spark program to fetch data from Snowflake. The program contains 1 simple SQL(join 2 tables, having <50000 records) as below.
val TABB = spark.sql("SELECT COL1, COL2 FROM XYZ ...
- 1,195
0
votes
1
answer
607
views
PySpark / Snowpark calculate running sum between two given dates
Using this sample table:
id
sales
sales_date
1
10
2020-04-30
1
6
2020-10-31
1
9
2020-09-30
1
2
2021-04-30
2
8
2020-08-31
2
7
2020-07-31
2
3
2021-06-30
2
2
2021-05-31
I would like to ...
- 663
0
votes
1
answer
526
views
PySpark / Snowpark random column name during left anti join problem
I am trying to compare two dataframes, to get new records to be inserted into an incremental table.
I am following previously asked questions, example How to compare two dataframes and extract ...
- 663
-1
votes
2
answers
498
views
Snowpark table creation failing even query executed successfully
I am trying to create a table from other table in snowpark using parameterized value
CREATE OR REPLACE PROCEDURE table_merge(DB varchar,SCHEMA varchar,TARGET_TABLE varchar, SRC_TABLE varchar)
...
- 115