Questions tagged [databricks-sql]
Questions about Databricks SQL
databricks-sql
500
questions
1
vote
2
answers
80
views
Unable to InferSchema with Databricks SQL
When I attempt to create a table with Databricks SQL I get the error:
AnalysisException: Unable to infer schema for CSV. It must be specified manually.
%sql
CREATE TABLE IF NOT EXISTS newtabletable ...
- 2,603
1
vote
0
answers
22
views
Stata ODBC - Why does declaring a temporary function in my query yield an error?
I'm trying to load data into Stata from a SQL database hosted in Databricks, via the odbc load command. For part of the query I need to declare and run a small Python function (Databricks SQL, or at ...
- 210
0
votes
0
answers
14
views
How to calculate date diff in the below scenario using databricks sql?
Input -
I want to derive a column based on below steps -
Check for same ServiceId and Name but Ref is different.
If so, then calculate day diff like below -
(take ActualEndTime from 1st row and ...
- 186
-1
votes
1
answer
30
views
Will CTE work with an INSERT in databricks / spark sql?
I don't have a databricks environment yet to run the queries and test stuff.
Could someone please help check if the below is a valid query :
insert into pqr.public.abc
with city_mapping as
(
select ...
- 159
0
votes
0
answers
30
views
Adding new columns to a Delta Live table in a CDC process
I am new to databricks and still learning.
I am trying to do a CDC on a table.
APPLY CHANGES INTO LIVE.table1
FROM schema2.table2
KEYS (Id)
SEQUENCE BY orderByColumn
COLUMNS * EXCEPT (col1, col2)
...
- 89
0
votes
1
answer
46
views
how to set call back for Databricks Statement Execution SQL API Query?
I m using https://docs.databricks.com/api/workspace/statementexecution.
using long running queries.my wait time is zero.
queries might take 1 hour and I don't want to do pooling
https://docs....
- 165
0
votes
1
answer
19
views
extract the value after a specific text that occurs multiple times
lets say I have a table
id
name
1
[{"name":"Herman"},{"name":"Iwan"]
and I would like to extract all the value.
the expected result should be in 2 rows,
id
...
- 1
0
votes
1
answer
71
views
UNSUPPORTED_FEATURE.SET_OPERATION_ON_MAP_TYPE The feature is not supported: Cannot have MAP type columns which calls set operations (INTERSECT, E
How can I fix the following error? I am getting error when running query in DataBrick:
[UNSUPPORTED_FEATURE.SET_OPERATION_ON_MAP_TYPE] The feature is not supported: Cannot have MAP type columns
in ...
-1
votes
1
answer
134
views
Databricks: managed tables vs. external tables
managed tables are fully managed by the Databricks workspace, where Databricks handles the storage and metadata of the table, including the lifecycle of the data. When you create or delete a managed ...
- 2,028
0
votes
1
answer
52
views
How to convert oracle regex pattern into Databricks compatible version?
Given this Oracle version with the goal to convert it so it is usable in Databricks SQL:
WITH pattern AS (
SELECT
/*+ INLINE */
'^([A-Z][A-Z0-9])([A-Z]\d{3,4})([T])?([A-D])?([W]\d)?([A-RT-Z][...
- 183
0
votes
1
answer
79
views
How to read system.information_schema.columns in databricks
How can I read the system table system.information_schema.columns, it take about 3 hours to read and only has 3 million rows, is there any hint or like in Oracle with(nolock) to read this table ...
0
votes
0
answers
30
views
Issue with BinaryType Data When Writing from S3 to Redshift Using Databricks API in Apache Spark
I encountered an issue while writing data from Amazon S3 to Amazon Redshift using Databricks API in Apache Spark. The issue arises due to a field with a BinaryType in the schema, which causes an ...
0
votes
1
answer
63
views
Databricks Azure SQL Merge - issue with % sign in column name
When I am using databricks sql merge to update target table I have issues with columns which contains: "%" percentage sign
Merge:
(
target_table.alias("trg")
.merge(...
- 43
0
votes
1
answer
83
views
Dynamic Transpose of Rows to Columns in Databricks SQL
I just have started to work with Databricks SQL.
Here I have a table that contains for each article a list of attributes and values following this structure:
Article Attribute Value
A Name Bike ...
- 331
0
votes
1
answer
20
views
Materialised View vs physical table
I'm really getting confused as why to use a materialised view instead of a normal table. if it is because of convinience of writing a complex query, precomputing and storing then we can do the same ...
- 35