Databricks Cluster "file not found" when accessing Volumes

Question

I am using a Databricks Shared Compute Cluster (DBR 15.2) to connect to an Oracle DB using the JDBC Driver. For the connection, I need to specify the trust store location. My preferred option would be to have the trust store in a Volume. I get an error stating the the file can not be found even though it is there.

This is my current code to connect to Oracle:

url = "jdbc:oracle:thin:@(DESCRIPTION=(ADDRESS=(PROTOCOL=TCPS)(PORT=xxxx)(HOST=xxxx))(CONNECT_DATA=(SID=xxx)))"

df = (
    spark.read.format("jdbc")
    .option("url", url)
    .option("dbTable", "schema.table")
    .option("user", "user")
    .option("password", dbutils.secrets.get(scope = "oracle", key = "password"))
    .option("driver", "oracle.jdbc.driver.OracleDriver")
    .option("fetchsize", 2000)
    .option("javax.net.ssl.trustStoreType", "JKS")
    .option("javax.net.ssl.trustStore", "/Volumes/test_catalog/test_schema/test_volume/oracle_truststore.jks")
    .option("javax.net.ssl.trustStorePassword", dbutils.secrets.get(scope = "oracle", key = "truststore_pw"))
    .load()
)

I get the following error:

(java.sql.SQLRecoverableException) IO Error: IO Error PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target, Authentication lapse 0 ms.

I can see the trust store when I use ls:

%sh
ls /Volumes/test_catalog/test_schema/test_volume
oracle_truststore.jks

I am the Owner of the file and have the needed permissions.

Does anyone know, what I am doing wrong or what might work?

As a workaround, I have stored the trust store in dbfs. But this solution only works with Personal Compute Clusters as the dbfs is not available on Shared Compute Cluster. Anyway I would prefer not to have the file in the dbfs as I cannot control the access.

I have tried different combinations with the following outcome. It is the same file in the dbfs and in the Volume.

x	personal compute	shared compute
dbfs	works	Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
Volumes	java.nio.file.FileSystemException: /Volumes/test_catalog/test_schema/test_volume/oracle_truststore.jks: Operation not permitted	Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

Answer 1

What is the Oracle JDBC driver version that you are using? You should be using the keystone as well. Can you add the below properties? Check out this blog for more details.

#FOLLOW THESE STEPS FOR USING JKS
#(1) Uncomment the following properties to use JKS.
#(2) Set the correct password for both trustStorePassword and keyStorePassword.
#It's the password you specified when downloading the wallet from OCI Console or the Service Console.

#javax.net.ssl.trustStore=${TNS_ADMIN}/truststore.jks
#javax.net.ssl.trustStorePassword=<password_from_console>
#javax.net.ssl.keyStore=${TNS_ADMIN}/keystore.jks
#javax.net.ssl.keyStorePassword=<password_from_console>
~                             

Answer 2

Usually the "PKIX path building failed" error means that the certificate from the database isn't signed by any authority found in the trust store file.

It's a different error from what you'd see if JDBC can't find the trust store file. In that case, the error would look like this:

java.sql.SQLException: ORA-17958: Unable to initialize the trust store.
... 
Caused by: java.nio.file.NoSuchFileException

For the PKIX path building error, I'd recommend double checking that the trust store does in fact contain the certificate of an authority which has signed the certificate from the database.

One way to do this is by running Java with -Djavax.net.debug=all. This will have a ton of debug output printed to stderr. There's a full guide on how to read the output here: https://docs.oracle.com/javase/8/docs/technotes/guides/security/jsse/ReadDebug.html

In your case, I'd check for a couple things in particular:

• Look for a message like "trustStore is: ...". You should be seeing the path to the file you've configured.
• Look for a message like "adding as trusted certificates (...". This will show you what certificates are found in the trust store file.
• Look for a message like "Consuming server Certificate handshake message (...". This will show you what certificates JDBC is receiving from the database.

You can then compare the certificates found in the trust store with those received from the database. If they're not matching, then the issue is likely one of the following:

1. Your JDBC URL is not correct. It is locating a database which presents a certificate that does not match what's in your trust store.
2. Or, your trust store is not correct. You are connecting to the right database, but the trust store doesn't contain a certificate authority which signed the certificate of that database.

Another possibility is that there is some man-in-the-middle attack happening here, and the trust store is doing it's job to alert you about that. But in my experience, the most common cause of this PKIX path building error is a trust store file not being in sync with the database.

I hope this helps.