Usage

This section goes over how to manipulate an encrypted DataFrame in either client or insecure mode.

Saving a DataFrame

Save the encrypted DataFrame to local disk. The encrypted data can then be uploaded to cloud storage of your choice for easy access.

Scala:

dfEncrypted.write.format("edu.berkeley.cs.rise.opaque.EncryptedSource").save("dfEncrypted")
// The file dfEncrypted/part-00000 now contains encrypted data

Python:

df_encrypted.write.format("edu.berkeley.cs.rise.opaque.EncryptedSource").save("df_encrypted")

Using the DataFrame interface

1. Users can load the previously persisted encrypted DataFrame.

   Scala:

   import org.apache.spark.sql.types._
   val dfEncrypted = (spark.read.format("edu.berkeley.cs.rise.opaque.EncryptedSource")
   .schema(StructType(Seq(StructField("word", StringType), StructField("count", IntegerType))))
   .load("dfEncrypted"))
   

   Python:

   df_encrypted = spark.read.format("edu.berkeley.cs.rise.opaque.EncryptedSource").load("df_encrypted")
   

2. Given an encrypted DataFrame, construct a new query. Users can use explain to see the generated query plan.

   Scala:

   val result = dfEncrypted.filter($"count" > lit(3))
   result.explain(true)
   // [...]
   // == Optimized Logical Plan ==
   // EncryptedFilter (count#6 > 3)
   // +- EncryptedLocalRelation [word#5, count#6]
   // [...]
   

   Python:

   result = df_encrypted.filter(df_encrypted["count"] > 3)
   result.explain(True)
   

Using the SQL interface

1. Users can also load the previously persisted encrypted DataFrame using the SQL interface.

   spark.sql(s"""
     |CREATE TEMPORARY VIEW dfEncrypted
     |USING edu.berkeley.cs.rise.opaque.EncryptedSource
     |OPTIONS (
     |  path "dfEncrypted"
     |)""".stripMargin)
   

2. The SQL API can be used to run the same query on the loaded data.

   val result = spark.sql(s"""
     |SELECT * FROM dfEncrypted
     |WHERE count > 3""".stripMargin)
   result.show