spark表缓存

2022年6月19日01:31:53 发表评论 387 views

实测可以从1s降低到76ms

scala> val df1 = spark.read.json("/testdata/emp.json") //需要上传到hdfs
df1: org.apache.spark.sql.DataFrame = [comm: string, deptno: bigint ... 6 more fields]

scala> df1.registerTempTable("emp")
warning: there was one deprecation warning; re-run with -deprecation for details

scala> spark.sql("select * from emp").show

//标记缓存某张表
scala> spark.sqlContext.cacheTable("emp")

//清理所有的缓存
scala> spark.sqlContext.clearCache
//清理某张表
scala> spark.sqlContext.uncacheTable("emp")

scala> spark.sql("select * from emp").show

发表评论

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen: