說明：withColumn用于在原有DF新增一列

1. 初始化sqlContext

val sqlContext = new org.apache.spark.sql.SQLContext(sc)?

2.導入sqlContext隱式轉換

import sqlContext.implicits._?

3. ?創建DataFrames

?

val df = sqlContext.read.json("file:///usr/local/spark-2.3.0/examples/src/main/resources/people.json")

4. 顯示內容

df.show() ??

| age| ? name|?+----+-------+|null|Michael|| ?30| ? Andy|| ?19| Justin|

5. 為原有df新加一列

df.withColumn("id2", monotonically_increasing_id()+1)?

6. 顯示添加列后的內容

?res6.show()?

+----+-------+---+| age| ? name|id2|+----+-------+---+|null|Michael| ?1|| ?30| ? Andy| ?2|| ?19| Justin| ?3|+----+-------+---+

?

完成的過程如下：

scala> val sqlContext = new org.apache.spark.sql.SQLContext(sc)?warning: there was one deprecation warning; re-run with -deprecation for detailssqlContext: org.apache.spark.sql.SQLContext = org.apache.spark.sql.SQLContext@2513155ascala> import sqlContext.implicits._import sqlContext.implicits._scala> val df = sqlContext.read.json("file:///usr/local/spark-2.3.0/examples/src/main/resources/people.json")2018-06-25 18:55:30 WARN ?ObjectStore:6666 - Version information not found in metastore. hive.metastore.schema.verification is not enabled so recording the schema version 1.2.02018-06-25 18:55:30 WARN ?ObjectStore:568 - Failed to get database default, returning NoSuchObjectException2018-06-25 18:55:32 WARN ?ObjectStore:568 - Failed to get database global_temp, returning NoSuchObjectExceptiondf: org.apache.spark.sql.DataFrame = [age: bigint, name: string]scala> df.show()+----+-------+| age| ? name|+----+-------+|null|Michael|| ?30| ? Andy|| ?19| Justin|+----+-------+

?

scala> df.withColumn("id2", monotonically_increasing_id()+1)res6: org.apache.spark.sql.DataFrame = [age: bigint, name: string ... 1 more field]scala> res6.show()+----+-------+---+| age| ? name|id2|+----+-------+---+|null|Michael| ?1|| ?30| ? Andy| ?2|| ?19| Justin| ?3|+----+-------+---+

?

轉載于:https://www.cnblogs.com/abcdwxc/p/9225855.html

總結

以上是生活随笔為你收集整理的spark dataFrame withColumn的全部內容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網站內容還不錯，歡迎將生活随笔推薦給好友。