How to achieve partial update on list of column

Help Connectors and Pipelines
1

Hi,

I have a dataframe which is having 10 columns.Out of 10 ,i want 5 columns values to be update in memsql table in one go.

I am aware about the script which can update single column value at time as i mentioned below.But i want to update 5 column at a time.
df.write
.format(“memsql”) .option(“onDuplicateKeySQL”,updateColumn=VALUES(updateColumn)).mode(SaveMode.Append)
.save(dbname.tablename)

i am using memsql version 7.1 and spark connector 3

Could you please suggest if any such option is available in spark-memsql connector.

Thanks.

2

Thank you for reaching out. Can something like this work for your use case? Here we’ve introduced 5 update columns (uc1-uc5). Let me know if you have any questions.

df.write.format(“memsql”)
.option(“onDuplicateKeySQL”, "uc1=VALUES(uc1), uc2=VALUES(uc2), uc3=VALUES(uc3), uc4=VALUES(uc4), uc5=VALUES(uc5)")
.mode(SaveMode.Append)
.save(dbname.tablename)
2 Likes
3

Thanks Roxanna for the Solution.
It is working as expected.

1 Like