Data Lake on Shaowen Chen's Websitehttps://www.chenshaowen.com/en/tags/data-lake/Recent content in Data Lake on Shaowen Chen's WebsiteHugo -- gohugo.ioen©2016 - {year}, All Rights Reserved.Thu, 12 Sep 2024 00:00:00 +0000weeklyProcessing Data on Kubernetes with Iceberg and Sparkhttps://www.chenshaowen.com/en/blog/use-iceberg-and-spark-on-kubernetes.htmlThu, 12 Sep 2024 00:00:00 +0000Thu, 12 Sep 2024 00:00:00 +0000https://www.chenshaowen.com/en/blog/use-iceberg-and-spark-on-kubernetes.html1. Data Processing Architecture It is mainly divided into four layers: Processing capability layer: Spark on Kubernetes provides streaming data processing capability Data management layer: Iceberg provides dataset access operations such as ACID and tables Storage layer: Hive MetaStore manages Iceberg table metadata, PostgreSQL serves as the storage backend for微信公众号SparkIcebergKubernetesBig DataData LakeOperationsData ProcessingLearning