Big Data

August 6-8, 2021

Online

Back to Event Home
 

Track Chairs : Duo zhang, Junping Du, Lidong Dai

Big Data is leading and changing various industries and is inseparable from our lives. Big Data is also a very important part of ASF. ASF has so many big data projects, such as Hadoop, Hive, Spark, HBase, Kylin, Ozone, CarbonData, Doris, Cassandra, etc. In this topic, you will learn the cutting-edge trends of these technologies and the practical experience, principles, architecture analysis and other exciting content from first-line users

2021-08-06 || ROOM : A

13:30 GMT+8

Scaling Impala - Common Mistakes and Best Practices

English Session

Apache Impala is a complex engine and requires a thorough technical understanding to utilize it fully. Without proper configuration or usage Impala’s performance becomes unpredictable, and end user experience suffers.

For many users/administrators the right configuration of Impala is still a mystery. During our work with some of the largest clusters in the world we’ve found a set of common mistakes in configuration and usage that lead to a lot of frustration.

In this talk, we will discuss ingestion best practices to keep an Impala deployment scalable and admission control configuration to provide consistent experience to end users. We will also take a high-level look at Impala’s query profile which is used as a first stop in any performance troubleshooting. In addition, we will discuss common mistakes users and BI tools make when interacting with Impala. Lastly, we will go over an ideal setup to show all of this in practice. Speakers:

Manish Maheshwari: I have 15+ years of experience building extremely large data warehouses and analytical solutions. He’s worked extensively on Apache Hadoop, DI, and BI tools, data mining and forecasting, data modeling, master and metadata management, and dashboard tools. He is proficient in Hadoop, SAS, R, Informatica, Teradata, and Qlikview.

Manish Maheshwari

14:10 GMT+8

How Security is implemented in Apache Ozone

English Session

Bharat Viswanadham, Shashikant Banerjee

Apache Ozone is a scalable, redundant, and distributed object store for Hadoop, which became Apache’s Top-level project in 2020. Apache Ozone has two metadata services one is Storage Container Manager(SCM) which manages the allocation of blocks/containers and replication, certificates, and node management and the other one is OzoneManager which manages metadata. In this talk, we shall discuss how security is implemented in ozone. From this talk, the audience would learn:

  • How security functions across Apache ozone
  • What security mechanisms are used across ozone components.
  • Security model for support of multiple protocols/endpoints, like S3, RPC, HCFS, etc
  • How security is handled with a CA distributing certificate to other components.
  • Storage Container Manager (SCM) HA security architecture with root CA and Sub CA.
  • OzoneManager HA Security and how tokens are validated.
  • How BlockTokens are verified on the datanode.

Speakers:

Bharat Viswanadham: Software engineering professional with 7+ years of experience in designing and building scalable and high performant distributed storage systems. I am currently, a committer and PMC member of Apache Hadoop and Apache Ozone.

Shashikant Banerjee: Software engineering professional with 8+ years of experience in designing and building scalable and high performant distributed storage systems. I am currently, a committer and PMC member of Apache Hadoop, Apache Ozone and Apache Ratis community.

14:50 GMT+8

OpenLooKeng heuristic index framework architecture analysis and application practice

English Session

Zheng Li

With the application and development of big data technologies, there are more types of data, more data distribution, and more complex query scenarios. This makes it difficult or impossible to process data easily. To improve the usability of big data, Huawei initiated the open source project openLooKeng, the data virtualization engine. openLooKeng uses the industry-renowned open source SQL engine Presto to provide basic interactive query and analysis capabilities, and continues to develop in cross-data center/cloud, data source expansion, performance, reliability, security etc., making big data simplified. This talk will focus on the openLooKeng heuristic index framework, then introduce severial indexing technologies based on the framework and its implementation and application challenges. Speakers:

Zheng Li: Zheng li got his Ph.D. from Wuhan Optoelectronics National Laboratory of Huazhong University of Science and Technology. He joined Huawei in June 2018. He is currently focusing on performance optimization research of openLooKeng, deeply involved in the design and implementation of big data query analysis engine architecture and other related work.

15:30 GMT+8

Data Analysis across Disparate Data Sources in CMB

English Session

Qiumin Wu

China Merchants Bank(CMB) has petabytes of data stored in RDBMS, NoSQL databases, object storage, big data frameworks – Apache Hadoop, Spark, Flink and so on. It is costly to transfer data from different data sources by ETL method. Hence, openLookeng is introduced to connect disparate data sources, and process data locally across datacenters and hybrid clouds.

This talk will provide an overview of the data processing engine in CMB which enables in-situ analytics on geographically remote data sources. And how we use the features of openLookeng, like high availability, auto-scaling, built-in caching and indexing support, etc. to meet enterprise workload demand with required reliability. Speakers:

Qiumin Wu: Qiumin Wu has 9-year big data experience in Fin-tech area, and is responsible for architectural design, implementation and maintenance of the big data platform in China Merchants Bank (CMB). Wu is also a PMC member of openLookeng.

16:10 GMT+8

Java-based machine learning solutions for big data

Chinese Session

Qing Lan

The success of Machine Learning (ML) application depends on utilizing the big data. Most of the big data is available in unstructured format. The availability of big data can be also offline and online. Although there are options available for ML tasks in python, Integration between Python application into the existing Java/Scala based Big Data pipeline are quite challenging. Apart from that, there are very few options in Java/Scala to bridge the gap of processing the big data and using the same library for ML workload.

To solve the issues above, we will demo a solution for Big Data ML in Java with DJL, a Machine Learning framework in Java. DJL offers a variety of ML engines, including TensorFlow, PyTorch, Apache MXNet (incubating). PaddlePaddle, ONNXRuntime and a lot more. Through using Apache Flink and Apache Spark, user can easily build their online/offline ML pipeline. At the end of the session, audience will be able to build an easy-to-use, high performance ML pipeline for all different scenarios. Speakers:

Qing Lan: Qing is a SDE in AWS Machine Learning Platform. He is one of the co-authors of DJL (djl.ai) and PPMC member of Apache MXNet. He graduated from Columbia University in 2017 with a MS degree in Computer Engineering. He holds the expertise in model training and inference.

16:50 GMT+8

Apache HUDI on AWS

Chinese Session

Lianghong Fei

An introduction to Apache Hudi on AWS and covers Apache Hudi introduction, Common Use Cases ,Hudi Storage Types,Writing Hudi DataSets, Querying Hudi DataSets and some of tips. Speakers:

Lianghong Fei: Lianghong Fei is a Principal developer advocate at Amazon Web Services, where he leverages his 20 years of experience to support innovation and to help startups and companies bring their ideas to life. Lianghong focuses on software development and cloud native architectures as well as the technical and business impact of Machine Learning and data analytics. Prior to joining AWS, he worked at Apple and Microsoft. Some of his interests include , artificial intelligence, data science and photography.

2021-08-07 || ROOM : A

13:30 GMT+8

analyzing transactional data in Apache druid

English Session

Vijay Narayanan

Apache druid is a platform for realtime analytics. Handling transactional data that changes around a constant primary key is challenging in druid as druid does not support row level updates. This session will focus on handling such data in druid and using druid to do both traditional OLAP analytics on such data and user behaviour/funnel analytics.

  • OLAP analytics on transactional data - two approaches can be thought off to do this (this usually focusses on queries like “How many transactions that were created last month completed?"). One is to use druid functions to retrieve latest/earliest information in time from such data. the second approach is to use kafka to hold current time information for transactions and use that as a lookup in druid.
  • Funnel analytics - This is used for user behaviour analytics and focusses on answering questions like “How many users completed all 4 steps in the sign up process?”. Storing events in time for transactional data allows such analytics to be done in druid. Some approaches will be laid out both for handling funnels where sequence is not important and those where sequence is important

Speakers:

Vijay Narayanan: Vijay has about 15+ years of experience in the data world. Vijay is currently a field engineer with Imply (Imply offers commercial enterprise support for open source druid). In this role Vijay is focussed on helping customers in APAC use the Imply platform (based on Apache druid). Before Imply Vijay was with cloudera for two years helping cloudera partners position and use the cloudera platform. Before cloudera Vijay spent 10 years with Informatica where he was part of the team that put together connectivity for informatica cloud.

14:10 GMT+8

OminiRuntime: A comon big data runtime framework

English Session

Jingfang Zhang

We analyze the data processing processes of several mainstream big data computing engines, identify some common problems, and use a unified solution to solve these optimization problems. The solution is OmniRuntime. OmniRuntime uses the JIT technology to dynamically generate high-performance operators. Enable vectorization execution with columnar memory data; Data can be exchanged efficiently. Reduce the network bandwidth between storage and computing by using the storage-computing collaboration technology. Speakers:

Jingfang Zhang: OpenLooKeng developer with 9 years of experience in big data development. Have been responsible for the design and development of massive storage active-active projects and deeply participated in the kernel design and development of the OpenLooKeng computing engine, involving the design and development of OmniRuntime. Cross Datacenter query technology exploration, and data transfer optimization work.

14:50 GMT+8

Apache Ozone: A High Performance Object Store for analytics workloads

English Session

Rakesh Radhakrishnan, Mukul Kumar Singh

Apache Ozone is a robust, distributed key-value object store for Hadoop with layered architecture and strong consistency. It provides Object Store semantics (like Amazon S3) and can handle billions of objects. Apache Ozone object store recently implemented a fast atomic rename and delete operation with O(1) complexity. This dramatic optimization lower the job latency equals lower total cost of ownership (TCO) for analytics workloads. As we know, most big data analytics tools like Apache Hive, Apache Spark, etc. often write output to temporary locations and then rename the location at the end of the job to become publicly visible. In the analysis of Object Store(like Amazon S3), rename is not a native Object Store operation, it is implemented using a costly copy and a delete operation. The rename operations can often take longer than the analytics process itself. These job committers demand atomic rename for improved performance as well as consistent listing operations. This talk will be a deep dive into the Apache Ozone architecture that describes the atomic rename and delete implementation, which greatly boost the analytics job performance. We will walk through performance benchmark results that show a consistent performance gain in various analytics workloads. Finally, we will also talk about a future roadmap to leverage this new design to achieve efficient lock management for namespace operations by avoiding global locks. Speakers:

Rakesh Radhakrishnan: Rakesh Radhakrishnan is a committer and a PMC in Apache Hadoop, Apache ZooKeeper, Apache BookKeeper projects and primarily focusing on open source big data technologies. Rakesh is currently working at Cloudera and actively contributing on the Apache Ozone project. He has more than 14 years of experience in large scale Distributed Software Platforms design and development. Prior to joining Cloudera, he worked as a Big Data Software Engineer in Intel Corporation.

Mukul Kumar Singh: Mukul is currently working with Cloudera where he is leading the Storage team working on both Apache Ozone and Apache HDFS. He has also been working on Storage Systems and File Systems for 12 years and has played various roles as open source contributor, Apache PMC member, researcher and a software developer.He also has worked with Nimble Storage and NetApp and worked on WAFL and CASL file systems respectively. He graduated from Carnegie Mellon University, where his thesis was on a file system for Shingled Magnetic recording disks.

15:30 GMT+8

Apache Atlas meets Apache Flink

English Session

Josh Yeh, Yan Liu

Apache Atlas has become the one of the rock star project for metadata management,where it can handle from data lineage to data tagging and terms. Apache Flink has also become the standard of streaming processing, while Apache Flink is powerful at processing data at scale, tracking the lineage became an problem for Apache Flink.

In this session, I would like to share the recent community progress on connecting Apache Atlas and Apache Flink, and how the community can benefit from tracking the Apache Flink application’s metadata. Speakers:

Josh Yeh: Cloudera Software Engineer currently working on streaming workflow governance with Apache Flink and Apache Atlas. Previous projects included developing Machine Learning Operations (MLOPS) focusing on Model lineage and Model Metrics on Cloudera multi-tenant SAAS platform and on-premise product Cloudera Data Science Workbench (CDSW), building data pipeline/workload automation with ML/DL/AI framework: keras, pytorch, tensorflow, CDSW Nvidia GPU support, and Cloudera Manager HDFS and Hive Backup and Disaster Restore (BDR).

Yan Liu: Been as a Solution Engineer at Cloudera for 5 years, I helped lots of customer on on the successful of adopting the Apache projects running in product

16:10 GMT+8

Apache Liminal (Incubating)—Orchestrate the Machine Learning Pipeline

English Session

Aviem Zur, Assaf Pinhasi

Apache Liminal (Incubating)—Orchestrate the Machine Learning Pipeline Apache Liminal is an end-to-end platform for data engineers & scientists, allowing them to build, train and deploy machine learning models in a robust and agile way. The platform provides the abstractions and declarative capabilities for data extraction & feature engineering followed by model training and serving; using standard tools and libraries (e.g. Airflow, K8S, Spark, scikit-learn, etc.). Apache Liminal’s goal is to operationalise the machine learning process, allowing data scientists to quickly transition from a successful experiment to an automated pipeline of model training, validation, deployment and inference in production, freeing them from engineering and non-functional tasks, and allowing them to focus on machine learning code and artifacts. Speakers:

Aviem Zur: Data tech Lead @ Natural Intelligence, PPMC Member, Apache Liminal, PMC Member, Apache Beam. Specializing in data frameworks and platforms as Well as open source software. Passionate about quality engineering, open source and Magic: The Gathering.

Assaf Pinhasi: Technology leader, highly experienced in building large scale systems and teams, specializing in Big Data and Machine Learning.

16:50 GMT+8

Bigtop 3.0: Rerising community driven Hadoop distribution

English Session

Kengo Seki, Masatake Iwasaki

Apache Bigtop provides comprehensive packaging, testing, and configuration of the leading open source big data components including, but not limited to, Hadoop, HBase and Spark. While Hadoop ecosystem provides rich toolset for processing big data, there is no clear picture of compatibility between them. Bigtop makes introduction easy by providing combination of products working as a whole.

We will introduce Bigtop 3.0.0, which is the first release based on Hadoop 3. Supported Linux distributions including Ubuntu, Debian, CentOS and Fedora are also updated. We will also talk about why it is hard to migrate compatibility issues and dependency conflicts between products, referring to on-going efforts in the ecosystem. Speakers:

Kengo Seki: Committer and PMC chair of Apache Bigtop. Committer and PMC member of Apache Airflow and Apache Yetus. Committer of Apache Avro and Apache Thrift.

Masatake Iwasaki: Committer and PMC member of Apache Hadoop and Apache Bigtop.

2021-08-08 || ROOM : A

13:30 GMT+8

Technical tips for secure Apache Hadoop cluster

English Session

Akira Ajisaka, Kei KORI

It is well known that Apache Hadoop is not secure by default, and it is easy to impersonate a privileged user if Kerberos is not enabled. In this talk, we will introduce further technical details about security. We will talk about the current “open-source” software capability for some security features in a vendor-neutral manner. We mainly introduce SSL/TLS for Hadoop and its related products (Apache Spark, Apache Hive, Apache ZooKeeper, etc.) and introduce HDFS data encryption. Regarding HDFS data encryption, we implemented Hadoop Credential Provider API to integrate Hadoop KMS with our internal credential store to store the secrets for encrypting/decrypting data encryption key. We also introduce the internals of HDFS data encryption and how to create your custom credential providers.

In addition, we will introduce the automatic keystore reloading, which is the latest feature of Apache Hadoop, and it enables updating SSL certificates without stopping the server. The feature is useful for operation. Speakers:

Akira Ajisaka: Akira Ajisaka is a senior software engineer at Yahoo Japan Corporation. He develops and validates some new features of Apache Hadoop for our use. Also, he troubleshoots and improves management/operations in our Hadoop clusters. He maintenances Apache Hadoop to improve its quality as an Apache Hadoop/Yetus committer and PMC member.

Kei KORI: Kei KORI is a data platform engineer on the Hadoop development team and Kubernetes team at Yahoo Japan Corporation, where he works to deliver secure Hadoop clusters and their user ecosystem. He has over years of experience in the administration of Hadoop clusters and Kubernetes clusters with expertise in automation and continuous delivery of distributed systems.

14:10 GMT+8

Data Lake accelerator on Hadoop-COS in Tencent Cloud

English Session

Li Cheng

New game-changer from Tencent Cloud, Data Lake accelerator GooseFS on Hadoop-COS Ever since COS, as Tencent Cloud Object Storage Solution, submitted Hadoop Capatible FS plugic Hadoop-COS to Hadoop community in 2019, Tencent Cloud has gone full speed to support Data Lake on Tencent COS. In 2021, Tencent COS weaponizes Hadoop-COS with multi-layer accelerator GooseFS, which not only greatly enhances COS performance in Hadoop eco-system, but also make Tencent COS more cohesive with big data and AI platforms. From the talk, audience would learn about:

  • How GooseFS helps the cohesion with EMR and K8s on Tencent Cloud platforms.
  • Translucent acceleration of IO performance brought by new Hadoop-COS.
  • How GooseFS loads namespace level cache and table level cache.
  • Data Lake solutions from Tencent Cloud Storage brought by new Hadoop-COS.

GooseFS in the new Hadoop-COS is a distributed cache solution, which can be natively deployed with EMR and K8s. GooseFS supports cache options as memory, SSD and HDD and multiple cache policies. GooseFS catalog also supports cache load table level Cache like Hive tables by table id and table partitions. Furthermore, GooseFS on Hadoop-COS would support Iceberg tables and load by versions. There are many more exciting features Tencent Cloud are building up to expand the Data Lake realm. Speakers:

Li Cheng: Current Senior Engineer who owns Big Data Storage in Tencent Cloud COS. Formerly works for AWS S3 and Huawei Storage team. Also active in open source community. Currently Apache Ozone PMC and Hadoop Committer.

14:50 GMT+8

Apache Kylin 4.0 : an architectural upgrade and a new path to tuning

Chinese Session

张智超

Apache Kylin 4.0 is a major architectural upgrade from Kylin 3, which replaces HBase storage with Parquet, a true columnar storage, to improve file scanning performance. In terms of build and query, the Spark-based build engine and query engine have been re-implemented, eliminating many of the drawbacks of previous versions and making it possible to separate compute and storage. For the majority of Kylin users, with the architecture upgrade, the tuning idea will also change from the original MR + HBase optimization to Spark + Parquet optimization, so this speaker will introduce the new tuning path of Apache Kylin 4.0 from several aspects, such as architecture upgrade, build tuning and query tuning. This session will introduce the new tuning path of Apache Kylin 4.0, helping users to quickly master the new Kylin 4.0. Speakers:

Zhichao Zhang: Apache Kylin Committer && Apache CarbonData Committer and PMC, working in Kyligence, mainly responsible for Apache Kylin development, optimization and promotion.

15:30 GMT+8

Big Data Format at Uber Data Infra

English Session

Xinli Shang, Pavi Subenderan

Big data format played an important role in data analytics for data storage efficiency, query performance, and security. In this talk, we will present our work in big data file format, Apache Parquet, at Uber for reducing storage size by ~20% to save millions of dollars, and encrypting columns to provide fine-grained access control. We will show the work we contributed to open source Parquet including ZSTD compression, high throughput column pruner, column size estimator, column encryption management, etc. On the query performance side, we will demonstrate our work related to Column Index, which is a new feature in Parquet 1.11.0.

Not only big data file format but also in-memory format, i.e. Apache Arrow, is one of the areas we look into. Use Apache Arrow as a cache layer would vastly improve the performance. We will show our vision at Uber for big data format. Speakers:

Xinli Shang: Xinli Shang is a tech lead on the Uber Data Infra team, Apache Parquet PMC Chair. He is passionate about big data file format for efficiency, performance, and security, tuning large-scale services for performance, throughput, and reliability. He is an active contributor to Apache Parquet. He also has many years of experience developing large-scale distributed systems like S3 Index, and the operating system Windows.

Pavi Subenderan: Pavi is a Software Engineer on Uber’s Data Infra team. His focus is on data security, privacy and open source big data technologies. He has been working on Parquet column encryption for 1.5 years and more recently on data masking.

Jianchun Xu: Jianchun is a Senior Software Engineer on Uber’s Data Infra team. He has mainly worked on big data infra and parquet since joined the data team about a year ago. He also had extensive experience in service deployment platform, developer tools and web/JavaScript engine.

16:10 GMT+8

Milvus: A Vector Database for Unstructured Data Processing

English Session

Xiaofan Luan

This presentation will talk about the increasing demands and challenges for unstructured data processing in in the AI era and how vector database comes to rescue. As the world’s most popular open source vector database, Milvus has been adopted by over 1000 organizations. We will present the technical insides about Milvus and also the forth coming execiting features and innovations in Milvus 2.0. We will also give examples and demos on how Milvus is helping various corporation with applications in computer vision, NLP, recommendation systems and AI drug discovery. Speakers:

Xiaofan luan, partner and director of engineering at Zilliz, member of Technical Advisory Committee of LF AI & data foundation. He worked successively in the US headquarters of Oracle and Hedvig, a software defined storage startup. He joined Alibaba cloud database team and was in charge of the development of NoSQL database HBase and Lindorm. Xiaofan Luan received his master degree in electronic computer engineering from Cornell University.

16:50 GMT+8

Apache Arrow based DataFrame for Data Processing in Python

English Session

Supun Kamburugamuve

Machine learning (ML) and deep learning (DL) fields have made amazing progress in the past few years. Modern ML/DL applications have outgrown resource requirements beyond a single node’s capability. However, this is just a small part of the overall data processing environment, which must also support a raft of data engineering for pre-and post-data processing, communication, and system integration. The data tools surrounding the ML/DL applications need to be able to easily integrate with existing ML/DL frameworks in a multitude of languages, which particularly increases user productivity and efficiency. All this demands an efficient and highly distributed integrated approach for data processing, yet many of today’s popular data analytics tools are unable to satisfy all these requirements at the same time.

This presentation introduces Cylon DataFrame, which is an open-source high-performance distributed Python API similar to Pandas. It is developed with a flexible C++ core on top of the Apache Arrow data format. The presentation discusses Cylon’s architecture and how it works at scale. Initial experiments show that Cylon outperforms popular tools such as Apache Spark and Dask with major performance improvements for key operations with the potential to integrate with them. Finally, we show how Cylon can enable high-performance data pre-processing in popular AI tools such as Pytorch, Tensorflow, and Jupyter notebook without taking away Data scientists’ productivity. Speakers:

Supun Kamburugamuve: Supun Kamburugamuve has a Ph.D. in computer science specializing in high-performance data analytics. He is working in the role of a principal software engineer at the Digital Science Center of Indiana University where he leads Twister2 and Cylon high-performance data analytics projects. Supun is an elected member of the Apache Software Foundation and has contributed to many open-source projects including Apache Web Services projects and Apache Heron. Before joining Indiana University, Supun worked on middleware systems and was a key member of the WSO2 ESB project, an open-source enterprise integration solution being widely used by enterprises. Supun has given many technical talks at research conferences and technical conferences including Strata NY, Big Data Conference, and Apache Con.

2021-08-06 || ROOM : B

13:30 GMT+8

How a DBS[Development Bank of Singapore] Data Platform Drives Real-time Insights & Analytics using Apache CarbonData

English Session

Ravindra Pesala, Kumar Vishal

DBS is a leading bank headquartered in Singapore.

As a major banking institution, DBS generates terabytes of structured and unstructured data that can be used to get the data insights and create strategies for the bank.

DBS decided in the year 2020 to invest in a CarbonData based data platform to drive real-time analytics and unlock insights on the available data from a variety of sources.

In this talk we’ll first explain why and how DBS embarked from a classical data warehouse to a CarbonData based Data-lake using Spark, Presto engines.

We’ll outline the series of use-cases and highlight for each use-case the advantages of CarbonData and how the team further tuned performance to truly deliver insights with continuous flow of streaming data and batch data. Speakers:

Ravindra Pesala: SVP, Big Data Platform Lead. Apache CarbonData PMC. Leading the Big Data Engineering Platform including ingestion, compute, data access, streaming and metadata

Kumar Vishal: AVP, Senior Big Data Engineer Apache CarbonData PMC Handling Big Data Engineering Platform including ingestion, compute, data access, streaming

14:10 GMT+8

Challenges of Building a Distributed Fault-Tolerant Scalable Analytics Stack

English Session

Nishant Bangarwa

As of today, the largest Apache druid cluster holds more than 50+ Trillion events amounting to over 500 petabytes of raw data and continuously ingesting new data streams growing at a never-before rate. There are many technical challenges, design decisions, cost vs performance tradeoffs, limiting factors that we faced on our journey in evolving druid to make it scale to handle petabytes of data without sacrificing performance.

In this talk, we will discuss general requirements and key challenges that anyone designing a production-ready scalable analytics stack is expected to encounter on their way due to various constraints. We will also discuss the learning and strategies we have developed over time and our path in evolving Apache Druid as a powerful distributed fault-tolerant scalable Analytics data store.

We hope the strategies we discuss in this session will help anyone struggling to keep up with the growing demands of data analytics. Speakers:

Nishant Bangarwa: Nishant is Co-founder and Head of Engineering at Rilldata. He is an active open-source contributor and a PMC member of Apache Druid and Apache Superset. He is also a committer in Apache Calcite and Apache Hive. Before starting Rilldata, he was part of Cloudera’s Data warehouse team and Metamarkets Druid team where was responsible for managing large-scale Apache Druid deployments. He holds a B.Tech in Computer Science from the National Institute of Technology, Kurukshetra, India.

14:50 GMT+8

Kyuubi: NetEase's Exploration and Practical Application of Serverless Spark Scenarios

Chinese Session

Kent Yao （姚琴）

This topic mainly covers the architecture, implementation principles and application scenarios of NetEase’s open source big data component Kyuubi project, and demonstrates Kyuubi’s ability to help businesses implement Serverless Spark within NetEase and the corresponding process and thinking through practical cases. Cases.

Implementing seamless migration from Hive QL to Spark SQL, achieving an overall 6~8x performance improvement
Mixed deployment of YARN and K8S clusters to achieve smooth business migration, elasticity to the cloud and converged scheduling requirements

At the same time, we will introduce how we are directly involved in the Spark open source community and complete the corresponding problem solving and feature optimization in the process." Speakers:

Kent Yao: Primary Author of Kyuubi Project / Apache Spark Committer / Apache Submarine Committer. @NetEase Bigdata Team

15:30 GMT+8

Inside Apache Druid's Storage and Query Engine

English Session

Gian Merlino

Apache Druid is an open-source columnar database known for high performance at scale; its largest deployments comprise thousands of servers. But no matter the scale, high performance starts with good fundamentals. This talk will dive into those fundamentals by exploring the inner workings of a single data server. We’ll cover how Apache Druid stores data, what kinds of compression it uses, how it indexes data, how the storage engine is linked with the query processing engine, and how the system handles resource management and multithreading. Together, all these pieces enable Apache Druid to process billions of records per second on a single data server. Speakers:

Gian Merlino: Gian is a co-founder and CTO of Imply. Gian is also one of the main committers of Druid. Previously, Gian led the data ingestion team at Metamarkets and held senior engineering positions at Yahoo. He holds a B.S. in Computer Science from Caltech.

16:10 GMT+8

Faster Bigdata Analytics by maneuvering Apache CarbonData’s Indexes

English Session

Akash R Nilugal, Kunal Kapoor

Data in the 21st Century is like Oil in the 18th Century: an immensely, untapped valuable asset if processed in an intelligent way. Storage and Analysis of Big Data can be challenging and expensive both in terms of cost and time. Analytical solutions need to keep adapting themselves to keep up with the challenge of exponential data growth rate. Apache CarbonData is a unified storage solution + File Format which aims to optimize the query performance thus decreasing the analytical cost. Apache CarbonData has been adapted by 100+ open source users. In databases, one of the major features is indexes, which basically helps to query without having to scan every row. Taking inspiration from the same concept Apache CarbonData supports custom indexes like min/max, bloom, Lucene, Secondary Index, and Materialized Views for faster row level updates, deletes, OLAP and point queries. This presentation emphasizes on CarbonData’s custom index architecture + Distributed Index Cache Server which helps to provide faster query results, the future challenges and scope. Speakers:

Akash R Nilugal: I am Apache carbondata PMC and Committer and Working as Senior Technical Lead at Cloud and AI/data platform team of Banglore Research center, Huawei. I have been working on Bigdata and mainly Apache carbondata for 5 years now and have worked and interested in areas like index support on bigdata, Materialized Views, CDC on bigdata, Spark SQL query optimizations, Spark structured streaming, data lake and data warehouse functionality, trino.cCurrently I am working on Carbondata CDC.

Kunal Kapoor: I am Apache carbondata PMC and Committer and Working as System Architect at Cloud and AI/data platform team of Banglore Research center, Huawei working on Bigdata technologies like Apache carbondata, Apache spark, Apache hive for 5 years now. Some of the major features include distributed index cache server, Hive + Carbondata integration, Pre-aggregation support, S3 support for carbondata, Secondary index on carbondata, Spark SQL query optimization in carbondata.

16:50 GMT+8

Insight into the secret of Open Source community —— the best practise for data-driven community operations

Chinese Session

Jun Zhong, Yikun Jiang, Lei Peng

The open source community is gradually becoming one of the most critical path in the software ecosystem. In the evaluating process of open source comunity, data-driven insight and analysis of the current state of the community is very meaningful for helping community grow healthy. Therefore, the data-driven operations plays a key role in community.

In this topic, we will introduce our best practices for data-driven community operations. This openrations management system helps several of the most active open source communities in China (such as openEuler, openGauss, MindSpore, etc.) to efficiently and scientifically measure the health, activity and other key metrics of the community.

This topic includes:

  • Why we need a data-driven operations system? It introduces the operations problem the community faced and also introduces the benefits of it for the open source community growthing.
  • The overview of the data-driven operations design. Introduced our thinking and practice in the process of technology selection, metrics data visualization and big data analysis.
  • With the real case of openEuler community, it talks about how data-driven operations system is implemented, how the powerful Apache bigdata project are used to build the first available version (including data storage, ananlysis, data insight and visualization). Finally, and introduced how we contribute our improvements to the upstream Apache projects.

Speakers:

Jun Zhong: Participated in the open source community for more than 6 years. Currently, she is responsible for the digital operation system of the openEuler, MindSpore, openGauss, and openLookeng projects. Served as a core contributor to multiple communities, such as the maintainer of the infra sig team of the openEuler open source community, the maintainer of the infra sig team of the openGauss open source community, and the core member of the OpenStack manila project.

Yikun Jiang: Yikun Jiang, is senior software engineer from Huawei opensource development team, participated in the open source community for more than 5 years, working on multi-arch support and improvement of projects in bigdata area. He has five years experience in cloud computing and bigdata optimization. Previously, he was the core member of OpenStack storage project.

Lei Peng: Lei Peng is senior software engineer from Huawei opensource development team, working on multi-arch support and improvement of MySQL. He has five years experience in SQL develop and bigdata use. Previously, he researched the MySQL kernel, including MySQL Group Replication and worked on kernel development of distributed databases. He has two years experience in using bigdata project such as Spark/Kafka/Hadoop.

2021-08-07 || ROOM : B

13:30 GMT+8

Cassandra powered workflows to automate at scale

English Session

Maciej Swiderski

Nowadays, more and more data is distributed across many geographical locations and as such must be available in almost any location where users are. This can be by applying „follow the sun” organization setup or by supporting various businesses that rely on „anywhere data accessibility”. Apache Cassandra brings the required distributed data mechanics to power up workflow based services and functions. Workflows provide data processing layers that can be expressed as services, functions or even function flows that are designed to run anywhere and that require access to data in a highly distributed manner. In this presentation you will see a practical use of Apache Cassandra that enables running workflow based business logic at scale - from traditional services, to functions and function flows running in serverless fashion where Apache Cassandra shows its features in the best possible way. Combining workflows with Apache Cassandra enables various use cases that were not available with traditional approaches mainly constrained by data access challenges. Speakers:

Maciej Swiderski: Maciej is an independent software engineer at OpenEnterprise. Since 2007 he is in business automation and workflow domain both from development point of view and helping to adopt business automation in different sectors. The last several years he spent at building and running workflows at scale utilising various cloud native solutions e.g. Kubernetes, KNative. He’s passionate about open source and tries to promote it wherever possible. He is also a creator of open source project Automatiko (https://autmatiko.io) that aims at building services and functions based on workflows. In his spare time he enjoys calm and relax life on country side and travels.

14:10 GMT+8

Advanced User Behavior Analysis System Based on Apache Impala & Kudu

Chinese Session

Qianqiong Zhang

SensorsData Co., LTD is now one of the largest BigData platform and martech product supplier in China. Today, At Sensors Data, to over 1000 companies, we deliver advanced user behavior analysis solutions that built on Impala and Kudu. More than 200 billion user events are ingested daily on 1000+ separated on-premise platforms. During the evolution of the platform, we have gained some practical experience.

The share is about:

Real-time data importing: multi-layer storage
Improvement of query robustness: resource estimation and scheduling strategy
Query performance improvement: query optimization

Speakers:

Qianqiong Zhang: Qianqiong Zhang is a big data architect at Sensorsdata, working in the infrastructure department, responsible for developing distributed storage and computing systems. He has very rich experience in system optimization of hadoop, kudu, impala, flink and other Apache systems in his work.

14:50 GMT+8

How Apache Ozone builds up High Availablity with Raft protocol

English Session

Li Cheng, Shashikant Banerjee, Nanda Kumar

Apache Ozone is a scalable, redundant, and distributed object store for Hadoop, which became Apache Top level project in 2020. S As an early adopter of Apache Ozone, Tencent Big Data platform has deployed an Ozone cluster with over 1000 nodes as backend storage for big data applications. Tencent also utilizes Ozone as a primary storage solution for some private data warehouse projects. As Ozone is being heavily used, Ozone high availability is put as top priority to support in production. In 2020, Tencent engineers collaborated with Cloudera engineers on implementing Ozone HA for SCM (Storage Container Manager) with the help of Apache Ratis (Raft implememtation). From this talk, audience would learn:

-What level of complexity Ozone SCM must do in order to maintain high availability. -Why Ozone decides to use Raft protocal to implement SCM HA. -How Ozone team uses Raft and Java reflection to replicate data across SCM group. -How Ozone team optimize the SCM performance after HA is enabled. This talk is also co-authored by Li Cheng from Tencent Cloud, Nanda and Shashikant from Cloudera. All of them are Apache Ozone PMCs and have been actively contributing to Apache Ozone community.

Speakers:

Li Cheng: Current Senior Engineer who owns Big Data Storage in Tencent Cloud COS. Formerly works for AWS S3 and Huawei Storage team. Also active in open source community. Currently Apache Ozone PMC and Hadoop Committer. Shashikant Banerjee: Software engineering professional with 8+ years of experience in designing and building scalable and high performant distributed storage systems. I am currently, a committer and PMC member of Apache Hadoop, Apache Ozone and Apache Ratis community. Nanda Kumar: Software engineering professional with 9+ years of experience in designing and building scalable distributed storage systems. Currently, a committer and PMC member of Apache Hadoop, and Apache Ozone community.

15:30 GMT+8

Running Realtime Analytics at scale with Apache Pinot at LinkedIn and Uber

English Session

Siddharth Teotia, Yupeng Fu

Apache Pinot (incubating) is a distributed columnar OLAP engine that can ingest data in real-time and serve analytical queries at low latency and high throughput. Pinot has evolved and matured over the last couple of years since it entered Apache Incubation. LinkedIn and Uber host the largest footprints of production Pinot installations and we leverage Pinot as the de-facto solution for high-speed analytical queries on both offline (batch) and real-time data. In this joint presentation, we will deep-dive into a few of the major features that have been contributed by LinkedIn and Uber. Specifically, we will go over the following features, briefly discuss the design and implementation, challenges encountered, and how they are being used at a huge scale within LinkedIn and Uber.

  • Support for text indexes in Pinot for efficient queries on arbitrary text data.
  • Solving audience reach estimation problem using theta sketch based approximate distinct count
  • Upsert allows the update on the existing record with the primary key
  • Geospatial support of the performant geo-related queries with the H3-based geo indexing
  • Deep-store bypass to support 7x24 real-time ingestion even when the 3rd party Cloud storage is unavailable

Speakers:

Siddharth Teotia: Siddharth works at LinkedIn in the Pinot team part of Systems and Infrastructure group Prior to LinkedIn, he worked at Oracle for 3.5 years in the Database kernel group on storage, indexing and in-memory columnar query processing. Prior to Oracle, Siddharth worked at Dremio for 2 years as one of the early engineers building out the distributed data lake query engine. He is also a PMC member for Apache Pinot and Apache Arrow.

Yupeng Fu: Yupeng is a Staff Engineer at Uber and he leads Uber’s Real-time Platform and Infrastructure. including multiple mission-critical services powered by several open-source technologies like Kafka/Flink/Pinot. Yupeng is an Apache Pinot committer.

16:10 GMT+8

Past, present and future of Doris

Chinese Session

Mingyu Chen（陈明雨）

In the presentation, I will give you a brief introduction to what Doris is and its development process. After that, it will focus on introducing many new features that Doris has developed in the past year, including the inheritance with Flink and Iceberg, the data update function and multi-tenant. Later, I will introduce the roadmap of Doris and some features work in progress. Speakers:

Mingyu Chen: PPMC of Apache Doris(incubating)

16:50 GMT+8

Apache InLong, a one-stop streaming data integration solution

Chinese Session

gosonzhang, leobiaoliu

In the big data scenario, it is very time-consuming to build a complete data stream service platform from scratch, and it is difficult to provide fast and stable services. Apache InLong, a one-stop streaming data integration solution, it provides the capabilities of publishing and subscribing to streaming data, based on this system, analysis and applications based on streaming data can be easily constructed. Speakers:

gosonzhang: Apache InLong project PPMC, working in the data storage group of Tencent Data Platform Department.

leobiaoliu: Working at real-time computing team of Tencent Data Platform Department.

2021-08-08 || ROOM : B

13:30 GMT+8

Building an authentication & authorization system using HashiCorp Vault

Chinese Session

guangning

Learn how to build an authentication & authorization system for Apache Pulsar using Hashicorp Vault. Vault provides a secure way to generate tokens and store sensitive data and Pulsar has a pluggable architecture for authentication, authorization, and secret management. This talk will walk through how to build an authentication and authentication system for pulsar based on vault, mainly including the following points:

build flexible authentication based on vault to ensure that pulsar clusters can easily access various systems, such as LDAP
how to implement service accounts based on app role of vault

Speakers:

guangning: I am an Apache Pulsar committer and the main contributor to Apache Pulsar IO and Apache Pulsar Manager. I work as a senior software engineer at StreamNative, where he specializes in the cloud platform, cloud computing, and big data-related fields.

14:10 GMT+8

Lambda Architecture for Big Data with Apache C *, Spark and Pulsar

Chinese Session

孟亚斌

The Lambda architecture is a generic data processing framework in Big Data. In this talk we first give a brief introduction to the Big Data Lambda architecture. We then focus on how the three top projects of the Apache Foundation (Cassandra database, Spark data processing engine, and Pulsar stream data processing) can be effectively integrated to achieve a distributed, highly available, linearly scalable Lambda architecture. Finally we will demonstrate the implementation of this architecture with a sample IoT sensor application. Speakers:

Yabin Meng: Yabin Meng is a leading architect at DataStax. In recent years, his focus has been primarily on the design and consulting of solutions for large, distributed databases and stream processing systems. Prior to joining DataStax, he spent most of his career designing, implementing, and consulting on systems in the areas of relational databases, data warehousing, business intelligence, NoSQL databases, and big data.

14:50 GMT+8

State of the Union with Apache YuniKorn (Incubating) - Cloud Native Scheduler for Big Data Usecases

English Session

Sunil Govindan, Julia Kinga Marton

Apache YuniKorn (Incubating) is a light-weight, universal resource scheduler for container orchestrator systems. It was created to achieve fine-grained resource sharing for various workloads efficiently on a large scale, multi-tenant, and cloud-native environment. YuniKorn brings a unified, cross-platform, scheduling experience for mixed workloads that consist of stateless batch workloads and stateful services.

As a custom K8s scheduler, YuniKorn can be easily deployable in K8s on-prem or on cloud. For this talk, we will talk about gaps in resource scheduling in Cloud-Native environment, and how YuniKorn can support running big data applications (like Spark/Flink/Tensorflow, etc.) on K8s.

We will share the advantages for onboarding Spark workloads in to K8s by using Yunikorn’s gang scheduling support. This is one of the biggest paintpoints for Big Data workload migration to Cloud-Native ecosystem. We will also share how YuniKorn being used in community partners such as Apple, Aplibaba, Cloudera, etc. Speakers:

Sunil Govindan: Sunil is contributing majorly towards resource scheduling and is looking into Kubernetes, YARN big data workload scheduling. He is contributing to Apache Hadoop projects since 2013 in various roles as Hadoop Contributor, Hadoop Committer, and a member of the Project Management Committee (PMC). Majorly working on YARN Scheduling improvements / Multiple Resource types support in YARN etc. He is also serving as Apache Submarine PMC member, Apache YuniKorn (incubating) PMC member.

Julia Kinga Marton: Kinga is a Software Engineer at Cloudera, where initially she joined the Apache Oozie team. She is an Apache Oozie committer and PMC member. In early 2020 she joined the YuniKorn team and in the summer of 2020 she became an Apache YuniKorn (Incubating) committer. Before Cloudera she worked at Lufthansa Systems where she was contributing into a weight & balance solution for airlines.

15:30 GMT+8

Flexible Optimizations and Efficient Execution of Data Processing on Apache Nemo

English Session

Won Wook SONG

Optimizing scheduling and communication of distributed data processing for resource and data characteristics is crucial for achieving high performance. In this talk, we introduce Apache Nemo, an optimization framework for distributed dataflow processing that provides fine control for high performance, and also ensures correctness for an easy user experience. In the talk, we provide a demo to show the execution flow of an Apache Beam program being run on our distributed data processing system. We demonstrate how the program is wrapped by the Nemo intermediate representation that enables compiler optimization passes and runtime extensions, and show how the optimizations can be easily and flexibly applied on dataflow applications. We also do a comparison of the evaluation results based on the optimizations. Speakers:

Won Wook SONG: Won Wook (pronounced as won-ook) is a PhD candidate in Seoul National University, advised by Prof. Byung-Gon Chun. His main research interests reside in big data and distributed systems, while he also has interests in machine learning systems. He is currently doing an internship at Microsoft Research Asia. He has been committing to the Apache Nemo project since 2017, and is one of the initial members that have built the system. He have been giving talks in the Apache Beam Summit and several domestic conferences, hosted by tech companies located in South Korea.

16:10 GMT+8

A Change-Data-Capture use-case: designing an evergreen cache

English Session

Nicolas Fränkel

When one’s app is challenged with poor performances, it’s easy to set up a cache in front of one’s SQL database. It doesn’t fix the root cause (e.g. bad schema design, bad SQL query, etc.) but it gets the job done. If the app is the only component that writes to the underlying database, it’s a no-brainer to update the cache accordingly, so the cache is always up-to-date with the data in the database.

Things start to go sour when the app is not the only component writing to the DB. Among other sources of writes, there are batches, other apps (shared databases exist unfortunately), etc. One might think about a couple of ways to keep data in sync i.e. polling the DB every now and then, DB triggers, etc. Unfortunately, they all have issues that make them unreliable and/or fragile.

You might have read about Change-Data-Capture before. It’s been described by Martin Kleppmann as turning the database inside out: it means the DB can send change events (SELECT, DELETE and UPDATE) that one can register to. Just opposite to Event Sourcing that aggregates events to produce state, CDC is about getting events out of states. Once CDC is implemented, one can subscribe to its events and update the cache accordingly. However, CDC is quite in its early stage, and implementations are quite specific.

In this talk, I’ll describe an easy-to-setup architecture that leverages CDC to have an evergreen cache. Speakers:

Nicolas Fränkel: Developer Advocate with 15+ years experience consulting for many different customers, in a wide range of contexts (such as telecoms, banking, insurances, large retail and public sector). Usually working on Java/Java EE and Spring technologies, but with focused interests like Rich Internet Applications, Testing, CI/CD and DevOps. Currently working for Hazelcast. Also double as a trainer and triples as a book author.

16:50 GMT+8

New Apache Bigtop 1.5 and Wikimedia: Empower BigData in the real world

English Session

Yuqi Gu, Luca Toscano

Bigtop is the Apache project for Infrastructure Engineers and Data Scientists who are looking for packaging, testing, and configuration of the open-source Big Data components. Its latest version 1.5.0 has been released as the Big Data Sw stack. From a real-world use case of Bigtop, the Wikimedia’s Analytics/Data-Engineering team collaborated with the project to move its Cloudera CDH Hadoop distribution to Apache Bigtop. In this session, the speakers will provide an overview of Bigtop’s new features and their implementation details, plus a brief introduction of the work done between Wikimedia and Bigtop to support the migration of their data infrastructure. Speakers:

Yuqi Gu: Yuqi Gu currently works for Arm and linaro. He is the committer and PMC member of Apache Bigtop. He is mainly focusing on performance optimization on Arm64.

Luca Toscano: Luca Toscano is a Site Reliability Engineer @ Wikimedia Foundation. He is interested in scalable systems design and reliability, process automation and parallel/distributed/low-level programming. He is passionate about software stacks able to manage and automate complex infrastructures.