Track Chairs : Yu Li
Streaming data processing is a big deal in big data these days, businesses crave ever-more timely insights into their data, and what was once a ‘batch’ mindset is quickly being replaced with stream processing. More and more companies, small and large, are rethinking their architecture with real-time context at the forefront, and starting to build their streaming platforms with powerful open source engines such as Apache Flink, Apache Spark, Apache Kafka, Apache Pulsar, Apache Storm, etc.
In this topic, you will not only learn about the practical experience of first-line users in applying these Apache projects to their in-production environment, but also the latest developments in the ecology of these Apache projects, and visions on where streaming technology is heading in the future.
2021-08-07
13:30 GMT+8
Flink's Greatest and Latest at Alibaba
Chinese Session
Yuan Mei
In this talk, we share some of the most exciting achievements of Flink at Alibaba in recent years, including two main topics: one is the architecture evolution of stream-batch unification; the other is the recent efforts to improve high availability for streaming processing.
Flink has been greatly improved since its first appearance at Alibaba in 2016 and has become the de facto real-time computing standard. Last year for the first time, Flink’s unified stream-batch processing was officially used in Tmall’s big-screen marketing analysis during Double 11 — one of the Double 11’s core scenarios. We will share the architecture evolution beneath to help us achieve this. In addition, Flink never stops expanding its application spectrum to extreme real-time processing. We will discuss some of the efforts in this area as well. Speakers:
Yuan Mei: Yuan Mei is currently the architect of Flink Engine at Alibaba, and she was a research scientist at Facebook before joining Alibaba. She was one of the main contributors of Turbine: Facebook’s Service Management Platform for Stream Processing (ICDE2020). She has various experiences building Stream Processing Systems (Puma & Stylus, VLDB2018) and many other data systems at Facebook (Presto). She holds a Ph.D. from MIT CSAIL, under the supervision of Prof. Samuel Madden & Prof. Michael Stonebraker.
14:10 GMT+8
Practice of Batch-Streaming Unification in Netease Music
Chinese Session
Lei Wang
In this talk, we will introduce the development of the streaming platform in Netease Music, the challenges and thoughts we are facing in order to realize batch streaming unification on the basis of existing business, and the current progress and future planning, with the below outline:
The development and current situation of stream computing in Netease Music
Target and challenge of batch streaming unification
Practice and throughts of batch streaming unification
Future work
Speakers:
Lei Wang: Data Development Specialist of Netease Cloud Music Data Function, Head of System Direction of Netease Cloud Music Data Platform Group, 8 years' experience in data related field
14:50 GMT+8
The Application of Spark Structured Streaming in the near real-time data lake scenarios
Chinese Session
ZhiweiPeng
The data lake is currently a popular big data storage and query solution. Based on the current mainstream data lake format such as DeltaLake, Hudi and Iceberg, it can realize the near real-time writing of large-scale data. It also supports Update and Delete operations, which can realize database bin-log, etc. CDC-type data sources enter the lake. Spark Structured Streaming is a stream processing framework based on the MinBatch execute mode, which can provide better throughput performance for near real-time writing for the data lake format. Through Spark Structured Streaming we can efficiently realize the near real-time write operation of CDC data. This article mainly want to talk about the application of Spark Structured Streaming in the scene of CDC data source entering the data lake, and the technical difficulties involved, including the performance optimization of real-time Merge, The multi-version problem of CDC data and the solution in the scenario of CDC schema change. Speakers:
ZhiweiPeng: Currently working in the Alibaba EMR team, mainly engaged in the research and development of data lake formats such as Hudi, and Delta.
15:30 GMT+8
Meituan's practice of building the real-time data warehouse platform
Chinese Session
Chuxi Tang
In this talk, we will introduce Meituan’s experience of developing the tool chain and setting up our real-time data warehouse platform Speakers:
Chuxi Tang: 2018-2021 Meituan - Data Platform Development Engineer
16:10 GMT+8
Evolution and typical scenes of Flink-based real time computing platform in Qihoo 360
Chinese Session
Fan Xinpu
We will introduce the development of Qihoo 360 real-time computing platform and some typical business scenarios of the real-time computing platform built on Flink in the near future. First, we will share the development history of the Qihoo 360 real-time computing platform and its recent usage. Secondly, we will share the experience and insights of the two larger versions in the platform construction. Next, we will share what useful attempts have been made based on the open source version of Flink. Finally, we would like to discuss with you the next step of platform construction. Speakers:
Fan Xinpu: Fan Xinpu is a real-time computing engineer from Qihoo 360’s system department. He is mainly responsible for platform construction for job based on Flink, and Flink technology promotion within the company.
2021-08-08
13:30 GMT+8
Advanced real-time and batch Analytics using Apache Druid
English Session
Tijo Thomas
Modern Streaming analytical technology excels in crunching live data in a real-time or near-real-time manner. In parallel, other big data technologies can query historical data very well. But these technologies often provide sub-optimal results when the queries involve real-time as well as historical data.
In the real world, this path often resulted in workarounds by splitting the query into real-time and historical. The former gets fired to a system handling real-time data whereas the latter is fired to historical data. The results are derived from subsequently merged output. This is suitable for scenarios where both the queries and the logic for merging the results are pre-defined. But this is not a scalable approach where the queries are ad hoc and analytical in nature and the aggregation logic spreads across both real-time and historical data.
Apache Druid overcomes the limitation of both batch and real-time systems. It will return the result on recent data combined with historical data with guaranteed correctness. This takes away the need for a merging logic. Thus Druid enables querying of real-time data in fast, flexible and with sub-second latency.
In this talk, we will review the limitations of existing analytic infrastructures built on Apache Kafka, Apache Flink, Apache Spark and other cloud-based streaming platforms for querying real-time and batch data. Finally, we go on to explain how Apache Druid helps to bridge this gap. Speakers:
Tijo Thomas: Tijo Thomas is a Sr. Solutions Architect at Imply and an experienced Data Engineer. He has over 18 years of experience in software development, mostly in big data and streaming technologies. He has been helping customers in setting up their stream processing infrastructures using Apache Druid over the last couple of years. During this time he has collected best practices, patterns and anti-patterns applied in production environments.
14:10 GMT+8
Structured Data Streaming
English Session
Shivji Kumar Jha
Type safety is extremely important in any application built around a stream / queue. Type definition and evolution can either be built in the application or relied upon the data layer to support it out of the box allowing the application to only concentrate on business logic, not how of data store and evolution. It is this property of the good old relational databases (among others) that make them a favourite among all the modern NoSQL databases. Modern software architectures requires asynchronous communication (via stream / queue). While the data store and query design changes with asynchronous communication, type safety is still equally important.
In this talk we will go over ways in which one can force structure (schema) over the streaming data. As an example, we will talk about Apache Pulsar. Apache pulsar offers server as well as client side support for the structured streaming. We have been using pulsar for asynchronous communication among microservices in our nutanix beam and flow security central apps for over 1.5 years in production. This talk presents the technical details on what is schema, how to represent schema, what is available in the apache pulsar server and client side, how we have used pulsar’s schema support to build our use cases and our learnings from them. Speakers:
Shivji Kumar Jha: Shiv is a senior software developer at Nutanix and works for the beam team helping Nutanix customers minimise cloud costs and security risks for hybrid cloud usage. Shiv loves spending time on data stores (databases, streams, analytics etc) and has contributed to MySQL and pulsar codebases. Shiv is an avid reader (tech, fiction, economics etc) and is always looking at ways to simplify software architectures.
14:50 GMT+8
FLaNK Stack with Flink For Streaming Use Cases
English Session
Timothy Spann
Today, data is being generated from devices and containers living at the edge of networks, clouds and data centers. We need to run business logic, analytics and deep learning at the edge before we start our real-time streaming flows. Fortunately using all the Apache FLaNK and FLiP stacks we can do this with ease! Streaming AI Powered Analytics From the Edge to the Data Center is now a simple use case. With MiNiFi we can ingest the data, do data checks, cleansing, run machine learning and deep learning models and route our data in real-time to Apache NiFi and Apache Pulsar for further transformations and processing. Apache Flink will provide our advanced streaming capabilities fed real-time via Apache Pulsar topics. Apache MXNet models will run both at the edge and in our data centers via Apache NiFi and MiNiFi.
Tools: Apache Flink, Apache Pulsar, Apache NiFi, MiNiFi, Apache MXNet Speakers:
Timothy Spann: Tim Spann is a Developer Advocate @ StreamNative where he works with Apache Pulsar, MiniFi, Apache NiFi, Apache Flink, Apache MXNet, TensorFlow, Apache Spark, big data, the IoT, machine learning, and deep learning. Tim has over a decade of experience with the IoT, big data, distributed computing, streaming technologies, and Java programming. Previously, he was a Principal Field Engineer @ Cloudera, Senior Solutions Architect at AirisData and a senior field engineer at Pivotal. He blogs for DZone, where he is the Big Data Zone leader, and runs a popular meetup in Princeton on big data, the IoT, deep learning, streaming, NiFi, the blockchain, and Spark. Tim is a frequent speaker at conferences such as IoT Fusion, Strata, ApacheCon, Data Works Summit Berlin, DataWorks Summit Sydney, and Oracle Code NYC. He holds a BS and MS in computer science.
https://www.datainmotion.dev/p/about-me.html
https://dzone.com/users/297029/bunkertor.html
15:30 GMT+8
REAL-TIME MACHINE LEARNING WITH PULSAR FUNCTIONS
English Session
David Kjerrumgaard
In this talk I will present a technique for deploying machine learning models to provide real-time predictions using Apache Pulsar Functions. In order to provide a prediction in real-time, the model usually receives a single data point from the caller, and is expected to provide an accurate prediction within a few milliseconds.
Throughout this talk, I will demonstrate the steps required to productionize a fully-trained ML that predicts the delivery time for a food delivery service based upon real-time traffic information, the customer;s location, and the restaurant that will be fulfilling the order. Speakers:
David Kjerrumgaard: David is the author of “Pulsar in Action” and also a Principal Software Engineer on Splunk’s messaging team that is responsible for Splunk’s internal Pulsar-as-a-Service platform Prior to Splunk, he was the Director of Solution Architecture at Streamlio where he was responsible for the development of best practices and solutions based on Apache Pulsar.
16:10 GMT+8
Interactive Realtime Dashboards on Data Streams using Apache Kafka, Druid and Superset
English Session
Nishant Bangarwa
When interacting with analytics dashboards in order to achieve a smooth user experience, two major key requirements are quick response time and data freshness. To meet the requirements of creating fast interactive BI dashboards over streaming data, organizations often struggle with selecting a proper stack.
This talk presents an open-source real-time data analytics stack using Apache Kafka, Apache Druid, and Apache Superset. The stack combines the low latency streaming and processing capabilities of Kafka with Druid which enables immediate exploration and provides low latency queries over the ingested data streams. Superset provides the visualization and dashboarding that integrates nicely with Druid. In this talk, we will discuss why this architecture is well suited to interactive applications over streaming data, present an end-to-end demo of the complete stack and discuss its key features and discuss performance characteristics from real-world use-cases.
Agenda of the Talk -
- Introduction and Ideal Use cases
- Architecture
- Demo
- Apache Kafka and Kafka Streams as Streaming Platform
- Druid as Serving Layer
- Superset as the Visualization layer
- Key features of Analytics Stack
- Performance benchmarks
Speakers:
Nishant Bangarwa: Nishant is Co-founder and Head of Engineering at Rilldata. He is an active open-source contributor and a PMC member of Apache Druid and Apache Superset. He is also a committer in Apache Calcite and Apache Hive. Prior to starting Rilldata, he was part of Cloudera’s Data warehouse team and Metamarkets Druid team where was responsible for managing large-scale Apache Druid deployments. He holds a B.Tech in Computer Science from the National Institute of Technology, Kurukshetra, India.
16:50 GMT+8
Apache Druid real-time ingestion challenges and best practices
English Session
Tijo Thomas
Modern businesses make real-time data-driven decisions with Apache Druid. One of the key challenges is to design a reliable ingestion pipeline. Some of these challenges are primarily because the nature of the queries varies from customer to customer and use case to use case.
In this talk, we will cover some of the best practices in setting up real-time ingestion Apache Druid with Apache Kafka. We will also discuss some of the advanced tips on optimizing real-time ingestion, query performance and reliability. Speakers:
Tijo Thomas: Tijo Thomas is a Sr. Solutions Architect at Imply and an experienced Data Engineer. He has over 18 years of experience in software development, mostly in big data and streaming technologies. He has been helping customers in setting up their stream processing infrastructures using Apache Druid over the last couple of years. During this time he has collected best practices, patterns and anti-patterns applied in production environments.