Apache Kafka News

• 

  Introducing Confluent Platform 5.0
  ksql confluent platform confluent control center replicator schema registry mqtt • • sorangutan  

  1
  1
  Posts
  71
  Views

  

  We are excited to announce the release of Confluent Platform 5.0, the enterprise streaming platform built on Apache Kafka^®. At Confluent, our vision is to place a streaming platform at the heart of every modern enterprise, helping infrastructure owners get the most out of Kafka and empowering developers to build powerful applications with real-time, streaming data. With a comprehensive streaming platform, companies can better integrate all their disparate data sources into a single source of truth and respond in real time to every event affecting the business.

  Since the release of Confluent Platform 1.0, we have provided an enterprise-class streaming platform that includes the latest in streaming technology based on Apache Kafka. Our latest release introduces significant new capabilities, from making infrastructure more secure, reliable and easier to manage, to enabling more powerful applications with streaming data.

  With Confluent Platform 5.0, operators can secure infrastructure using the new, easy-to-use LDAP authorizer plugin and can deliver faster disaster recovery (DR) thanks to automatic offset translation in Confluent Replicator. In Confluent Control Center, operators can now view broker configurations and inspect consumer lag to ensure that they are getting the most out of Kafka and that applications are performing as expected.

  We have also introduced advanced capabilities for developers. In Confluent Control Center, developers can now better understand the data in Kafka topics due to the new topic inspection feature and Confluent Schema Registry integration. Control Center presents a new graphical user interface (GUI) for writing KSQL, making stream processing more effortless and intuitive as well. The latest version of KSQL itself introduces exciting additions, such as support for nested data, user-defined functions (UDFs), new types of joins and an enhanced REST API. Furthermore, Confluent Platform 5.0 includes the new Confluent MQTT Proxy for easier Internet of Things (IoT) integration with Kafka. The latest release is built on Apache Kafka 2.0, which features several new functionalities and performance improvements.

  Download Confluent Platform 5.0

   

  Operate more secure, reliable and performant Apache Kafka

  Operationalizing enterprise infrastructure is hard, and Apache Kafka is no exception. Our mission at Confluent is to make managing Kafka easy, and the latest release of Confluent Platform helps operators do just that.

  Secure Kafka using the new LDAP authorizer plugin and control feature access in Confluent Control Center

  As customers adopt Confluent Platform for their mission-critical data, security becomes an important consideration. Operators need to be able to easily manage user access to sensitive data, but doing so in Kafka has historically been difficult. Because security is top-of-mind for many of our customers, we have made several investments to improve security management in Confluent Platform and will continue providing new security features in later releases.

  The majority of enterprises standardize on AD/LDAP for identity-related services. The latest release of Confluent Platform introduces an LDAP authorizer plugin to set up access control for Kafka, allowing customers to use the user-group mappings in their LDAP server. The plugin works with any security protocol by mapping an authenticated user principal to one or more groups and allowing user-principal-based ACLs, as well as group-based ACLs. This simplifies access control management because a smaller number of rules can be used to manage access to groups or organizations.

  Enterprises also need to ensure that end users cannot access sensitive data via Control Center for security and compliance reasons. In order to let administrators control application-wide access to features that reveal topic data, we have included feature access controls for topic inspection, schemas, and KSQL. When customers restrict access to a feature via the configuration file, Control Center’s UI will reflect this change upon startup, and users cannot circumvent these protections in any way.

  Minimize disaster recovery time with automatic offset translation in Confluent Replicator

  The disaster recovery (DR) response process is complex. Operators need to ensure that the data is replicated, applications failover, and applications then switch back when the original data center (DC) has recovered. Replicator supports operators along each step of the process by providing automation that minimizes the work needed during a DR response.

  Replicator already automates the replication of topic messages and related metadata. With Confluent Platform 5.0,  it also ensures that consumer offsets are available on the secondary cluster. When consumer client applications failover to the secondary cluster, Replicator handles consumer offset translation so that they can resume consuming near the last point they stopped consuming at in the original cluster. This minimizes the reprocessing that consumers needs to do in a disaster scenario without skipping messages.

  To facilitate application failover and switch back in Confluent Platform 5.0, Replicator adds support for protection against circular replication. This guarantees that if two Replicator instances are configured to run, one replicating from DC1 to DC2, and the second instance configured to replicate from DC2 to DC1, Replicator will ensure that messages replicated to DC2 are not replicated back to DC1, and vice versa. As a result, Replicator safely runs in each direction.

  Monitor more and manage better using Confluent Control Center

  We built Control Center to deliver understanding and insight into the inner workings of Apache Kafka and streaming data, helping operators monitor and manage their Kafka installations as well as possible. Confluent Platform 5.0 introduces a broker configuration view that ultimately improves this experience. It empowers operators to see broker configurations across multiple Kafka clusters, check configuration values for specific brokers, compare differences within a cluster to identify potential risks such as mismatched security configurations in different brokers, and even download configurations. This new feature helps operators ensure that their brokers are set up correctly, enabling them to get the most out of Kafka.

  

  Once you have configured Kafka the way you want and have applications consuming data from topics, it’s important to understand how those applications are performing. Monitoring application performance is something that both operators and developers care about. Since its genesis, Control Center has included an advanced stream monitoring feature, which provides insight into end-to-end performance of the produce-consume cycle. In the latest release, we introduce consumer lag, which allows users to view how consumers are performing based on offset, spot potential issues at a glance, and take proactive steps to keep performance high. Users can also set alerts to monitor lag behind-the-scenes and notify them if issues arise.

  

  Build more powerful streaming applications with Kafka

  Having a secure, reliable, and performant Kafka infrastructure is the first part of being successful with streaming data, but it’s not enough. Along with helping enterprises better monitor and manage Kafka, we strive to make building applications with streaming data straightforward, seamless and fast. Confluent Platform 5.0 delivers new capabilities for application developers in both Control Center and KSQL.

  Understand and process streaming data with Confluent Control Center

  Control Center now allows users to gain insight into the actual data in Kafka topics via a new topic inspection feature. Users can see the streaming messages in topics and read key, header, and value data for each message. Thanks to integration with Schema Registry, topic inspection supports JSON, string, and Avro data formats.

  

  In addition to viewing message data, developers also need to understand how data is organized in order to build applications with Kafka. This need to understand data structure and manage compatibility quickly grows in importance as an enterprise scales up with Kafka. Confluent Schema Registry is a valuable tool for coordinating teams of developers, and makes managing a real-time streaming platform with multiple connected applications seamless. With the latest release of Confluent Platform, Control Center integrates with Schema Registry and allows users to see the key and value schemas for topics. Users can also view older schema versions and compare them against current versions in a git-like UI, which makes finding differences in evolving schemas and building applications on streaming data simple.

  

  Once developers can view message data and understand how data is structured, they have the basic building blocks to start building applications. Last year, we announced KSQL, the streaming SQL engine for Apache Kafka, to help developers build powerful applications with streaming data. While Confluent Platform 4.1 presented an experimental web interface for developers to run KSQL, in the latest release, Control Center includes KSQL as part of a new development area. With this new GUI for KSQL, stream processing has never been more accessible. Developers can create streams and tables from topics, experiment with transient queries, and run persistent queries to filter and enrich data. What’s more, they can do all this with the power of autocompletion, which lowers the bar to getting started with stream processing. No more trying to figure out what streams and tables exist, or what syntax is right for a command—autocomplete guides developers along the way. KSQL in Control Center lowers the bar for both developers building streaming applications as well as operators who wish to debug cluster issues by filtering and consuming data from topics.

  

  
  Build more powerful applications with KSQL

  Along with providing better GUI support for building applications in Control Center, the latest release includes several highly requested KSQL functionalities.

  The most commonly requested feature for KSQL has been support for nested data. Today, KSQL includes a STRUCT type that allows developers to work with nested data in both Avro and JSON formats. This means that you can use KSQL to process data from a broader variety of sources, and tasks such as data extraction and data conversion are even easier.

  Another highly requested feature has been the ability to define your own computations that go beyond what KSQL provides out-of-the-box. The newly added User-Defined Functions (UDF) and User-Defined Aggregate Functions (UDAFs) open the door for many novel use cases, for which you need to perform custom computations over your data when KSQL’s built-in scalar functions or built-in aggregation functions are insufficient. Applying a machine learning model in real time to a stream of data is an example of an exciting new use case this allows. UDFs take one input row to produce one output row (e.g., ABS, SUBSTRING), whereas UDAFs take numerous input rows to produce one output row (e.g., SUM, MAX, COUNT).

  KSQL also now supports stream-stream, table-table, and stream-table joins. Further, you can execute inner and outer joins in addition to left joins where appropriate for the joined entities. This means that KSQL now covers all of the available join operations in Kafka Streams, and this expanded scope lets you cover more use cases with KSQL. For example, in retail you can detect late shipments by joining a stream of orders with a stream of shipments; in advertising, you can match ad impressions with ad clicks.

  
  Using KSQL’s new INSERT INTO syntax, events from different source streams and queries can be written to the same output stream. This can be useful in various cases, including situations where you have the same logical data arriving on multiple physical topics.

  While users enjoy the ability to define stream processing logic simply through SQL-like queries, developers often want to interact with KSQL programmatically in order to compose and run such queries. Through the KSQL REST API, you can leverage your favorite programming language, whether it’s Python, Go, .NET, Java, JavaScript, and even shell scripting, to work with KSQL. This lets you compose KSQL queries at runtime and submit them to the server when you cannot know the query ahead of time, making KSQL and Confluent Platform an even better fit for non-Java virtual machine and/or polyglot environments.

  Streamline IoT integration with Kafka

  More and more customers of all industries are integrating their IoT devices and gateways with Apache Kafka using the MQTT standard for applications, as in the cases of connected cars, assembly lines in manufacturing, and predictive maintenance. Until today, project teams had to insert an MQTT broker into the middle as an intermediary before building a custom ingestion pipeline into Kafka.

  Confluent Platform 5.0 debuts MQTT Proxy to reduce these complexities and streamline IoT integration. It permits users to build a scalable proxy architecture similar to Confluent REST Proxy for HTTP(S) communication with Kafka. MQTT Proxy enables the replacement of third party MQTT brokers with Kafka-native MQTT proxies in order to remove the additional cost and complexity of intermediate storage and lag. The introduction of MQTT Proxy means that fewer services are required and the need to write a custom communication mechanism between MQTT and Kafka is eliminated.

  Using Transport Layer Security encryption and basic authentication, MQTT Proxy supports the widely used MQTT 3.1.1 protocol and allows publishing messages in all three MQTT quality of service levels. Future releases will add bidirectional communication and more security options.

  New in Apache Kafka 2.0

  Like previous releases, Confluent Platform 5.0 is built on the most recent version of Apache Kafka in order to provide customers with the latest in streaming technology. Confluent Platform 5.0 includes Apache Kafka 2.0, which contains a number of new features, performance improvements, bug fixes, and support changes, including:

  • 40 KIPs and 246 resolved JIRAs
  • Prefixed wildcard access control lists (ACLs), fine-grained ACLs for CreateTopics
  • Simple Authentication and Security Layer/OAuthBearer implementation
  • Improved quota communication and customization of quotas
  • Efficient memory usage for down conversion
  • Fix log divergence between leader and follower during fast leader failover
  • Improved timeout behaviour in the Java consumer, eliminating indefinite blocking when brokers are not available
  • Kafka Connect REST extension plugin, support for externalizing secrets, and improved error handling with dead-letter queue functionality
  • Scala API for Kafka Streams and other Streams API improvements

  Apache Kafka 2.0 also drops support for Java 7 and removes the previously deprecated Scala producer and consumer. More details are available in the Apache Kafka 2.0 Release Notes.

  Get started

  Here are the best ways to get started with Confluent Platform:

  • Try our free 30-day trial of Confluent Enterprise, which includes all of these additional capabilities.
  • Alternatively, you can install Confluent Open Source, which includes only open source components. Details for both Confluent Enterprise and Confluent Open Source can be found in the documentation.
  • Receive expert training and technical consulting to get up and running quickly. Confluent Enterprise 5.0 is backed by our subscription support.

  As always, we are happy to hear your feedback. Please post your questions and suggestions to the public Confluent Platform mailing list, join our community Slack channel, or contact us directly!

  Special thanks to Apurva, Dan, Derrick, Erica, Ewen, Hojjat, Ismael, Jo, Joseph, Kai, Mikkin, Neha, Nick, Ran, Randall, Robin, Rohan, Rohit, Tim, Vahid, Victoria, and Vivian for their contributions to this post.

  The post Introducing Confluent Platform 5.0 appeared first on Confluent.

  
  
  https://www.confluent.io/blog/introducing-confluent-platform-5-0/
• 

  Highlights of the Kafka Summit San Francisco 2018 Agenda
  apache kafka kafka summit • • sorangutan  

  1
  1
  Posts
  99
  Views

  

  Honestly, it seems like I just filed my expense report for Kafka Summit London, and now we are announcing the program for Kafka Summit San Francisco. (CONFIDENTIAL TO CONFLUENT FINANCE: I did just file my expense report for Kafka Summit London. Three months isn’t so bad, right? Right?)

  But hey, here we are, just under three months out from the third annual marquee Apache Kafka^® community event. Kidding aside, Kafka Summit has established itself as the premier streaming systems event for people like you: developers, DevOps people, software architects, and anyone else who builds things with Kafka. Happily, people talk about Kafka in lots of different venues, but this is the place for the community to rally around our next-generation plans, share experiences, and plot the future of streaming technologies.

  Because Kafka has become an increasingly strategic platform, it has also attracted the attention of business leaders. To help welcome this new constituency, we are rolling out a new track at the 2018 summit: The Business Case.

  The Kafka Summit Program Committee just finished sorting through more than 250 excellent submissions and sent out formal acceptance emails last week. We’ll be publishing the full agenda (50+ sessions) soon, so keep an eye on the website or sign up for updates, and we’ll let you know as soon as the deed is done.

  Some highlights I can share from the agenda so far:

  Internals Track

  • Kafka Security 101 & Real-World Tips (Stephane Maarek, DataCumulus)
  • Connecting Kafka Across Multiple AWS VPCs (Benoit Carrière, Expedia)
  • Kafka on ZFS: Better Living Through Filesystems (Hugh O’Brien, Jet.com)

  Pipelines Track

  • Data Streaming Ecosystem Management at Booking.com (Alex Mironov, Booking.com)
  • More Data, More Problems: Scaling Kafka Mirroring Pipelines at LinkedIn (Celia Kung, LinkedIn)
  • Move the Needle of the Pin—Pinterest’s Story of Streaming Hundreds of Terabytes of Pins from MySQL to S3/Hadoop Continuously (Henry Cai, Pinterest)

  Streams Track

  • Deploying Kafka Streams Applications with Docker and Kubernetes (Gwen Shapira + Matthias J. Sax, Confluent)
  • How Big Fish Games Developed Real-Time Analytics Using Kafka Streams and Elasticsearch (Kalah Brown, Big Fish Games)
  • Real-Time Market Data Analytics Using Kafka Streams (Lei Chen, Bloomberg L.P.)

  The Business Case Track

  • Life is a Stream of Events (Bjørne Kvernstuen, Norwegian Directorate for Work and Welfare)
  • Kafka for Real-Time Event Processing in Serverless Environments (Jeff Sharpe, Capital One)
  • Using Kafka in Your Organization with Real-Time User Insights for a Customer 360 View (Chris Maier, Western Monroe Partners)

  The full speaker list will be posted soon.

  As always, permit me to thank the program committee for all their hard work and investment in putting together the agenda:

  • Becket Qin, LinkedIn
  • Ben Lorica, O’Reilly Media
  • Helena Edelson
  • Jun Rao, Confluent
  • Michael Noll, Confluent
  • Neha Narkhede, Confluent
  • Rajini Sivaram, Confluent
  • Sam Newman
  • Tim Berglund, Confluent
  • Todd Palino, LinkedIn

  Oh, hey one more thing: early bird pricing ends Aug. 1! “Oh, that’s August; I have plenty of time,” you think, but you and I both know that’s essentially tomorrow.

  Register now to save $200 off the standard conference pass.

  I look forward to seeing you at the summit!

  The post Highlights of the Kafka Summit San Francisco 2018 Agenda appeared first on Confluent.

  
  
  https://www.confluent.io/blog/highlights-of-the-kafka-summit-san-francisco-2018-agenda/
• 

  Ansible Playbooks for Confluent Platform
  confluent platform ansible playbooks • • sorangutan  

  1
  1
  Posts
  137
  Views

  

  Here at Confluent, we want to see each Confluent Platform installation be successful right from the start. That means going beyond beautiful APIs and spectacular stream processing features. We want to give engineers running the services a great experience, too, which is why I’m pleased to introduce our first set of Ansible playbooks to help set up and install Confluent Platform. These playbooks help you get a proof-of-concept cluster off the ground that is production-ready. To get started, just put your hostnames into the example hosts.yml file and run the full playbook with a simple `ansible-playbook -i hosts.yml all.yml`. The result is a secured Confluent Platform in less than five minutes.

  What makes these playbooks special? First, they deploy Confluent Platform services with the newly packaged systemd service unit files, giving administrators the familiar look and feel of a production system. We’ve also made the hard decisions for you by taking the liberties of setting well-documented default usernames and log locations to help installations of new environments be repeatable and recognizable. The systemd integration even provides a quite-lovely integration with journalctl for logging.

  The second feature is optional automated authentication and encryption setup. The supported modes are PLAINTEXT, SSL, and SASL_SSL. We often hear from users about how they want to get started quickly on a proof-of-concept to show the business value of a streaming data platform, but struggle to get past the first hurdle of enabling SASL and SSL. This setup generates simple SSL certificates and uses SASL authentication with a plaintext mechanism by  choosing the `sasl_ssl` template. The automated setup is designed to show the security capabilities of the platform simply and quickly. The best part is that when it’s time to go to full production, you can replace the provided self-signed SSL certificates and plaintext SASL with your own production grade and Kerberos principals without even having to worry if you missed a configuration.

  Finally, the setup isn’t limited to deploying some Confluent Platform services and leaving others as an exercise to the user. You can easily launch every component: Apache ZooKeeper, Kafka brokers, Confluent Schema Registry, Confluent REST Proxy, Kafka Connect workers, KSQL server, and Confluent Control Center. If you launch the whole platform, you also get the awesome topic and connector management features available in Confluent Control Center 4.1, and can get data flowing directly from the UI. In fact, you can check out Robin’s blog post for some great ideas on what to get connected to right away.

  The bottom line is, at Confluent, we want each of our users to be successful. Whether they are just getting started on a laptop, setting up their first proof-of-concept to show the business, or getting ready to launch their 15th production installation of the platform. We think that these playbooks are a great way to get those proof-of-concept installs off the ground in a way that follows best practices directly from our team.

  The repository is open! We would love to hear your feedback and welcome pull requests.

  Interested in more?

  If you’d like to know more, here are some resources for you:
  

  • You can download Confluent Platform
  • Read our documentation
  • Confluent Professional Services offers advice and help with deployments
  • Confluent Training can get your team ready for development and deployment of Confluent Platform

   

  The post Ansible Playbooks for Confluent Platform appeared first on Confluent.

  
  
  https://www.confluent.io/blog/ansible-playbooks-for-confluent-platform/
• 

  Apache Kafka® vs. Enterprise Service Bus (ESB)—Friends, Enemies, or Frenemies?
  apache kafka etl integration enterprise architecture enterprise service bus • • sorangutan  

  1
  1
  Posts
  90
  Views

  

  Typically, an Enterprise Service Bus (ESB) or other integration solutions like Extract-Transform-Load (ETL) tools have been used to try to decouple systems. However, the sheer number of connectors, as well as the requirement that applications publish and subscribe to the data at the same time, mean that systems are always intertwined. As a result, development projects have lots of dependencies on other systems and nothing can be truly decoupled.

  This blog post shows why so many enterprises leverage the open source ecosystem of Apache Kafka for successful integration of different legacy and modern applications, and how this differs but also complements existing integration solutions like ESB or ETL tools.

  The need for integration—a never ending story

  No matter in which enterprise you work, no matter when your company was founded, you will have the requirement to integrate your applications with each other to implement your business processes.

  This includes many different factors:

  • Technologies (standards like SOAP, REST, JMS, MQTT, data formats like JSON, XML, Apache Avro or Protocol Buffers, open frameworks like Nginx or Kubernetes, and proprietary interfaces like EDIFACT or SAP BAPI)
  • Programming languages and platforms like Cobol, Java, .NET, Go, or Python
  • Application architectures like Monolith, Client Server, Service-oriented Architecture (SOA), Microservices, or Serverless
  • Communication paradigms like batch processing, (near) real time, request-response, fire-and-forget, publish subscribe, continuous queries, and rewinding

  Many enterprise architectures are a bit messy—something like this:

  Every company needs to solve these spaghetti architectures. Depending on the decade, you either bought something like an ETL tool to build batch pipelines or an ESB to design a SOA. Some products also changed their names. Today, you are offered things like middleware messaging, an integration platform, microservice gateway, or API management. The branding and product name do not matter. You always see the same picture as a solution to move away from your spaghetti architecture to a central integral box in the middle, like this:

  

  This rarely worked well in practice, unfortunately. Most SOA projects in the last two decades failed. Instead of using an ETL tool or ESB for this, enterprises are now moving on to a streaming platform to solve this issue. Is this the next bubble on the market? Just a new term? Or, did something really change to allow successful integration across an enterprise—whether you integrate legacy mainframes, standard applications like CRM and ERPs, modern microservices built with any programming platform, or public cloud services? Why are companies now migrating to Apache Kafka to build this streaming platform? Why is everybody happy and talking about this at conferences, tech talks, and blog posts? How does it compare to an ESB or ETL tool?

  The next sections will answer all these questions, and explain the reason and differences between the open source ecosystem of Apache Kafka and other existing integration solutions.

  Event-driven processing and streaming as a key concept in the enterprise architecture

  A streaming platform (you can also enter another buzzword here) leverages events as a core principle. You think in data flows of events and process the data while it is in motion.

  Many concepts, such as event sourcing, or design patterns such as Enterprise Integration Patterns (EIPs), are based on event-driven architecture. The following are some characteristics of a streaming platform:

  • Event-based data flows as a foundation for (near) real-time and batch processing. In the past, everything was built on data stores (data at rest), making it impossible to build flexible, agile services to act on data while it is relevant.
  • Scalable central nervous system for events between any number of sources and sinks. Central does not mean one or two big boxes in the middle but a scalable, distributed infrastructure, built by design for zero downtime, handling the failure of nodes and networks, and rolling upgrades. Different versions of infrastructure (like Kafka) and applications (business services) can be deployed and managed in an agile, dynamic way.
  • Integrability of any kind of applications and systems. Technology does not matter. Connect anything: programming language, APIs like REST, open standards, proprietary tools, and legacy applications. Speed does not matter. Read once. Read several times. Read again from the beginning (e.g., add new application, train different machine learning models with the same data).
  • Distributed storage for decoupling applications. Don’t try to build your own streaming platform using your favorite traditional messaging system and in-memory cache/data grid. There is a lot of complexity behind this and a streaming platform simply has it built-in. This allows you to store the state of a microservice instead of requiring a separate database, for example.
  • Stateless service and stateful business processes. Business processes typically are stateful processes. They often need to be implemented with events and state changes, not with remote procedure calls and request-response style. Patterns like event sourcing and CQRS help implement this in an event-driven streaming architecture.

  Benefits of a streaming platform in the enterprise architecture

  A streaming platform establishes huge benefits for your enterprise architecture:

  • Large and elastic scalability regarding nodes, volume, throughput—all on commodity hardware, in any public cloud environments, or via hybrid deployments.
  • Flexibility of architecture. Build small services, big services, sometimes still even monoliths.  
  • Event-driven microservices. Asynchronously connected microservices model complex business flows and move data to where it is needed.
  • Openness without commitment to a unique technology or data format. The next new standard, protocol, programming language, or framework is coming for sure. The central streaming platform is open even if some sources or sinks use a proprietary data format or technology.
  • Independent and decoupled business services, managed as products, with their own lifecycle regarding development, testing, deployment, and monitoring. Loose coupling allows for independent speed of processing between different producers and consumers, on/offline modes, and handling backpressure.
  • Multi-tenancy to ensure that only the right user can create, write to, and read from different data streams in a single cluster.
  • Industrialized deployment using containers, devops, etc., deployed where needed, whether on premise, in the public cloud or in a hybrid environment.

  These characteristics build the foundation of a streaming platform, the beginning of your successful digital transformation. With services implementing a limited set of functions, and services being developed, deployed, and scaled independently, you get shorter time to results and increased flexibility. This is only possible with a streaming platform having the above characteristics.

  Use cases for a streaming platform

  Here are some generic scenarios for how you can leverage a streaming platform with the characteristics discussed above:

  • Event-driven processing of big data sets (e.g., logs, IoT sensors, social feeds)
  • Mission-critical, real-time applications (e.g., payments, fraud detection, customer experience)
  • Decoupled integration between different legacy applications and modern applications
  • Microservices architecture
  • Analytics (e.g., for data science, machine learning)

  Producer and consumers of different applications are really decoupled. They scale independently at their speed and requirements. You can add new applications over time, both on the producer and consumer side. Often, one event is required to be consumed by many independent applications to complete the business process. For example, a hotel room reservation needs immediate payment fraud detection in real time, the ability to process the booking through all backend systems in near real time, and overnight batch analytics to improve customer 360, aftersales, hotel logistics, and other business processes.

  While some processes need real-time processing, you also need to be capable of supporting batch processes. You even need re-consumption of data more often than you would think in the beginning, such as in cases of an application being down for some time, A/B testing with different versions of an application, adding a new application that needs to consume the data from scratch, or building different analytic models via machine learning based on the same data sets.

  Think about some more use cases that you can build easily with a real decoupled system that is still a well-integrated and scalable streaming platform:

  • Selling before the customer left the store
  • Aborting a transaction before the fraud happened
  • Replacing a part of a manufacturing machine before it breaks
  • Informing customers if a flight or train is late (plus sending updates, rebooking, or a voucher)   
  • You name it—the list goes on.

  Big bang from batch to real time?

  Now, you understand the added value of a real decoupled, scalable streaming platform. So, do I have to introduce this as a central data platform for all of our applications?

  Caution! No mature enterprise can do a big bang successfully. Legacy applications exist everywhere. Go step by step from pre-streaming to streaming platform. If you come from the mainframe ages, then you might even have batch and non-streaming applications forever (or realistically at least for the next 20-30 years). That’s fine. You just need to bring the events from these systems into the event-driven central nervous system.

  The following shows the streaming maturity model that we use to identify the current situation and planning in large enterprises:

  

  Where you are today?

  1. Pre-Streaming (Batch or legacy)
  2. Interest (First proof-of-concepts or pilots)
  3. Early Production (Some independent projects in production)
  4. Integrated Streaming (Streaming platform with different projects in production)
  5. Streaming Platform (Streaming enterprise with mostly event-based applications)

  Most traditional enterprises start their journey in the pre-streaming phase. That’s totally fine. The next section explains why almost any successful transformation into a streaming platform leverages the Apache Kafka ecosystem as a key architectural component.

  Introducing Apache Kafka ecosystem as streaming platform

  Often people are familiar with Apache Kafka, as it has been a hugely successful open source project, created at LinkedIn for big data log analytics. That was the beginning of Kafka, and just one of many use cases today. Kafka evolved from a data ingestion layer to a real-time streaming platform for all the use cases previously discussed. Many projects focus on building mission-critical applications around Kafka. It has to be up and performant 24/7. If Kafka is down, their business processes stop working.

  Kafka is unique because it combines messaging, storage, and processing of events all in one platform. It does this in a distributed architecture using a distributed commit log and topics divided into multiple partitions, as seen below:

  
  With this distributed architecture, Kafka is different from existing integration and messaging solutions. Not only is it massively scalable and built for high throughput but different consumers can also read data independently of each other and in different speeds.

  Applications publish data as a stream of events while other applications pick up that stream and consume it when they want. Because all events are stored, applications can hook into this stream and consume as required—in batch, real time or near-real-time. This means that you can truly decouple systems and enable proper agile development. Furthermore, a new system can subscribe to the stream and catch up with historic data up until the present before existing systems are properly decommissioned.

  The uniqueness of having messaging, storage, and processing in one distributed, scalable, fault-tolerant, high-volume, technology-independent streaming platform is the reason for the global success of Apache Kafka in almost every bigger company on this planet.

  So, what is Apache Kafka and its open source ecosystem? Let’s take a high-level look at its components:

  • Apache Kafka as core for distributed messaging and storage. High throughput, low latency, high availability, secure.
  • Kafka Connect as an integration framework for connecting external sources/destinations into Kafka.
  • Kafka Streams as a simple library that enables streaming application development within the Kafka framework.
  • Additional clients for non-Java programming languages, including C, C++, Python, .NET, Go, and several others.
  • Confluent REST Proxy to provide universal access to Kafka from any network connected device via HTTP(S).
  • Confluent Schema Registry as a central registry for the format of Kafka data, guaranteeing that all data is always consumable, including schema evolution.
  • Confluent KSQL as the streaming SQL engine that enables scalable, high-volume stream processing natively against Apache Kafka without writing source code.

  All these open source components are based on top of the core messaging and storage layer of Apache Kafka, and all leverage its features of high scalability, high volume/throughput, and failover. In addition, Confluent offers 24/7 support and enterprise tooling for end-to-end monitoring, management of Kafka clusters, multi-datacenter replication, and more.

  Okay, so we should start evaluating Kafka in a proof of concept (POC)?

  POC? Well, the Apache Kafka ecosystem is already battle-tested at a scale that you will probably not reach in the next 5-10 years. Many notable organizations are Kafka users:

  • LinkedIn processes over 4.5 trillion messages per day in 2018.
  • Netflix processes over 6 petabytes of data per day in 2018.
  • Ebay, Uber, Paypal, you-name-it. Just Google your favorite tech giant + Apache Kafka to see how they use Kafka. It is everywhere!
  • Even “traditional” companies in banking, insurance, telco, retail, automotive, and manufacturing use Kafka to transform and revolutionize their core business. Public references, including ING Group (banking), Audi (automotive), and Target (retail), to name a few, are available on the web.

  Don’t waste your time with POCs about performance, maturity, or scalability. Instead, evaluate how to integrate Kafka into your enterprise architecture to evolve business scenarios and innovate towards digital transformation.

  Should we replace our existing MQ and ESB deployments?

  Younger companies like Netflix, LinkedIn, and Zalando built their whole infrastructure on Kafka. Older companies are not that fortunate because they have plenty of mainframes, monoliths, and legacy technology. However, as discussed, a big bang replacement is not the right way to be successful. It’s a lot like transforming your home. Although it might make sense in theory to rebuild it from the ground up, oftentimes it is more practical to extend the house, change certain rooms, or redecorate.

  It’s difficult to just rip and replace mission-critical applications that are deeply integrated with other apps that work and involve a skilled team operating them. That said, sometimes it may be more cost effective to replace legacy architecture, just like it sometimes does make sense to remodel a house from the ground up. This was the case with Sberbank, the biggest bank in Russia, which built their complete core banking system around Kafka as central nervous system.

  Apache Kafka and other middleware as complementary components

  Legacy apps are typically based on complex data formats like EDIFACT, use complex interfaces like CORBA, or are built with an unmaintainable and inflexible programming language like Cobol. You cannot simply turn it off, or cut it out and replace it. This has to be done step by step. Legacy and modern applications co-exist to run the existing business and add new offerings to augment it.

  Innovate by integrating your old systems via a streaming platform—i.e., the Apache Kafka ecosystem. Use concepts like change data capture (CDC) and integration tools, such as an ESB or ETL, with great graphical tooling and connectors for legacy applications. With this foundation, you can build new applications with modern technologies, big data systems, machine learning, etc., natively around Apache Kafka, and at the same time keep access to your legacy events, which you still need for added value in new projects.

  Real decoupling and technology independence also means that you have dumb pipes and smart endpoints. If you build all the integration logic (or, even worse, some business logic) into the central integration layer, then all your scalability, agility, and independence of the different systems are gone. This is a key difference to traditional integration solutions in which you put all the logic in the middle layer ESB. It creates dependency on this (proprietary) technology/API, as well as inflexibility.

  Smart endpoints can be anything. You can leverage the Kafka ecosystem to build applications around Kafka with Kafka Streams, KSQL, or any Kafka client like Java, .Net, Python, or Go. You can also use any other application to integrate other applications with Kafka. The secret to long-term success is that the infrastructure is open to any technology and architectural pattern.

  ETL and ESB have excellent tooling, including graphical mappings for doing complex integration with SOAP, EDIFACT, SAP BAPI, COBOL, etc. (Trust me, you don’t want to write the code for this.) It is already running, paid, and integrated. Therefore, existing MQ and ESB solutions, which already integrate with your legacy world, are not competitive to Apache Kafka. Rather, they are complementary! Leverage them like you did in the past to integrate with the old world. You can use the following:

  • Confluent JMS connector for IBM MQ and other JMS brokers
  • Kafka Connect CDC connectors for mainframe and databases
  • ESB or ETL tools, which integrate with legacy protocols (SOAP, EDIFACT, etc.) and applications (SAP, Siebel, etc.). All these tools also have Kafka connectors in the meantime.
  • Traditional messaging brokers also have a proxy to Kafka in the meantime. Even though some time ago they told everybody that you cannot build a universal messaging platform, traditional messaging brokers now integrate to Kafka and other standards like MQTT to not become forgotten
  • Kafka’s official low-level client APIs like Java, .NET, Go, and Python implement a direct integration if no other more feasible integration option is available.

  As a last note, after explaining the usage of ESB and ETL tools in the Kafka world, I quote ThoughtWorks to remind you not to try building a new ESB around Kafka:

  Some organizations recreate ESB antipatterns with Kafka by centralizing the Kafka ecosystem components—such as connectors and stream processors—instead of allowing these components to live with product or service teams. This reminds us of seriously problematic ESB antipatterns, where more and more logic, orchestration and transformation were thrust into a centrally managed ESB, creating a significant dependency on a centralized team. We’re calling this out to dissuade further implementations of this flawed pattern.

  Source: https://www.thoughtworks.com/radar/techniques/recreating-esb-antipatterns-with-kafka

  Apache Kafka and its ecosystem is designed as a distributed architecture with many smart features built-in to allow high throughput, high scalability, fault tolerance and failover! Let the product or service teams build their applications with Kafka Streams, KSQL, and any other Kafka client API. Integrate Kafka with ESB and ETL tools if you need their features for specific legacy integration. An ESB or ETL process can be a source or sink to Apache Kafka like any other Kafka producer or consumer API. Oftentimes, the integration with legacy systems using such a tool is built and running already anyway. Currently all these tools also have a Kafka connector because the market drives them this way. So, you just need to combine the existing integration with the Kafka connector, and there you have it: flexible, scalable and highly available integration between legacy and future ecosystems through Kafka.

  Apache Kafka as streaming platform between legacy and the new modern world

  Apache Kafka is an open source streaming platform that allows you to build a scalable, distributed infrastructure that integrates legacy and modern applications in a flexible, decoupled way. It is already battle-tested for processing trillions of messages and petabytes of data per day. Simply leverage the Apache Kafka ecosystem in your enterprise architecture to make integration of your various systems successful and dynamic. But whatever you do, do not try to build an ESB around Kafka—it is an anti-pattern that will create inflexibility and unwanted dependencies. Instead, leverage the distributed architecture of the Apache Kafka ecosystem to build a flexible, event-driven streaming infrastructure with high throughput, high scalability, fault tolerance, and failover.

  As a next step, you can get further proof by looking at past Kafka Summits in London, New York, and San Francisco, where plenty of tech giants, established companies, and software vendors explained how they digitize their business and innovate their products leveraging the Apache Kafka ecosystem. All video recordings and slide decks are available online.

  Interested in more?

  If you’d like to know more, here are some resources for you:

  • You can download Confluent Platform
  • Read our documentation
  • Confluent Professional Services offers advice and help with deployments
  • Confluent Training can get your team ready for development and deployment of Confluent Platform

  The post Apache Kafka® vs. Enterprise Service Bus (ESB)—Friends, Enemies, or Frenemies? appeared first on Confluent.

  
  
  https://www.confluent.io/blog/apache-kafka-vs-enterprise-service-bus-esb-friends-enemies-or-frenemies/
• 

  June Preview Release: Packing Confluent Platform with the Features You Requested!
  ksql confluent platform confluent control center preview release • • sorangutan  

  1
  1
  Posts
  74
  Views

  

  We are very excited to announce Confluent Platform June 2018 Preview. This is our most feature-packed preview release for Confluent Platform since we started doing our monthly preview releases in April 2018.

  Read on to learn more, and remember to share your feedback to help improve the Apache Kafka ecosystem. You can do that by visiting the Confluent Community Slack channel (particularly the #ksql and #control-center channels) or by contributing to the KSQL project on GitHub, where you can file issues, submit pull requests, and contribute to discussions.

  Download The Preview Release

   

  Confluent Control Center

  Confluent Schema Registry Support for Topic Inspection

  Confluent Control Center’s topic inspection feature already supports JSON and string message formats, but starting with this release, it will have full support for Avro data through the Confluent Schema Registry.

  Schema Registry, as we all should know, is a component of Confluent Platform that stores a versioned history of all schemas and works with clients to support predictable schema evolution. To configure Schema Registry in Control Center, simply add confluent.controlcenter.schema.registry.url=<your_schema_registry_server> in control-center.properties file before starting up the application. The message viewer in the topic inspection tab will automatically detect and deserialize Avro records.

  

  Feature Access Control

  In this release, we’re also introducing feature access toggles within Confluent Control Center for KSQL, Topic Inspection, and Schema Registry. In many deployments, the end users of Control Center are not allowed to view records in topics for security and compliance reasons. In order to allow admins to control application-wide access to features that reveal topic data in Control Center, we added the option to control access at start-up time. When you restrict access to a feature, Control Center’s UI will reflect this change, and users cannot circumvent these protections in any way.

  To enable/disable a feature, change the following configs in the Control Center properties file before starting up:

  confluent.controlcenter.ksql.enable=[true|false]

  confluent.controlcenter.schema.registry.enable=[true|false]

  confluent.controlcenter.topic.inspection.enable=[true|false]

   

  KSQL

  Working with Nested Data

  One of the most requested enhancements for KSQL has been support for complex nested data types. Starting with the June preview release, you can now declare streams and tables with columns using a STRUCT data type in your CREATE STREAM and CREATE TABLE statements, and then access the internal fields of these columns in your SELECT queries as you do in any other expression. STRUCT support is available for both JSON and Avro data formats. Nested fields in the STRUCT can be any valid type in KSQL, including STRUCT,  ARRAY and MAP. For instance, you can have a column that is an ARRAY of STRUCTs, where one of the fields in the STRUCT is a MAP. This means KSQL now supports querying data with many nested levels.

  The following is an example of declaring an ORDERS stream, where the ADDRESS column has a STRUCT type with nested fields for CITY, STATE, and ZIPCODE:

  
  You can refer to fields in a STRUCT using the `->` operator:

   

  Beyond SQL with User Defined Functions

  Another highly requested feature for KSQL has been the ability to define your own functions that go beyond what’s provided by KSQL out of the box. User Defined Functions (UDFs) and User Defined Aggregate Functions (UDAFs) open the door for many novel use cases where you need to perform custom computations over your data. UDFs take one input row to produce one output row (e.g., ABS and SUBSTRING) whereas UDAFs take n input rows to produce one output row (e.g., SUM, MAX, COUNT).

  Using the new UDF and UDAF features,you can define custom computations and make them available to your KSQL queries in case the built-in scalar functions or the built-in aggregation functions are not sufficient.

  For now, UDF(A)Fs need to be implemented in Java. You simply write the function code and mark the classes with @Udf or @UdafFactory annotations. To make the UDF(A)Fs available to KSQL, you must create an uberjar of the UDF(A)Fs code including any transitive dependencies, and then place the jar file in the `ext/` folder of a KSQL Server. To better secure your KSQL deployment environment when using UDF(A)Fs you can also provide an optional blacklisting configuration that will prevent the use of certain Java classes and packages such as `java.io.*`. Finally, you can use the SHOW FUNCTIONS statement to get a listing of all available functions including UDFs and UDAFs.

  Here is an example UDF that returns the length of a string:

  
  Here is an example of calling the `STRING_LENGTH` function:

  
  Even more interesting is the ability to bring your own custom aggregation functions. Here we define a UDAF for computing the sum of string lengths:

  
  This UDAF can then be used in your KSQL queries:

   

  Stream-Stream and Table-Table Joins

  Beyond the existing Stream-Table joins, the June preview release introduces Stream-Stream and Table-Table joins.  For each of these joins, KSQL supports inner, full outer, and left join types. This means KSQL now covers all of the available join operations in Kafka Streams.

  A Stream-Stream join enables KSQL users to join two streams based on the desired message key within a given join window. Consider the following scenario where we have two streams, ORDERS and SHIPMENTS. Assuming that an order is late if it is shipped more than 2 hours after the order was placed, we can create a new stream for the late orders by joining the two streams and filtering the orders that were not shipped in the given time window. The following query creates the stream of late orders:

  The above query will match every shipment row with the order rows that are within the 2-hour window. If there is no match, the right-hand side of the join result will be NULL, indicating that the given order was not shipped within the expected time. Note that we used the `WITHIN` keyword in the Stream-Stream join to specify the join window.  

  Where to go from here

  Try out the new Confluent Platform June 2018 Preview release and share your feedback! Here’s what you can do to get started:

  • Download Confluent Platform June 2018 preview release.
  • Check out the Docker images for KSQL Server and KSQL CLI.
  • Visit the Confluent Community Slack channel to ask questions or share feedback. To discuss KSQL, visit the #ksql channel. For Control Center, visit the #control-center channel.
  • If you are interested in contributing to KSQL, visit the KSQL project on GitHub. Feel free to file issues, submit pull requests, and contribute to discussions there. Remember that KSQL is open source and already includes some great community contributions – the next one could be yours!

   

  The post June Preview Release: Packing Confluent Platform with the Features You Requested! appeared first on Confluent.

  
  
  https://www.confluent.io/blog/june-preview-release-confluent-plaform/
• 

  We ❤️ syslogs: Real-time syslog processing with Apache Kafka and KSQL—Part 3: Enriching events with external data
  ksql apache kafka stream processing mongodb kafka connect syslog • • sorangutan  

  1
  1
  Posts
  129
  Views

  

  Using KSQL, the SQL streaming engine for Apache Kafka®, it’s straightforward to build streaming data pipelines that filter, aggregate, and enrich inbound data. The data could be from numerous sources, including applications, log files, databases, and IoT and network devices. In the two previous articles (1 | 2) in this series I’ve shown how to use Apache Kafka and its Connect API to ingest syslog data from multiple sources. Using KSQL, it’s easy to filter it as well as use aggregations to drive simple anomaly detection. The output from KSQL stream processing is a Kafka topic, which can you stream with Kafka Connect to a data store such as S3, or use to drive applications directly such as push-notification alerting.

  

  In this article we’re going to conclude our fun with syslog data by looking at how we can enrich inbound streams of syslog data with reference information from elsewhere to produce a real-time enriched data stream. The syslog data in this example comes from various servers and network devices, and the additional information with which we’re going to enrich it is from MongoDB, which happens to be the datastore used by Ubiquiti network devices. With the enriched data we’re going to drive some real-time analytics through Elasticsearch and Kibana, as well as trigger push notifications based on activity of certain devices on the network.

  Streaming syslog data from Ubiquiti devices

  

  I’m a big fan of Ubiquiti’s network devices, including their Unifi access points. I could wax lyrical here as to why, but I will instead refer you to Troy Hunt’s thorough write-up of Ubiquiti. One of the nice characteristics of them is that there is a ton of configuration that you can do to do them, of the sort that typical consumer-grade networking equipment just doesn’t support.

  Configuring the Unifi controller to write syslog data for all devices is easily done through the GUI:

  

  The syslog server and port should be those where you’re running the Kafka Connect syslog agent (see here for setup instructions). Run the console consumer against the topic to validate that you’re getting data:

  $ kafka-avro-console-consumer \
   --bootstrap-server localhost:9092 \
   --property schema.registry.url=http://localhost:8081 \
   --topic syslog
  
  {"date":{"long":1522059085000},"facility":{"int":3},"host":{"string":"(\"U7PG2,xxxxxxxxxx,v3.7.40.6115\")"},"level":{"int":6},"message":{"string":"(\"U7PG2,xxxxxxxxxx,v3.7.40.6115\") hostapd: ath3: STA xx:xx:xx:xx:xx:xx IEEE 802.11: associated\n"},"charset":{"string":"UTF-8"},"remote_address":{"string":"/192.168.10.71:45584"},"hostname":{"string":"192.168.10.71"}}

  Working with Ubiquiti syslog data in Apache Kafka and KSQL

  From the KSQL prompt, we can inspect the topic, using PRINT:

  ksql> PRINT 'syslog';
  Format:AVRO
  3/4/18 9:14:35 AM GMT, /(/192.168.10.87:50442, {"date": 1520154874000, "facility": 3, "host": "asgard02", "level": 5, "message": "asgard02 syslogd[4134]: --- syslogd restarted ---\n", "charset": "UTF-8", "remote_address": "/192.168.10.87:50442", "hostname": "192.168.10.87"}
  3/4/18 9:14:35 AM GMT, /(/192.168.10.87:50442, {"date": 1520154874000, "facility": 5, "host": "asgard02", "level": 5, "message": "asgard02 syslogd[4134]: Configuration Notice:\n\tASL Module \"com.apple.cdscheduler\" claims selected messages.\n\tThose messages may not appear in standard system log files or in the ASL database.\n", "charset": "UTF-8", "remote_address": "/192.168.10.87:50442", "hostname": "192.168.10.87"}

  You’ll note that KSQL determines automagically that the data is in Avro format, and displays the timestamp, key, and message value. Declaring a KSQL stream on top of these syslog events is simple:

  ksql> CREATE STREAM SYSLOG WITH (KAFKA_TOPIC='syslog',VALUE_FORMAT='AVRO');

  Looking at the data some more, we can inspect the host names:

  ksql> SELECT HOST FROM SYSLOG WHERE HOST !='asgard02' LIMIT 50;
  ("U7PG2,f09fc2000000,v3.7.40.6115")
  rpi-03
  proxmox01
  asgard02
  ("BZ2,24a43c000000,v3.7.40.6115")

  Whilst most devices on my network use a standard hostname when sending their syslog data, Ubiquiti ones send a hostname that looks like this:

  (\"U7PG2,f09fc2000000,v3.7.40.6115\")

  Where this appears to be a concatenation of device model, MAC, and firmware version. I can use this ‘fingerprint’ to pick out all syslog traffic relating to these devices in KSQL:

  ksql> SELECT MESSAGE FROM SYSLOG WHERE HOST LIKE '(%';
  ("U7PG2,f09fc2000000,v3.7.40.6115") hostapd: ath0: STA 60:01:94:xx:xx:xx IEEE 802.11: disassociated
  ("U7PG2,f09fc2000000,v3.7.40.6115") kernel: [7142078.740000]  ieee80211_ioctl_kickmac[14643]
  ("U7PG2,f09fc2000000,v3.7.40.6115") libubnt[9577]: dpi.dpi_stainfo_notify(): dpi not enable
  ("U7PG2,f09fc2000000,v3.7.40.6115") libubnt[9577]: wevent.ubnt_custom_event(): EVENT_STA_LEAVE ath0: 60:01:94:xx:xx:xx / 0

  (NB: a better pattern would be (%,%,%) but KSQL currently only supports single wildcard LIKE predicates)

  

  Let’s persist this stream of Ubiquiti syslog data to make it easier to work with in subsequent querying and processing:

  ksql> CREATE STREAM UBNT_SYSLOG AS SELECT * FROM SYSLOG WHERE HOST LIKE '(%';
  
   Message
  ----------------------------
   Stream created and running
  ----------------------------
  ksql> DESCRIBE UBNT_SYSLOG;
  
   Field          | Type
  --------------------------------------------
   ROWTIME        | BIGINT           (system)
   ROWKEY         | VARCHAR(STRING)  (system)
   DATE           | BIGINT
   FACILITY       | INTEGER
   HOST           | VARCHAR(STRING)
   LEVEL          | INTEGER
   MESSAGE        | VARCHAR(STRING)
   CHARSET        | VARCHAR(STRING)
   REMOTE_ADDRESS | VARCHAR(STRING)
   HOSTNAME       | VARCHAR(STRING)
  --------------------------------------------
  For runtime statistics and query details run: DESCRIBE EXTENDED <Stream,Table>;
  ksql>

  From this stream, we can examine our data and look for certain conditions and events—using standard SQL predicates. For example, we can see devices connecting to an Access Point:

  ksql> SELECT MESSAGE from UBNT_SYSLOG WHERE MESSAGE LIKE '% associated%';
  ("U7PG2,f09fc2000000,v3.7.40.6115") hostapd: ath3: STA 38:f7:3d:xx:xx:xx IEEE 802.11: associated

  So this in itself it pretty neat, as we can filter in realtime the syslog data that’s coming in. But…how about being able to identify the access point and user device names? At the moment we can only see their MAC addresses. Read on to see how!

  Accessing Ubiquiti data from MongoDB via KSQL and Debezium

  

  To get additional metadata from our network devices, including things like the access point and user device names, we’re going to stream this data from its source into Kafka. Ubiquiti uses MongoDB to hold this data, and there’s a great change-data-capture (CDC) tool built on Kafka Connect called Debezium that supports several source databases—including MongoDB! We can use this to stream the current contents of the database into a Kafka topic, as well as all and every subsequent changes to that data. This means that in effect we maintain a live mirror of our MongoDB database in Kafka itself. We don’t need to do some kind of convoluted lookups from Kafka out to a database each time we want to look up some reference data, since we have now that reference data within Kafka itself.

  For the purposes of this blog, I cloned my MongoDB instance from the Ubiquiti controller onto a separate server. For a real-life example then we’d use the live controller itself – since working against a clone means that any new changes to the data (such as new access points being created) won’t be available to us.

  You can use a tool such as Robo 3T to explore the data that Ubiqiuti has within it:

  

  Setting up Debezium to stream the data from MongoDB is straightforward – you can follow the steps in this guide. After doing it, you’ll have a bunch of Kafka topics, each reflecting the data in each of the MongoDB collections. The two that we’re interested in for this article are:

  • device – metadata for access points
  • user – metadata for user devices (i.e. stuff that connects to the access points—phones, computers, etc)

  Let’s take a look at the data we’ve brought in, using KSQL. Create a stream over the inbound Device data topic (which is metadata about the access points), and use the EXTRACTJSONFIELD function to show specific fields:

  ksql> CREATE STREAM DEVICE_RAW (after VARCHAR) WITH (KAFKA_TOPIC='ubnt.ace.device',VALUE_FORMAT='AVRO');
  ksql> SELECT EXTRACTJSONFIELD(after,'$.name'),EXTRACTJSONFIELD(after,'$.ip') FROM DEVICE_RAW;
  Unifi AP - Study | 192.168.10.68
  Unifi AP - Attic | 192.168.10.67
  ubnt.moffatt.me | 77.102.5.159
  Unifi AP - Pantry | 192.168.10.71

  Now let’s declare all the columns of interest in our schema, and using CREATE STREAM AS SELECT (“CSAS”) generate an Avro topic based on the source stream. This resulting topic will hold not only the transformed data that’s currently on the source topic from MongoDB, but also any subsequent changes to that data.

  ksql> CREATE STREAM UBNT_DEVICE_SRC AS \
  SELECT EXTRACTJSONFIELD(after,'$._id') as id, \
  EXTRACTJSONFIELD(after,'$.name') as name, \
  EXTRACTJSONFIELD(after,'$.ip') as ip, \
  EXTRACTJSONFIELD(after,'$.mac') as mac, \
  EXTRACTJSONFIELD(after,'$.model') as model, \
  EXTRACTJSONFIELD(after,'$.version') as version \
  FROM DEVICE_RAW;
  
  ksql> SELECT id, name, ip, mac, model, version FROM UBNT_DEVICE_SRC;
  {"$oid":"58385328e4b001431e4e497a"} | Unifi AP - Study | 192.168.10.68 | xx:xx:xx:xx:xx:xx | BZ2 | 3.7.40.6115
  {"$oid":"583854cde4b001431e4e4982"} | Unifi AP - Attic | 192.168.10.67 | xx:xx:xx:xx:xx:xx | BZ2 | 3.7.40.6115
  {"$oid":"58b3fb48e4b0b79e50242621"} | ubnt.moffatt.me | 77.102.5.159 | xx:xx:xx:xx:xx:xx | UGW3 | 4.3.34.4943823
  {"$oid":"58b406f1e4b0e334d74c46e4"} | Unifi AP - Pantry | 192.168.10.71 | xx:xx:xx:xx:xx:xx | U7PG2 | 3.7.40.6115

  Because we want to join this data (which is, in effect, reference data) we need to ensure that the topic’s messages are keyed on the column on which we want to join, which is the IP address.

  Let’s see what the key currently is, courtesy of the system column ROWKEY:

  ksql> SELECT ROWKEY FROM UBNT_DEVICE_SRC;
  {"id":"{ \"$oid\" : \"58b3fb48e4b0b79e50242621\"}"}
  {"id":"{ \"$oid\" : \"58385328e4b001431e4e497a\"}"}
  {"id":"{ \"$oid\" : \"58b406f1e4b0e334d74c46e4\"}"}

  The source topic is keyed on the id field, which traces back to the source MongoDB collection. We want to rekey the topic to use ip instead. KSQL is a simple way to do this, using the PARTITION BY clause:

  ksql> CREATE STREAM UBNT_DEVICE_SRC_REKEY AS SELECT * FROM UBNT_DEVICE_SRC PARTITION BY IP;
  
   Message
  ----------------------------
   Stream created and running
  ----------------------------

  Let’s verify that the message key is now the same as the ip field in the message value:

  ksql> SELECT ROWKEY, ip FROM UBNT_DEVICE_SRC_REKEY;
  192.168.10.68 | 192.168.10.68
  77.102.5.159 | 77.102.5.159
  192.168.10.67 | 192.168.10.67

  Perfect! The final part of this is to take the resulting Kafka topic from this KSQL streaming transformation and use it as the basis for our KSQL table which will be joined to the syslog data.

  ksql> CREATE TABLE UBNT_DEVICE WITH (KAFKA_TOPIC='UBNT_DEVICE_SRC_REKEY',VALUE_FORMAT='AVRO',KEY='IP');

  It’s crucial that our table’s topic messages are keyed on the join column which we will be using, so let’s verify again that it is indeed the case in our new table:

  ksql> SELECT ROWKEY, IP, NAME FROM UBNT_DEVICE;
  192.168.10.68 | 192.168.10.68 | Unifi AP - Study
  192.168.10.71 | 192.168.10.71 | Unifi AP - Pantry
  192.168.10.67 | 192.168.10.67 | Unifi AP - Attic
  77.102.5.159 | 77.102.5.159 | ubnt.moffatt.me

  Q: Why did we create a STREAM of device data, and then a TABLE? When should we use STREAM and when should we use TABLE? A: Great question! Logically, we are using the device data as a TABLE. That is, we want to join an inbound stream of events to our device data in order to enrich it. We want to know for a given key, what the corresponding values are. For a given device row, what’s its name, it’s model, it’s version etc. So it is most definitely a TABLE. But, in order to join to a TABLE, that TABLE must be keyed on the join column. And as we saw from inspecting ROWKEY above, this was not the case. So we utilised KSQL’s powerful re-keying functionality to rekey the topic automagically. And to do that, we treat the inbound data as a STREAM. Why? Because it’s simply an inbound Kafka topic of events, partitioned on one column and on which we want to partition another. Each event (in this context, a change to the source devicedata on MongoDB) simply needs re-routing to the output topic with the new partitioning key.

  Joining Ubiquiti syslog events with access point reference data

  

  Having defined our inbound stream of Ubiquiti events (captured through Kafka Connect’s syslog connector), and device reference data sourced from MongoDB (snapshotted and changes both streamed through Kafka Connect from Debezium), we can join the two, which is remarkably straightforward to anyone familiar with SQL. Here we’re showing each syslog message, enriched with the name of the access point from which it originated:

  ksql> SELECT D.NAME, L.MESSAGE \
  FROM UBNT_SYSLOG L \
  LEFT JOIN UBNT_DEVICE D \
  ON L.HOSTNAME=D.IP;
  
  Unifi AP - Pantry | ("U7PG2,f09fc2000000,v3.7.40.6115") kernel: [5255656.050000] ieee80211_ioctl_set_ratelimit: node with aid 8 and mac cc:2d:b7:xx:xx:xx has been tagged non rate-limiting
  Unifi AP - Attic | ("BZ2,dc9fdbxxxxxx,v3.7.40.6115") hostapd: ath1: STA 3c:2e:f9:xx:xx:xx IEEE 802.11: associated
  Unifi AP - Attic | ("BZ2,dc9fdbxxxxxx,v3.7.40.6115") syslog: dpi.dpi_stainfo_notify(): dpi not enable

  In the syslog data from Ubiquiti is a whole wealth of events, many of them more low-level than we may be interested in. One of the things that is useful to track is user devices connecting to access points, and we can easily expose this here using a SQL predicate. We can also expose the timestamp at which this happened, made human-readable with the TIMESTAMPTOSTRING function:

  ksql> SELECT TIMESTAMPTOSTRING(L.ROWTIME, 'yyyy-MM-dd HH:mm:ss') AS CONNECT_TS, D.NAME, L.MESSAGE \
  FROM UBNT_SYSLOG L \
  LEFT JOIN UBNT_DEVICE D \
  ON L.HOSTNAME=D.IP \
  WHERE L.MESSAGE LIKE '% associated%';
  
  2018-03-28 09:37:33 | Unifi AP - Attic | ("BZ2,dc9fdbxxxxxx,v3.7.40.6115") hostapd: ath1: STA 3c:2e:f9:xx:xx:xx IEEE 802.11: associated
  2018-03-27 17:15:11 | Unifi AP - Study | ("BZ2,24a43cde91a0,v3.7.40.6115") hostapd: ath1: STA fc:a1:83:xx:xx:xx IEEE 802.11: associated

  So this is pretty cool, as we can now see as events occur more information about them—in this case the access point that a device is joining. Let’s persist it as a stream, and add in the derivation of the MAC address of the connecting device, extracted using the SUBSTRING function:

  ksql> CREATE STREAM UBNT_SYSLOG_AP_CONNECTS AS \
  SELECT D.NAME AS AP_NAME, \
  SUBSTRING(L.MESSAGE,53,70) as USER_MAC \
  FROM UBNT_SYSLOG L \
  LEFT JOIN UBNT_DEVICE D \
  ON L.HOSTNAME=D.IP \
  WHERE L.MESSAGE LIKE '% associated%';
  
   Message
  ----------------------------
   Stream created and running
  ----------------------------
  ksql> SELECT TIMESTAMPTOSTRING(ROWTIME, 'yyyy-MM-dd HH:mm:ss') , AP_NAME, USER_MAC \
  FROM UBNT_SYSLOG_AP_CONNECTS;
  2018-03-28 09:37:33 | Unifi AP - Attic | 3c:2e:f9:xx:xx:xx
  2018-03-27 17:15:11 | Unifi AP - Study | fc:a1:83:xx:xx:xx

  But what about the user device—the third column in the above output? All we have is a MAC address (obfuscated in the examples above). Wouldn’t it be nice to see the name of the device as well as the access point? Well, you know what’s coming now!

  Joining Ubiquiti syslog events with access point and device reference data

  

  In the same way as we saw above how to manipulate the inbound stream of snapshotted device (access point) data and subsequent changes (streamed from MongoDB via Debezium), we’ll do the same here for the device information, which is found in the user collection. The steps are:

  1. Declare a raw STREAM on source user MongoDB collection data, in which all data resides in the after field

     ksql> CREATE STREAM USER_RAW (AFTER VARCHAR) WITH (KAFKA_TOPIC='ubnt.ace.user',VALUE_FORMAT='AVRO');
     
      Message
     ----------------
      Stream created
     ----------------

  2. Declare a derived STREAM in which columns are extracted from the source after field

     ksql> CREATE STREAM UBNT_USER_SRC AS SELECT EXTRACTJSONFIELD(after,'$.mac') as mac, \
     EXTRACTJSONFIELD(after,'$.oui') as oui, \
     EXTRACTJSONFIELD(after,'$.name') as name, \
     EXTRACTJSONFIELD(after,'$.hostname') as hostname, \
     EXTRACTJSONFIELD(after,'$.is_guest') as is_guest \
     FROM USER_RAW;
     
      Message
     ----------------------------
      Stream created and running
     ----------------------------

  3. Re-key the STREAM by join column (MAC address)

     ksql> CREATE STREAM UBNT_USER_SRC_REKEY AS SELECT * FROM UBNT_USER_SRC PARTITION BY MAC;
     
      Message
     ----------------------------
      Stream created and running
     ----------------------------

  4. Declare a TABLE on top of the Kafka topic resulting from the re-key operation

     ksql> CREATE TABLE UBNT_USER WITH (KAFKA_TOPIC='UBNT_USER_SRC_REKEY',VALUE_FORMAT='AVRO',KEY='MAC');
     
      Message
     ---------------
      Table created
     ---------------

  5. Check that the message key (ROWKEY) matches the intended join column (MAC):

     ksql> SELECT ROWKEY, MAC, NAME FROM UBNT_USER;
     b8:27:eb:xx:xx:xx | b8:27:eb:xx:xx:xx | rpi-01.moffatt.me
     ac:bc:32:xx:xx:xx | ac:bc:32:xx:xx:xx | asgard
     1e:7e:60:xx:xx:xx | 1e:7e:60:xx:xx:xx | media02.moffatt.me
     fc:a1:83:xx:xx:xx | fc:a1:83:xx:xx:xx | Fire 01 (Red)
     60:01:94:xx:xx:xx | 60:01:94:xx:xx:xx | Attic lights plug
     00:04:20:xx:xx:xx | 00:04:20:xx:xx:xx | Squeezebox - Kitchen
     b8:ac:6f:xx:xx:xx | b8:ac:6f:xx:xx:xx | crashplan.moffatt.me

  Now we can join between Ubiquiti syslog events and reference information for both access points (persisted above in the UBNT_SYSLOG_AP_CONNECTS stream), and user devices (UBNT_USER):

  ksql> SELECT TIMESTAMPTOSTRING(L.ROWTIME, 'yyyy-MM-dd HH:mm:ss') AS CONNECT_TS, \
  L.AP_NAME, U.NAME AS USER_DEVICE_NAME , U.HOSTNAME as HOSTNAME, U.IS_GUEST  \
  FROM UBNT_SYSLOG_AP_CONNECTS L \
  LEFT JOIN UBNT_USER U \
  ON L.USER_MAC = U.MAC ;
  2018-06-11 15:05:06 | Unifi AP - Study | null | ItStillnsWhenIP | false
  2018-06-11 15:05:06 | Unifi AP - Study | null | ItStillnsWhenIP | false
  2018-06-11 15:13:46 | Unifi AP - Study | Fire 02 (Yellow) | amazon-ca6091cfe | false

  So every time a user’s device connects to an access point, we get to see the name of the access point, the name of the user device, and the type of the device. We’ll persist this as a stream, because we’ve not quite finished this exploration yet!

  ksql> CREATE STREAM UBNT_AP_USER_DEVICE_CONNECTS AS \
  SELECT TIMESTAMPTOSTRING(L.ROWTIME, 'yyyy-MM-dd HH:mm:ss') AS CONNECT_TS, \
  L.AP_NAME, U.NAME AS USER_DEVICE_NAME , U.HOSTNAME as HOSTNAME, U.IS_GUEST  \
  FROM UBNT_SYSLOG_AP_CONNECTS L \
  LEFT JOIN UBNT_USER U \
  ON L.USER_MAC = U.MAC ;

  

  Taking this enriched stream of data we can use a tool such as Elasticsearch with Kibana on top to provide an easy visualisation of the real-time data, as well as aggregate analysis based upon it:

  

  Streaming the Kafka topic to Elasticsearch is easy using the Kafka Connect Elasticsearch plugin. Take a look at the Elasticsearch mapping template and Kafka Connect configuration I am using.

  Enriched streams of data are valuable for analysis that we want to consume and look at, but even more valuable is event-driven alerting on conditions that we’re interested in. We saw in the previous article how we can use something like a simple Python script to drive push-based notifications in response to events on a Kafka topic. Using this pattern we can use a further KSQL expression to send notifications to a topic when a connection is made to an access point from a certain type of device. For example, if I wanted to track whenever my wifi-enabled plugs (yes, really!) reconnect to an access point, I can filter on the device type of “Espressi”:

  ksql> SELECT CONNECT_TS, AP_NAME, USER_DEVICE_NAME \
  FROM UBNT_AP_USER_DEVICE_CONNECTS \
  WHERE DEVICE_TYPE='Espressi';
  2018-03-04 14:07:43 | Unifi AP - Study | Wifi Plug - Sitting Room
  2018-03-04 14:07:48 | Unifi AP - Study | Attic lights plug
  2018-03-04 14:07:42 | Unifi AP - Study | Wifi Plug - Sitting Room
  2018-03-04 14:07:42 | Unifi AP - Study | Study light

  

  Persisting this to a target stream that the python application is listening to results in a nice push notification every time the device reconnects…

  CREATE STREAM KSQL_NOTIFY_DEVICE WITH (VALUE_FORMAT='JSON') AS \
  SELECT CONNECT_TS + '    ' + USER_DEVICE_NAME + ' connected to ' + AP_NAME AS TEXT, \
         'device_connections' AS CHANNEL \
  FROM UBNT_AP_USER_DEVICE_CONNECTS5 \
  WHERE DEVICE_TYPE='Espressi';

  …on both desktop:

  

  …and mobile

  

  Recap

  We’ve come a long way! Let’s remind ourselves what we’ve built:

  

  1. Streaming syslog events from multiple devices into Kafka
  2. Streaming contents and subsequent changes from MongoDB into Kafka
  3. Filtering syslog events to just include those from Ubiquiti devices
  4. Enriching Ubiquiti syslog events with both (i) access point name and (ii) user device name and device type
  5. Streaming enriched Ubiquiti syslog data to Elasticsearch for analysis in Kibana
  6. Filtering Ubiquiti syslog data to include only wifi connections from devices which are wifi-plugs
  7. Triggering push notification from Kafka topic to Slack

  Phew!

  Summary

  In this mini-series of blog articles we’ve seen the power of Apache Kafka—including Kafka Connect—and KSQL to build applications and analytics on Apache Kafka as a streaming platform. Using KSQL, it’s simple to create scalable real-time stream processing applications, using just SQL—no other coding required! Kafka Connect provides a configuration-file based method for powerful streaming integration between sources of data into Kafka, and from Kafka out to targets such as Elasticsearch.

  If you’re interested in what KSQL can do, check out the rest of this:

  • Download Confluent Platform and follow the Quick Start to get started with KSQL
  • Watch the introductory video at the KSQL home page
  • Check out the clickstream demo that’s available as part of KSQL’s GitHub repo
  • Questions? Ask them in the #ksql channel on our community Slack group.

  The post We ❤️ syslogs: Real-time syslog processing with Apache Kafka and KSQL—Part 3: Enriching events with external data appeared first on Confluent.

  
  
  https://www.confluent.io/blog/real-time-syslog-processing-apache-kafka-ksql-enriching-events-with-external-data/
• 

  The Role of Service Partners with Confluent Cloud
  • sorangutan  

  1
  1
  Posts
  10
  Views

  

  Through Confluent Cloud’s service partners, our users can easily build scalable data streaming applications with fully managed Apache Kafka, connectors, security, and more.
  
  https://www.confluent.io/blog/benefits-of-confluent-clouds-service-partners/
• 

  The 5 Tenets of Data Integration
  • sorangutan  

  1
  1
  Posts
  13
  Views

  

  Data integration is the process of combining data into a single, unified view. Learn how it works, its benefits, use cases, and 5 principles for getting started.
  
  https://www.confluent.io/blog/data-integration-benefits-challenges-considerations/
• 

  Top 4 Reasons Confluent Needs to Be on Your AWS re:Invent 2022 Agenda
  • sorangutan  

  1
  1
  Posts
  11
  Views

  

  Learn to leverage Confluent for real-time business intelligence, build streaming data pipelines in minutes, meet with industry Kafka leaders, and end the night with DJ’s and dancing.
  
  https://www.confluent.io/blog/visit-confluent-at-aws-reinvent-2022/
• 

  Finding Inclusion & Career Opportunity at Confluent
  • sorangutan  

  1
  1
  Posts
  18
  Views

  

  How Meshach Cisero (Senior Growth Marketing Manager) found not only career growth, but opportunities to add value to the entire organization.
  
  https://www.confluent.io/blog/diversity-inclusion-and-career-opportunities-at-confluent/
• 

  Connect Data From Legacy Databases Anywhere to MongoDB in Real Time With Confluent and Apache Kafka®
  • sorangutan  

  1
  1
  Posts
  15
  Views

  

  Connect any database to MongoDB using Confluent's cloud-native data streaming platform. Modernize any database, build streaming data pipelines, and empower real-time data in minutes.
  
  https://www.confluent.io/blog/connect-legacy-databases-to-stream-data-with-confluent/