Cloudera News

• 

  Integrating Machine Learning Models into Your Big Data Pipelines in Real-Time With No Coding
  • sorangutan  

  1
  1
  Posts
  48
  Views

  

  [Editor’s note: This article was originally published on the Hortonworks Community Connection, but reproduced here because CDSW is now available on both Cloudera and Hortonworks platforms.]

  Using Deployed Models as a Function as a Service

    

  Using Cloudera Data Science Workbench with Apache NiFi, we can easily call functions within our deployed models from Apache NiFi as part of flows. I am working against CDSW on HDP (https://www.cloudera.com/documentation/data-science-workbench/latest/topics/cdsw_hdp.html), 

  Read more

  The post Integrating Machine Learning Models into Your Big Data Pipelines in Real-Time With No Coding appeared first on Cloudera Engineering Blog.

  
  
  https://blog.cloudera.com/blog/2019/02/integrating-machine-learning-models-into-your-big-data-pipelines-in-real-time-with-no-coding/
• 

  Using Sqoop to Import Data from MySQL to Cloudera Data Warehouse
  • sorangutan  

  1
  1
  Posts
  59
  Views

  

  Cloudera Data Warehouse offers a powerful combination of flexibility and cost-savings. Using Cloudera Data Warehouse, you can transform and optimize your current traditional data warehouse by moving select workloads to your CDH cluster. This article shows you how to transform your current setup into a modern data warehouse by moving some initial data over to Impala on your CDH cluster.

  

  Prerequisites

  To use the following data import scenario,

  Read more

  The post Using Sqoop to Import Data from MySQL to Cloudera Data Warehouse appeared first on Cloudera Engineering Blog.

  
  
  https://blog.cloudera.com/blog/2019/02/using-sqoop-to-import-data-from-mysql-to-cloudera-data-warehouse/
• 

  Scalability Improvement of Apache Impala 2.12.0 in CDH 5.15.0
  • sorangutan  

  1
  1
  Posts
  53
  Views

  

  Key Takeaways

  We have significantly improved Impala in CDH 5.15.0 to address some of the scalability bottlenecks in query execution. 64 concurrent streams of TPC-DS queries at 10TB scale in a 135-node cluster now run at 6x query throughput compared to previous releases. In addition to running faster, the query success rate also improved from 73% to 100%. Overall, Impala in CDH 5.15.0 provides massive improvements in throughput and reliability while reducing the resource usage significantly.

  Read more

  The post Scalability Improvement of Apache Impala 2.12.0 in CDH 5.15.0 appeared first on Cloudera Engineering Blog.

  
  
  https://blog.cloudera.com/blog/2019/01/scalability-improvement-of-apache-impala-2-12-0-in-cdh-5-15-0/
• 

  Faster Swarms of Data : Accelerating Hive Queries with Parquet Vectorization
  • sorangutan  

  1
  1
  Posts
  44
  Views

  

  Background

  Apache Hive is a widely adopted data warehouse engine that runs on Apache Hadoop. Features that improve Hive performance can significantly improve the overall utilization of resources on the cluster. Hive processes data using a chain of operators within the Hive execution engine. These operators are scheduled in the various tasks (for example, MapTask, ReduceTask, or SparkTask) of the query execution plan. Traditionally, these operators are designed to process one row at a time.

  Read more

  The post Faster Swarms of Data : Accelerating Hive Queries with Parquet Vectorization appeared first on Cloudera Engineering Blog.

  
  
  https://blog.cloudera.com/blog/2018/12/faster-swarms-of-data-accelerating-hive-queries-with-parquet-vectorization/
• 

  What’s New in Cloudera Altus Director 6.1?
  • sorangutan  

  1
  1
  Posts
  55
  Views

  

  Cloudera Altus Director helps you deploy, scale, and manage Cloudera clusters on AWS, Microsoft Azure, or Google Cloud Platform. Altus Director both enables and enforces the best practices of big data deployments and cloud infrastructure. Altus Director’s enterprise-grade features deliver a mechanism for establishing production-ready clusters in the cloud for big data workloads and applications in a simple, reliable, automated fashion. In this post, you will learn about new functionality and changes in release 6.1.

  Read more

  The post What’s New in Cloudera Altus Director 6.1? appeared first on Cloudera Engineering Blog.

  
  
  https://blog.cloudera.com/blog/2018/12/whats-new-in-cloudera-altus-director-6-1/
• 

  Cloudera Enterprise 6.1.0 is Now Available
  • sorangutan  

  1
  1
  Posts
  58
  Views

  

  We are pleased to announce the general availability of Cloudera Enterprise 6.1.0, the modern platform for machine learning and analytics optimized for the cloud. This release delivers several new capabilities, improved usability, and better performance.

  As usual, the release includes a number of quality enhancements, bug fixes, and other improvements across the stack. Here is a partial list of what’s included (see the Release Notes for a full list):

  Data Engineering

  Cloudera Enterprise 6.1 now supports Spark Structured Streaming and enables micro-batch processing at ~100ms increments enabling ingest to query latencies in the Cloudera platform measured in seconds.

  Read more

  The post Cloudera Enterprise 6.1.0 is Now Available appeared first on Cloudera Engineering Blog.

  
  
  https://blog.cloudera.com/blog/2018/12/cloudera-enterprise-6-1-0-is-now-available/
• 

  Assessment of Apache Impala Performance using Cloudera Manager Metrics – Part 1 of 3
  metrics cdh performance cloudera manager impala apache impala queries tuning • • sorangutan  

  1
  1
  Posts
  100
  Views

  

  For a user-facing system like Apache Impala, bad performance and downtime can have serious negative impacts on your business. Given the complexity of the system and all the moving parts, troubleshooting can be time-consuming and overwhelming.

  In this blog post series, we are going to show how the charts and metrics on Cloudera Manager (CM) can help troubleshoot Impala performance issues. They can also help to monitor the system to predict and prevent future outages.

  Read more

  The post Assessment of Apache Impala Performance using Cloudera Manager Metrics – Part 1 of 3 appeared first on Cloudera Engineering Blog.

  
  
  https://blog.cloudera.com/blog/2018/12/assessment-of-apache-impala-performance-using-cloudera-manager-metrics-part-1-of-3/
• 

  Cloudera Altus Director SDX Integration
  cloud cdh altus cloudera director director data warehouse sdx • • sorangutan  

  1
  1
  Posts
  83
  Views

  

  Cloudera now provides a pathway for migrating an Altus Director managed cluster to a cluster that takes advantage of other Cloudera Altus cloud services. This blog post will outline how to do that.

  SDX for Cloudera Altus persists both Apache Hive metadata and Apache Sentry data access policies independently from clusters in SDX namespaces. In this way, SDX for Cloudera Altus provides the missing link to share Altus Director managed workload with Altus cloud services.

  Read more

  The post Cloudera Altus Director SDX Integration appeared first on Cloudera Engineering Blog.

  
  
  https://blog.cloudera.com/blog/2018/12/cloudera-altus-director-sdx-integration/
• 

  Learn for Free How to Deploy Cloudera Enterprise on Microsoft Azure
  • sorangutan  

  1
  1
  Posts
  63
  Views

  

  At Cloudera, we spend our time helping customers benefit from data. We help them with different types of data—structured, semi-structured, or raw unstructured. We also help them implement solutions for storing, tracing, securing, processing, enriching, analyzing, and visualizing it.

  Over the past several years, we’ve observed that customers are increasingly working with their data in the cloud, and Microsoft’s Azure cloud service is a popular deployment option. Cloudera University is pleased to announce a free course,

  Read more

  The post Learn for Free How to Deploy Cloudera Enterprise on Microsoft Azure appeared first on Cloudera Engineering Blog.

  
  
  https://blog.cloudera.com/blog/2018/11/learn-for-free-how-to-deploy-cloudera-enterprise-on-microsoft-azure/
• 

  Proactive Data Pipeline Alerting with Pulse
  events cdh search logs guest apache solr cloudera search phdata pulse • • sorangutan  

  1
  1
  Posts
  109
  Views

  

  In mid-2017, we were working with one of the world’s largest healthcare companies to put a new data application into production. The customer had grown through acquisition and in order to maintain compliance with the FDA, they needed to aggregate data in real-time from dozens of different divisions of the company. The consumers of this application, of course, did not care how we built the data pipeline. However, they cared greatly that if it broke,

  Read more

  The post Proactive Data Pipeline Alerting with Pulse appeared first on Cloudera Engineering Blog.

  
  
  http://blog.cloudera.com/blog/2018/11/proactive-data-pipeline-alerting-with-pulse/
• 

  Protecting Hadoop Clusters From Malware Attacks
  • sorangutan  

  1
  1
  Posts
  66
  Views

  

  Two new strains of malware–XBash and DemonBot–are targeting Apache Hadoop servers for Bitcoin mining and DDOS purposes. This malware is scanning the internet so vigorously for Hadoop clusters that an infection can occur within minutes of an insecure cluster being placed on the open internet. This blog post describes the mechanism this malware uses and offers specific actions to protect your Hadoop-based clusters.

  A History of Hadoop Malware

  Roughly two years ago there were a spate of attacks against the open source database solution MongoDB,

  Read more

  The post Protecting Hadoop Clusters From Malware Attacks appeared first on Cloudera Engineering Blog.

  
  
  https://blog.cloudera.com/blog/2018/11/protecting-hadoop-clusters-from-malware-attacks/
• 

  New in Cloudera Enterprise 6: Apache Hive 2.1
  cdh hive hive-on-spark • • sorangutan  

  1
  1
  Posts
  122
  Views

  

  We recently released Cloudera Enterprise 6.0 featuring significant improvements across a number of core components. In this blog post, we’re going to focus on Apache Hive 2.1.

  Hive’s Approach to Rebase: Stability and Quality Most Important

  Prior to the release of Cloudera Enterprise 6.0, Cloudera’s supported platform included Apache Hive 1.1 augmented with numerous features, enhancements and fixes from the later Apache Hive releases—all of which were included only after rigorous quality criteria were met.

  Read more

  The post New in Cloudera Enterprise 6: Apache Hive 2.1 appeared first on Cloudera Engineering Blog.

  
  
  http://blog.cloudera.com/blog/2018/10/new-in-cloudera-enterprise-6-apache-hive-2-1/
• 

  Network Security with Cloudera Altus and Apache Spot
  security altus platform security & cyber apache spot spot streamsets • • sorangutan  

  1
  1
  Posts
  137
  Views

  

  Introduction

  In the last few years, IT security threats to enterprise systems have increased, which has necessitated installing log ingestion and analysis solutions in any enterprise network.

  This blog post illustrates how Cloudera built its own scalable solution for log ingestion and analytics using Apache Spot and Cloudera Altus. By leveraging transient workloads in the cloud, Cloudera reduced the solution’s operational costs by 50% when compared to traditional, persistent cluster approaches.

  Use Case

  At Cloudera,

  Read more

  The post Network Security with Cloudera Altus and Apache Spot appeared first on Cloudera Engineering Blog.

  
  
  http://blog.cloudera.com/blog/2018/10/network-security-with-cloudera-altus-and-apache-spot/
• 

  Third-Party Libraries in C6
  security development cdh open source general platform security & cyber dependencies libraries • • sorangutan  

  1
  1
  Posts
  115
  Views

  

  Cloudera has put a significant amount of work into upgrading the third-party libraries used in our just-released C6 version. This major upgrade of our software has given us the opportunity to upgrade many of the libraries we use. These upgrades allow us to avoid security vulnerabilities, use modern versions of libraries, and to standardize versions of libraries across CDH.

  Modern software development relies on reusing other people’s code. Code reused in this fashion is called a “third-party library.”

  Read more

  The post Third-Party Libraries in C6 appeared first on Cloudera Engineering Blog.

  
  
  http://blog.cloudera.com/blog/2018/10/third-party-libraries-in-c6/
• 

  Cloudera Altus Cloud Services—You’re in Control (Part 2)
  aws cloud infrastructure altus • • sorangutan  

  1
  1
  Posts
  161
  Views

  

  In part 1 of this two-part series, we discussed how Cloudera Altus cloud services put you, the Cloudera Altus customer, in control of your cloud resources. In that blog post, we described the problems and solutions in a cloud provider independent manner and then showed the solutions with Azure. In this second blog post, we’ll look at the solutions using AWS.

  We previously introduced two control problems: the infrastructure construction problem (how do Cloudera Altus cloud services allow you to have precise control over how they build and manage a cluster for you);

  Read more

  The post Cloudera Altus Cloud Services—You’re in Control (Part 2) appeared first on Cloudera Engineering Blog.

  
  
  http://blog.cloudera.com/blog/2018/10/cloudera-altus-cloud-services-youre-in-control-part2/
• 

  Cloudera Altus Cloud Services—You’re in Control (Part 1)
  • sorangutan  

  1
  1
  Posts
  100
  Views

  

  Cloudera Altus is a suite of products which enable you to deploy and consume Cloudera Enterprise clusters in the public cloud environment of your choice. With Cloudera Altus cloud services such as Altus Data Engineering or Altus Data Warehouse Cloudera creates and manages clusters on your behalf, using your cloud provider’s infrastructure resources, freeing you up to focus on your workloads and their business value. Cloudera Altus cloud services creates resources and manages them in your account,

  Read more

  The post Cloudera Altus Cloud Services—You’re in Control (Part 1) appeared first on Cloudera Engineering Blog.

  
  
  http://blog.cloudera.com/blog/2018/10/cloudera-altus-cloud-services-youre-in-control-part1/
• 

  What’s New in Cloudera Altus Director 6.0?
  cloud altus cloudera director director administration • • sorangutan  

  1
  1
  Posts
  146
  Views

  

  Cloudera Altus Director 6.0 adds support for deploying CDH 6 clusters on public cloud infrastructure. Altus Director also features improved integration with SDX configured namespaces, allows encryption of its configuration properties, and includes a tool for copying its database. Altus Director’s interfaces have been streamlined, removing deprecated features and old API endpoints.

  Cloudera Altus Director, formerly known as Cloudera Director, helps you deploy, scale, and manage Cloudera clusters in AWS,

  Read more

  The post What’s New in Cloudera Altus Director 6.0? appeared first on Cloudera Engineering Blog.

  
  
  http://blog.cloudera.com/blog/2018/09/whats-new-in-cloudera-altus-director-6-0/
• 

  Support Best Practices Video
  customer success best practices communication video customer experience support • • sorangutan  

  1
  1
  Posts
  103
  Views

  

  Cloudera Support recently received Service Capability & Performance (SCP) certification for the third consecutive year. In order to earn this industry-leading mark of distinction, a support organization must demonstrate an ongoing commitment to exemplary standards and methodologies in over 100 major performance areas. At the heart of this achievement is the fundamental mission of Cloudera Support: Enable total customer success.

  Cloudera strongly believes that timely, accurate, and healthy communication has to exist between the customer and our support organization in order to result in a successful outcome for both parties.

  Read more

  The post Support Best Practices Video appeared first on Cloudera Engineering Blog.

  
  
  http://blog.cloudera.com/blog/2018/09/support-best-practices-video/
• 

  Robust Message Serialization in Apache Kafka Using Apache Avro, Part 3
  cdh avro streaming use case kafka how-to • • sorangutan  

  1
  1
  Posts
  120
  Views

  

  Part 3: Configuring Clients

  Earlier, we introduced Kafka Serializers and Deserializers that are capable of writing and reading Kafka records in Avro format. In this part we will going to see how to configure producers and consumers to use them.

  Setting up a Kafka Topic for use as a Schema Store

  KafkaTopicSchemaProvider works with a Kafka topic as its persistent store. This topic will contain at most thousands of records: the schemas. It does not need multiple partitions,

  Read more

  The post Robust Message Serialization in Apache Kafka Using Apache Avro, Part 3 appeared first on Cloudera Engineering Blog.

  
  
  http://blog.cloudera.com/blog/2018/08/robust-message-serialization-in-apache-kafka-using-apache-avro-part-3/
• 

  Robust Message Serialization in Apache Kafka Using Apache Avro, Part 2
  cdh avro streaming use case kafka how-to • • sorangutan  

  1
  1
  Posts
  171
  Views

  

  Implementing a Schema Store

  In Part 1, we saw the need for an Apache Avro schema provider but did not implement one. In this part we will implement a schema provider that works with Apache Kafka as storage.

  In-Memory SchemaStore

  First we can implement an in-memory store for schemas. This is useful to understand the requirements for such a store and as the cache of the Kafka backed store. A SchemaStore has to be quick in looking up VersionedSchema entries.

  Read more

  The post Robust Message Serialization in Apache Kafka Using Apache Avro, Part 2 appeared first on Cloudera Engineering Blog.

  
  
  http://blog.cloudera.com/blog/2018/07/robust-message-serialization-in-apache-kafka-using-apache-avro-part-2/