Cloudera News

• 

  CDH6.2 – Cloudera Search Attribute Based Access Control Part 2
  • sorangutan  

  1
  1
  Posts
  15
  Views

  

  Cloudera Search is a highly scalable and flexible search solution based on Apache Solr which enables exploration, discovery and analytics over massive, unstructured and semi-structured datasets (for example logs, emails, dna-strings, claims forms, jpegs, xls sheets, etc). It has been adopted by a large number of Cloudera customers across a wide range of industries for high ROI and SLA-bound workloads, with many of those having strict requirements around security and compliance.

  In CDH6.2 we introduce two new features to Cloudera Search relating to document-level security.

  Read more

  The post CDH6.2 – Cloudera Search Attribute Based Access Control Part 2 appeared first on Cloudera Engineering Blog.

  
  
  https://blog.cloudera.com/blog/2019/06/cdh6-2-cloudera-search-attribute-based-access-control-part-2/
• 

  CDH6.2 – Cloudera Search Attribute Based Access Control Part 1
  • sorangutan  

  1
  1
  Posts
  13
  Views

  

  Cloudera Search is a highly scalable and flexible search solution based on Apache Solr which enables exploration, discovery and analytics over massive, unstructured and semi-structured datasets (for example logs, emails, dna-strings, claims forms, jpegs, xls sheets, etc). It has been adopted by a large number of Cloudera customers across a wide range of industries for high ROI and SLA-bound workloads, with many of those having strict requirements around security and compliance.

  In CDH 5.1.0 we introduced document-level security for Cloudera Search using Apache Sentry.

  Read more

  The post CDH6.2 – Cloudera Search Attribute Based Access Control Part 1 appeared first on Cloudera Engineering Blog.

  
  
  https://blog.cloudera.com/blog/2019/05/cdh6-2-cloudera-search-attribute-based-access-control-part-1/
• 

  The Case for MirrorMaker 2
  • sorangutan  

  1
  1
  Posts
  13
  Views

  

  Apache Kafka has become an essential component of enterprise data pipelines and is used for tracking clickstream event data, collecting logs, gathering metrics, and being the enterprise data bus in a microservices based architectures. Kafka is essentially a highly available and highly scalable distributed log of all the messages flowing in an enterprise data pipeline. Kafka supports internal replication to support data availability within a cluster. However, enterprises require that the data availability and durability guarantees span entire cluster and site failures.

  Read more

  The post The Case for MirrorMaker 2 appeared first on Cloudera Engineering Blog.

  
  
  https://blog.cloudera.com/blog/2019/05/the-case-for-mirrormaker-2/
• 

  Small Files, Big Foils: Addressing the Associated Metadata and Application Challenges
  • sorangutan  

  1
  1
  Posts
  36
  Views

  

  Small files are a common challenge in the Apache Hadoop world and when not handled with care, they can lead to a number of complications. The Apache Hadoop Distributed File System (HDFS) was developed to store and process large data sets over the range of terabytes and petabytes. However, HDFS stores small files inefficiently, leading to inefficient Namenode memory utilization and RPC calls, block scanning throughput degradation, and reduced application layer performance. In this blog post,

  Read more

  The post Small Files, Big Foils: Addressing the Associated Metadata and Application Challenges appeared first on Cloudera Engineering Blog.

  
  
  https://blog.cloudera.com/blog/2019/05/small-files-big-foils-addressing-the-associated-metadata-and-application-challenges/
• 

  Partition Management in Hadoop
  • sorangutan  

  1
  1
  Posts
  40
  Views

  

  Guest blog post written by Adir Mashiach

  In this post I’ll talk about the problem of Hive tables with a lot of small partitions and files and describe my solution in details.

  

  A little background

  In my organization,  we keep a lot of our data in HDFS. Most of it is the raw data but a significant amount is the final product of many data enrichment processes.

  Read more

  The post Partition Management in Hadoop appeared first on Cloudera Engineering Blog.

  
  
  https://blog.cloudera.com/blog/2019/05/partition-management-in-hadoop/
• 

  Fine-Grained Authorization with Apache Kudu and Impala
  • sorangutan  

  1
  1
  Posts
  14
  Views

  

  Apache Impala supports fine-grained authorization via Apache Sentry on all of the tables it manages including Apache Kudu tables. Given Impala is a very common way to access the data stored in Kudu, this capability allows users deploying Impala and Kudu to fully secure the Kudu data in multi-tenant clusters even though Kudu does not yet have native fine-grained authorization of its own. This solution works because Kudu natively supports coarse-grained (all or nothing) authorization which enables blocking all access to Kudu directly except for the impala user and an optional whitelist of other trusted users.

  Read more

  The post Fine-Grained Authorization with Apache Kudu and Impala appeared first on Cloudera Engineering Blog.

  
  
  https://blog.cloudera.com/blog/2019/04/fine-grained-authorization-with-apache-kudu-and-impala/
• 

  Demystifying Spark Jobs to Optimize for Cost and Performance
  • sorangutan  

  1
  1
  Posts
  16
  Views

  

  Apache Spark is one of the most popular engines for distributed data processing on Big Data clusters. Spark jobs come in all shapes, sizes and cluster form factors. Ranging from 10’s to 1000’s of nodes and executors, seconds to hours or even days for job duration, megabytes to petabytes of data and simple data scans to complicated analytical workloads. Throw in a growing number of streaming workloads to huge body of batch and machine learning jobs —

  Read more

  The post Demystifying Spark Jobs to Optimize for Cost and Performance appeared first on Cloudera Engineering Blog.

  
  
  https://blog.cloudera.com/blog/2019/04/demystifying-spark-jobs-to-optimize-for-cost-and-performance/
• 

  A Guide to Learning with Limited Labeled Data
  • sorangutan  

  1
  1
  Posts
  39
  Views

  

  This was originally published on the Fast Forward Labs blog

  We are excited to release Learning with Limited Labeled Data, the latest report and prototype from Cloudera Fast Forward Labs.

  Being able to learn with limited labeled data relaxes the stringent labeled data requirement for supervised machine learning. Our report focuses on active learning, a technique that relies on collaboration between machines and humans to label smartly.

  Read more

  The post A Guide to Learning with Limited Labeled Data appeared first on Cloudera Engineering Blog.

  
  
  https://blog.cloudera.com/blog/2019/04/a-guide-to-learning-with-limited-labeled-data/
• 

  What’s New in Cloudera Altus Director 6.2?
  • sorangutan  

  1
  1
  Posts
  32
  Views

  

  Cloudera Altus Director helps you deploy, scale, and manage Cloudera clusters on AWS, Microsoft Azure, or Google Cloud Platform. Altus Director both enables and enforces the best practices of big data deployments and cloud infrastructure. Altus Director’s enterprise-grade features deliver a mechanism for establishing production-ready clusters in the cloud for big data workloads and applications in a simple, reliable, automated fashion. In this post, you will learn about new functionality and changes in release 6.2.

  Read more

  The post What’s New in Cloudera Altus Director 6.2? appeared first on Cloudera Engineering Blog.

  
  
  https://blog.cloudera.com/blog/2019/04/whats-new-in-cloudera-altus-director-6-2/
• 

  What’s new in the Hue Data Warehouse Editor in Cloudera 6.2
  • sorangutan  

  1
  1
  Posts
  31
  Views

  

  Self-service exploratory analytics is one of the most common use cases we see by our customers running on Cloudera’s Data Warehouse solution.

  With the recent release of Cloudera 6.2, we continue to improve the end user query experience with Hue, focusing on easier SQL query troubleshooting and increased compatibility with Hive. Read on to learn more and try it out in one-click at demo.gethue.com.

  Easier SelfService Query Troubleshooting

  Hue has great assistance for finding tables in the Data Catalog and getting recommendations on how to write (better) queries with the smart autocomplete,

  Read more

  The post What’s new in the Hue Data Warehouse Editor in Cloudera 6.2 appeared first on Cloudera Engineering Blog.

  
  
  https://blog.cloudera.com/blog/2019/04/whats-new-in-the-hue-data-warehouse-editor-in-cloudera-6-2/
• 

  Testing Apache Kudu Applications on the JVM
  • sorangutan  

  1
  1
  Posts
  35
  Views

  

  Although the Kudu server is written in C++ for performance and efficiency, developers can write client applications in C++, Java, or Python. To make it easier for Java developers to create reliable client applications, we’ve added new utilities in Kudu 1.9.0 that allow you to write tests using a Kudu cluster without needing to build Kudu yourself, without any knowledge of C++, and without any complicated coordination around starting and stopping Kudu clusters for each test.

  Read more

  The post Testing Apache Kudu Applications on the JVM appeared first on Cloudera Engineering Blog.

  
  
  https://blog.cloudera.com/blog/2019/03/testing-apache-kudu-applications-on-the-jvm/
• 

  Transparent Hierarchical Storage Management with Apache Kudu and Impala
  • sorangutan  

  1
  1
  Posts
  45
  Views

  

  When picking a storage option for an application it is common to pick a single storage option which has the most applicable features to your use case. For mutability and real-time analytics workloads you may want to use Apache Kudu, but for massive scalability at a low cost you may want to use HDFS. For that reason, there is a need for a solution that allows you to leverage the best features of multiple storage options.

  Read more

  The post Transparent Hierarchical Storage Management with Apache Kudu and Impala appeared first on Cloudera Engineering Blog.

  
  
  https://blog.cloudera.com/blog/2019/03/transparent-hierarchical-storage-management-with-apache-kudu-and-impala/
• 

  SMM 1.2 Released with Powerful New Alerting and Topic Lifecycle Management Features with Schema Registry Integration
  • sorangutan  

  1
  1
  Posts
  52
  Views

  

  [Editor’s note: Now that the recent merger is complete, the Cloudera Engineering blog will expand to cover products, such as this, originally developed for the Hortonworks platform. Please stay tuned for future product announcements regarding availability of these products on the Cloudera platform.]

  Since the release of Streams Messaging Manager (SMM) at the end of last summer, our customers have started to cure the Kafka Blindness within their organizations by using SMM to monitor their Kafka clusters and streaming microservices applications.

  Read more

  The post SMM 1.2 Released with Powerful New Alerting and Topic Lifecycle Management Features with Schema Registry Integration appeared first on Cloudera Engineering Blog.

  
  
  https://blog.cloudera.com/blog/2019/02/new-kafka-alert-management-features-in-smm-12/
• 

  Enable MKL Library for Spark ML Applications: Methods and Benchmarks
  • sorangutan  

  1
  1
  Posts
  52
  Views

  

  Spark ML is one of the dominant frameworks for many major machine learning algorithms, such as the Alternating Least Squares (ALS) algorithm for recommendation systems, the Principal Component Analysis algorithm, and the Random Forest algorithm. However, the complexity of configuring it optimally  means that frequently, Spark ML is underutilized.. Using native math libraries for Spark ML can help unlock the full potential of Spark ML.

  This article discusses how to accelerate model training speed by using native libraries for Spark ML.

  Read more

  The post Enable MKL Library for Spark ML Applications: Methods and Benchmarks appeared first on Cloudera Engineering Blog.

  
  
  https://blog.cloudera.com/blog/2019/02/enable-mkl-library-for-spark-ml-applications-methods-and-benchmarks/
• 

  Integrating Machine Learning Models into Your Big Data Pipelines in Real-Time With No Coding
  • sorangutan  

  1
  1
  Posts
  48
  Views

  

  [Editor’s note: This article was originally published on the Hortonworks Community Connection, but reproduced here because CDSW is now available on both Cloudera and Hortonworks platforms.]

  Using Deployed Models as a Function as a Service

    

  Using Cloudera Data Science Workbench with Apache NiFi, we can easily call functions within our deployed models from Apache NiFi as part of flows. I am working against CDSW on HDP (https://www.cloudera.com/documentation/data-science-workbench/latest/topics/cdsw_hdp.html), 

  Read more

  The post Integrating Machine Learning Models into Your Big Data Pipelines in Real-Time With No Coding appeared first on Cloudera Engineering Blog.

  
  
  https://blog.cloudera.com/blog/2019/02/integrating-machine-learning-models-into-your-big-data-pipelines-in-real-time-with-no-coding/
• 

  Using Sqoop to Import Data from MySQL to Cloudera Data Warehouse
  • sorangutan  

  1
  1
  Posts
  58
  Views

  

  Cloudera Data Warehouse offers a powerful combination of flexibility and cost-savings. Using Cloudera Data Warehouse, you can transform and optimize your current traditional data warehouse by moving select workloads to your CDH cluster. This article shows you how to transform your current setup into a modern data warehouse by moving some initial data over to Impala on your CDH cluster.

  

  Prerequisites

  To use the following data import scenario,

  Read more

  The post Using Sqoop to Import Data from MySQL to Cloudera Data Warehouse appeared first on Cloudera Engineering Blog.

  
  
  https://blog.cloudera.com/blog/2019/02/using-sqoop-to-import-data-from-mysql-to-cloudera-data-warehouse/
• 

  Scalability Improvement of Apache Impala 2.12.0 in CDH 5.15.0
  • sorangutan  

  1
  1
  Posts
  53
  Views

  

  Key Takeaways

  We have significantly improved Impala in CDH 5.15.0 to address some of the scalability bottlenecks in query execution. 64 concurrent streams of TPC-DS queries at 10TB scale in a 135-node cluster now run at 6x query throughput compared to previous releases. In addition to running faster, the query success rate also improved from 73% to 100%. Overall, Impala in CDH 5.15.0 provides massive improvements in throughput and reliability while reducing the resource usage significantly.

  Read more

  The post Scalability Improvement of Apache Impala 2.12.0 in CDH 5.15.0 appeared first on Cloudera Engineering Blog.

  
  
  https://blog.cloudera.com/blog/2019/01/scalability-improvement-of-apache-impala-2-12-0-in-cdh-5-15-0/
• 

  Faster Swarms of Data : Accelerating Hive Queries with Parquet Vectorization
  • sorangutan  

  1
  1
  Posts
  43
  Views

  

  Background

  Apache Hive is a widely adopted data warehouse engine that runs on Apache Hadoop. Features that improve Hive performance can significantly improve the overall utilization of resources on the cluster. Hive processes data using a chain of operators within the Hive execution engine. These operators are scheduled in the various tasks (for example, MapTask, ReduceTask, or SparkTask) of the query execution plan. Traditionally, these operators are designed to process one row at a time.

  Read more

  The post Faster Swarms of Data : Accelerating Hive Queries with Parquet Vectorization appeared first on Cloudera Engineering Blog.

  
  
  https://blog.cloudera.com/blog/2018/12/faster-swarms-of-data-accelerating-hive-queries-with-parquet-vectorization/
• 

  What’s New in Cloudera Altus Director 6.1?
  • sorangutan  

  1
  1
  Posts
  55
  Views

  

  Cloudera Altus Director helps you deploy, scale, and manage Cloudera clusters on AWS, Microsoft Azure, or Google Cloud Platform. Altus Director both enables and enforces the best practices of big data deployments and cloud infrastructure. Altus Director’s enterprise-grade features deliver a mechanism for establishing production-ready clusters in the cloud for big data workloads and applications in a simple, reliable, automated fashion. In this post, you will learn about new functionality and changes in release 6.1.

  Read more

  The post What’s New in Cloudera Altus Director 6.1? appeared first on Cloudera Engineering Blog.

  
  
  https://blog.cloudera.com/blog/2018/12/whats-new-in-cloudera-altus-director-6-1/
• 

  Cloudera Enterprise 6.1.0 is Now Available
  • sorangutan  

  1
  1
  Posts
  57
  Views

  

  We are pleased to announce the general availability of Cloudera Enterprise 6.1.0, the modern platform for machine learning and analytics optimized for the cloud. This release delivers several new capabilities, improved usability, and better performance.

  As usual, the release includes a number of quality enhancements, bug fixes, and other improvements across the stack. Here is a partial list of what’s included (see the Release Notes for a full list):

  Data Engineering

  Cloudera Enterprise 6.1 now supports Spark Structured Streaming and enables micro-batch processing at ~100ms increments enabling ingest to query latencies in the Cloudera platform measured in seconds.

  Read more

  The post Cloudera Enterprise 6.1.0 is Now Available appeared first on Cloudera Engineering Blog.

  
  
  https://blog.cloudera.com/blog/2018/12/cloudera-enterprise-6-1-0-is-now-available/