1/3/2024 0 Comments Redshift ra3 pricing![]() ![]() ![]() The following chart shows what the profile of our test set looked like. It should be noted that we only took into consideration SELECT and FETCH query types (to simplify this first stage of performance tests). A set of queries from the production cluster – This set can be reconstructed from the Amazon Redshift logs ( STL_QUERYTEXT) and enriched by metadata ( STL_QUERY).A reference cluster snapshot – This ensures that we can replay any tests starting from the same state.This strategy aims to replicate a realistic workload in different RA3 cluster configurations and compare them with our DC2 configuration. The following figures illustrate the approach we took to evaluate the performance of RA3. To assess the nodes and find an optimal RA3 cluster configuration, we collaborated with AllCloud, the AWS premier consulting partner. In order to be confident with the performance of the RA3 nodes, we decided to stress test them in a controlled environment before making the decision to migrate. In this section, we discuss how we conducted a performance evaluation of RA3 nodes with an Amazon Redshift cluster. Evaluating the performance of Amazon Redshift clusters with RA3 nodes This cluster’s performance was generally good, ETL (extract, transform, and load) and interactive queries barely had any queue time, and 80% of them would finish in under 5 minutes. 10,000 queries per hour, 30 queries in parallel.250 monthly active users, consistently increasing.Our data warehouse had the following configuration before the migration: We saw this as a temporary solution, and we mainly did it to buy some time to explore other cost-effective alternatives, such as RA3 nodes.Īmazon Redshift RA3 nodes along with Redshift Managed Storage (RMS) provided separation of storage and compute, enabling us to scale storage and compute separately to better meet our business requirements. However, this solution wasn’t cost-efficient enough because we were adding compute capacity to the cluster even though computation power was underutilized. Overall, we reached a cluster size of 18 nodes. Our usual method to solve storage problems used to be to simply increase the number of nodes. Almost every day we would get an alert that our disk space was close to 100%, which was about 40 TB worth of data. However, we kept facing challenges with storage demand due to having more users, more data sources, and more prepared data. We started with a small Amazon Redshift cluster of 7 DC2.8xlarge nodes, and as its popularity and adoption increased inside the OLX Group data community, this cluster naturally grew.īefore migrating to RA3, we were using a 16 DC2.8xlarge nodes cluster with a highly tuned workload management (WLM), and performance wasn’t an issue at all. Here at OLX Group, Amazon Redshift has been our choice for data warehouse for over 5 years. Hopefully you can learn from our experience in case you are considering doing the same. In this post, we share how we modernized our Amazon Redshift data warehouse by migrating to RA3 nodes and how it enabled us to achieve our business expectations. ![]() As such, it’s crucial to maintain a cluster with high availability and performance while also being storage cost-efficient. Our data warehouse is built using Amazon Redshift and is used by multiple internal teams to power their products and data-driven business decisions. This scenario is very familiar to us in OLX Group. However, a common data warehouse issue with ever-growing volumes of data is storage limitations and the degrading performance that comes with it. These analyses are often done using data warehouses. We live in a data-producing world, and as companies want to become data driven, there is the need to analyze more and more data. We help people buy and sell cars, find housing, get jobs, buy and sell household goods, and much more. OLX Group is one of the world’s fastest-growing networks of online marketplaces, operating in over 30 countries around the world. This is a guest post by Miguel Chin, Data Engineering Manager at OLX Group and David Greenshtein, Specialist Solutions Architect for Analytics, AWS. ![]()
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |