Large companies have a lot of data available with them, however most of this is in silos. Building pipelines for these and getting insightful analytics from this is a time consuming process with the delays being costly from a business perspective. Our client has designed a solution that makes it possible for every company to do advanced analysis without the overhead that comes with traditional data infrastructure.Their solution is an end-to-end enterprise data platform for analysts.
Primary project location: Israel
About Sela Software Labs Ltd
MORE (by Sela group) is an international team of Multi-Cloud Experts, delivering the best Multi-Cloud services to its customers and helping them grow.
Enterprise Data Company
We, as partners, helped modernize their data management solution. They wanted to migrate from AWS Redshift to BigQuery for better scale and we helped them with design for the solution.
As the client’s customer base increased, the amount of data they needed to handle also began increasing. They were looking for a solution where the overhead of managing data and customer access to their data would be much reduced. Since each customer accessed their data independently, it was critical to managing customer data access securely. They were also looking at ways to scale quickly and this would be possible only with a solution that would require minimal to no configuration.
The obvious solution was to use BigQuery for the data warehouse. BigQuery is a data warehouse as a service , which means there is no absolutely no configuration or monitoring needed. It can handle large amounts of data and produce results in milliseconds. From a security point of view, each customer needed to be given access only to their data and the safest way to do this was to let each customer have their own datasets in their own projects in BigQuery. While one project was created to help manage the admin processes, for each customer a separate project was created. Creation of projects and bigquery datasets as well as imports into bigquery were all automated using terraform. This made onboarding customers very simple and at the same time, offered complete security.
Google Cloud Features Used:
|Compute Engine||Cloud IAM|
The end result was an easy-to-manage, secure and scalable solution. The solution resulted in overall cost savings at the infrastructure level and also due to operational efficiency. Automation of asset creation and no-management services such as BigQuery make management and maintenance very efficient. So the main areas of benefit were
- Cost saving – BigQuery charges are based on data storage and query time.
- Operational Efficiency – achieved by automation of asset creation
- Scalability – achieved by use of BigQuery that is a managed data warehouse