Category: Technology

Technology

Data Engineering Series – The Architecture

It is given that, Big data eco-system is the life blood of most organizations. But it is also true that there are a ton of…

Rob Stokes December 15, 2020

0 Comments

Recent Comments

No comments to show.

Guide, Grow and Uplift

Donate
FAQs
Privacy Policy
Terms of Service
Contact Us
About Us

Forum Description

It is given that, Big data eco-system is the life blood of most organizations. But it is also true that there are a ton of organizations that face daunting challenges when it comes to handling Big Data correctly. Reflecting back on my experience over the years in fortune 500 companies including the most stringent of the environments where "data is the company" as well as having talked to many small and large companies in the industry, opportunity to improve the big data ecosystem are huge.

Do you find yourself asking the following questions...

Technology - Engineering Challenges... Old arcane architecture (Hadoop?), running in your data center? Scaling, performance, maintainability, Data Quality, Data Governance and Observability issues? List of technology challenges is very long...
People - Too many people doing the same thing and still cant get it right?
Process - Manual processes and as a result delayed and inaccurate data? or communication break down between organizations such that the ownership of data is not clear?

Good news is Big Data, Machine Learning, Data Science ecosystem has evolved tremendously in recent years and there are tools that handle pretty much any problem in Data Engineering. But like they say... with power comes responsibility and with so many tools comes the complexity. If you misuse or pick the wrong tool for the job, you are in constant conundrum of data hell.

In this series of articles covering Data Engineering, I am planning on covering various technologies and components of a very generic data ecosystem architecture.

Without further ado... Fig. 1 covers a very high level end to end modern data ecosystem. I will cover more details and commentary on the components and technologies in more detail in coming articles.

Fig 1. Generic End to End Architecture for Data Ecosystem

Always welcome thoughts.

Report

There was a problem reporting this post.

Harassment Harassment or bullying behavior

Inappropriate Contains mature or sensitive content

Misinformation Contains misleading or false information

Offensive Contains abusive or derogatory content

Suspicious Contains spam, fake content or potential malware

Other

Block Member?

Please confirm you want to block this member.

You will no longer be able to:

See blocked member's posts
Mention this member in posts
Invite this member to groups
Message this member
Add this member as a connection

Please note: This action will also remove this member from your connections and send a report to the site admin. Please allow a few minutes for this process to complete.

Report

You have already reported this .