Emerging Architectures for Modern Data Infrastructure by Matt Bornstein, Jennifer Li, Martin Casado via @future

This detailed look at the modern data stack was initially created in 2020 and has been recently updated to show how things are evolving. Covers best-in-class architectures for both analytic and operational systems and lays out a hypothesis for why specific changes are happening. The article came out of discussions with dozens of practitioners.

I found this very interesting - and the fact that it highlights the changes from the original version gives us all a great idea as to how things have changed. Use it for discussion in your own teams and what needs to be changed.

Data Mesh & Its Distributed Data Architecture by Yash Mehta via @kdnuggets

Going forward, data professionals have found a new way to address the scalability of sources through data mesh.

I like that this is distributed and that you do not need to build a huge data warehouse or data lake.

WEBINAR: Unlocking Data Value Faster with a Modern Open Data Architecture - 20 January 2022

 

See Dremio Cloud in action and ask our product experts questions. Join Wednesdays at 10:00 AM PT. Register Here.           Unlocking Data Value Faster with a Modern Open Data Architecture       REGISTER             Thursday, January 20, 2022   9:00 a.m. PT 12:00 p.m. ET         Speaker         Matt Aslett VP & Research Director Ventana Research Billy Bosworth Chief Executive Officer Dremio         Ivan Alvarez VP Big Data and Enterprise Analytics NCR Corporation       Hi, A truly modern open data architecture means you can unleash the query engine/s of your choice on the whole dataset in the public cloud of your choice, allowing access for the right data personas in a highly price-performant way. That's how a modern and open architecture shortens the value chain of data while also accelerating the innovation associated with rich open ecosystems. BI and analytics goals can now be achieved directly on this open data architecture - no need to extract, transform, and load (ETL/ELT) the data into proprietary data warehouses. Recent advancements such as these modern & open technologies are making it feasible for organizations that don't have armies of data engineers to use these building blocks of the modern data stack:    Semantic Layer to easily unlock data access for data consumers such as data analysts, business analysts, and data scientists     Apache Arrow Flight, an open source data connectivity technology that provides 10X faster data transfer rates than ODBC, JDBC, and pyodbc     Apache Iceberg that came out of a need Netflix had for a more consistent, performant, and end-user friendly table format for their Amazon S3 environment     Project Nessie, which Dremio open sourced that extends and leverages table formats such as Iceberg, bringing multi-table transactions and Git-like version control to the data lake

Data modelling vs. data architecture: What's the difference? by/via David Loshin

Data modellers and data architects have distinctly different roles, but they work in a complementary fashion to help enterprises unlock and capitalize on data's business value.

It's a pretty important job definition difference and a key one for people to understand.

The 3 Things to Keep in Mind While Building the Modern Data Stack by Raghotham Murthy via @TDataScience

Building Your Data Stack is Confusing. It Doesn’t Have to Be.

I really enjoyed reading this and it should be quite helpful to read this when you are thinking of creating a data stack.

What happened to Hadoop? by @derrickharris via @Medium

It was the next big thing...until it wasn’t. Derrick Harris explains, “Hadoop’s path to ubiquity intersected a host of other technology shifts that as a whole would prove to be more impactful in the long run, in part by peeling off the most valuable promises of big data and making them more consumable.”

Definitely, a question that needed to be answered!  Thank you Derrick for the great answer - to thank him give him applause and a follow.

GDPR compliance the perfect opportunity to modernise data architecture by Amandeep Khurana via @infomgmt

Compliance with the data privacy and security mandate enables organisations to become more agile in their product and service development and rollouts, and more efficient and effective in their ability to respond to market trends and competitive threats.

Yes this is exactly right - everything has to turn onto it's head and be data centric not application centric. I think we need to concentrate on:

WHERE is the data created
WHERE is it also stored (so where is it interfaced to)
HOW it is updated
WHAT changes when it is updated
HOW do you delete the data in ALL systems?

I would suggest you do something like a data flow diagram so you can document all of this for every piece of data.

Mastering data architecture to enable digital transformation by Joshua Satten and Nicolas Papadakos via @infomgmt

The information model is the gas powering the engine, enabling organisations to more effectively communicate and reach their specific goals.

Some good points. I would add to it that you need to try and balance the needs of the business, the availability of data and as mentioned in the article you need to focus of granularity in order to get that right.  Business knowledge is key.  Also be aware that documentation can sometimes we scarce so be prepared to have to generate your own for some older data/systems.

19 top paying Internet-of-Things jobs by Bob Violino via @infomgmt

The Internet of Things remains one of the hottest trends in technology. Here's how the demand is translating to salaries for experienced professionals.

Good if you want to know what skills and roles you need to aim towards in your own career.

From Lambda to Kappa: A Guide on Real-Time Big Data Architectures by Michael Verrilli via @DZone

This is a discussion of real-time big data architectures as there are options now.

A great article and a huge help in understanding the architectures. I'm really looking forward to his follow up articles where he goes into them in more detail.

The information management challenges of data monetisation by David M. Raab via @infomgmt

Organisations must adjust to blurred lines between known and anonymous customer identity information.

I agree with this article - nowadays the data available will contain data that can but also cannot be connected to known data. All organisations need to have a strategy on this data . I recall in the past the unknown data was ignored and deleted.  Nowadays this is missing a valuable source of information and potential customers.  My advice would be to have default values for all fields so that you do not have any empty fields and do not find that joins will not work or have to be outer ones.  That way you can use everything as much as you can.

Data Lineage Demystified: The What, Why, and How by Michelle Knight via @Dataversity

Trusting Big Data requires understanding its Data Lineage. Without Data Lineage, Big Data becomes synonymous with the last phrase in a game of telephone.

Michelle is right - you have to know the system of record and the data flow for the data that you are using, and you need to know that for ALL data that you use.  You need to also understand the quality of that data and what to do when values are missing (preferably that should never happen but we all live in the real world with legacy systems).

SLIDESHOW: Data Pay Days: The 12 Top Paying Noncertified Data Skills via @infomgmt

Data Pay Days: The 12 Top Paying Noncertified Data Skills by David Wedon via +Information Management - There seems to be no end in sight when it comes to high demand for data professionals. But just how well an individual can cash in on that trend depends on their job experience, location, and acquired skills. This week we look at how all those factors impact the paycheck. Today we review the pay premiums being paid in 2016 for the top noncertified data skills.