The Information Lab

Data Processing Solutions - The Information Lab Nederland

Geschreven door Priyadarshini Ramabadran | Sep 21, 2021 5:34:33 PM

There are 2 data processing solutions:

  1. Transaction Systems (OLTP)
  2. Analytical Systems (OLAP)

Online Transaction Processing AKA OLTP is a process where data from different sources such as banking activity, retail checkout etc., and stores the transaction data into a database.

Online Analytical Processing AKA OLAP is a process applies complex queries to large amounts of historical data, aggregated from OLTP databases and other sources, for data mining, analytics, and business Intelligence projects

OLTP OLAP
Handles day to day transactions that results from enterprise operations Analysis of information in the database for the purpose of making management decisions.
Small, discrete unit of work Big picture view of the information held in DB
Often high volume Can be high/low depending on the requirement
Data is processed very quickly Seconds, minutes, or hours depending on the amount of data to process
Control and run essential business operations in real time Plan, solve problems, support decisions, discover hidden insights
Normalized databases for efficiency Denormalized databases for analysis

DATA INGESTION

  • Getting the data from multiple data sources and storing it into one centralised location is called DATA INGESTION. It is the process of obtaining and importing data.
  • This data can arrive as continuous stream or batches.
  • Raw data can be stored in DBMS as files or other forms for fast, easily accessible storage.
  • Data Ingestion might perform
    • Filtering: Eg, Reject suspicious, corrupt or duplicated data
    • Simple Transformation: Converting data into standard form

NOTE: During ingestion we can make only simple transformations and not complex critical transformation

DATA PROCESSING

  • Data processing takes data in raw form, cleans it and converts it into a more meaningful format.
  • The result is a databases /data warehouse that you can use to perform queries and generate visualisations.
  • Data Processing might perform
    • Data Cleaning: Removing anomalies, applying filters and transformations
    • Data Wrangling: Capture, filter, clean, combine and aggregate data

NOTE: We can make more complex transformation. Eg: Azure synapse Analytics is used to store cleaned and transformed data

DATA EXPLORATION

  • Data exploration is the process of trying to put together the pieces of puzzle in the journey to find a message in the data working with feedback cycles of defining hypotheses, analyzing data, and visualizing results.
  • Exploration is a deep dive analysis of data in search for better insights
  • The explored data can be visualized using tools such as Tableau, Power BI etc.,
Volledig bericht weergeven