Data Engineering Training

Course Duration : 5 Weeks

Level : All Levels

Instructor-Led Training : Yes

Live Classroom: Yes

Live-Online : Yes

Data Engineering

This course will nurture and transform you into a highly-skilled data engineer who builds and maintains data structures and architectures for data ingestion, processing, and deployment for large-scale, data-intensive applications. It’s a promising career for both new and experienced professionals with a passion for data, including graduates from all streams.

Vetted curriculum by Industry experts

Real-Time Experienced Trainers

Hands-On Learning

Real-Time Experienced Trainers

Weeks Duration

Hr/Week Therory

Hr/Week Lab

Students

Introduction To Data Engineering

Students Big Data Introduction, Hadoop Introduction
What is Big Data

Bigdata Analytics

Bigdata Challenges

Technologies for Bigdata

What is Hadoop?

History of Hadoop

Basic Concepts

Future of Hadoop

The Hadoop Distributed File System

Future of Hadoop

Breakthroughs of Hadoop

Breakthroughs of Hadoop

Apache Hadoop

Cloudera Hadoop (CDH)

Horton Networks Hadoop (HDP)

MapR Hadoop (mapR)

AWS – EMR
Hadoop Daemon Processes, HDFS (Hadoop Distributed File System)
Hadoop Architecture

High Availability

Name Node

DataNode

Secondary Name Node

Job Tracker/Resource Manager

Task Tracker/Node Manager

Blocks and Input Splits

Data Replication

Hadoop Rack Awareness

Hadoop Cluster Architecture and Block Placement

Accessing HDFS

CLI Approach

HDFS basic file operations

Basic Administration commands
Hadoop Installation Modes, YARN
Local Mode

Pseudo-distributed Mode

Fully distributed mode

What is YARN

How YARN Works

Resource Manager

Node Manager

Application Master

Containers

Advantages of YARN

Fail Over Mechanizm
Big Data On Cloud, Course Deliverables
Introduction to Cloud – Models, Service Categories, AWS security, IAM

AWS platform

EC2

S3

Databases on AWS

AWS EMR

Workshop style coaching

Interactive approach

Course material

Hands on practice exercises for each topic

Quiz at the end of each major topic

Tips and techniques on Certification Examinations

Linux concepts and basic commands on demand
Introduction to Spark, Spark Basics
What is Spark?

Spark Overview

Setting up PySpark environment

Using Spark Shell

Resilient Distributed Datasets (RDDs)

Spark Context

Spark Ecosystem

In-Memory Computations in Spark
Working with RDDs
Creating, Loading and Saving RDD

Transformations on RDD

Actions on RDD

Key-Value Pair Transformation on RDDs

RDD Partitioning

RDD Persistence
Spark Streaming
Spark Streaming with DStreamAPI

Spark Streaming Architecture

Spark Streaming Transformations

Stateless

Stateful Transformations

Rolling Window and Check pointing

Integrating Spark with Kafka Streaming Data

Structured Streaming

Integrating Spark with Twitter Streaming Data

Spark Streaming Performance Considerations

Structured Streaming

Structured Streaming Overview

Advantages of Structured Streaming

Other Streaming system Vs. Structured Streaming

Stateful operations

Spark Structured Streaming

Output Modes

Spark Structured Streaming Example

Window API

Event Time

Late Events

Watermark

Structured Streaming and Kafka integrations

Integrating Spark with Twitter Streaming Data

Spark Streaming Performance Considerations
Performance Tuning and Debugging
Shared Variables: Broadcast Variables

Shared Variables: Accumulators

Common Performance Issues

Performance Tuning Tips

Spark WebUI

Monitoring Driver and Executor Logs

Testimonial's

I have completed the Data Engineering Training at Invictus. I would like to express my gratitude to the trainer for their knowledge and expertise which has helped me gain a strong understanding of the subject matter. Thank you, Invictus

Aman Y

This was a very good hands-on data engineering cloud learning experience. The training sessions and detailed course curriculum were both helpful. The trainer did a great job of covering each and every topic with detailed hands-on sessions

Aditya R

I found the data engineering sessions to be both informative and interactive. I learned about the in-depth workings of data engineering on the cloud and got to do several hands-on assignments with the help of the trainer. I found the whole experience to be very valuable and would recommend it to others.

Vidhi A

Sathyajith N

The training was excellent. The explanations were clear and easy to understand. The trainer did a great job of explaining the theory and then demonstrating it with practical examples. This was really helpful in learning and understanding Data Engineering on cloud.

Arya

This is a very comprehensive training. it deserves 5 stars. To anyone who wants to get this course, I totally recommend it. It was a brilliant presentation that explicitly focused on time-series analysis

Isaac

Start your journey to becoming an IT expert with our comprehensive courses