GCP Data Engineering Training in Hyderabad – Kalyan IT Hub

• The need for cloud computing in modern businesses.
• Key features and offerings of Google Cloud Platform (GCP).
• Overview of core GCP services and products.
• Benefits and advantages of using cloud infrastructure.
• Step-by-step guide to creating a free-tier account on GCP.

✅Console

• Navigating the GCP Console
• Configuring the GCP Console for Efficiency
• Using the GCP Console for Service Management

✅ Shell

• Introduction to GCP Shell
• Command-line Interface (CLI) Basics
• GCP Shell Commands for Service Deployment and Management

✅SDK

• Overview of GCP Software Development Kits (SDKs)
• Installing and Configuring SDKs
• Writing and Executing GCP SDK Commands

✅Regions

• Understanding GCP Regions
• Selecting Regions for Service Deployment
• Impact of Region on Service Performance

✅Zones

• Exploring GCP Zones
• Distributing Resources Across Zones
• High Availability and Disaster Recovery Considerations

✅Importance

• Significance of Choosing the Right Location
• Global vs. Regional Resources
• Factors Influencing Location Decisions

✅ Identities

• Introduction to Identity and Access Management (IAM)
• Users, Groups, and Service Accounts
• Best Practices for Identity Management

✅ Roles

• GCP IAM Roles Overview
• Defining Custom Roles
• Role-Based Access Control (RBAC) Implementation

✅ Policy

• Resource-based Policies
• Understanding and Implementing Organization Policies
• Auditing and Monitoring Policies

✅ Resource Hierarchy

• GCP Resource Hierarchy Structure
• Managing Resources in a Hierarchy
• Organizational Structure Best Practices

✅Getting started with Linux

✅Linux Installation

✅Basic Linux Commands

• Cloud shell tips
• File and Directory Operations (ls, cd, pwd, mkdir, rmdir, cp, mv, touch, rm, nano)
• File Content Manipulation (cat, less, head, tail, grep)
• Text Processing (awk, sed, cut, sort, uniq)
• User and Permission related (whoami, id, su, sudo, chmod, chown)

✅ Data Types

• Strings
• Operators
• Numbers (Int, Float)
• Booleans

✅Data Structures

• Lists
• Tuples
• Dictionaries
• Sets

✅ Python Programming Constructs

• if, elif, else statements
• for loops, while loops
• Exception Handling
• File I/O operations

✅ Modular Programming in Python

• Functions & Lambda Functions
• Classes

• Overview of Cloud Storage as a scalable and durable object storage service.
• Understanding buckets and objects in Cloud Storage.
• Use cases for Cloud Storage, such as data backup, multimedia storage, and website
  content.
• Creating and managing Cloud Storage buckets.
• Uploading and downloading objects to and from Cloud Storage.
• Setting access controls and permissions for buckets and objects.
• Data Transfer and Lifecycle Management
• Versioning and Object Versioning
• Integration with Other GCP Services
• Implementing best practices for optimizing Cloud Storage performance.
• Securing data in Cloud Storage with encryption and access controls.
• Monitoring and logging for Cloud Storage operations.

• Introduction to Cloud SQL
• Creating and Managing Cloud SQL Instances
• Configuring database settings, users, and access controls.
• Connecting to Cloud SQL instances using Cloud SQL studio, Shell, Workbenches
• Importing and exporting data in Cloud SQL.
• Backups and High Availability
• Integration with Other GCP Services
• Managing database user roles and permissions.
• Introduction to DMS
• End to End Database migration Project 

‣Offline: Export and Import method
‣Online: DMS method

• Introduction to BigQuery
• BigQuery Architecture
• Use cases for BigQuery in business intelligence and analytics.
• Various method of creating table in BigQuery
• BigQuery Data Sources and File Formats
• Native table and External Tables
• SQL Queries and Performance Optimization
• Writing and optimizing SQL queries in BigQuery.
• Understanding query execution plans and best practices.
• Partitioning and clustering tables for performance.

✅ Data Integration and Export

• Loading data into BigQuery from Cloud Storage, Cloud SQL, and other sources.
• Exporting data from BigQuery to various formats.
• Real-time data streaming into BigQuery.

✅Configuring access controls and permissions in BigQuery.

✅ BigQuery Views:

• Views
• Materialized Views
• Authorized Views

✅ Integration with Other GCP Services

• Integrating BigQuery with Dataflow for ETL processes.
• Building data pipelines with BigQuery and Composer.

✅ Case Study-1: Spotify

✅ Case Study-2: Social Media 

✅Introduction to Hadoop and Apache Spark

✅Understanding the difference between Spark and MapReduce

✅What is Spark and Pyspark.

✅Understanding Spark framework and its functionalities

✅Overview of DataProc as a fully managed Apache Spark and Hadoop service.

✅Use cases for DataProc in data processing and analytics.

✅Cluster Creation and Configuration

• Creating and managing DataProc clusters.
• Configuring cluster properties for performance and scalability.
• Preemptible instances and cost optimization. 

✅Running Jobs on DataProc

• Submitting and monitoring Spark and Hadoop jobs on DataProc.
• Use of initialization actions and custom scripts.
• Job debugging and troubleshooting.

✅ Integration with Storage and BigQuery

• Reading and writing data from/to Cloud Storage and BigQuery.
• Integrating DataProc with other storage solutions.
• Performance optimization for data access.

✅ Automation and scheduling of recurring jobs.

✅ Case Study-1: Data Cleaning of Employee Travel Records

✅End to End Batch Pyspark pipeline using Dataproc, BigQuery, GCS 

✅ What is Databricks lakehouse platform

✅ Databricks architecture and components

✅ Setting up and Administering a Databricks workspace

✅ Managing data with Delta Lake

✅ Databricks Unity Catalog

✅ Note books and clusters

✅ ELT with Spark SQL and Python

✅ optimize performance within Databricks.

✅ Incremental Data Processing

✅ Delta Live tables

✅ Case study: creating end to end workflows

✅ Introduction to DataFlow

✅ Use cases for DataFlow in real-time analytics and ETL.

✅ Understanding the difference between Apache Spark and Apache Beam

✅ How Dataflow is different from Dataproc

✅ Building Data Pipelines with Apache Beam 

• Writing Apache Beam pipelines for batch and stream processing.
• Custom Pipelines and Pre-defined pipelines
• Transformations and windowing concepts.

✅ Integration with Other GCP Services

• Integrating DataFlow with BigQuery, Pub/Sub, and other GCP services.
• Real-time analytics and visualization using DataFlow and BigQuery.
• Workflow orchestration with Composer.

✅ End to End Streaming Pipeline using Apache beam with Dataflow, Python app, PubSub,
BigQuery, GCS.

✅ Template method of creating pipelines

✅ Introduction to Pub/Sub

✅ Understanding the role of Pub/Sub in event-driven architectures.

✅ Key Pub/Sub concepts: topics, subscriptions, messages, and acknowledgments.

✅ Creating and Managing Topics and Subscriptions

• Using the GCP Console to create Pub/Sub topics and subscriptions.
• Configuring message retention policies and acknowledgment settings.

✅ Publishing and Consuming Messages 

• Writing and deploying code to publish messages to a topic.
• Implementing subscribers to consume and process messages from subscriptions. 

✅ Integration with Other GCP Services

• Connecting Pub/Sub with Cloud Functions for serverless event-driven computing.
• Integrating Pub/Sub with Dataflow for real-time stream processing.

✅ Streaming use-case using Dataflow

✅ Introduction to Composer/Airflow

✅Overview of Airflow Architecture

✅ Use cases for Composer in managing and scheduling workflows.

✅ Creating and Managing Workflows

• Creating and configuring Composer environments.
• Defining and scheduling workflows using Apache Airflow.
• Monitoring and managing workflow executions. 

✅ Integration with Data Engineering Services

• Orchestrating workflows involving BigQuery, DataFlow, and other services.
• Coordinating ETL processes with Composer.
• Integrating with external systems and APIs. 

✅ Error Handling and Troubleshooting

• Handling errors and retries in Composer workflows.
• Debugging and troubleshooting failed workflow executions.
• Logging and monitoring for Composer workflows.

✅ Level-1-DAG: Orchestrating the BigQuery pipelines

✅ Level-2-DAG: Orchestrating the DataProc pipelines

✅ Level-3-DAG: Orchestrating the Dataflow pipelines

✅ Implementing CI/CD in Composer Using Cloud Build and GitHub 

✅ Introduction to Data Fusion 

• Overview of Data Fusion as a fully managed data integration service.
• Use cases for Data Fusion in ETL and data migration.

✅ Building Data Integration Pipelines

• Creating ETL pipelines using the visual interface.
• Configuring data sources, transformations, and sinks.
• Using pre-built templates for common integration scenarios.

✅ Integration with GCP and External Services

• Integrating Data Fusion with BigQuery, Cloud Storage, and other GCP services.

✅ End to End pipeline using Data fusion with Wrangler, GCS, BigQuery

✅ Cloud Functions Introduction

✅ Setting up Cloud Functions in GCP

✅ Event-driven architecture and use cases

✅ Writing and deploying Cloud Functions

✅ Triggering Cloud Functions:

• HTTP triggers
• Pub/Sub triggers
• Cloud Storage triggers

✅ Monitoring and logging Cloud Functions

✅  Usecase-1: Loading the files from GCS to BigQuery as soon as it is uploaded.

✅ Terraform Introduction

✅ Installing and configuring Terraform.

✅ Infrastructure Provisioning

✅ Terraform basic commands

• Init, plan, apply, destroy

✅ Create Resources in Google Cloud Platform

• GCS buckets
• Dataproc cluster
• BigQuery Datasets and tables
• And more resources as needed