Spark Programs

A collection of Apache Spark programs and practical examples created for learning and practicing Spark concepts using Scala.

This repository focuses specifically on Apache Spark and its commonly used APIs and operations for data processing.

📚 Spark Concepts Covered

Spark Fundamentals

• SparkSession
• SparkContext
• RDD basics
• Spark architecture and execution concepts

DataFrames

• Creating DataFrames
• Selecting columns
• Filtering data
• Adding and modifying columns
• Renaming columns
• Dropping columns
• Removing duplicate records
• Sorting data
• Combining DataFrames

Transformations

• select()
• filter()
• where()
• withColumn()
• withColumnRenamed()
• drop()
• distinct()
• dropDuplicates()
• union()
• Other commonly used DataFrame transformations

Actions

• show()
• count()
• collect()
• first()
• take()
• Other commonly used Spark actions

Aggregations

• groupBy()
• agg()
• sum()
• min()
• max()
• avg()
• count()

Joins

• Inner Join
• Left Join
• Right Join
• Full Outer Join
• Left Semi Join
• Left Anti Join
• Cross Join

Window Functions

• row_number()
• rank()
• dense_rank()
• lag()
• lead()
• Window specifications and partitioning

Conditional & Built-in Functions

• when() / otherwise()
• String functions
• Date and timestamp functions
• Null handling functions
• Mathematical functions
• Other commonly used Spark functions

Spark SQL

• Creating temporary views
• Running SQL queries using Spark
• SQL-based DataFrame processing

🎯 Purpose

This repository is created to:

• Practice Apache Spark programming
• Understand Spark DataFrame operations
• Learn transformations and actions
• Practice aggregations and joins
• Work with Spark window functions
• Strengthen practical Spark skills
• Prepare for Apache Spark and Big Data development roles

🛠️ Technology

• Apache Spark
• Scala

👨‍💻 Author

Ajay Somarthy