Senior Big Data & Databricks Engineer - PySpark, Hadoop & Real-Time Analytics Job at Artmac Soft LLC, Atlanta, GA

YnpWbXNXTnVRTUZTYnlhRmViajYxZzkwS1E9PQ==
  • Artmac Soft LLC
  • Atlanta, GA

Job Description

Who We Are


Artmac Soft is a technology consulting and service-oriented IT company dedicated to providing innovative technology solutions and services to customers.



Job Description


Job Title : Senior Big Data & Databricks Engineer - PySpark, Hadoop & Real-Time Analytics

Job Type : C2C


Experience : 10-12 years


Location : Atlanta, New York

Responsibilities :

  • Proven experience as a Big Data Engineer with expertise in the Hadoop ecosystem and real-time analytics.
  • Strong proficiency in Spark (PySpark/Scala), Hive, and related big data technologies.
  • Experience in building and deploying data pipelines and stream processing applications.
  • Familiarity with job scheduling and optimization techniques within the Hadoop ecosystem.
  • Solid understanding of Unix/Linux environments, including Shell scripting.
  • Experience with cloud technologies, specifically Azure, is preferred.
  • Excellent problem-solving skills and the ability to work independently and collaboratively in a fast-paced environment.
  • Experience with MongoDB or similar NoSQL databases.
  • Knowledge of data warehousing concepts and methodologies.
  • Familiarity with machine learning frameworks and applications.
  • Collaborate with business stakeholders to gather and understand requirements, translating them into technical specifications for data solutions.
  • Design, develop, and implement complex data pipelines utilizing technologies such as PySpark, Scala Spark, Hive, Hadoop CLI, MapReduce, Storm, Kafka, and Lambda Architecture.
  • Create and submit Spark jobs while ensuring high-performance tuning and scalability of data processes.
  • Work with real-time stream processing technologies, including Spark Structured Streaming and Kafka, to deliver timely insights.
  • Leverage expertise in Python/Spark and their related libraries and frameworks to optimize data processing tasks.
  • Manage job scheduling challenges in Hadoop, ensuring reliability and efficiency in data workflows.
  • Develop and execute unit and integration testing for data pipelines, handling large data volumes to derive actionable insights.
  • Optimize code for efficiency to meet stipulated SLAs, focusing on performance and resource management.
  • Demonstrate strong Unix/Linux expertise, comfortable with the Linux operating system and Shell Scripting.
  • Utilize Azure cache to enhance performance and speed of data processing tasks.

Qualification:

  • Bachelor's degree in Computer Science, Information Security, or a related field.

Job Tags

Similar Jobs

Nesco Resource

Janitorial Job at Nesco Resource

 ...Job Summary: The Janitorial position consists of keeping the building in a clean and orderly condition by performing cleaning duties such as cleaning floors, washing walls, glass, removing and disposing of trash. It also consists of cleaning snow or debris from sidewalks... 

State Utility Contractors

Concrete Finisher (Plant Division) Job at State Utility Contractors

 ...Position DescriptionCandidates for the Concrete Finisher position should have the skills and knowledge to perform the following:-Place, vibrate, level and finish concrete surfaces are the focal job duty. Check the concrete forms for appropriate depths and angles... 

Christian Career

Pastor of Worship - Eastview Christian Church Job at Christian Career

 ...Pastor of Worship -Eastview Christian Church Position Scope: Full-time; Sunday-Thursday Position Status: Exempt Position Summary: The Pastor of Worship oversees the ministry operations of the department, promotes the spiritual and professional development... 

Gold Star Feed And Grain LLC

CDL Class A - Feed Delivery Truck Driver Job at Gold Star Feed And Grain LLC

 ...Class A driver needed to organize, perform and complete the delivery of commercial bulk and bagged feeds in a safe and efficient manner. Operate truck in a safe and courteous manner. Safety is first in importance in every delivery task. Maintain truck to include... 

Lignetics Inc.

CDL Driver Job at Lignetics Inc.

 ...Description: Job Type: Full-time CDL Class A Driver Hours: Monday - Friday 6 AM tp 4 PM Occasional Saturday work Compensation: Based on experience - $27-31per hour CDL Driver Job Description Lignetics, Inc. is in search of a skilled, Full-time CDL...