Data Engineer - Databricks

Bachelor degree

Key Responsibilities:

• Collaborate with stakeholders to gather data requirements and design Databricks-based solutions aligned with business goals.
• Engineer and document scalable, secure, and high-performance data architectures and models.
• Provide expert guidance to cross-functional teams including data engineers, analysts, and scientists, ensuring adherence to best practices.
• Develop and manage complex data pipelines, ETL workflows, and data integration strategies within Databricks.
• Optimize and monitor Databricks clusters, workloads, and queries for performance and cost efficiency.
• Implement strict security controls and ensure compliance with data privacy standards within Databricks solutions.
• Create and maintain comprehensive technical documentation including architecture diagrams, data flow representations, and guides.
• Act as the primary technical point of contact, providing updates to stakeholders and ensuring smooth communication of project progress.
• Collaborate closely with cross-functional teams to ensure end-to-end data solutions are met with project objectives.
• Stay up-to-date with trends in data engineering and Databricks, recommending and implementing new methodologies and tools.

Required Skills & Experience:

• Minimum of 5+ years of hands-on experience in data engineering and analytics.
• Proficiency in Databricks, with a focus on Spark-based data processing and analytics.
• Strong understanding of data engineering concepts, including data modeling, ETL processes, and data pipeline development.
• Advanced skills in SQL for querying, transforming, and analyzing data.
• Expertise in Python programming for automation and data manipulation within Databricks.
• Strong experience with Apache Airflow for orchestrating and scheduling ETL workflows.
• Familiarity with big data technologies such as Spark, Hadoop, and related frameworks.
• Strong understanding of data security and compliance best practices.
• Proficient in data modeling principles and data warehousing concepts.
• Experience using version control systems like Git.
• Excellent communication and collaboration skills, with the ability to work with cross-functional teams.