Senior Data Engineer
About the job
The Senior Data Engineer position at NCS Australia is all about designing, building, and optimizing scalable data pipelines on Azure-based cloud platforms. This hands-on job plays a crucial role in delivering robust, production-grade data engineering solutions, enabling reliable, high-quality data at scale. The team thrives on collaboration, innovation, and a shared commitment to creating a better tomorrow.
You'll be responsible for
🔗
Designing data pipelines
Designing, building, and optimizing scalable, reliable data pipelines using Azure Databricks, PySpark, and Delta Lake to support large-scale data transformation and analytics needs.💻
Developing SQL and Python code
Developing and maintaining complex SQL and Python code for data ingestion, transformation, validation, and quality checks across cloud data platforms.🚀
Implementing end-to-end pipelines
Implementing end-to-end pipelines leveraging Azure services such as ADLS and Azure Data Factory, while configuring and managing Databricks notebooks, workflows, jobs, and clusters for performance and cost efficiency.Key criteria
🔧
Azure Databricks expertise
Strong hands-on experience with Azure Databricks functionalitiesView
📊
Proven PySpark skills
Demonstrated expertise in PySpark for large-scale data processingView
🗄️
Delta Lake experience
Solid experience with Delta Lake for data management and optimizationView
View more
.png?w=1640&q=75)
