Develop and maintain scalable data pipelines and ETL processes using Azure Databricks and other Azure services. These processes are designed to extract data from multiple sources, transform it into meaningful formats, and load it into data lakes or data warehouses.
A key aspect of this responsibility is ensuring that data is accurately cleaned, transformed, and loaded to enable consistent and reliable analytics and reporting.
Implement data storage solutions leveraging Azure Databricks and Azure Data Lake Storage
Your role involves designing data storage systems that scale, perform well, and meet the organization’s requirements.
Implement monitoring and alerting systems to proactively detect and resolve issues within data pipelines.
Continuously oversee data storage and processing activities, ensuring optimal performance and addressing any operational challenges.