Teranet is seeking an experienced Data Engineer for its Enterprise Data Analytics team within the IT line of business. This is a pure data engineering role focused on processing large-scale data and optimizing Spark performance, not a Data Analyst or BI Developer position.
In this role, you will:
Design, build, and orchestrate data pipelines using tools like Airflow.
Integrate data from diverse sources such as RDBMS systems (via Debezium) and files (CSV, JSON, XML, text) using Python.
Be responsible for transforming and processing large volumes of data, optimizing Spark execution plans for performance and cost efficiency and writing complex SQL and Python code.
Work closely with product owners, data analysts, and other data engineers to develop and enhance data pipelines and data models that support data delivery, AI development, and BI insights.
Configure data ingestion frameworks for database...