锦鲤求职岗位描述
Job ID: 64398 Design, develop and optimize scalable data pipelines, ETL/ELT processes and data integration solutions on Azure. Build and maintain lakehouse / data warehouse layered models, such as Raw/ODS/DWD/ADS or Bronze/Silver/Gold. Develop batch and incremental data solutions using Azure Databricks, Azure Data Factory, Azure storage, Delta Lake and Azure DevOps. Use prophecy to develop the ETL pipeline with logical design. Manage complex CICD platform which is used by multiple teams together. Perform task orchestration, scheduling, pipeline monitoring and production issue troubleshooting. Apply dimensional modeling, star schema and wide-table design to support analytics and downstream data consumption. Ensure data quality, accuracy, consistency and compliance across multiple systems and data sources. Support data warehouse migration projects, including data validation and test reporting. Optimize data pipeline and query performance with Spark big data platform tunning skills. Collaborate with cross-functional and cross-regional teams, communicate with business stakeholders and translate business requirements into technical solutions. Participate in python coding, code reviews, technical discussions and best-practice sharing for data development. Bachelor's degree in Computer Science, Information Technology or a related field. 5+ years of hands-on experience in data engineering related roles. Hands-on, real-project experience with the Azure data stack, including Azure Databricks, Azure Data Factory, Azure storage, Delta Lake. Strong proficiency in SQL and experience with relational databases. Solid programming skills in Python and PySpark, with experience in data processing frameworks. Experience with ETL/ELT tools and data integration technologies. Understanding of data modeling, data warehousing concepts, dimensional design. Familiar with multi-parties CICD design and management. Mandatory English capability: able to communicate with international business and technical teams, gather and clarify requirements, write clear technical documentation and present technical concepts to non-technical stakeholders. Experience working with business stakeholders, translating business requirements into technical solutions, managing the end to end delivery.