Job Overview:
The Senior/Lead Data Engineer is responsible for designing, developing, and optimizing scalable data solutions on Alibaba Cloud’s big data platforms, including MaxCompute and Hologres. This role will take a hands-on approach in building robust data pipelines, ensuring data quality and integrity, and supporting the organization’s data-driven initiatives. The ideal candidate has deep experience with Alibaba Cloud’s big data ecosystem and is proficient in leading technical efforts in complex data warehousing and processing environments.
Responsibilities:
1. Data Pipeline Development and Optimization
- Design and implement efficient ETL/ELT pipelines on Alibaba Cloud to support ingestion, transformation, and loading of large-scale data
- Integrate data from multiple systems into a centralized data warehouse, ensuring consistency, accuracy, and accessibility for analytics and reporting
- Continuously optimize and monitor data pipelines to ensure high performance, scalability, and efficient resource usage within Alibaba Cloud environments
2. Data Warehouse Management
- Develop, manage, and optimize big data warehouses on Alibaba Cloud, using tools like Hologres, MaxCompute, and DataWorks
- Implement data modeling best practices to structure and organize data for optimal accessibility, flexibility, and query performance
- Establish data validation, monitoring, and quality assurance practices to maintain high data quality standards across the data warehouse
3. Collaboration and Team Management
- Work closely with data analysts, data scientists, and business stakeholders to understand data needs and provide tailored data solutions
- Translate business requirements into technical specifications for data pipelines, storage, and processing solutions
- Provide guidance on data engineering best practices and data warehouse solutions to team members and support their technical growth
4. Data Governance and Compliance
- Maintain clear and comprehensive documentation for data pipelines, data warehouse structures, and data governance practices
- Ensure compliance with data governance policies and standards for data security and privacy, working closely with security and governance teams
Requirements:
- Bachelor’s or Master’s degree in Computer Science, Data Engineering, Information Technology, or related field
- 5+ years in data engineering, with strong hands-on experience in Alibaba Cloud, particularly Hologres, MaxCompute, and related big data tools
- Proficiency in big data technologies (e.g., Hologres, MaxCompute, DataWorks, Flink), Strong SQL, Python, or Spark skills for data manipulation and processing
- Ability to work independently and take initiative on complex data engineering tasks
- Capacity to lead technical efforts, mentor junior engineers, and set standards for data engineering best practices
- Team player with strong collaboration skills, able to work effectively with both internal and external teams