Big Data Engineer (Libra) - Data Platform

职位描述

About the team

Libra is a large-scale online one-stop A/B testing platform developed by Data Platform. Some of its features include:

- Provides experimental evaluation services for all product lines within the company, covering solutions for complex scenarios such as recommendation, algorithm, function, UI, marketing, advertising, operation, social isolation, causal inference, etc.

- Provides services throughout the entire experimental lifecycle from experimental design, experimental creation, indicator calculation, statistical analysis to final evaluation launch.

- Supports the entire company's business on the road of rapid iterative trial and error, boldly assuming and carefully verifying.

Responsibilities

- Responsible for data system of experimentation platform operation and maintenance.

- Construct PB-level data warehouses, participate in and be responsible for data warehouse design, modeling, and development, etc.

- Build ETL data pipelines and automated ETL data pipeline systems.

- Build an expert system for metric data processing that combines offline and real-time processing.

任职要求

Minimum Qualifications

- Bachelor's degree in Computer Science, a related technical field involving software or systems engineering, or equivalent practical experience.

- Proficiency with big data frameworks such as Presto, Hive, Spark, Flink, Clickhouse, Hadoop, and have experience in large-scale data processing.

- Minimum 1 year of experience in Data Engineering.

- Experience writing code in Java, Scala, SQL, Python or a similar language.

- Experience with data warehouse implementation methodologies, and have supported actual business scenarios.

Preferred Qualifications

- Knowledge about a variety of strategies for ingesting, modeling, processing, and persisting data, ETL design, job scheduling and dimensional modeling.

- Expertise in designing, analyzing, and troubleshooting large-scale distributed systems is a plus (Hadoop, M/R, Hive, Spark, Presto, Flume, Kafka, ClickHouse, Flink or comparable solutions).

- Work/internship experience in internet companies, and those with big data processing experience are preferred.