About this role
The primary focus of Data Engineer will be on choosing optimal solutions to use for collecting, storing, processing, and analyzing of huge sets of data, then maintaining, implementing, and monitoring them. Data Engineer will also be responsible for integrating them with the architecture used across the company.
What you will be doing :
Designing, constructing, maintaining, and scaling our data infrastructure. Our data infrastructure is data lake for all industry under Kompas Gramedia.Designing and constructingETL process as data pipeline for each data source inside Kompas GramediaAs a person who solve business case using data technology, such as Kafka, Spark, DataFlow, BigQuery, Airflow, etc.Ensure systems meet business requirements and industry practicesEmploy a variety of languages and tools (e.g. scripting languages) to marry systems togetherCollaborate with other Data Team members on project goals Passion in big data, software engineering, and systems.Ability to design and architect distributed data systemsKnowledge of data warehousing and data pipeline concepts, such as ETL, data integration, data streamingIn-depth knowledge of SQL and other database solutions (No SQL), such as MongoDB, Elastic Search, and so onKnowledgeable in Linux-based system administratingPractical experience in planning, sizing & migrating large enterprise workloads from on-premises to cloud platform.Working knowledge of big data components like Hadoop v2, MapReduce, HDFS, Hive, MapReduce, Spark, Apache Beam, Storm is preferredExperience with various messaging systems, such as Kafka, RabbitMQ, NSQ is preferredExperience with Big Data ML toolkits, such as Mahout, SparkML, or H2O is preferredGood knowledge on big data platform such as Cloudera, MapR, Hortonworks, or ELK (Elastic Search – Log stash – Kibana) is an added advantageExperience in DevOps tools such as Ansible, Docker, Kubernetes, Chef, Vagrant is a plusKnowledge of other open source database technologies is an added advantage.A bachelor degree in technical majors is preferred (e.g. Computer Science, Information Technology) Require at least 2 years of work experience