Big data engineer
As a Big Data Engineer, you'll play a pivotal role in designing, implementing, and optimizing large-scale data processing systems. Leveraging cutting-edge technologies such as Kafka, Apache Hadoop, Hive, and Spring Boot, you'll architect robust solutions to handle vast volumes of data and facilitate data-driven decision-making processes within the organization.
-
Architecture and Design:
Design and implement scalable data architectures using Apache Hadoop ecosystem components such as HDFS, YARN, and MapReduce.
Utilize Kafka for building real-time data streaming pipelines, ensuring high throughput and low latency for processing streaming data. -
Data Processing and Transformation:
Develop and optimize data pipelines using Apache Spark for efficient data processing and transformation.
Implement Hive for querying and analyzing large datasets stored in Hadoop Distributed File System (HDFS), providing users with SQL-like interfaces for data exploration. -
Integration with Spring Boot:
Integrate Kafka and Apache Hadoop components with Spring Boot applications, enabling seamless communication and data exchange between microservices and big data platforms.
Develop RESTful APIs and web services using Spring Boot to expose data processing capabilities and enable integration with other systems and applications. -
Performance Optimization:
Optimize data processing workflows and algorithms to enhance system performance and scalability.
Implement caching mechanisms and data partitioning strategies to improve query response times and reduce resource utilization. -
Monitoring and Maintenance:
Implement monitoring and alerting solutions to track system performance, resource utilization, and data quality.
Conduct regular maintenance activities such as software upgrades, patch management, and data backup to ensure system reliability and availability. -
Security and Compliance:
Implement data encryption, access controls, and authentication mechanisms to ensure data security and compliance with regulatory requirements.
Define and enforce data governance policies to maintain data integrity and confidentiality.
Title salary range:
15000 L.E.
Average time to complete:
10 Weeks
Percentage of risk:
30 %
Minimum price to pay:
1000 L.E.