R&D Data Engineer in AI and Computer Vision Ref. No: 516696
Location: Madrid, M, ES, 28037
Eviden, part of the Atos Group, with an annual revenue of circa € 5 billion is a global leader in data-driven, trusted and sustainable digital transformation. We are seeking a skilled and motivated Data Engineer to complete our team on end-to-end data pipeline implementation and data lake operation.
Responsibilities: Build and maintain robust data pipelines to ingest, transform, and load data from various sources. Ensure data quality, consistency, and reliability throughout the pipeline. Implement data transformation logic to clean, enrich, and transform raw data into structured formats suitable for analysis and reporting. Leverage ETL/ELT processes as needed. Manage the data platform infrastructure, including choosing appropriate storage technologies, optimizing storage utilization, and ensuring data accessibility. Implement and enforce data security measures, access controls, and compliance standards to maintain the integrity and privacy of the data. Develop mechanisms for efficient data search and retrieval, considering relevancy, query performance, and user experience. Monitor and optimize the performance of data pipelines and storage systems to ensure efficient data processing and retrieval. Maintain clear and comprehensive documentation of data pipeline designs, processes, and configurations to facilitate knowledge sharing and future maintenance. Implement workflows to automate the building, testing and deployment of data lake components following DevOps practices. Implement unit and integration tests following guidelines and propagate knowledge across the team. Manage AI assets (i.e., datasets, models) properly and securely. Integrate meta-data extraction components leveraging AI models and third-party tools such as labelling frameworks. Collaborate effectively with cross-functional teams including data scientists, data engineers, frontend and backend developers, and product owners to align on project goals and requirements. Education Bachelor's, Master's, or PhD in Computer Science, Electrical Engineering, or a related field. Essential knowledge and professional experience Proven experience (3+ years) in designing, building, and maintaining large-scale data pipelines and data lake infrastructure. Strong proficiency in programming languages such as Python. Hands-on experience in REST API development. Hands-on experience with Elasticsearch, including data ingestion, indexing, and search capabilities. Familiarity with Elasticsearch Query DSL and search relevancy concepts. Knowledge of data modelling, schema design, and ETL/ELT processes. Prepare software applications for their deployment using Docker and Kubernetes. Proficiency in the usage of Git and GitHub Actions. Practice of agile methodology. Proficient user of Linux environments (bash or shell). Additional knowledge Experience in MLOps tools, e.g., MLFlow, Kubeflow. Experience with Google Cloud Platform (GCP). CPU vs GPU programming. General knowledge about clusters. Competences Autonomy: capacity to seek and read documentation. Ability to collaborate: provide constructive comments and embrace best practices and guidelines. Fluency in English. Good writing and presentation skills. Strong personal soft skills set communicative, enthusiastic, highly collaborative, proactive, and self-driven. Ability and enthusiasm to learn new technologies quickly.
#J-18808-Ljbffr
