.Atradius, a globally recognized leader in credit insurance and risk management, is currently looking for a skilled Data Engineer to join our dynamic Data Platform & MI team.As a member of the team, you will report to the Lead Data Engineer and work on a cloud-based LakeHouse platform where your expertise in Apache Spark will be a game-changer. The ideal candidate should have at least three years of hands-on experience in PySpark, experience in Databricks, and a proven track record of success in developing big data applications. This is a fantastic opportunity for anyone with the required skillset!Key Responsibilities:Development of reusable and standardized components to efficiently process, automate and optimize data engineering end-to-end solutions using PySpark.Build and maintain data ingestion pipelines to integrate various raw data sources (internal and external) into our cloud-based platform using Oracle GoldenGate.Design, maintain and operate Databricks workflows by ensuring the expected run behavior and results.Ensure data quality, accuracy, and reliability throughout the entire Medallion architecture.Collaborate seamlessly with cross-functional teams (Platform engineers, Data Modelers, Data Architects and Power BI engineers) to comprehend data requirements and interpret business needs, translating them into technically feasible developments.Support the Lead Data Engineer with solution design, decision making and implementation phase.Contribute to the team by proactively suggesting new ideas for improvement in the data engineering domain. Stay up to date on trends and best practices.Apply testing methodologies (TDD) and code quality rules in your developments to ensure robust and maintainable data solutions.Requirements:Bachelor's or Master's degree in Computer Science, Engineering, or a related field.Fluent spoken and written EnglishMinimum of 3 years of hands-on experience as a Data Engineer.Proficiency in Spark/PySpark and SQLHands-on experience with Databricks, particularly with Medallion architecture, to build and manage scalable data pipelines from bronze to gold layer.Source repository management and version control (Git), CI/CD knowledge (Azure DevOps) Excellent problem-solving skills and attention to detail.Strong communication and collaboration skills.Takes responsibility and feels accountable for own and team.Proven ability to work with speed and accuracy to deadlines is essential.Nice-to-have:Data Modeling experience in Medallion architecture, especially Gold layer read optimizations.Cluster tuning and optimization of Spark applications.Experience in Azure Cloud data-related services such as ADF or Azure Functions.Databricks native optimization features knowledge.Previous experience working in an international team environment
