Software Engineer 3

The worldwide data management software market is massive (According to IDC, the worldwide database software market, which it refers to as the database management systems software market, was forecasted to be approximately $82 billion in 2023 growing to approximately $137 billion in 2027. This represents a 14% compound annual growth rate). At MongoDB we are transforming industries and empowering developers to build amazing apps that people use every day. We are the leading developer data platform and the first database provider to IPO in over 20 years. Join our team and be at the forefront of innovation and creativity.

The Data Pipelines Engineering team is responsible for building ETL pipelines that populate the Internal Data Platform, which drives analytics that help the company run more efficiently. Our team builds highly performant and scalable processes that extract massive datasets and makes those datasets available for querying in an optimal way. We are also building a Generative AI framework that will help teams within the company tap into the data that we store in their Retrieval-Augmented Generation (RAG)-based applications.

We are looking to speak to candidates who are based in Gurgaon, India for our hybrid working model.

What you’ll do

• Design, build and maintain efficient, scalable ETL/ELT pipelines using Python and Spark across batch, file-based, and streaming architectures
• Ensure data quality, reliability and timeliness across pipelines by following established data engineering best practices
• Model and store large datasets using modern file formats (Parquet, JSON, Avro) and table formats (Iceberg, Hive)
• Deploy pipelines on cloud infrastructure, leveraging Cloud-based technologies (mostly AWS, some GCP) to build and deploy data pipelines
• Partner with Data Analysts and Data Scientists to understand their needs and deliver the datasets that drive their work
• Work with Security and Compliance teams to ensure that datasets have appropriate permissions and regulations in place
• Work with our Data Platform, and Governance sibling teams to make data scalable, consumable, and discoverable

We’re looking for someone with

• 4+ years of building ETL pipelines for a Data Lake/Warehouse
• Expertise in Python, Spark, SQL, Airflow
• Experience with data warehousing and engineering concepts, analytical data modeling, data quality validation, monitoring, and pipeline reliability practices
• Hive, Iceberg, Glue, or other technologies that expose big data as tables
• Familiarity with different big data file types such as Parquet, Avro, JSON etc
• Background in building data platforms on Cloud (e.g. AWS, GCP, Azure)
• Exposure to real-time or streaming data technologies is a plus

Success Measures

• In 3 months, you'll have collaborated with stakeholde