Software Engineering Lead - Data Collection

View full description

Requirements:

• 5+ years of experience in software development, experience in a high-growth B2B SaaS startup is strongly preferred.
• Proven track record of leading other engineers to deliver projects that customers love.
• Organized and detail oriented with project management experience.
• Experience collecting data from the web using parsing and scraper tools like Beautiful Soup, proxy usage, and Playwright/Puppeteer.
• Experience building and scaling batch-style data ingestion systems.
• Experience with data pipeline tools similar to Apache Airflow and dbt.
• Experience using cloud services such as AWS: RDS, ECS, S3, Lambda, OpenSearch.
• Experience with Python, Go, Ruby on Rails, or similar languages and frameworks.
• Experience building and implementing robust monitoring and observability solutions, including performance metrics, error monitoring, and distributed tracing.
• Excellent communication and interpersonal skills, with the ability to collaborate effectively with cross-functional teams and influence stakeholders.
• Demonstrated ability to thrive in a dynamic, rapidly changing startup environment.

Responsibilities:

• Lead a small team of skilled engineers to design, develop, and maintain data pipelines, ensuring scalability, reliability, and performance.
• Allocate resources effectively to ensure that projects are completed efficiently.
• Provide technical guidance and support to team members as needed.
• Monitor project progress and address any issues or risks that arise.
• Make impactful individual contributions while setting a high standard for code quality.
• Lead the design and implementation of data ingestion processes using asynchronous workers and store data in modern warehousing solutions.
• Monitor daily ingest processing and utilize anomaly detection to identify issues quickly.
• Foster a culture of innovation and continuous improvement within the engineering department.
• Stay current with industry trends and best practices in engineering management.
• Anticipate, identify, and resolve technical issues that may inhibit product enhancements and project goals.
• Fostering a culture of mentoring, learning, and information sharing across the broader engineering team.