Data Engineering Team Lead - Azure Focused (AWS/GCP Experience a Plus)
Protecting the World’s Critical Infrastructure
OPSWAT, a global leader in IT, OT, and ICS critical infrastructure cybersecurity, delivers an end-to-end platform that gives public and private sector organizations and enterprises the critical advantage needed to protect their complex networks, secure their devices, and ensure compliance. Over the last 20 years our commitment to innovative technology has earned the trust of more than 1,700 organizations, governments, and institutions globally, solidifying our role in protecting the world’s critical infrastructure and securing our way of life.
The Position
The Data Engineering team is a vital component of OPSWAT's technology division, responsible for building and maintaining the foundational data infrastructure that supports the entire organization. One of OPSWAT Core Principles is “Data is King”. Enterprise Data team's mission is to create a scalable, reliable, and secure data platform, enabling data-driven decision-making across all business units. This involves managing existing data warehouses and lakes, developing robust data integration pipelines, and ensuring the quality and accessibility of data assets. A key strategic initiative for the team is the development of a Master Data Management (MDM) layer and the evolution of our architecture to serve data close to real-time for internal applications.
What You Will Be Doing
- Platform Stewardship & Maintenance: Manage and maintain the existing Microsoft Fabric platform, ensuring the performance, reliability, and security of OneLake, Lakehouses, and Warehouses through proactive monitoring and capacity optimization.
- Infrastructure Enhancement: Propose and implement continuous improvements to the existing Fabric infrastructure and lead the design of new build roadmaps at least three quarters in advance.
- Real-Time Data Architecture: Gather requirements and design data architectures that serve data close to real-time for internal applications.
- Data Quality & Governance: Design and implement data governance frameworks and data quality rules to ensure a "single source of truth" across all connected systems.
- Cost & Resource Optimization: Optimize data storage & computing resources for cost efficiency
- Risk Management: Ensure data platform availability and resiliency, including disaster recovery planning, handling service outages, and mitigating external security threats.
- Collaboration & Communication: Report to Head of Enterprise Data and work with Analytics, Product, Digital Transformation and other Operation teams... to translate complex technical concepts into layman's terms and deliver solutions that meet stakeholder needs.
- Team Leadership: Manage team resources, define measurable KPIs for team members, and implement strategies to mitigate "Key Man Dependency."
- Innovation: Stay abreast of industry trends and technologies to propose appropriate solutions that align with the company’s evolving needs.
- Hands-on Execution: Balance team management responsibilities with hands-on technical work in pipeline development and architectural design.
What We Need From You
- Bachelor’s degree in Computer Science, Data Engineering, or related field.
- Minimum of 5 years of proven experience in data warehouse/data lake design and implementation.
- Hands-on experience maintaining and optimizing a Microsoft Fabric environment (including OneLake and Fabric Capacity management).
- Extensive experience with ETL/ELT tools and the ability to build complex ingestion frameworks for API-based data sources.
- Experience in proposing and designing near real-time data solutions to support internal application requirements.
- Strong understanding of database architecture principles, encompassing relational (SQL) and NoSQL concepts.
- Proven project management skills with the ability to lead a technical project from inception to completion.
- Direct experience in building, managing, and mentoring a data engineering team.
- Experience integrating data from multiple SaaS applications (e.g., Salesforce, Hubspot, Netsuite, Workday, ZoomInfo).
- Experience researching and implementing Master Data Management (MDM) solutions.
- Strong problem-solving, analytical, and troubleshooting skills.
It Would Be Nice If You Had
- Good written and verbal communication skills for effective cross-functional collaboration.
- Experience with real-time data streaming technologies such as Azure Event Hubs or Kafka.
- Experience with DevOps practices for automating data pipelines, specifically using Fabric Git integration and CI/CD.
- Experience working within a Scrum or Agile framework
OPSWAT is an equal opportunity employer. We celebrate diversity and are committed to providing an environment where equal employment opportunities are extended to all employees and applicants, free of discrimination and harassment of any type. All employment decisions are based on individual qualifications, job requirements, and business needs without regard to race, color, religion, age, sex, national origin, disability status, genetics, protected veteran status, sexual orientation, gender identity or expression, or any other category protected by federal, state, or local laws.
Recruiting Agencies: we do not accept unsolicited resumes from third party agencies for any of our open positions. To submit resumes for our jobs, there must be a recruiting contract approved by our legal team and endorsed by both parties. We are currently not accepting additional 3rd party agencies at this time.
