Lead Data Engineer
Brown & Brown Insurance · Plano, Texas, United States
Built on meritocracy, our unique company culture rewards self-starters and those who are committed to doing what is best for our customers.
Brown & Brown is seeking a Lead Data Engineer to join our growing team in Plano, TX!
As a Lead Data Engineer, you will play a key role in advancing our Data & AI platform capabilities by designing, building, and operating modern data solutions that power Artificial Intelligence, Machine Learning, analytics, business intelligence, and enterprise insights. This role combines hands-on engineering with technical leadership. You will partner with product owners, architects, business stakeholders, and engineering teams to translate business needs into pragmatic, production-ready data solutions.
You will own Data solutions across the engineering lifecycle, from architecture and design through development, deployment, and steady-state operations. You will establish engineering patterns and standards, improve platform capabilities, mentor engineers, resolve complex technical challenges, and help the team deliver reliable, governed, scalable, and cost-effective data products.
What Will You Do?
- Lead the design, development, deployment, and operational support of scalable batch and streaming data solutions.
- Design and build Databricks-native data engineering solutions using Delta Lake, Lakeflow Declarative Pipelines, Databricks Jobs, Auto Loader, Structured Streaming, and Unity Catalog.
- Develop production-grade ingestion, transformation, and data-processing pipelines using Python, PySpark, Apache Spark, and SQL.
- Define technical designs, engineering standards, and reusable patterns that improve scalability, reliability, maintainability, and developer productivity.
- Build and maintain governed data solutions that improve the accuracy, consistency, accessibility, and trustworthiness of enterprise data.
- Develop reusable frameworks, libraries, templates, and automation that accelerate data ingestion, transformation, feature engineering, and consumption.
- Implement data-quality validation, monitoring, observability, and remediation capabilities for critical data assets.
- Integrate data solutions with enterprise governance capabilities, including metadata, lineage, classification, certification, ownership, and access controls.
- Develop governed data assets supporting AI, Machine Learning, advanced analytics, semantic models, APIs, Genie Spaces, and AI agents.
- Troubleshoot and optimize production workloads using Spark diagnostics, job logs, query plans, partitioning strategies, join optimization, and Adaptive Query Execution.
- Operate and continuously improve data solutions through performance monitoring, incident resolution, migrations, upgrades, observability, and cost optimization.
- Implement automated infrastructure and environment-aware deployments using Infrastructure as Code, Git, and CI/CD practices.
- Partner with product owners, architects, operations teams, and engineering teams to prioritize capabilities, address technical dependencies, and expand platform adoption.
- Lead technical design discussions, perform code and design reviews, drive architectural alignment, and proactively identify and remove technical blockers.
- Mentor engineers and foster a culture of engineering excellence, collaboration, continuous learning, and accountability.
Required Qualifications
- Bachelor’s degree in computer science, Engineering, Mathematics, Information Systems, or a related STEM field, or equivalent education and professional experience.
- 8+ years of experience in data engineering or related software/data engineering disciplines, including significant hands-on experience building and supporting production data solutions.
- Strong hands-on experience with Databricks, Apache Spark/PySpark, Delta Lake, Python, and SQL. Experience designing and operating production-grade batch and streaming data pipelines. Experience with Databricks capabilities such as Lakeflow Declarative Pipelines, Databricks Jobs, Auto Loader, Structured Streaming, Delta Lake, and Unity Catalog.
- Experience designing reliable workflow orchestration, including scheduling, dependencies, parameters, retries, notifications, and operational monitoring.
- Experience implementing data governance and access controls with Unity Catalog, including catalogs, schemas, tables, views, external locations, storage credentials, lineage, and fine-grained permissions.
- Experience with Terraform, Databricks Asset Bundles, or comparable Infrastructure as Code approaches, along with Git and CI/CD practices.
- Strong experience troubleshooting and optimizing Spark workloads using tools and techniques such as Spark UI, job logs, query plans, partitioning, join optimization, and Adaptive Query Execution.
- Strong understanding of conceptual, logical, relational, dimensional, and physical data modeling.
- Experience delivering ETL/ELT solutions through the complete engineering lifecycle, from requirements and design through deployment and production support.
- Demonstrated ability to balance engineering quality, delivery speed, scalability, resiliency, and cost while making pragmatic technical trade-offs.
- Strong problem-solving skills with demonstrated ability to diagnose and resolve complex data and production issues.
- Demonstrated technical leadership, including solution design, code reviews, engineering standards, mentoring, and team unblocking.
- Ability to influence technical direction, facilitate alignment, and build consensus across engineering teams and stakeholders without relying on formal authority.
- Experience working within Agile delivery models such as Scrum, Kanban, or SAFe.
Preferred Qualifications
- Prior experience serving as a technical lead, lead engineer, or senior engineer with responsibility for technical design and engineering standards.
- Advanced Python and/or Scala development experience.
- Experience with cloud platforms such as Microsoft Azure, AWS, or Google Cloud.
- Experience with cloud storage technologies such as Azure Data Lake Storage Gen2, Amazon S3, OneLake, or Google Cloud Storage.
- Experience with modern data platforms and open table technologies such as Snowflake, Apache Iceberg, Apache Hudi, or BigQuery.
- Experience with messaging and event-streaming technologies
- Experience with NoSQL technologies such as Cassandra or HBase.
- Experience designing or consuming REST APIs and web services.
- Experience developing governed data products supporting AI, Generative AI, Machine Learning, semantic models, APIs, advanced analytics, Genie Spaces, or AI agents.
- Experience implementing enterprise-scale data governance, data quality, metadata, and lineage capabilities.
- Experience working with insurance, Health data.
- Practical experience using AI-assisted development tools such as GitHub Copilot, Claude, or Databricks AI-assisted development capabilities to improve engineering productivity.
What Will Put You Ahead
- Deep experience designing, building, and optimizing production-scale Databricks Lakehouse solutions.
- Experience building high-volume or near-real-time ingestion solutions
- Advanced Python experience developing reusable engineering frameworks, libraries, automation, and platform capabilities.
- Experience building governed, reusable data products for AI and Generative AI applications.
- Demonstrated ability to simplify complex technical architectures and establish reusable engineering patterns across teams.
- A track record of raising engineering standards while helping teams deliver incrementally and predictably.
- Strong technical judgment and the ability to identify the right balance between speed, simplicity, scalability, reliability, governance, and cost
Teammate Benefits & Total Well-Being
We go beyond standard benefits, focusing on the total well-being of our teammates, including:
- Health Benefits: Medical/Rx, Dental, Vision, Life Insurance, Disability Insurance
- Financial Benefits