Job Summary
As an Advanced Data Engineer, you will lead the development of cutting-edge data solutions, enabling data-driven decision-making across the organization. You will be responsible for designing, building, and maintaining scalable and efficient data pipelines using tools like GCP (Google Cloud Platform), Vertex AI, and BigQuery. This role will emphasize expertise in real-time data workflows, advanced feature engineering for machine learning pipelines, and collaboration with cross-functional teams. If you are passionate about leveraging data as a strategic asset and have a proven track record in cloud-based solutions, this position is for you!
Key Responsibilities
Provide Technical Leadership: Offer guidance and technical leadership to ensure clarity between ongoing projects and facilitate collaboration across teams.
Build and Maintain Data Pipelines: Design, build, and maintain scalable, efficient, and reliable data pipelines for data ingestion, transformation, and integration using tools such as Kafka, Databricks, and other toolsets.
Drive Digital Innovation: Leverage innovative technologies to modernize and extend core data assets, including SQL-based, NoSQL-based, and real-time streaming data platforms.
Implement Feature Engineering: Develop and manage feature engineering pipelines for machine learning workflows, utilizing tools like Vertex AI, BigQuery ML, and custom Python libraries.
Automated Testing: Design and implement automated testing frameworks (unit, integration, and performance testing) to ensure high data quality, reliability, and compliance with organizational standards.
Optimize Data Workflows: Improve data workflows for performance, cost efficiency, and scalability across large datasets and complex environments.
Mentor Team Members: Guide and mentor team members in best practices, data principles, and processes to promote team growth and knowledge sharing.
Documentation: Draft and review architectural diagrams, interface specifications, and other design documents to ensure clear and effective communication of data solutions.
Cost/Benefit Analysis: Present technical solutions with cost/benefit analysis, guiding leadership on scalable and efficient data solution architectures.
Required Qualifications
Education:
Bachelor's or Master's degree in Computer Science, Engineering, Information Technology, or a related technical field.
Experience: 4+ years of professional experience in Data Engineering or similar roles.
4+ years of experience with SQL and NoSQL technologies.
3+ years of hands-on experience building and maintaining data pipelines and workflows using Kafka, Databricks, and similar tools.
5+ years of experience developing with Java.
2+ years of experience developing with Python.
3+ years of experience working with Kafka for real-time data solutions.
2+ years of experience in feature engineering for machine learning pipelines.
Solid experience with GCP services such as BigQuery, Vertex AI, Cloud Storage, AutoMLOps, and Dataflow.
Experience with CI/CD pipelines, automated unit, integration, and performance testing.
Proficient in Git for version control and collaboration.
Strong understanding of ETL and Data Warehousing concepts.
Deep knowledge of Agile principles (Scrum) and a proven track record working in Agile environments.
Preferred Qualifications Experience with Structured Streaming technologies such as Spark, Kafka, or EventHub.
Familiarity with GitHub SaaS or GitHub Actions.
Understanding of Databricks concepts.
Experience with PySpark and Spark development.
Certifications (if any)
Google Cloud Professional Data Engineer certification (preferred).
AWS Certified Solutions Architect or equivalent cloud certifications (optional).
TensorFlow, PyTorch, or related AI/ML certifications (preferred).
Top 5 Skills to Prioritize Expertise in GCP Services: Strong experience with Vertex AI, BigQuery, AutoMLOps, and Dataflow is critical.
Data Pipelines and Real-Time Streaming: Experience building and maintaining real-time data pipelines with Kafka, PySpark, or Databricks.
Programming Skills: Advanced proficiency in Java (5+ years) and Python (2+ years). Experience with CI/CD, automated testing, and Git.
Machine Learning and Feature Engineering: At least 2 years of experience in feature engineering for ML pipelines using tools like Vertex AI or BigQuery ML.
Agile Thinking and Communication: Understanding of Agile principles (Scrum) and the ability to communicate technical concepts effectively to both technical and non-technical stakeholders.
Why Join Us?
Innovative Projects: Work on impactful and innovative data solutions that drive digital transformation.
Collaboration: Be part of a highly collaborative environment, working closely with cross-functional teams to solve complex challenges.
Growth Opportunities: Opportunity to mentor junior team members and take your career to the next level.
Cutting-Edge Technologies: Get hands-on experience with advanced technologies in GCP, AI/ML, and real-time data pipelines.
Competitive Benefits: Enjoy a comprehensive benefits package including health, wellness, and professional development programs.
Education: Bachelors Degree
Receive real-time job alerts and never miss the right job again
US - KY - KY - Lexington
January 9, 2025
&Requirements • Pediatrics - Pediatric Nephrology • StartDate: • ASAP Pay Rate: • $173500.00 - $250000.00 • The University of Kentucky's Department of Pediatrics seeks a Pediatric Nephrologist at the Assistant or Associate Professor level for a rewarding academic opportunity at a premier hospital. •...
US - KY - KY - Louisville
January 11, 2025
TRANSLATING IN THE AIR • When we receive or intercept data in the air, it's critical for us to understand it in order to act accordingly. Responsible for translating intelligence communications, Airborne Cryptologic Language Analysts utilize fluent foreign language skills to analyze messages...
US - KY - KY - Middletown
January 11, 2025
ASSEMBLING OUR AMMO • As the world's leading air force, we work with some of the most advanced weapons known to man. It's the responsibility of Munitions Systems specialists to manage all aspects of nonnuclear munitions processes. Working with a high attention to detail, they store, assemble,...
US - KY - KY - Florence
January 11, 2025
ASSEMBLING OUR AMMO • As the world's leading air force, we work with some of the most advanced weapons known to man. It's the responsibility of Munitions Systems specialists to manage all aspects of nonnuclear munitions processes. Working with a high attention to detail, they store, assemble,...
US - KY - KY - Florence
January 11, 2025
TRANSLATING IN THE AIR • When we receive or intercept data in the air, it's critical for us to understand it in order to act accordingly. Responsible for translating intelligence communications, Airborne Cryptologic Language Analysts utilize fluent foreign language skills to analyze messages...
US - KY - KY - Lexington
January 11, 2025
ASSEMBLING OUR AMMO • As the world's leading air force, we work with some of the most advanced weapons known to man. It's the responsibility of Munitions Systems specialists to manage all aspects of nonnuclear munitions processes. Working with a high attention to detail, they store, assemble,...
US - KY - KY - Covington
January 11, 2025
ASSEMBLING OUR AMMO • As the world's leading air force, we work with some of the most advanced weapons known to man. It's the responsibility of Munitions Systems specialists to manage all aspects of nonnuclear munitions processes. Working with a high attention to detail, they store, assemble,...
US - KY - KY - Ashland
January 11, 2025
ASSEMBLING OUR AMMO • As the world's leading air force, we work with some of the most advanced weapons known to man. It's the responsibility of Munitions Systems specialists to manage all aspects of nonnuclear munitions processes. Working with a high attention to detail, they store, assemble,...
US - KY - KY - Louisville
January 11, 2025
ASSEMBLING OUR AMMO • As the world's leading air force, we work with some of the most advanced weapons known to man. It's the responsibility of Munitions Systems specialists to manage all aspects of nonnuclear munitions processes. Working with a high attention to detail, they store, assemble,...
US - KY - KY - Middletown
January 11, 2025
TRANSLATING IN THE AIR • When we receive or intercept data in the air, it's critical for us to understand it in order to act accordingly. Responsible for translating intelligence communications, Airborne Cryptologic Language Analysts utilize fluent foreign language skills to analyze messages...
US - KY - KY - Hopkinsville
January 11, 2025
ASSEMBLING OUR AMMO • As the world's leading air force, we work with some of the most advanced weapons known to man. It's the responsibility of Munitions Systems specialists to manage all aspects of nonnuclear munitions processes. Working with a high attention to detail, they store, assemble,...
US - KY - KY - Georgetown
January 11, 2025
TRANSLATING IN THE AIR • When we receive or intercept data in the air, it's critical for us to understand it in order to act accordingly. Responsible for translating intelligence communications, Airborne Cryptologic Language Analysts utilize fluent foreign language skills to analyze messages...
US - KY - KY - Florence
January 15, 2025
At Jabil we strive to make ANYTHING POSSIBLE and EVERYTHING BETTER. We are proud to be a trusted partner for the world's top brands, offering comprehensive engineering, manufacturing, and supply chain solutions. With over 50 years of experience across industries and a vast network of over 100 sites...
US - KY - KY - Owensboro
January 11, 2025
TRANSLATING IN THE AIR • When we receive or intercept data in the air, it's critical for us to understand it in order to act accordingly. Responsible for translating intelligence communications, Airborne Cryptologic Language Analysts utilize fluent foreign language skills to analyze messages...
US - KY - KY - Richmond
January 11, 2025
TRANSLATING IN THE AIR • When we receive or intercept data in the air, it's critical for us to understand it in order to act accordingly. Responsible for translating intelligence communications, Airborne Cryptologic Language Analysts utilize fluent foreign language skills to analyze messages...
US - KY - KY - Elizabethtown
January 11, 2025
TRANSLATING IN THE AIR • When we receive or intercept data in the air, it's critical for us to understand it in order to act accordingly. Responsible for translating intelligence communications, Airborne Cryptologic Language Analysts utilize fluent foreign language skills to analyze messages...
US - KY - KY - Bowling Green
January 11, 2025
ASSEMBLING OUR AMMO • As the world's leading air force, we work with some of the most advanced weapons known to man. It's the responsibility of Munitions Systems specialists to manage all aspects of nonnuclear munitions processes. Working with a high attention to detail, they store, assemble,...
US - KY - KY - Florence
January 10, 2025
At Jabil we strive to make ANYTHING POSSIBLE and EVERYTHING BETTER. We are proud to be a trusted partner for the world's top brands, offering comprehensive engineering, manufacturing, and supply chain solutions. With over 50 years of experience across industries and a vast network of over 100 sites...
US - KY - KY - Hopkinsville
January 11, 2025
TRANSLATING IN THE AIR • When we receive or intercept data in the air, it's critical for us to understand it in order to act accordingly. Responsible for translating intelligence communications, Airborne Cryptologic Language Analysts utilize fluent foreign language skills to analyze messages...
US - KY - KY - Bradfordsville
January 6, 2025
About the facility A hospital is looking for CRNA coverage. About the facility location The facility is located 1 hour away from Lexington, KY. About the clinician's workday This position operates Monday through Friday, 7 AM to 3 PM, with a call ratio of approximately 1:3 and a 20-minute beeper...
US - KY - KY - Georgetown
January 11, 2025
ASSEMBLING OUR AMMO • As the world's leading air force, we work with some of the most advanced weapons known to man. It's the responsibility of Munitions Systems specialists to manage all aspects of nonnuclear munitions processes. Working with a high attention to detail, they store, assemble,...
US - KY - KY - Lexington
January 11, 2025
TRANSLATING IN THE AIR • When we receive or intercept data in the air, it's critical for us to understand it in order to act accordingly. Responsible for translating intelligence communications, Airborne Cryptologic Language Analysts utilize fluent foreign language skills to analyze messages...
US - KY - KY - Bowling Green
January 11, 2025
TRANSLATING IN THE AIR • When we receive or intercept data in the air, it's critical for us to understand it in order to act accordingly. Responsible for translating intelligence communications, Airborne Cryptologic Language Analysts utilize fluent foreign language skills to analyze messages...
US - KY - KY - Covington
January 11, 2025
TRANSLATING IN THE AIR • When we receive or intercept data in the air, it's critical for us to understand it in order to act accordingly. Responsible for translating intelligence communications, Airborne Cryptologic Language Analysts utilize fluent foreign language skills to analyze messages...
US - KY - KY - Pikeville
January 9, 2025
&Requirements • Urology • StartDate: ASAP Pay Rate: $810000.00 - $900000.00 • A reputable practice that seeks to provide world-class quality is seeking a qualified candidate who is board-certified or board-eligible in urology to join its team. This position offers highly competitive compensation and...
US - KY - KY - Pikeville
December 18, 2024
&Requirements • Internal Medicine - Endocrinology, Diabetes&Metabolism • StartDate: ASAP Pay Rate: $347785.00 - $450000.00 • A reputable practice that seeks to provide world-class quality is seeking a qualified candidate who is board-certified or board-eligible in endocrinologist to join its team....
US - KY - KY - Elizabethtown
January 11, 2025
ASSEMBLING OUR AMMO • As the world's leading air force, we work with some of the most advanced weapons known to man. It's the responsibility of Munitions Systems specialists to manage all aspects of nonnuclear munitions processes. Working with a high attention to detail, they store, assemble,...