Advanced Data Engineer

JobRialto
US - Kentucky - Union
January 13, 2025

Job Summary

As an Advanced Data Engineer, you will lead the development of cutting-edge data solutions, enabling data-driven decision-making across the organization. You will be responsible for designing, building, and maintaining scalable and efficient data pipelines using tools like GCP (Google Cloud Platform), Vertex AI, and BigQuery. This role will emphasize expertise in real-time data workflows, advanced feature engineering for machine learning pipelines, and collaboration with cross-functional teams. If you are passionate about leveraging data as a strategic asset and have a proven track record in cloud-based solutions, this position is for you!

Key Responsibilities
Provide Technical Leadership: Offer guidance and technical leadership to ensure clarity between ongoing projects and facilitate collaboration across teams.
Build and Maintain Data Pipelines: Design, build, and maintain scalable, efficient, and reliable data pipelines for data ingestion, transformation, and integration using tools such as Kafka, Databricks, and other toolsets.
Drive Digital Innovation: Leverage innovative technologies to modernize and extend core data assets, including SQL-based, NoSQL-based, and real-time streaming data platforms.
Implement Feature Engineering: Develop and manage feature engineering pipelines for machine learning workflows, utilizing tools like Vertex AI, BigQuery ML, and custom Python libraries.
Automated Testing: Design and implement automated testing frameworks (unit, integration, and performance testing) to ensure high data quality, reliability, and compliance with organizational standards.
Optimize Data Workflows: Improve data workflows for performance, cost efficiency, and scalability across large datasets and complex environments.
Mentor Team Members: Guide and mentor team members in best practices, data principles, and processes to promote team growth and knowledge sharing.
Documentation: Draft and review architectural diagrams, interface specifications, and other design documents to ensure clear and effective communication of data solutions.
Cost/Benefit Analysis: Present technical solutions with cost/benefit analysis, guiding leadership on scalable and efficient data solution architectures.
Required Qualifications

Education:

Bachelor's or Master's degree in Computer Science, Engineering, Information Technology, or a related technical field.

Experience: 4+ years of professional experience in Data Engineering or similar roles.
4+ years of experience with SQL and NoSQL technologies.
3+ years of hands-on experience building and maintaining data pipelines and workflows using Kafka, Databricks, and similar tools.
5+ years of experience developing with Java.
2+ years of experience developing with Python.
3+ years of experience working with Kafka for real-time data solutions.
2+ years of experience in feature engineering for machine learning pipelines.
Solid experience with GCP services such as BigQuery, Vertex AI, Cloud Storage, AutoMLOps, and Dataflow.
Experience with CI/CD pipelines, automated unit, integration, and performance testing.
Proficient in Git for version control and collaboration.
Strong understanding of ETL and Data Warehousing concepts.
Deep knowledge of Agile principles (Scrum) and a proven track record working in Agile environments.
Preferred Qualifications Experience with Structured Streaming technologies such as Spark, Kafka, or EventHub.
Familiarity with GitHub SaaS or GitHub Actions.
Understanding of Databricks concepts.
Experience with PySpark and Spark development.
Certifications (if any)

Google Cloud Professional Data Engineer certification (preferred).

AWS Certified Solutions Architect or equivalent cloud certifications (optional).

TensorFlow, PyTorch, or related AI/ML certifications (preferred).

Top 5 Skills to Prioritize Expertise in GCP Services: Strong experience with Vertex AI, BigQuery, AutoMLOps, and Dataflow is critical.
Data Pipelines and Real-Time Streaming: Experience building and maintaining real-time data pipelines with Kafka, PySpark, or Databricks.
Programming Skills: Advanced proficiency in Java (5+ years) and Python (2+ years). Experience with CI/CD, automated testing, and Git.
Machine Learning and Feature Engineering: At least 2 years of experience in feature engineering for ML pipelines using tools like Vertex AI or BigQuery ML.
Agile Thinking and Communication: Understanding of Agile principles (Scrum) and the ability to communicate technical concepts effectively to both technical and non-technical stakeholders.

Why Join Us?

Innovative Projects: Work on impactful and innovative data solutions that drive digital transformation.

Collaboration: Be part of a highly collaborative environment, working closely with cross-functional teams to solve complex challenges.

Growth Opportunities: Opportunity to mentor junior team members and take your career to the next level.

Cutting-Edge Technologies: Get hands-on experience with advanced technologies in GCP, AI/ML, and real-time data pipelines.

Competitive Benefits: Enjoy a comprehensive benefits package including health, wellness, and professional development programs.

Education: Bachelors Degree

Read the full job description and apply online on the recuiter's web-site

Advanced Data Engineer

Advanced Data Engineer

Find Jobs Hiring Now Near You!

Get Jobilize Job Search Mobile App Now

Related Job Postings

CDL-A Drivers / Lease Opportunities / Avg 80K Yearly

CDL-A Drivers / Lease Opportunities / Avg 80K Yearly

CDL-A Drivers / Lease Opportunities / Avg 80K Yearly

CDL-A Drivers / Lease Opportunities / Avg 80K Yearly

CDL-A Drivers / Lease Opportunities / Avg 80K Yearly

CDL-A Drivers / Lease Opportunities / Avg 80K Yearly

CDL-A Drivers / Lease Opportunities / Avg 80K Yearly

RN/Registered Nurse

CDL-A Drivers / Lease Opportunities / Avg 80K Yearly

CDL-A Drivers / Lease Opportunities / Avg 80K Yearly

Registered Nurse

CDL-A Drivers / Lease Opportunities / Avg 80K Yearly

CDL-A Drivers / Lease Opportunities / Avg 80K Yearly

CDL-A Drivers / Lease Opportunities / Avg 80K Yearly

CDL-A Drivers / Lease Opportunities / Avg 80K Yearly

CDL-A Drivers / Lease Opportunities / Avg 80K Yearly

CDL-A Drivers / Lease Opportunities / Avg 80K Yearly

CDL-A Drivers / Lease Opportunities / Avg 80K Yearly

CDL-A Drivers / Lease Opportunities / Avg 80K Yearly

CDL-A Drivers / Lease Opportunities / Avg 80K Yearly

CDL-A Drivers / Lease Opportunities / Avg 80K Yearly

CDL-A Drivers / Lease Opportunities / Avg 80K Yearly

CDL-A Drivers / Lease Opportunities / Avg 80K Yearly

Urology

CDL-A Drivers / Lease Opportunities / Avg 80K Yearly

CDL-A Drivers / Lease Opportunities / Avg 80K Yearly

Internal Medicine - Endocrinology, Diabetes&Metabolism