Overview

Hudson Valley Credit Union is currently recruiting for the position of Senior Data Engineer. Primary Functions: Design, build, and optimize our data infrastructure, leveraging the power and scalability of Databricks within our Azure environment. Maintain a deep expertise in Databricks and its ecosystem in order to build high-performance data pipelines, ensure data quality, and enable advanced analytics and machine learning initiatives.

Responsibilities

Design, develop, and maintain robust and scalable data pipelines primarily using Databricks, leveraging Spark (PySpark or Scala), Delta Lake, and Databricks Workflows
Build and optimize data lakehouse solutions on Azure Data Lake Storage Gen2 integrated with Databricks
Implement and enforce data quality frameworks and data governance policies within Databricks using tools and features like Delta Lake's ACID properties and data validation techniques
Identify and resolve performance bottlenecks in Databricks-based data pipelines and Spark applications
Collaborate closely with analysts to understand their data requirements for advanced analytics, providing them with curated and performant datasets within Databricks
Automate data ingestion, transformation, and deployment processes, potentially integrating with CI/CD tools and Databricks MLOps
Provide technical leadership and mentorship to junior data engineers on Databricks best practices and performance tuning
Document Databricks notebooks, data pipelines, data models, and configurations
Ensure data security and compliance within the Databricks environment, adhering to relevant security standards and best practices
Troubleshoot and resolve complex data-related issues within the Databricks ecosystem
Work independently and contribute to immediate and cross-functional teams
Participate in design discussions
Analyze problem statements, break down problems, and provide solutions
Follow an agile-based approach to deliver data products
Take ownership of large tasks, delivering on time, and mentoring team members
Oversee the maintenance and enforcement of SDLC and ITSM processes, change management procedures, departmental standards, style requirements and all other procedures necessary for data and system integrity
Adhere to all Credit Union policies, procedures, and regulatory agency requirements. Participate in all required and recommended training and development including, but not limited to, Bank Secrecy Act training (BSA) and demonstrate attained knowledge. Participate on Credit Union teams, projects and strategic initiatives when the opportunity arises. Perform additional duties and special projects as assigned.
Embrace and apply HVCU’s guiding principles to all activities and responsibilities. This includes the Credit Union’s Mission, Vision, Core Values, Employee and Member Value Propositions, Sales and Service Model, and commitment to Lean Six Sigma practices. Support the Credit Union’s initiatives by demonstrating teamwork and professionalism.
Responsible for regular and predictable attendance including punctuality.

Qualifications

Bachelor’s Degree or equivalent in Computer Science or related field required.
Master's Degree or equivalent in Computer Science or related field preferred.
License/Certification Databrick Data Engineer Associate required.
License/Certification Databrick Data Engineer Professional preferred.
License/Certification Mircrosoft Azure Data Engineer preferred.
Minimum 5 years of experience as Data Engineer required.
Minimum 5 years of experience Data Wharehousing, Data Lakes and big date platforms required.
Minimum 5 years Python experience required.
Minimum 3 years Databricks experience required.

Click here to view full job description

Why Choose HVCU?

Join a team where your growth matters. At Hudson Valley Credit Union, we offer a supportive, community-focused culture, competitive benefits, and opportunities to grow your career while making a meaningful impact. Come be part of a Great Place to Work!

Employee Perks:

Base & Variable Pay Structures

Paid Personal, Holiday & Volunteer Time Off

Professional Development & Training

Student Loan Repayment & Tuition Reimbursement Programs

Medical, Dental & Vision Coverage

401(k) With Employer Match & Non-Elective Employer Contribution

Life, Short-term & Long-term Disability Insurances

Discounted Loan Rates & Fees

Hudson Valley Federal Credit Union is an AA/EEO employer committed to equal opportunity and employee diversity. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, age, sexual orientation, gender identity, national origin, disability, military or veteran status, genetic information, marital or familial status, domestic violence victim status, or any other characteristic protected by law.

Hudson Valley Credit Union

Latest Jobs

Senior Data Engineer