Senior Data Engineer

We are seeking a skilled and mission-driven Senior Data Engineer to support NYC Opportunity's government-led research on poverty, income inequality, and social mobility. The successful candidate will design, build, and maintain robust data pipelines and infrastructure to enable high-quality empirical research. The primary responsibility focuses on managing data collection, integration, and transformation activities in support of producing the NYCgov poverty measure. Responsibilities include providing data technology guidance, design support, task automation, and scripting. The position requires collaboration within the Poverty Research Unit and with cross-functional teams to extract insights from various data sources, build and maintain data pipelines, and develop impactful visualizations.


  Responsibilities

Data Engineering Leadership

  • Develop data architecture solutions to enable reproducible research and support long-term storage of historical and current data sets.
  • Design and implement scalable ETL processes for large datasets from surveys, administrative sources, and third-party providers.
  • Implement data validation routines to ensure data quality, accuracy, and reliability standards are met.
  • Coordinate and manage data across income components and version control of poverty metrics.
  • Streamlining the production of poverty research data to ensure fast access to metrics.
  • Documenting research databases, including description of content, context, and changes; enhancing the ability to promote discovery through the use of data.

Data pipeline management and collaboration

  • Build and maintain various indicator data pipelines with city agencies and external partners, ensuring efficient data flow and integrity.
  • Own the development and maintenance of data pipelines, including new data on economic security and risks.

Efficiency Optimization

  • Guide data architecture towards modern and efficient approaches.
  • Manage query efficiency and adjust data structures and toolsets as needed to meet evolving requirements.
  • Develop automation of data processes, minimizing the need for manual intervention.

Data Analysis and Visualization

  • Collaborate with analysts on predictive models for benefit take-up, income, and expenditures to contribute to the annual production of the NYCgov poverty measure.
  • Responding to rapid data requests.
  • Supporting the modeling of policy alternatives and their comparative outcomes (anti-poverty impacts and distributional consequences).

Poverty Dashboard

  • Maintain, optimize, and document the data pipeline feeding into the poverty dashboard.
  • Work closely with the digital product team to enhance and deploy the poverty dashboard. This dashboard is designed to be a valuable resource, offering the public clear insights into poverty issues and empowering them to advocate for change.

Data Insights and Strategies

  • Collaborate with the product team to create user-friendly applications to help the city agencies and the general public use our data offerings and analytical reporting.
  • Other duties as assigned.

  Skills We're Seeking

We understand that candidates may not have all of the skills we are seeking below. We encourage those interested to apply even if they do not meet all of them. We are looking for talented individuals to join our team.

  • Extensive experience with AWS cloud services: in particular, S3, Redshift, Lambda, ETL services such as Data Pipeline or Glue, CI/CD service (Code Pipeline), and CodeCommit.
  • Proficiency in scripting languages, including R, Python, and Bash, is required. Proficiency in JavaScript is a plus.
  • Certifications in cloud platforms, such as AWS Certified Solution Architect or Data Engineer, are preferred.
  • Knowledge and experience with ML engineering is a plus.
  • Experience with database technologies such as MySQL, PostgreSQL, MongoDB, and Amazon Redshift.
  • Experience with a version control system (preferably GitHub).
  • Familiarity with anti-poverty policies and benefit programs at the federal, state, or local level, such as SNAP, the Earned Income Tax Credit, or housing assistance programs.
  • Experience in government or non-profit work, particularly with low-income and vulnerable youth, families, and immigrant populations, is a plus.
  • Familiarity with administrative data or large datasets such as the American Community Survey, Current Population Survey, or the NYC Housing and Vacancy Survey.
  • Strong problem-solving and analytical skills with a keen attention to detail.
  • Must be able to work both independently and in a collaborative setting.
  • Excellent verbal and written skills.
  • Ability to work under occasional deadline pressure.
  • Proven experience in architecting and managing data systems is required.

  Qualifications

Must have a Bachelor's degree in computer science, engineering, information technology, data science, or a related field, plus a minimum of four years full-time experience as a data engineering role in designing building, and managing scalable and reliable data system or a cloud engineering role with specialization in cloud architects, automation, and cloud software development that build and maintain software features and functions, databases and applications for cloud technologies;

A master's degree in the aforementioned fields can be considered an equivalent substitute for two years of professional experience.

An advanced degree, such as an MS or PhD in the above fields, is preferred.

Please note: Candidates must be authorized to work in the United States. Sponsorship is not available for this position.

  Benefits and Compensation

This is a contracted, full-time position offering competitive salary and benefits. The salary range for this position is $115,000.00 - $130,000.00 commensurate with experience and qualifications.

Location: Downtown Brooklyn, NY

  How to Apply

To apply, please fill out the application. The form will prompt you to submit a resume and cover letter.

Due to the volume of applications received we may not be able to respond to every application.