Senior Data Engineer
New Hyde Park, NY 11040 | Contract
The Data Engineer is responsible for processing structured and unstructured data, validating data quality, and developing and supporting data products. The Data Engineer also plays a role in Agile planning, providing advice and guidance, and monitoring emerging technologies.
-Gathers and processes raw, structured, semi-structured, and unstructured data using batch and real-time data processing frameworks.
-Understands and enforces appropriate data master management techniques.
-Ensures data quality and implements tools and frameworks for automating the identification of data quality issues.-
-Work with internal and external data providers on data validation providing feedback and making customized changes to data feeds and data mappings.
-Understands the challenges that the analytics organization faces in their day-to-day work and partner with them to design viable data solutions.
-Recommends improvements to processes, technology, and interfaces that improve the effectiveness of the team and reduce technical debt.
-Implements and optimizes data solutions in enterprise data warehouses and big data repositories.
-Installs, maintains, monitors, and supports business intelligence, distributed computation, and big data analytics tools.
-Provides ongoing support, monitoring, and maintenance of deployed products.
-Actively works with less experienced data engineers providing technical guidance and oversight.
-Experience in design, development, and implementation of highly scalable, high-volume Analytics Systems
-Data modelling methodologies (e.g. Star, Snowflake, Relational), data modelling tools (e.g. Erwin), data access, and data storage techniques
-Experience in performing data profiling, analyzing data flows, and setting up best practices / processes for the Development and QA teams
-Working as a liaison between Product Management and Data Engineers to translate business rules to technical transformations
-Setting up batch & real-time data processing frameworks for high volume heterogeneous data using MS SSIS
-Development and tuning of MS SQL Server Database (e.g. Tables, Indexes, Views, Stored Procedures, Functions)
-Experience working with Relational databases (e.g. Oracle, DB2) and NoSQL databases (e.g. MongoDB)
-Knowledge of Business intelligence tools and platforms (e.g. MS SSRS, Tableau)
-Working experience with application lifecycle methodologies (e.g. waterfall, agile, iterative)
-Bachelor’ s degree in Computer Science or related field or equivalent combination of industry related professional experience and education. Masters may be preferred in some areas.
-5 – 8 years of working experience
-Working experience with Python and Amazon Web Services (e.g. S3, EC2, RDS, Lambda, EMR, Data Pipeline, DynamoDB)