Data Engineer

Company: Brooksource
Location: Minneapolis
Posted on: April 10, 2021

Job Description:

Do you love big data and the challenges it entails? Do you want to create applications that people love?
As a Data Engineer, you will work cross-functionally with data scientists, data analysts, product managers, and stakeholders to understand business needs and develop, maintain and optimize the data sets, data models and large-scale data pipelines. You'll primarily leverage the Azure Databricks Spark cloud stack used for data science models and visualizations. You will partner with various teams to drive best practices and set standards for data engineering patterns and optimization. You are a key influencer in data engineering strategy.
This is a unique, high visibility opportunity for someone who wants to have business impact, dive deep into large scale data pipeline and work closely with cross functional team. This position would be fully remote, and report directly to the Director of Data Science.
If you feel that this describes you, keep reading and apply!
Required Qualifications:

• 5+ years of experience optimizing and managing modern large-scale data pipelines ETL/ELT processing to support data integration for analytics, machine learning features and predictive modelling.
• Expertise within Azure Databricks, Delta Lake and Spark
• Experience writing advanced / complex SQL with performance tuning and optimization.
• Experience designing and building data service APIs.
• Interest in and proven ability to mentor and coach other data engineers.
  Responsibilities:
  • Design and develop ETL/ELT solutions on Azure Databricks, Delta Lake and Spark to support Digital MBO's.
  • Develop, implement, and deploy large scale data pipelines powering machine learning algorithms, insights generation, business intelligence dashboards, reporting and new data products.
  • Create and maintain the technical architecture of the Enterprise Delta Lake to consolidate data from many systems into a single source for machine learning and reporting analytics.
  • Consume data from a variety of sources (RDBMS, APIs, FTPs and other cloud storage) & formats (Excel, CSV, XML, JSON, Parquet, Unstructured)
  • Identify ways to improve data reliability, data integrity, system efficiency and quality.
  • Participate in architectural evolution of data engineering patterns, frameworks, systems, and platforms including defining best practices and standards for managing data collections and integration.
  • Work with data scientists to deploy machine learning models to real-time analytics systems.
  • Design and build data service APIs.
  • Mentor other data engineers and provide significant technical direction by teaching other data engineers how to leverage cloud data platforms.
    
    Eight Eleven Group provides equal employment opportunities (EEO) to all employees and applicants for employment without regard to race, color, religion, national origin, age, sex, citizenship, disability, genetic information, gender, sexual orientation, gender identity, marital status, amnesty or status as a covered veteran in accordance with applicable federal, state, and local laws.--

Keywords: Brooksource, Minneapolis , Data Engineer, Engineering , Minneapolis, Minnesota