Responsible for developing, implementing, and operating stable, scalable, low cost solutions to source data from client systems into the data lake, data warehouse and end-user facing BI applications. Responsible for ingestion, transformation and integration of data to provide a platform that supports data analysis, data enrichment and data science as well as making data operationally available for analysis.
- Build and maintain scalable automated data pipelines. Support critical data pipelines with a highly scalable distributed architecture - including data ingestion (streaming, events and batch), data integration, data curation.
- Deploy, automate, maintain and manage Azure cloud-based production system, to ensure the availability, performance, scalability and security of productions systems.
- Good architectural understanding to build and ensure customer success when building new solutions and migrating existing data applications on Azure platform.
- Conduct full technical discovery, identifying pain points, business and technical requirements, “as is” and “to be” scenarios.
- Design and arrangement of scalable, highly attainable, and fault tolerant systems on Azure platform.
- Ownership and responsibility for end-to-end design and development, testing, release of key components.
- Understand and implement best practices in management of data, including master data, reference data, metadata, data quality and lineage.
- Experience with code versioning tools and a command of configuration management concepts and tools, CI-CD including DevOps.
- Other duties as assigned.
Bachelor’s degree (BA or BS) from an accredited college or university plus a minimum of four (4) years of experience in the specific or related field. Or High School Diploma or equivalent plus a minimum of eight (8) years of experience in the field.
- Expert level SQL knowledge and experience.
- Expert level experience with Python/Pyspark.
- Experience with Microsoft SQL Server Integration Services, or other ETL/ELT tools.
- Experience with streaming integration and cloud-based data processing systems such as Streamsets, Kafka, and Databricks.
- Experience with Snowflake/SnowSQL preferred.
- Hands-on knowledge of cloud-based data warehouse solutions.
- Experience with cloud architecture and solutions.
- Experience with cloud-based data storage such as Azure data lake and blob storage
- Experience with Microsoft SQL Server database and Oracle database.
- Experience with Microsoft Windows and Linux virtual servers.
- Experience implementing data pipelines.
- Experience deploying developed code.
- Moderate skill in Power BI.