About the Role
As a Senior Data Engineer at Kiana AI, you will be responsible for designing, developing, and maintaining robust data pipelines and architectures that support our AI and machine learning initiatives. You will work closely with data scientists, software engineers, and other stakeholders to ensure the availability, reliability, and scalability of our data infrastructure.
Requirements
Key Responsibilities:
Design, develop, and maintain robust data pipelines and ETL processes using Azure Data Factory, Databricks, and other Azure services.
Collaborate with data scientists, analysts, and other stakeholders to understand data requirements and deliver high-quality data solutions.
Optimize and maintain data storage solutions, including Azure Data Lake, SQL Database, and Cosmos DB.
Implement and manage data integration solutions to ensure data consistency, accuracy, and availability.
Monitor and troubleshoot data pipeline performance, ensuring reliability and scalability.
Develop and enforce data governance and security standards to protect sensitive data.
Mentor junior data engineers and provide technical leadership within the team.
Qualifications:
Bachelor's degree in Computer Science, Engineering, or a related field (Master's degree preferred).
5+ years of experience in data engineering, with a focus on cloud-based solutions.
Proficiency in Azure Cloud services, including Azure Data Factory, Databricks, Data Lake, SQL Database, and Cosmos DB.
Strong programming skills in Python, SQL, and experience with ETL tools.
Familiarity with data modeling, data warehousing, and big data technologies.
Experience with CI/CD pipelines and version control systems (e.g., Git).
Excellent problem-solving skills and attention to detail.
Strong communication and collaboration skills.
Preferred Skills:
Experience with data orchestration tools (e.g., Apache Airflow, Luigi).
Knowledge of machine learning and AI frameworks.
Familiarity with containerization and orchestration tools (e.g., Docker, Kubernetes).
Experience with data visualization tools (e.g., Tableau, Power BI).
Experience with machine learning and data science frameworks.
Certification in Microsoft Azure (e.g., Azure Data Engineer Associate).
About the Company
Kiana AI is a leading generative AI consulting firm dedicated to helping businesses harness the power of artificial intelligence to drive innovation and success. We are seeking a highly skilled Senior Data Engineer to join our team and contribute to our advanced data projects.