Data Engineer- ModelOps
Typical Day in Role:
• Building, architecting, designing, and implementing highly distributed global cloud-based systems in Google Cloud Platform
• Crafting and improving workflows in GCP using standard, modern components such as Dataflow, BigQuery, PubSub, GKE, etc.
• Developing applications using Google Cloud Platform
• Developing data pipelines using Spark
• User of Docker and on-prem Kubernetes clusters
• Optimization improves query speed and reduces cost in BigQuery
• Monitor the run time of the models, work closely with Data Scientist on continuous optimization
• Bring relevant information from different data sources to allow initial data exploration and model training
• Data integration through collaboration and standardization in Cloud Data Fusion
Candidate Requirements/Must-Have skills:
1. Background in Data Engineering
2. 4+ years of working experience with Google Cloud Platform such as PubSub, BigQuery, Dataflow (Apache Beam), Airflow, Looker Studio
3. 4+ years of working experience on the following tools and programing languages:
o Google cloud Platform, Python, Scala, Spark, Linux and Unix familiarity
4. 4+ years of working experience with CI/CD tech stack:
o Google Cloud Source Repositories/Git/BitBucket, Jenkins, Artifactory, Kubernetes
5. 4+ years of working experience with GCP products for Data Science and Engineering:
o Vertex | Workbench AI, AutoML, Cloud Natural Language, Dialogflow, Speech-to-Text
o Timeseries Insights API, Deep Learning VM Image
Nice-To-Have Skills:
• Working experience with MinIO Data Storage, Airflow and Trino is an asset – It would be good to have to support our legacy systems
Soft Skills:
• Excellent written, presentation, and verbal communication skills to be able to work well with technical peers and business stakeholders at different levels within the organization
• Strong decision making, forward thinking and creative problem-solving skills to anticipate and respond quickly to technological/market influences
• Outstanding analytical thinking
• Ability to work as part of a team, as well as work independently or with minimal direction
Education:
• University education in Computer Sciences, Software Engineering, or equivalent experience