Data Engineer OKR Examples

Design and implement data pipelines

The goal of this objective is to design and implement new data pipelines to keep our business scaling

• Run 1 POC for a data acquisition pipeline and create a final report for the CTO
• Run 1 POC for a data transformation pipeline and create a final report for the CTO
• Run 1 POC for a data visualization pipeline and create a final report for the CTO

Run a POC on AWS Glue for cost reduction

The goal of this objective is to cut data processing costs

• Run for 1 week in parallel our current pipeline from AWS EMR on AWS Glue
• Migrate parts of the pipeline to AWS Lambda to save 20% of the current costs for transformation
• Prepare a final report and get approval on the migration decision from AWS EMR to AWS Glue

Create a roadmap for migration from Redshift to Snowflake

The goal of this objective is to remove the overhead of managing the data infrastructure

• Run consultation session with Snowflake's success team and develop a migration guide
• Implement a backup strategy to guarantee zero data loss during and after migration by the end of January
• Finish the migration and shut down all Redshift clusters by the end of February

Improve the accuracy of our NLP models

Develop a new NLP pipeline and benchmark it against OpenAI

• Develop and implement a new NLP pipeline with at least 90% accuracy on benchmark tests
• Benchmark the new NLP pipeline against OpenAI's models and achieve comparable or better results
• Integrate the new NLP pipeline into our existing SaaS platform and ensure seamless integration with other features

Increase the range of tasks our NLP models can handle

Expand the capabilities of our NLP models

• Research and identify new tasks that our NLP models could potentially handle
• Develop and implement methods to expand the capabilities of our NLP models to handle the identified tasks
• Test the expanded NLP models on a variety of real-world data and demonstrate improved performance on the new tasks

Reduce the time and resources required to run our NLP models

Optimize the efficiency of our NLP pipeline

• Implement optimization techniques such as parallel processing and batching to improve the runtime of our NLP pipeline
• Test the optimized NLP pipeline on a variety of real-world data and demonstrate a significant reduction in runtime and resource usage
• Integrate the optimized NLP pipeline into our existing SaaS platform and ensure it does not negatively impact other features or performance.

Improve infrastructure

Responsible for collaborating with data scientists and analysts to build and improve data and ML products that drive revenue for the company

• Define new infrastructure programmatically in Terraform
• Convert existing infrastructure to scalable solutions on GCP
• Assess and implement appropriate technologies such as Kubernetes

Improve model deployment and monitoring

Responsible for deploying, maintaining, and monitoring machine learning models used by the company

• Develop a system for deploying ML models efficiently and reliably
• Implement monitoring tools to track model performance and accuracy
• Create processes for regularly evaluating and updating models

Improve data pipeline performance and reliability

Responsible for building performant and reliable data pipelines using tools like Airflow and BigQuery

• Optimize existing data pipelines for improved performance
• Implement measures to ensure data pipeline reliability, such as error handling and alerting
• Collaborate with data scientists to design and implement efficient data pipelines for new projects

Improve cloud infrastructure

Responsible for maintaining and improving the cloud infrastructure used by the team

• Monitor and maintain existing cloud infrastructure to ensure stability and uptime
• Implement improvements to infrastructure, such as scalability and cost optimization
• Research and evaluate new technologies and services to incorporate into the cloud infrastructure