Staff Data Platform Engineer, Self-Driving

Company: Rivian Automotive
Location: Palo Alto
Posted on: January 8, 2022

Job Description:

Staff Data Platform Engineer, Self-Driving Rivian -is on a mission to keep the world adventurous forever. This goes for the emissions-free Electric Adventure Vehicles we build, and the curious, courageous souls we seek to attract. - As a company, we constantly challenge what's possible, never simply accepting what has always been done. We reframe old problems, seek new solutions and operate comfortably in areas that are unknown. Our backgrounds are diverse, but our team shares a love of the outdoors and a desire to protect it for future generations. - Rivian---is on a mission to keep the world adventurous forever. This goes for the emissions-free Electric Adventure Vehicles we build, and the curious, courageous souls we seek to attract.--- As a company, we constantly challenge what's possible, never simply accepting what has always been done. We reframe old problems, seek new solutions and operate comfortably in areas that are unknown. Our backgrounds are diverse, but our team shares a love of the outdoors and a desire to protect it for future generations.--- The -ADAS/Self Driving -org at -Rivian -is seeking a Staff -Cloud -Data Platform Engineer to join the -Self Driving Cloud -team, reporting to the -Self Driving -Sr. -Cloud Manager. We are looking for data engineers with strong software engineering backgrounds, a "platform engineering mindset"---and deep passion for data -and automating data processing in the cloud. You will be a part of -the -Rivian -Self Driving team helping to realize fully autonomous driving in our vehicles. Responsibilities :

• Build scalable real-time as well as batch -vehicle -data ingestion and data processing pipelines. Choose the appropriate purpose-built technology based on the data patterns and use-case needs.
• Build -generic ingestion frameworks as well as individual pipelines using Kafka, Kinesis, serverless lambdas with API Gateway, HTTP 2, GRPC streams, Change Data Capture tools, -GraphQL, -etc.
• Assist the data science ecosystem with data cleansing, wrangling and feature engineering on AWS -SageMaker.
• Build modular set of data services for data transformation and business logic processing using container microservices, Apache Spark on AWS EMR, ETL technologies like AWS Glue, -dbt -and---SQL.
• Build automated unit and integration testing pipelines using frameworks like Cucumber, -PySpark.
• Create and manage CICD pipelines -and data infrastructure, experience with Terraform -highly preferred.
• Manage the orchestration of code artifacts with Kubernetes using helm charts and job orchestration with Apache Airflow, Kubeflow and/or -SageMaker.
• Partner with the BI team to load right data sets into AWS Redshift or Snowflake -to support visualizations.
• Design and build data quality management services with Apache -Deequ -or Amazon DQAF.
• Provide a variety of query services with REST, -gRPC, -Athena/Presto, server sent events.
• Run static and dynamic code scans with Prisma, -Checkmarx, SonarQube etc.
• Configure and setup the enterprise data lineage and meta data management and data catalog support using tools like Collibra/Atalion.
• Build appropriate logging of metrics and diagnostics in code to enable effective monitoring and data observability using Splunk, -DataDog, Grafana or Prometheus.
• Perform code reviews with peers and ensure green builds for deployment.
• Use Agile -to manage projects and tasks. Qualifications:
  • At least---8-10 -years building software engineering platforms and applications using Java, Scala, Go, Python and SQL.
  • 8-10 -years of experience with a modern data engineering platform on AWS using MSK, Kinesis, AWS EMR,---AWS Athena, -AWS Glue, lambdas, API Gateway, -GraphQL -etc.
  • Experience with native cloud development on the AWS ecosystem and event-driven architectures and pub-subs like Kafka, Kinesis, AWS Event Bridge, -Flink, web sockets and SSE is a must.
  • Experience with batch data processing using Apache Spark on AWS EMR is a -like to have.
  • Experience with structured Spark streaming is highly desired.
  • Ability to assist the BI team in creating the right set of materialized views in a SQL data warehouse like Redshift/Snowflake.
  • Experience in building highly scalable data architectures---using modern OLAP cube like -Kylin, Apache Druid is a plus.
  • Experience -with -Kubernetes -is highly desired.
  • Familiarity with machine learning and data science ecosystems especially AWS -SageMaker -or Databricks is highly preferred.
  • Experience -designing relational -DB -and/or -NOSQL -solutions. Rivian is an equal opportunity employer and complies with all applicable federal, state, and local fair employment practices laws. All qualified applicants will receive consideration for employment without regard to race, color, religion, national origin, ancestry, sex, sexual orientation, gender, gender expression, gender identity, genetic information or characteristics, physical or mental disability, marital/domestic partner status, age, military/veteran status, medical condition, or any other characteristic protected by law. Rivian is committed to ensuring that our hiring process is accessible for persons with disabilities. If you have a disability or limitation, such as those covered by the Americans with Disabilities Act, that requires accommodations to assist you in the search and application process, please email us at -accessibility@rivian.com . Please note that we are currently not accepting applications from third party application services.

Keywords: Rivian Automotive, Palo Alto , Staff Data Platform Engineer, Self-Driving, Engineering , Palo Alto, California