Senior Software Engineer, Site Reliability
We are looking for an experienced systems engineer to design and implement deployment automation, automatic failover, reliable backup/restoration, and scalable data storage strategies that can be distributed globally.
Our technology stack is based on Haskell, Postgres, NixOS and we deploy to AWS with NixOps. Our ideal candidate is well versed in all these technologies, but most importantly has experience managing, scaling and monitoring high performance web services with any stack.
Additionally, we desire candidates with continuous deployment/delivery experience.Candidates should be familiar with being part of an on-call rotation, and developing a runbook. We rely on our continuous integration server to build, run tests, and provide binary caches of compiled assets to accelerate compilation. We automate anything we can. We know the value of a two line shell script and a one step build.
A typical week might include
- Adding Varnish collectd plugin and useful alerts from the metrics.
- Designing an automatic failover strategy for our database setup.
- Working with backend engineers to plan out a database sharding strategy.
- Setting up auto scaling of web hosts based on cpu idle.
- Investigating production issues.
- Adding a log aggregation system, so logs are easily searchable.
- Developing an offline database job system, to facilitate zero downtime migrations.
- Investigate anomalies in our metrics.
- Experience using Haskell in some capacity is required.
- Aware of NixOS and a desire to learn how to administer systems with it.
- Experience with running large-scale web service on AWS.
- An understanding of network protocols and theory.
- An understanding of linux kernel and shell systems.
- BA/BS in Computer Science, related technical field or equivalent practical experience.
- Minimum 4 years experience writing scalable, reliable software.
- Expertise in designing, analyzing and troubleshooting large-scale distributed systems.
To apply, send your cover letter/resume to firstname.lastname@example.org