Replacing a live telemetry pipeline without breaking the dashboards

What I owned

Amazon's delivery associates run an app on their routes. It reports telemetry from the device, and operations teams watched that data on dashboards. I owned the pipeline that carried the telemetry and the client library on the device that produced it. The fleet grew from 70K+ associates to about 1M registered and 600K+ active in that time, and I re-architected both for the growth.

Moving live traffic

Operations depended on dashboards built on this data. So old and new ran in parallel, and live traffic moved onto the new pipeline while the old one kept running: about 1.1 TB a day, into a Redshift store of more than 100 TB. The dashboards kept working through the migration.

The query layer

I scaled the Redash query layer so analysts and engineers could query the whole 100+ TB store.

Engineers and product managers asked for a second, bigger database for their ad-hoc queries. I said no. A second store meant paying twice and maintaining twice. Their ad-hoc questions were waiting behind the scheduled dashboard queries, so I added priority queuing to Redash.

Then everyone marked every query as priority. I restricted the priority option to on-call engineers and managers, and the complaints stopped.