Writing a server and getting it on the internet can be pretty complicated by itself. But what happens once people start using your service: how do you update your running server without dropping requests?

You can’t just turn things off and turn them back on again like in development, especially if your service provides a critical utility that users expect to be working, because you’ll drop in-flight requests as soon as you turn things off until your service comes back up.

Here’s what that would look like:

(requests are shown as arrows above server states)

You can see how any requests that overlap with the intermediate transition phase…

All apologies.

for (( ; ; )) do pod2kill=$(kubectl get pods — namespace=YOUR_NAMESPACE | tail -n +2 | awk ‘{print $1}’ | sort -R | tail -1); echo “Killing pod: ${pod2kill}…” && kubectl — namespace=YOUR_NAMESPACE delete pod ${pod2kill} && sleep 20; done

This is not a very good chaos-monkey for a million reasons (it always kills machines politely, it only hits in one namespace, happens on a regular period, etc, etc) but it has been a useful tool for me as we make our app more resilient to server restarts :)

One thing I struggled with initially while using Kubernetes (k8s from here on out, — lazy fingers) was knowing… what was running where? Because I wasn’t really proficient with the kubectl tool yet I was constantly poking around the GKE UI looking for my workload and pod and then logs, which was a really slow way to do things!

One of the first things that helped me was this little bash snippet to see and switch between all my GKE clusters. …

More than ten years have passed since I seriously wrote words on the internet! (It’s not even called blogging anymore, is it?) How wild is that?

Last January, after 7 years of working for the man, I left Google to build something new. It feels good to be working on something I can talk about again :)

We’re making developer tools that provide a “Google-style” development environment for *anyone* who wants it. We believe that everyone can benefit from having a team solely focused on improving your developer experience.

Our first product is BuildBuddy: an open-core cache, UI, and remote execution platform for Bazel.

Anyway, lots more to come, I hope!

Tyler Williams

14k gold slum computer wizard

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store