Methods for Vector Display of Internet Artifacts

One way to learn about distributed systems in depth is to creatively misuse them, and find the limits of their flexibility. Thus, one year when April Fool's came along, I took the opportunity to do just that - Google as a whole has quite a history of April Fools' jokes.

I am a frequent user of Monarch, Google's planet-scale monitoring system, and Panopticon (or PCon), its user interface. PCon displays monitoring graphs that contain up to 127 lines of information on how your system is doing. Monarch collects and stores metric data, and allows arbitrary querying using a rich query language: the python-based, domain-specific language called Mash. Google Cloud users may also recognize the Monarch Query Language.

A Mash query looks like so:

Fetch(Raw('example.production.Servers', '/my/service/errors'),
     {'user': 'stepand',
      'server': 'rickroll_server'})
 | Window(Align('1h'))
 | GroupBy([], Sum())

Now I wanted to take this Monarch system and creatively misuse it-that is, use it in an unusual way. What better way to do this than-wait for it-Rickroll Monarch?

For those that don't know, "Rickrolling is a bait-and-switch prank that involves posting a hyperlink that is supposedly relevant to the topic at hand . . . but redirects the viewer to the music video of Never Gonna Give You Up, a 1987 dance-pop single by English singer-songwriter Rick Astley."

So let's put Monarch and Rickroll together! How can we Rickroll Monarch?

Aside from the April Fool's joke, serious learning occurred, too. This project taught me about problem decomposition and being principled at cutting corners in exactly the right places. It helped me move from instinctive to deliberate when explaining why I felt this or that corner should be cut. And it gave me more familiarity with our monitoring system. And this prank was also a good opportunity to practice a thing or two about graphics and animation, subjects I find interesting in their own right.

Let's get started

Okay, so how do you get Rick Astley to show up dancing on your monitoring dashboard?

Bonus points: Don't melt Monarch in the process! Everything done here is an abuse of the system, of course, so I want to be careful. I need to learn a lot about the system first

From Rick Astley Video to Rick Silhouete

$ mkdir frames
$ ffmpeg -i klqi_h9FElc.mkv -r 4 frames/output_%04d.png

Now that I have each individual frame, I only care about the red part of every frame (thanks, whoever made this Webdriver Torso video!). Shell script using Imagemagick makes quick work of that and leaves only the parts I care about. I found this Imagemagick invocation on the internet and just adjusted the colors:

$ cd frames
$ mkdir processed
$ for i in *.png; do convert "$i" -level 25%,75% -fill white -fuzz 10% +opaque "#f90000" "processed/$i.pnm"; done

This is when I notice that Potrace supports "GeoJSON", a format I've never heard of before, but it conveniently dumps data into JSON. In addition, GeoJSON approximates Bezier curves by eight straight line segments. Score! Double victory!

Next, let's use CSV as a simple data interchange format. My data format is a CSV with three columns: frame number, point x coordinates, and point y coordinates. A single frame has a single silhouette, and the points are the outline of that silhouette.

I run a simple Python script and get this generated!

import csv
import json
import subprocess
import sys

with open('result.csv', 'w') as output_csv:
  writer = csv.writer(output_csv, lineterminator='\n')
  for index, fn in enumerate(sys.argv[1:]):
    print '[%d] Tracing PNM %s' % (index, fn)
    with open(fn) as input_pnm:
      p = subprocess.Popen(['/usr/bin/potrace', '-b', 'geojson'],
                           stdin=subprocess.PIPE,
                           stdout=subprocess.PIPE,
                           close_fds=True)
      stuff_from_pipe = p.communicate(input_pnm.read())[0]
      p.wait()
    print '[%d] Writing frame' % index
    data = json.loads(stuff_from_pipe)
    for item in data['features'][0]['geometry']['coordinates'][0]:
      writer.writerow([index]+item)

Woohoo, I got my CSV file with the animation!

From Rick Silhouete to Monarch Astley

Each picture now has width, which is equivalent to a time duration. It also has height, which is the value of the time series.

Originally, I'd wanted to bulk-write one frame per second into Monarch, slightly into the future according to the PCon viewport. Each second would have been the one frame I'm interested in, and I would have narrowed the PCon viewport to a one-second width. Unfortunately, this was not possible. Streamz API prevented me from writing two data points for the same metric, in the same write. I also could not write quickly, because I was hitting issues with reordered writes (recall Monarch wants them in chronological order) and my naive implementation couldn't write fast enough because of this requirement.

I still have other options:

1) Work around the reordering problem by writing one data point per one stream.

2) Create new targets or new metric fields.

3) Find another solution to the whole animation question.

I start with the first option but quickly realize that the total number of points for all frames is around 20,000. Having one stream per data point is a very inefficient use of Monarch, because it is a mismatch to a typical time series (with one stream running for a long time), and goes counter to the physical storage model. I conclude that I can't render a frame very fast this way.

Therefore, the other solution is to write slowly. Writing the entire image takes about 20 minutes now, and the viewport is one hour wide. Instead of writing just one frame, it now writes all frames. Each frame has a different target, so they can be filtered individually.

On-Screen, Data

Animate!

Okay, on to the last step: how do we animate? Well, PCon has a feature where it automatically refreshes the graph as often as you'd like.

We write individual frames to Monarch, and set the refresh interval to one second. The last thing we need is a way to decide which frame to render. Essentially, we need a clock with a one-second precision.

This simple query gives us the number of seconds since the start of the hour:

Fetch(Raw('experimental.users.stepand.and.RickAstley', '/presence/found'),
{'give': 0, 'gonna': 0, 'never': 'global', 'up': 0, 'you': 0})
 | Filter(False)
 | Window(Align('1h'), '1h')
 | GroupBy(['never', 'gonna', 'give'], PickAny())
 | JoinWithLiteralTable(target_schema_name='experimental.users.stepand.and.RickAstley',
fields=('never', 'gonna', 'give'), streams=[('global', 0, 0, True)], input_default=True)
 | Point(TimestampMicros())
 | Window(Align('1h'))
 | Point(Floor(VAL / 1000000) % 3600)

We use this query and join it with the stored data, and only filter the frame we want. Simply take the stream value modulo number of frames (which is 21), turn it into a field, and join it against the original data.

It turns out, we can also use this query to solve another problem. When displaying time series, we always see only a limited time window. After a while the image starts to slide to the left, out of the left edge of the window, and we only get a part of Rick Astley. By using time shift, however, we don't even need to write twice: we just move a single set of frames to the right! Therefore, we use this timing information to decide whether to shift right or not. The entire animation is approximately 30 minutes wide (quite a great width unit)! When the leftmost edge of the data hits the leftmost edge of the viewed window, we just shift it by 30 minutes to the right. After another 30 minutes, we remove the shift, but load data from the new data push.

Here's how it looks, all pulled together: check out the result here!

Writing On The Wall

I worry that the Rick Astley silhouette may not be immediately recognizable, so I decide to add some writing to drive the point home.

Now that we have a general vector display, the lettering is really the easiest part. I manually prepare a bunch of letters as vector paths, and then the code scales them and writes them to Monarch, just like the other vectors. There are just minor improvements, then, to the lettering over time, to make it more legible.

Final Thoughts

This was a fun project, and there were many details I had to solve which I don't go into here.

Pcon Rickroll was a good exercise in project decomposition: how to go from a bizarre and ambiguous problem statement, to concrete subproblems and their concrete solutions, while staying within the limited time budget I'm willing to dedicate to a joke. It helped me be more principled in identifying where to cut corners to maximize impact while minimizing effort: I could have rotoscoped the whole video, had a more robust vectorization, or better graphing-a colleague recommended using histograms, which can be used to construct grayscale bitmaps. However, recognizing what was "good enough" for each step helped me actually pull off this April Fool's joke. I could not spend an infinite amount of time, so this strategy made the difference between making this project feasible or not.

On a final note, I'd like to thank the outstanding engineers working on Monarch. The Monarch system is one of the most exciting services I have the pleasure to use.