Livestreaming done three ways

Like almost everyone else on the planet, my church has recently and rapidly become acquainted with a host of new technologies designed to let us keep in touch despite the necessary partial lockdown the UK finds itself under. During this time, people are being asked to remain in their homes unless necessary (for definitions of "necessary" that vary in strictness); this includes church staff and production team on Sundays! In the hope that someone, somewhere might find it useful - or at the very least, mildly interesting - I'm sharing the approaches we've taken as a church to continue being the body of Christ even in our social isolation.

It's fair to say that there's been a large amount of figuring things out as we go, and our processes have been changing almost as much as the UK government's advice; by no means are the things I write about here definitive - there are many, many ways to skin this particular bantha, and I'm sure we'll continue to make changes as time goes on, until eventually we can get back to "normal" - whatever that means!

Week 1: Same same, but different

Initially, UK government advice was to avoid gatherings of more than 20 people. This meant we couldn't meet in church as normal - we get in the region of 1,000 people a Sunday, spread over three services - but we could still bring a small team to the church building and produce a live-streamed service from there (while each maintaining a suitable physical distance to the others, of course).

This was useful for a number of reasons, mostly because it meant we could make use of our existing production infrastructure: our sound desk and video mixer, and all the equipment that comes with that setup; we could have used our regular PTZ cameras, but our media and communications manager set up a couple of manually-operated cameras instead for more precise control. The increased latency of the image from those cameras would have been a problem in an IMAG setting, but there are no such concerns for the live stream; I eyeballed an approximate 100ms delay and added that to the audio capture.

The output of the video mixer was then fed into a Blackmagic Design Web Presenter, and from there into a laptop running OBS Studio. The OBS setup was pretty minimal; one scene with a "service starting soon" static image, and one with the Web Presenter's output full screen. Song lyrics were overlaid on the camera image in the video mixer, following our normal process, so we could literally stream the unaltered output to YouTube.

The Web Presenter is only capable of 720p, but on a fundamental level - who really cares it's not 1080p? Our video system (run over old cabling) is only capable of 1080p25 anyway; and the limiting factor would almost certainly become the church network which was already being taxed by the lower-resolution stream, leading to a couple of stutters during the service.

Lots of thought went into staging and framing of camera angles to avoid the "big empty building" feel; and, well, the rest of the room wasn't at its most presentable, owing to a storage room being deep-cleaned that week!

Sunday morning live stream: the "when we could still get in the building" edition.

As well as the morning live-stream, church set up a Sunday evening meeting on Zoom, intended as a less formal and more interactive "call to prayer" event, with clergy hosting from home. Zoom was also adopted to facilitate the day-to-day operations of the church with its staff spread out across the city.

I had no formal involvement with running this evening meeting, but I did run an "unofficial" lyrics stream during the sung worship: capturing the output of JustWords into OBS and rendering to a "virtual webcam", set as my camera for Zoom. (Actually, it would have been better to just use Zoom's screen-sharing features, since that would have allowed viewing the lyrics side-by-side with the worship leader.)

Week 2: When all you have is a hammer

Within a few days of that first stream, Government advice changed to more closely match the rest of the world: stay at home, unless necessary (subsequent days would see various Government ministers disagree on what "necessary" meant, but this isn't supposed to be a politics blog!). In the face of contradictory advice from Government, the Archbishop of Canterbury (in charge of the Church of England of which my church is a part) ordered church buildings to be closed entirely, which meant that our previous solution was no longer viable.

With little time to come up with a more comprehensive solution, and having some experience with it already, church decided to use Zoom for the morning service too, with our rector presenting from his study at home, and our media manager inserting pre-recorded segments via Zoom screen-share. Our worship pastor had recorded a selection of songs, and someone overlaid captions, so they were ready to be rolled out in whichever order was chosen for a service. Other contributors were also incorporated live via Zoom; and, exceptionally, the wider church were able to view proceedings either via YouTube or the Zoom call itself, much more like the Sunday evening meetings.

Zoom has the facility built-in to stream to YouTube (hint: if you want to stream to a pre-scheduled YouTube event, you need to use the "custom live streaming service" option and not the "YouTube" one!). And, you know what? It works, and I'll take a working system over a broken one any day. It did leave some room for improvement, though:

• The massive Zoom watermark in the bottom of the screen that can't be removed, only changed (if you have special permission to do so) - as someone said, "I was surprised to see the watermark considering what we are paying/month!"
• Screen-sharing for videos was a little ropey, with quality loss and stuttering (much like a standard Zoom meeting); and it also appeared as a split-screen view with whoever spoke last appearing in the corner of the screen. Not only was that a bit distracting, but it also meant we could see them talking or texting during the songs! (Almost certainly on the WhatsApp group set up to orchestrate these live meetings - but there's no way for the congregation to know that!)
• There was no control over which participants appeared in the stream output - it was stuck on Zoom's slightly unpredictable "most recent speaker" view, and someone therefore had to be vigilant about muting and unmuting participants throughout. There was no option to remove the name tags from the corner of people's screens, either.

Playback of a prerecorded video clip during a Zoom meeting streamed to YouTube.

Could we improve on this setup? Well, I like to think so, and it'd be a much shorter blog post otherwise...

Week 3: The kitchen (virtual audio) sink

I was asked for suggestions on how to take the same basic service "elements" and accomplish a similar stream "in a way that makes it easier or increases quality". Whenever coming up with a technical solution, it's important to bear in mind the specific goals: here, we're not aiming to produce a radically different output, but instead make iterative process and technical improvements to achieve those two core goals, ease of operation and quality of output. This process of continuous evaluation and refinement is common in the software industry, where a "minimum viable" product with a bare-bones set of features can slowly, incrementally gain more functionality over subsequent releases.

I'm going to go into more detail here than for the previous two weeks, partly because it's necessary to encapsulate some of the complexity, but also because it's the current marker of "we are here" and might serve as a useful milestone.

At a glance

The fundamentals of the current solution are:

• Live participants join a private Zoom meeting
• Screen- and audio-capture from Zoom into OBS
• Pre-recorded content streamed from OBS
• Slides run through screen-capture into OBS
• Audio and video foldback routed from OBS to Zoom input via virtual webcam output

I'll look at each point in turn, saving the most complex until last.

During the live stream. Left-hand screen: OBS advanced audio settings. Right-hand screen: OBS in Studio Mode. The mug of tea is an essential component. This is after I had tidied my desk.

Live participants join a private Zoom meeting

I mentioned that Week 2's stream allowed the congregation (indeed, anybody with the meeting number) to join the call alongside the service participants. Even before Zoom's now-well-documented security problems were brought into sharper focus, this had been identified as less than ideal; if nothing else, it was splitting the congregation across two different media, so those on the Zoom call couldn't see the YouTube community chat and vice versa. It's also less possible to curate the experience for someone on the Zoom call, since what they see and hear will depend on their settings, their choices within the meeting, and their platform (I'm reasonably sure that Zoom's clients behave differently on iOS and Android than desktop).

The wise decision was made to limit the Sunday morning Zoom meeting to active participants in the service only, plus one person acting as administrator, and the AV operator - that was me! 

Capture Zoom output in OBS

OBS has built-in tools to capture the output of a specific window, and to capture desktop audio, adding them into scenes as required. This was the mechanism used to send video and audio from Zoom participants to the YouTube live stream.

There are some settings in Zoom that made this easier. 

• Unchecking "Always display participant names on the video" cleaned up that part of the output. 
• Tucked away under "Share Screen" is a setting to "Use dual monitors" whose behaviour is counter-intuitive but ended up providing us exactly what we needed. 

With the "dual monitors" option selected, joining a meeting will open up two video windows: the first, as normal, with Zoom's meeting controls and the option to switch between gallery and speaker views; and the second whose behaviour seems to change depending on how many participants there are in the meeting - which really threw me at first! With two people, it will show you your own video - not particularly useful. But with three or more participants, it will show the current speaker's video, but stripped of all the UI elements Zoom normally renders over the top. It was this second window that was captured by OBS.

A few more things to consider. Zoom doesn't restrict the video window to the aspect ratio of the source content, so if the window is anything other than a perfect 16:9 it will either add black bars or crop the video (perhaps both). I needed a couple of rounds of tweaking the window size until it perfectly lined up with OBS's output canvas. Making the window full-screen wouldn't have helped me as my monitors are 16:10!

Unlike the "main" Zoom window, the secondary window won't display a toolbar when the mouse moves over it or someone speaks in chat. Nevertheless, since OBS can still capture the window from a different virtual desktop, I set it up on its own and left it there during the stream.

In this configuration, Zoom also gives you the ability to "pin" participants' videos to either window. We could have used this to override the "current speaker" detection, but I didn't really have the cognitive bandwidth to want to do that; plus it starts adding additional UI chrome to the otherwise clean video, so I chose to avoid it. "Current speaker" was good enough. (I could have cropped the video in OBS to exclude these elements if I'd really wanted to.)

I set up a couple of scenes in OBS with both the Zoom window video and desktop audio: one full-screen, which we used most of the time for the live content, and one with a shrunk-down Zoom window and space for slide content. Each of these scenes needed both the audio and video source in. (I could probably have made those two an OBS group and copied them over, but there wasn't really a need.)

Pre-recorded content streamed from OBS

This was quite straightforward, but did highlight a shortcoming of OBS: the lack of any sort of media controls when playing back videos.

I set up one scene per video, with the video full-screen and set to restart playback when it becomes visible. I needed to adjust the audio level for most of them (and in one case add a compressor) because the levels were a bit uneven and in many cases too close to clipping; a normalisation to -5dB or so when rendering would have been useful. One of the videos had been shot vertically, so I composited it over a generic background image.

I numbered the scenes and videos sequentially - at least, they were sequential until I got a final copy of the script with an additional video inserted halfway through! - and annotated the running order with these identifiers, along with video duration and, for the ones with spoken content, the final sentence as a cue to move to the next scene. It would have been really useful if OBS were able to display a progress bar or, ideally, a timer against a playing video.

Production notes for the livestream, with video identifiers, timecodes and in/out cues.

It would also have been useful - in rehearsal at least - to be able to seek within the video as it played. That way, we would have discovered that the video that was supposed to contain two worship songs back-to-back in fact contained three! A quick exchange of messages with the rector, and a (hopefully) well-timed crossfade, cut the interloper short without it standing out too much...

I could have run an external media player and captured that in the same way I was doing with OBS, which would have given me greater control; but I wasn't sure if that would reduce the quality of the stream output (remember our goals) and would have made a complicated audio setup even more so, since I was already capturing desktop audio from Zoom and didn't want to be broadcasting sound from the Zoom call during the pre-recorded segments.

Slides run through screen-capture into OBS

Another straightforward component once the initial content preparation work had been done: I'd been provided with some content slides as a PDF document, but out of preference I converted those into a series of images and dropped them into a presentation (LibreOffice Impress, as it happens, but other slide-based presentational media packages are available).

I set up the usual two-screen presentation options, with "presenter view" on one and the slide show on the other, then captured the slide show window in OBS. This sat on another virtual desktop which I switched to only to change slides - and there was little enough slide content that I didn't spend much time here at all during the service.

Audio and video foldback into Zoom

All of the above was relatively straightforward, but left one significant hole in the setup: there was no way for the participants of the Zoom call to be able to see and hear the prerecorded media as it played out to the stream. (Watching the stream wasn't an option, since there was about a 25-second delay on it.) It's also useful to have a way to communicate between the "control room" (err, me) and the participants, to cue in/out of video and as a reassuring familiar face running the show.

The video was the easier of the two parts. I set up a "virtual webcam" clone of OBS's output, and set that as the video source in Zoom's settings. Since I run Linux, that meant compiling the v4l2loopback kernel module (Ubuntu's package is an out-of-date version that doesn't work properly) and installing the OBS v4l2sink plugin. Windows users can make use of obs-VirtualCam by the same author. (Sorry, Mac users, you're on your own in the walled garden; there's an alternative later that might work for you.)

Audio was more complicated. The solution ended up as:

• Set the main audio mix of the computer as OBS's monitor device
• Set the monitor of that mix as OBS's input
• Make sure the audio for all prerecorded content was set to "monitor and output" in OBS' advanced audio settings window
• Ensure that the "desktop audio capture" device was NOT set to monitor, or else it would have created a feedback loop
• Rely on the fact that Zoom never plays you your own audio, so audio sent via monitor wouldn't end up on the stream output

There are almost certainly better ways of doing this, but nobody wants to research PulseAudio's virtual sink and loopback devices on a Saturday evening, and that wouldn't help Windows users anyway.

It also doesn't help that a bug in OBS means the first point is impossible (at least on Linux) - it lists only audio sources as potential monitor devices, when you need to be able to select an audio sink! So I needed to leave it on "default" and set it using the PulseAudio volume control instead. 

Likewise, Zoom's device selection menus weren't up to the task, so I ran it wrapped by padsp (not sure that was actually needed), set its own settings to "default", then changed what the default was from PulseAudio.

What would have been really useful is a second monitor output from OBS. That would have let me set up one virtual audio sink receiving Zoom audio, which I could monitor locally as well as send to OBS' main output, plus a second one to receive OBS' other monitored audio to feed into Zoom.

Before you ask: I don't use Windows, so I have no experience of setting this sort of thing up there; Virtual Audio Cable looks like a good place to start.

One thing to make participants aware of is that the foldback video they see will have been through Zoom's compression at least once, if not twice: in their words, it will look "fuzzy" - and it will also be delayed, which might throw them if they're not used to seeing themselves on screen while presenting. Reassure them that the real live stream will look much better than that, and it's a technical limitation of the system being used to show them a copy of it.

I had an extra scene in OBS with a local webcam/mic so that I could participate in conversations during the rehearsal and setup, with the audio from the mic configured to only be sent to OBS's monitor output - even if I'd accidentally sent the scene live, nobody would have been able to hear me swear as I realised and changed back!

Alternatives?

Ostensibly, an alternative would have been to say to all participants, "watch the videos in advance so you know what you're linking to/from". It's not a good alternative, but it is the most straightforward!

A feature of OBS I've not played around with too much are "projectors". It's possible to take a copy of OBS's rendered output and display it in a window. It might be worth trying to use Zoom's screen-sharing feature with that window to provide the foldback channel, rather than a virtual camera; I suspect that might have a detrimental effect on the dual-screen Zoom setup we introduced earlier, but it's an option where a virtual webcam isn't a possibility, and it's worth investigating. (I think it might replace the video output of the second screen with the shared screen content; but I could be wrong, and it could do that in the primary window, in which case it's all good.)

If your computer has a line-in (not a microphone in) you could try setting up a second computer on the same Zoom call to act as an audio source, and see if the jitter is low enough that a constant delay one way or another is enough to match it up to the video from the first computer, but I wouldn't hold out much hope. You'd also need to feed OBS's monitor output to that computer so that the foldback audio was from the same Zoom participant as the video - or else it'd cut to the wrong user in its "current speaker" view.

Actually running the stream

All of the above discussion focuses on the system design and setup, which as always takes the lion's share of the time. With all of that sorted, actually running the stream ran fairly smoothly, mostly a case of following the script and clicking appropriate buttons at the right time. Since my audio bus was in use for exchanging audio with Zoom, I wasn't able to monitor the YouTube stream preview for audio levels (or at least, I wasn't supposed to - as I realised a second after unmuting it!) but fortunately my wife was watching the stream downstairs, so I could pop down there quickly to check that output levels were sensible. (Would be really nice for OBS to have audio meters for its main output mix.)

In rehearsal, I'd noticed that sometimes switching to Zoom would cause frame rates to tank and system load to shoot up, so I avoided switching to it during the stream, just leaving it on a virtual desktop on its own.

There were a couple of mid-service instructions from our rector, which I wasn't always able to accommodate. In particular, I'd set up the video for the offering song to include text detailing how to give; I was asked to take that text down halfway through the song, but I didn't want to transition to/from the scene since I wasn't sure if that would restart the video. Turns out, I could quite straightforwardly have done so - but wasn't brave or foolish enough to try and find out in the middle of the livestream! 

(For the record: a source set to "restart playback when it becomes visible" doesn't restart if it was already visible - even if that's on a different scene. So I could have had the same video in two scenes, one with text and one without, and transitioned between them without affecting the playback.)

Back to the goals

In system design, as in software design, it's important to keep going back to your original goals and checking how closely you're tracking to them. To save you scrolling back, here was the original goal statement, said in comparison with the "week 2" solution:

[accomplish a similar stream] in a way that makes it easier or increases quality

Let's break that into two parts.

Makes it easier

On the surface, you might consider that anything requiring as much exposition as the above can't possibly have succeeded in achieving an "easier" goal.

But much of the effort and difficulty was in coming up with a working solution in the first place. That done, it'll be straightforward enough to replicate in future weeks - I'll just load up OBS and Zoom, check the audio routing is still working, make sure I'm capturing the right window, and set up that week's videos as needed. (Again, the setup takes time, but none of the individual steps involved were particularly difficult).

Even then, I think there's still a big win in terms of in-stream ease of use. Here's the procedure our media manager sent describing what he did to show a video in week 2:

...making the transition smooth requires two screens and adhering carefully to the below simple steps (I practised this for about 15 min)

Getting ready:

1. Open video in video software (don't play yet) move it to a screen were you don't have anything else, and go into fullscreen mode
2. in zoom chose 'share screen' and select the video software, also tick both tick boxes (these will be ticked by default from now on)  don't click share button yet! 

When service leaders gives the cue:

1. Play video in video software, (i've left a few seconds of quiet at the beginning of each song)
2. click share screen  

When video ends

1. click 'stop sharing'
2. close video player window
3. get next clip ready asap using above steps

Transitioning between scenes in OBS is considerably easier than that!

But the production team aren't the only people (or person) whom this process needs to serve. How easy things are for the non-production team and participant "talents" is also important to consider. From the point of view of most of the participants, very little will have changed - they still just need to sign into Zoom and work out when they need to talk (and when they need to not talk).

From the point of view of the person orchestrating the call, things are much easier as they don't need to worry about constantly muting and unmuting participants during video segments (and indeed it's advantageous not to, since then the next speaker can count themselves - and me - in when starting the event or ending a video early, since I wasn't routing their audio to the stream output at those times).

None of which is to say that it's "easy"! I most definitely needed the Saturday rehearsal to straighten out all the kinks in my own mind about the process, and even then it wasn't until partway through that I started to relax into it. New things can be stressful!

Increases quality

The facets that make a "quality" livestream can be pretty subjective. Here's one subjective opinion:

From the WhatsApp group used to orchestrate the stream: a screenshot accompanied by the text "Just so you all know, this is how everyone can see it on YouTube. It's high quality and sounds [some emoji I'm assuming is positive]"

From my perspective, the main "quality" wins were:

• Greater control over who appeared on screen, when and where (no forced split screen)
• Videos no longer subjected to Zoom's compression
• Lack of Zoom UI clutter on the stream

In summary

Well done for making it this far!

Hopefully this post has given an insight, not just into the process we've developed for putting together our livestreams during this unusual time, but also into the evolution of that process and the key decision points along the way.

With all of that said, it's worth making the point that this isn't the final version of that process. I'm not even sure there is, or ever will be, a final version! As our Worship and Creative Director posted this morning:

It's important to pay attention to the process and not just the project...every song, project or completion is a step in the right direction but never the destination. None of us have arrived or made it.

Now, perhaps more so than ever, we're pretty much making it up as we go along, hoping that each tweak, each change, each new piece of technology is solving a part of the puzzle, a step in the right direction, helping us "be" church while we're apart a little more easily.

Eventually this time shall pass, and something resembling normality will return - and when it does, we should remember to ask ourselves, "what have we learned over this time that we should hold on to going forward?" Answering that might be a little easier if we've paid heed to not just the end results we can catch up on via YouTube, but the process we took to get there, too.

Delving into Renault's new API

(Geek level: High - you have been warned!)

I've been driving a Renault Zoe for over a year now. It's a great car, but the companion mobile application - allowing you to turn on the heater or air conditioning remotely, or to set a charge schedule to make the most of cheap overnight electricity - has been lacklustre at best.

At the tail end of last year, Renault decided to push an update to the ZE Services app that effectively removed the "app" part, instead redirecting users to their website (which works even more poorly on mobile devices). Renault promised a new "MY Renault application [with] an improved interface with new and easy-to-use services".

Nearly eight months later, still no sign of the new "MY Renault" app in the UK, but some countries on the continent have it in their hands. I decided to take a look and see how different the new API was to the previous one, and how much work I'd have to do to update my charge scheduling script (it takes half-hourly price data from Octopus, works out the cheapest window in which to charge overnight, and schedules the car to charge in that window.)

I'm not going to spend any time looking at registering for MY Renault; it's boring, and I went through the process in French, so all the following assumes you have a MY Renault account. I'm going to focus on the area of most interest to me: functions to interact with the electric-vehicle-specific parts of the API.

The first time I introduce a named piece of data, I'll make it bold so it's easier to skim back to. Where a parameter needs substituting in, it'll be in {italics, and probably monospace}. There are lots of scattered bits of information you'll need to pull together!

Update: This post has been updated with corrections from kind folk in the comments.

Logging in

Authentication has now been parcelled out to Israeli SAP subsidiary Gigya, who have extensive API documentation available online. The first thing to note is that you'll need the correct Gigya API key - this is embedded in the MY Renault app's configuration. Once you have that, you can log in by POSTing your MY Renault credentials to the appropriate endpoint. This will yield your Gigya session key (returned as sessionInfo.cookieValue). It's not clear when, or even if, this session key expires, so keep hold of it - you're going to need it a lot.

Once you've logged in, you'll need to extract a couple more pieces of information from the Gigya API before you can start to talk to Renault's servers. The first is your person ID, which ties your Gigya account to your MY Renault one (or specifically, ties you to a set of MY Renault accounts). You'll need use your Gigya session key as the oauth_token field value to pull the person ID from the Gigya accounts.getAccountInfo endpoint, but it's a fair bet the value won't change for a particular user.

You'll then need to request your Gigya JWT token from accounts.getJWT, again using your Gigya session key as the oauth_token, and you need to include data.personId,data.gigyaDataCenter as the value of fields - Renault's servers need that data to be in the token. It looks like you can pick the expiry of your choice here - Renault's app uses 900 seconds. When this token expires, you'll need to hit this endpoint again to get a new one.

OK, we're done talking to Gigya, now we can start the second part of the authentication process - this time with Renault's servers. Or, more precisely, the Nissan-Renault Alliance's shared platform Kamereon. Here, you'll need a Kamereon API key - again, this is embedded in the MY Renault app. The root URL for this API is https://api-wired-prod-1-euw1.wrd-aws.com/commerce/v1.

You don't yet have your Kamereon account ID, so you'll need to get it using your person ID from earlier. You'll need to pass your Gigya JWT token in the x-gigya-id_token header (note the funky mix of hyphens and underscore), and the Kamereon API key in the apikey header, in a GET request to the endpoint at persons/{person ID}?country=XX, inserting your person ID and two-letter country code. I'm not sure that the latter makes the slightest bit of difference, as I've tried exchanging FR for GB and not seen any effects, but the whole thing blows up if it's not there.

Looking at the data returned from that endpoint, you'll notice that it contains an array of accounts, not just a single account. I'm not sure in which scenarios one might have multiple accounts (multiple cars can be added to a single account), but it looks like it's possible. Not for me, though, since there was only a single account here; the value of accountId is what we'll need to go forward.

We're not done yet! The last thing you'll need to start pulling data is a Kamereon token. These are short-lived and are obtained from the endpoint at /accounts/{Kamereon account ID}/kamereon/token?country=XX, again with the apikey and x-gigya-id_token headers. The one you want is the accessToken. I've not looked into using the refreshToken - you might as well just repeat this request when the token expires - and the idToken returned is a copy of your Gigya JWT token (I think).

Phew! At the end of that process we should have:

• A Gigya JWT token
• A Kamereon account ID
• A Kamereon token

and the means to regenerate those tokens when they expire.

Listing vehicles

I mentioned you can add more than one vehicle to an account. Before you can do much you'll need the VIN of the vehicle you're interested in. You might well have that to hand if it's your vehicle, but in the general case you'll need to get the list from /accounts/{Kamereon account ID}/vehicles?country=XX. You'll need three auth headers:

• apikey: the Kamereon API key
• x-gigya-id_token: your Gigya JWT token
• x-kamereon-authorization: Bearer {kamereon token}

You'll note that the vehicle's registration plate is also included in the data now. That's a nice feature that ought to make it easier for end-users to identify vehicles in multi-vehicle accounts. As with the previous API, though, everything is keyed off the VIN (which makes sense, since the registration plate could change - though I've no idea if Renault's systems would reflect that sort of change).

Interacting with a vehicle

So, what can we do?

Each of these endpoints is under /accounts/kmr/remote-services/car-adapter/v1/cars/{vin}, and each requires the same three auth headers described above. The server expects a Content-type of application/vnd.api+json. For the most part, the returned data is self-explanatory.

Reading data

• /battery-status (GET): plugged in or not, charging or not, percentage charge remaining, estimated range (given in km). Presumably, as with the previous API when charging, information about the charge rate etc - I've not tried this yet.
• /hvac-status (GET): Air conditioning on or off, external temperature, and scheduled preconditioning start time. I've not seen it report anything other than AC off, even when preconditioning was running - possibly as a result of caching somewhere? External temperature seems accurate.
• /charge-mode (GET): Always charging or on the scheduler? (There's possibly a third state for the in-car "charge after X" mode - I've not investigated this yet. The "charge after" in-car setting is represented here as always_charging.)
• /charges?start=YYYMMDD&end=YYYYMMDD (GET): Detail for past charges. This isn't currently returning any useful data for me. end cannot be in the future.
• /charge-history?type={type}&start={start}&end={end} (GET): aggregated charge statistics. type is the aggregation period, either month or day; for 'month' start and end should be of the form YYYYMM; for 'day' it should be YYYYMMDD. Again, this is not supplying any useful data for me right now.
• /hvac-sessions?start=&end= (GET): Preconditioning history
• /hvac-history?type=&start=&end= (GET): Same as charge history but for preconditioning stats.
• /cockpit (GET): odometer reading (total mileage, though it's given in kilometers).
• /charge-schedule (GET): the current charge schedule - see later.
• /lock-status (GET): The server returns 501 Not Implemented.
• /location (GET): The server returns 501 Not Implemented.
• /notification-settings (GET): Settings for notifications (d'uh) - as well as email and SMS, there's also now an option for push notifications via the app, for each of "charge complete", "charge error", "charge on", "charge status" and "low battery alert" / "low battery reminder".

Writing data

Each of these expects a JSON object body of the form:

{

  "data": {
    "type": "SomeTypeName",
    "attributes": {
      (... the actual data ...)
    }

  }
}

In many cases, you'll be re-POSTing a similar object to that which you received for the corresponding GET method. A success response from the server tends to be the same object you just POSTed, with an ID added. I've listed the required attributes where I know them.

• /actions/charge-mode (POST): sets the charge mode. (Type ChargeMode)
• action: either schedule_mode or always_charging (possibly a third value - see above)
• /actions/hvac-start (POST): Sets the preconditioning timer, or turns on preconditioning immediately. (Type HvacStart)
• action: start
• targetTemperature: in degrees Celsius. The app is hard-coded to use 21°C.
• startDateTime: if supplied, should be of the form YYYY-MM-DDTHH:MM:SSZ - I'm not sure what would happen if you tried to use a different timezone offset. If not supplied, preconditioning will begin immediately.
• /actions/charge-schedule (POST): Set a charge schedule. See later. (Type ChargeSchedule)
• /actions/notification-settings (POST): Sets notification settings. (Type ZeNotifSettings)
• /actions/send-navigation (POST): The much-vaunted "send a destination to your car and start navigating". I've not explored this one much but parameters include:
• name
• latitude
• longitude
• address
• latitudeMode
• longitudeMode
• locationType
• calculationCondition

The charge scheduler

Perhaps unsurprisingly, given the need to interoperate with the existing fleet of current Zoes, the charge scheduler functions in exactly the same way, and with the same limitations:

• You must specify exactly one charging period per day, for every day of the week
• Charge start and duration must be in intervals of 15 minutes
• Charge duration must be at least 15 minutes
• Charging periods must not overlap
• Charge start time is specified as a four-character digit string e.g. "0115" (because that's how everyone represents time, right?)
• Charge duration is specified as an integer, rather than a digit string as it was in the previous API

Interestingly, looking at the data structure, there's scope here to support multiple charging periods per day: each day has an array of periods against it. I wonder if the Zoe ZE50 might have different onboard software that isn't quite so arse-backwards as the Atos/Worldline system?

Conclusions

The new API is definitely different, and it's probably better than the old one. It certainly seems a lot faster to respond (it no longer takes several seconds to log in, for one). What it can't do is change the capabilities of the software deployed on vehicles already in the wild - hence the crappy charge scheduler remains, and no doubt people will still be annoyed at the infrequency of the battery state updates, especially on a rapid charger. (Aside: it's possible to tweak the parameters of your car's TCU to increase the frequency at which it reports its state to the server.)

I'm not sure why the login process is quite so convoluted, except that perhaps it needed to be given the constraints of interacting with a third-party authentication gateway (Gigya) and the Alliance Kamereon API which has its own set of requirements. I do feel that requiring three different tokens on each request is a little excessive!

We've lost a few odds and ends, none of which seemed particularly important:

• It doesn't appear possible to cancel a preconditioning request
• It doesn't appear possible to request a state update from the car (though it's possible this is now handled behind the scenes, as it was a bit cludgy anyway)

In each case, maybe it is possible, and I've simply not discovered that function yet.

What is clear, though, is that neither of these APIs were designed to be public - the requirement to have an API key for both Gigya and Kamereon makes that apparent, and that's the reason I've not included these keys in this post. If you've read this far, chances are you'll know exactly how and where in the MY Renault APK to find the configuration resource that lists them - you don't even need any specialist software to do so.

Update: Or you could grab the keys from Renault's own website where they've been uploaded in plain text for all to stumble upon.

Also clear is that the scope of these APIs extend far beyond the ZE-specific capabilities. I've not detailed them, as they're very much of secondary interest, but there's information available via the new app on all sorts, from the type of fuel your vehicle uses, to its service schedule and any optional extras you added when you bought it. I guess that's a natural part of moving to the "MY Renault" platform.

An implementation

The first cut of my CLI tool / thin Python API wrapper is now on GitHub.

The long weekend: a retrospective

The Le Mans 24 Hours is the world's greatest motor race.

One of the toughest tests in motorsport, the race pits 180 drivers in 60 cars against the 8.5 mile Circuit de la Sarthe, and against the clock, with 24 hours straight of racing through the French countryside.

It's not just the drivers and cars that are put to the test; teams, too, face a battle to stay alert and react to the changing phases of the race. It's an endurance challenge for fans as well - at the track or around the world, staying awake for as long as possible to keep following a race which rarely disappoints in terms of action and (mis)adventure. The 2017 running of the 24 Hours was also something of a technical endurance challenge for this fan in particular...

Several years ago, unhappy with the live timing web pages made available by race organisers the WEC and ACO, I decided to start playing around with the data feed and see what I could come up with. Over the course of the 2015 race, I developed a prototype written in JavaScript that would later start to evolve and grow into something much bigger...

Fast-forward to 2017, and the Live Timing Aggregator was soft-launched before the 6 Hours of Silverstone via /r/wec and the Midweek Motorsport Listeners' Collective on Facebook. Despite having to debug the integration with a new format of WEC data from the grandstands of the International Pit Straight, and the system conking out a few hours into the race being held inconveniently on my wedding anniversary, feedback was overwhelmingly positive, and a few generous individuals even asked if they could donate money as a thank-you. The money let me move the system away from my existing server (which was becoming increasingly busy with other projects!) and onto a VPS of its own.

Sadly, though, the performance of the new VPS left a lot to be desired. On regular occasions, even loopback network connections were dropped, and when simply issuing ls would sometimes take more than ten seconds to execute, I decided that for the Big Race an alternative solution would be the safe bet; I took advantage of the AWS free tier to try and minimise my expenditure, and since the system isn't particularly CPU-intensive I didn't feel the restrictions on nano instances would be too arduous.

The AWS setup was ready in time for the Le Mans test day - the first opportunity the racing teams have to run on the full 24 Hours circuit, and the first opportunity for me to test the new setup. In all, over 1,500 unique users visited my timing pages that day, almost three times the previous high-water mark - helped by the inclusion of the per-sector timings that, while included in the official site's data feed, are inexplicably not displayed on the WEC timing pages.

In the following weeks, visitors enjoyed the "replay" functionality, giving them "as-live" timing replays of both test day sessions and the entire 2016 race, plus extensive live timing of a single-seater championship considered a "feeder" series into the WEC. Then into Le Mans week itself - and things started to get a bit nuts.

More and more people had caught word of the timing pages, and I was seeing steady traffic - as well as frequent messages via Twitter and email, some carrying thanks, some feature requests. One of the commentators at a well-known broadcaster even got in touch to say that they had no official timing from the circuit and that my site had made their TV production a whole lot easier! Many of the feature requests were already on my backlog, and there were a few I could sneak in as low-effort (although, deployments in race week seem a pretty bad idea in general).

Signs of not all being well were starting to become apparent - though not at my end. Rather, the WEC timing server seemed to be creaking under the load a little bit, and rather than updating every ten seconds (itself an age in motorsport terms!) there were five, ten, sometimes fifteen minutes between update cycles. I started research into an alternative data source which, at that stage at least, seemed to be more reliable. The modular nature of the timing system meant that it only took about an hour to get this alternative source (which I badged "WEC beta") up and running. (I ended up running both systems in parallel during the race, and once the WEC systems had calmed down there wasn't much difference between them.)

Peak visitors over practice and qualifying was about the same as for the test day. At this point, I had no idea of what was going to happen on Saturday afternoon...

Then real life interfered. For reasons I couldn't avoid, I had to be out of my house for the start of the race. Not only that, but the place I had to be had no mobile signal; and I ended up missing the first three hours of the race entirely.

I got home to find the world was on fire.

The WEC website had buckled under its own load (later claimed to be a DOS attack), which drove more and more visitors to my timing site. At some point, various processes reached their limits for open file handles. CPU usage had hit 100%, and stayed there. To make it a proper comedy of errors, I'd managed to leave my glasses, and my work laptop, at the friend's house at which I'd been at the start of the race - so I could only work by squinting an inch from the screen, or with sunglasses that rendered the monitors very dim. Nevertheless, I persevered...

First task was to get the load on the node under control. I took nginx offline briefly, and upped its allowed file handles. I also restarted the timing data collection process (which can be done while preserving its state). This helped very briefly - but after a few minutes, the number of connections was such that the data collection process itself was losing connection to the routing process, so no timing data could get in or out.

It was then that I had a brainwave - I could shunt the routing process (a crossbar.io instance) onto its own node, reducing the network and CPU load on the webserver and data collection node. I still had the code on the slow VPS, so I just needed to reactivate it, and patch the JavaScript client to connect to the timing network via the VPS rather than the AWS node. I also removed nginx as a proxy to the crossbar process, reducing the overhead - crossbar is capable of managing a large number of connections itself.

It turns out network IO on the VPS is adequate for the task, and over the next hour or so, things started to stabilise. I'd also decided to reduce network load by disabling the analysis portion of the site - which is a shame, as the stint-length and drive-time analyses were written with Le Mans in mind. I'll need to re-architect that portion somewhat, as the pub/sub model has proved to be an expensive one compared to request-response, especially with a large number of cars.

I'm grateful to those on Twitter and Reddit who, at this point, started to encourage me to not forget actually watching and enjoying the race! Thankfully, after another round of file handle limit increases (turns out that systemd completely ignores /etc/security/limits.conf and friends) - and my loving and patient spouse having retrieved my spectacles - I could do just that, only occasionally needing to hit it with a hammer to get it running again.

I also have some ideas to work with to improve function under load in future. Separating the data-collection process from the WAMP router one was a good idea, but still the former can be squeezed out of connectivity with the latter. Some method of ensuring that "internal" connections are given priority will keep up performance of the service for those users still connected. Upping the file handle limit and opening Crossbar directly helped increase the concurrent user count - around 10,000 over the course of the race - but a way of spreading that load over multiple nodes is going to be needed to go much beyond that.

The official WEC timekeepers, Al Kamel Systems, publish on their website a "chronological analysis" - a 3MB CSV file containing every lap and sector time set during the race. I wonder what effort will be involved to reconstruct the missing timing data from the first part of the race, into a replay-able format for my site...