Jan Holthuishttps://homepage.ruhr-uni-bochum.de/jan.holthuis/2021-12-22T23:00:18+01:00How Does Timecode Vinyl Actually Work? (Pt. 2)2021-12-22T23:00:18+01:002021-12-22T23:00:18+01:00Jan Holthuistag:homepage.ruhr-uni-bochum.de,2021-12-22:/jan.holthuis/how-does-timecode-vinyl-actually-work-pt-2.html<p><em>This is a mirror of a blog post that was published on the website of Mixxx, an open-source DJ software. You can read the original post over at <a href="https://mixxx.org/news/2021-12-22-dvs-internals-pt2/">mixxx.org</a>.</em></p>
<p>In the <a href="https://homepage.ruhr-uni-bochum.de/jan.holthuis/how-does-timecode-vinyl-actually-work-pt-1.html">last post</a>, I explained how a basic relative-mode Digital Vinyl System (DVS) works.
But there are two problems …</p><p><em>This is a mirror of a blog post that was published on the website of Mixxx, an open-source DJ software. You can read the original post over at <a href="https://mixxx.org/news/2021-12-22-dvs-internals-pt2/">mixxx.org</a>.</em></p>
<p>In the <a href="https://homepage.ruhr-uni-bochum.de/jan.holthuis/how-does-timecode-vinyl-actually-work-pt-1.html">last post</a>, I explained how a basic relative-mode Digital Vinyl System (DVS) works.
But there are two problems that are still left to solve:</p>
<ol>
<li>Seeking inside a track by picking up the needle from the record and dropping it somewhere else will not work.</li>
<li>The basic relative-mode implementation will likely suffer from so-called <em>sticker drift</em>.
This means that if you <a href="https://djtechtools.com/2015/11/04/marking-vinyl-records-with-stickers-throwback-thursday-dj-technique/">put a sticker on the vinyl record as visual aid</a> and then scratch the record back and forth so that the sticker is in the same position it had before the scratch, the track should also be at the same position.
If you just use the pitch information, it will likely be too inaccurate to make this work, and track position and sticker position drift apart (hence the name).</li>
</ol>
<p>Both of these problems can be solved by throwing position information into the mix.
This is why the timecode signal also contains information that can be used to detect the needle position on the record.</p>
<h2>Getting the position out of the analog signal</h2>
<p>From a steady sine waveform alone, it is hard to determine in which groove the needle is and at which rotation degree the platter is moving. One can imagine to count cycles of the wave from the beginning of the control track. Unfortunately this is error prone due to crackling or a skipped groove. The counted number becomes void after such an event. To restore the counted number, a periodical position information is required.
For Serato timecode, this information is added to the sine wave by amplitude modulation (AM) representing the position as a series of low and high amplitudes, digital bits, where 1 is a relatively high peak, and 0 is a relatively low peak.</p>
<p>The right channel is just a phase-shifted version of the left one, so we only need to look at the positive peaks of the left channel.</p>
<p><img alt="Serato Timecode Signal" src="https://homepage.ruhr-uni-bochum.de/jan.holthuis/images/timecode-signal.png"></p>
<p>If we look closely, we notice that the peaks occur every time the right channel crosses zero and the signal value goes from the positive part of the signal to negative part.</p>
<p>This is very convenient, because that way we don't have to lose any sleep over how to detect the peaks in the left channel, we just detect if there was a positive-to-negative zero crossing in the <em>right</em> channel, then take the current value of the left one and check if it's a <code>0</code> or <code>1</code> by comparing it with a threshold.</p>
<p>Now it's clear how the analog timecode signal can be converted to a stream of bits.
But how can these bits be interpreted as positions?
Or rather: How did the engineers that created the timecode format encode the positions as bits?</p>
<h2>Encoding strategies</h2>
<h3>Simple Base-2 Encoding</h3>
<p>The intuitive approach is to encode positions as a sequence of ascending numbers, e.g. the first position is 0, the second position is 1, the third position is 2, etc.
Since we're working with bits, we could simply express these decimal numbers in the base-2 (binary) numeral system, like this:</p>
<table>
<thead>
<tr>
<th>Position</th>
<th>Bit Sequence</th>
</tr>
</thead>
<tbody>
<tr>
<td>0</td>
<td><code>000</code></td>
</tr>
<tr>
<td>1</td>
<td><code>001</code></td>
</tr>
<tr>
<td>2</td>
<td><code>010</code></td>
</tr>
<tr>
<td>3</td>
<td><code>011</code></td>
</tr>
<tr>
<td>4</td>
<td><code>100</code></td>
</tr>
<tr>
<td>5</td>
<td><code>101</code></td>
</tr>
<tr>
<td>6</td>
<td><code>110</code></td>
</tr>
<tr>
<td>7</td>
<td><code>111</code></td>
</tr>
</tbody>
</table>
<p>Of course, that is just a toy example, but even if we only had 8 different positions, we need use 3 bits (= 3 cycles) to represent each position.
In reality, we want to encode <em>a lot</em> more different positions, which means the resulting binary number will be longer, meaning we need more bits and therefore cycles per position we want to encode.</p>
<p>The corresponding signal would look like this:</p>
<p><img alt="Simple bitstream with 3 bits per position" src="https://homepage.ruhr-uni-bochum.de/jan.holthuis/images/timecode-3-bit-bitstream.svg"></p>
<p>The first 3 cycles encode position 0, the next 3 cycles encode position 1, and so on.
Simple, right?
Unfortunately, this approach has a major problem.</p>
<p>Let's assume you seek to a random position in the track (by picking up the needle and dropping it somewhere) and the
next 3 bits are <code>001</code>.
Do you know what position or cycle number we are at?</p>
<p>We can look up <code>001</code> in the table above and see that these bits encode position 1 (cycles 3-6).
So after reading these bits we should be at cycle 6.
And that is <em>one</em> possibility, but not the only one.</p>
<p><img alt="Simple bitstream with 3 bits per position and all occurrences of 001 highlighted" src="https://homepage.ruhr-uni-bochum.de/jan.holthuis/images/timecode-3-bit-bitstream-highlight.svg"></p>
<p>So there are multiple possible positions and we don't know which one is the correct one.
We can mitigate this by reading the next 3 cycles too, but this doubles the amount of cycles we have to read (6 instead of 3 cycles) until we know the position for sure.</p>
<p>Digital Vinyl Systems need to be responsive and minimize the latency (i.e. the delay between moving the vinyl and seeing/hearing the action in the software) to be usable for scratching.
Therefore, this approach is not sufficient.</p>
<p>The problem with the simple base-2 encoding is that may end up in the middle of a number and we don't know where a number begins or ends.</p>
<h3>Using Sentinel Values</h3>
<p>To detect the beginning of a new position mark reliably, we need something special, that can be clearly distinguished from the position values.
Such a value is called a <a href="https://en.wikipedia.org/wiki/Sentinel_value">sentinel value</a> and could be used to indicate "A new position value starts here".</p>
<p>After a seek, we have to keep reading bits until we encounter the sentinel value, then we can start reading the actual position bits.
In fact, that's exactly what the early <a href="https://en.wikipedia.org/wiki/Final_Scratch#Vinyl/CD_time_code">Final Scratch DVS</a> did.</p>
<p>In the Final Scratch timecode, each position was prefixed by <code>0001</code>, followed by 16 bit of position information.
Although using a sentinel value ensures that we know when a position starts, this approach is also problematic.
Let me explain why.</p>
<p>As we already stated, we want the latency between moving the needle and detecting the position to be as small as possible.
Among other things, the latency is determined by how many bits we have to read until we know the position, so we want this to be as small as possible in all cases.
And this is where using sentinel values fail.</p>
<p>Let's assume the DJ dropped the needle directly in front of a sentinel value:</p>
<div class="highlight"><pre><span></span><code> 0 0 0 1 [-------16 bit position-------] 0 0 0 1 [-------16 bit position-------] 0 0 0 1 ...
^ ^
Needle dropped here At this point we know the position
</code></pre></div>
<p>After reading the next 20 bits (4 bit sentinel value + 16 bit position value), we know the position.
That's fine, but it's only the <em>best case</em>.</p>
<p>What if the DJ dropped the needle after the first bit of the sentinel value?
That would be the <em>worst case</em> and we'd need to read a lot more bits before know the position:</p>
<div class="highlight"><pre><span></span><code> 0 0 1 [-------16 bit position-------] 0 0 0 1 [-------16 bit position-------] 0 0 0 1 ...
^ ^
Needle dropped here At this point we know the position
</code></pre></div>
<p>The DVS can't detect that the needle was dropped on the sentinel value (because the first bit is missing).
Therefore, it need to wait for the next one and just ignore the remaining 3 bits of the sentinel value and the subsequent position value.
Then it can detect the next sentinel value and position.</p>
<p>So in that case it would be necessary to read 39 bits before the position can be known.
Needing so many bits to reliably detect a position is bad for latency and might make the system feel "sluggish".</p>
<h3>Using an LFSR</h3>
<p>It would be great if we could reduce the number of bits that we need to read to detect the position reliably - even in the worst case.
In the DJ community, Serato's take on DVS is widely considered one of the best solutions.
What are they using for position detection?
We can get a basic idea by taking a look at the back of the Serato Control Vinyl (CV2):</p>
<p><img alt="Serato Control Vinyl (CV2) back" src="https://homepage.ruhr-uni-bochum.de/jan.holthuis/images/timecode-serato-lfsr.png"></p>
<blockquote>
<p>Deep in a New Zealand laboratory, the founders of Serato set out to create a control record to give professional DJs superior control. Standard timecode records just didn't feel enough like vinyl to make the cut. After months of nerdy, mathematical investigation, they created a maximal-length pseudo random bit sequence using a linear feedback shift register. Or, in English, the Serato NoiseMap™ - a unique control tone for digital DJs that offers unparalleled sensitivity and tightest, most authentic vinyl feel.</p>
</blockquote>
<p>So Serato is using a Linear Feedback Shift Register (LFSR) its control signal.
Sounds scary?
Don't worry, it's not as complicated as it may sound.</p>
<h4>What is an LFSR and how does it work?</h4>
<p>A Linear Feedback Shift Register (LFSR) is a shift register that uses a linear feedback function.
LFSRs are common in modern computing (e.g. for generating pseudo-random numbers), but using them for a vinyl control signal an interesting and unusual application.
I'll outline the basic principle and then try to explain step-by-step using an example.</p>
<p>For our purposes, it suffices to know that a register is an array that contains a fixed number of bits.
We are working with a <em>Shift</em> Register, which means means that the contents of the register are shifted to the left or right in each step.</p>
<p><img alt="3-bit Shift Register" src="https://homepage.ruhr-uni-bochum.de/jan.holthuis/images/3-bit-shift-register.svg"></p>
<p>If we shift the register's contents to the right, we need a new value that we insert at the leftmost position.
LFSRs use a (linear) function that takes the current content of the register as input.
The resulting value is then fed back into the register.</p>
<p>Let's have a look at a step-by-step example.
Here's a 3-bit LFSR that generates a maximal-length pseudo random bit sequence:</p>
<p><img alt="3-bit maximal-length LFSR" src="https://homepage.ruhr-uni-bochum.de/jan.holthuis/images/3-bit-lfsr.svg"></p>
<p>In each step the following happens:</p>
<ol>
<li>We take the current contents of the register to calculate the feedback bit. In this case, we calculate <em>x = s<sub>1</sub> + s<sub>0</sub></em>.</li>
<li>Then shift the register to the right.</li>
<li>The bit that is "pushed out" of the register (i.e. the rightmost bit <em>s<sub>0</sub></em>) is the output bit</li>
<li>The leftmost (empty) space is filled with the feedback bit (<em>x</em>) we calculated.</li>
</ol>
<p>Let's assume that the initial state of the register is <em>(1, 0, 0)</em>.</p>
<p>First, we need to calculate the feedback bit <em>x</em> which is defined as the sum of <em>s<sub>1</sub></em> and <em>s<sub>0</sub></em>.
In this step, both bits have the value <code>0</code>, so <em>x = s<sub>1</sub> + s<sub>0</sub> = 0 + 0 = 0</em>
Next, we need to shift everything to the right, and insert <em>x = 0</em> on the left.
The output bit is <em>0</em>, because that is the rightmost bit that is "pushed out" of the register.</p>
<p>In the next step, the feedback bit is <em>x = s<sub>1</sub> + s<sub>0</sub> = 1 + 0 = 0</em>.
Note that this time <em>s<sub>1</sub></em> has the value <em>1</em> because we shifted the register to the right in the previous step and <em>1</em> moved from <em>s<sub>2</sub></em> to <em>s<sub>1</sub></em>.
Now that we calculated <em>x</em>, we again shift the whole register to the right and write the feedback bit into the leftmost position.
The output bit is <em>0</em>.</p>
<p>We can now continue this a few more times.
In step 5, the feedback bit is <em>x = s<sub>1</sub> + s<sub>1</sub> = 1 + 1</em>.
Usually, the result of that calculation would be <em>2</em>, but in this case it's <em>0</em>.
The reason for that is that we're working with bits (which are either <em>0</em> or <em>1</em>), so <em>2</em> is not a valid value.</p>
<p>Consider a 24-hours clock:
If it's 23:00 and you wait 2 hours, it's <em>01:00</em> and not <em>25:00</em>.
That time doesn't exist, so you just subtract 24, which is the number of possible hours from 0 to 23, to make it valid (25:00 - 24:00 = 01:00).
In the same way 2 becomes 0, when you calculate <em>2 - 2 = 0</em>, because <em>2</em> is not a valid bit value.</p>
<p>After the first 6 steps we get the following table:</p>
<table>
<thead>
<tr>
<th>Step</th>
<th><em>s<sub>2</sub></em></th>
<th><em>s<sub>1</sub></em></th>
<th><em>s<sub>0</sub></em></th>
<th>Feedback bit <em>x = s<sub>1</sub> + s<sub>0</sub></em></th>
<th>Output bit <em>x = s<sub>0</sub></em></th>
</tr>
</thead>
<tbody>
<tr>
<td>1.</td>
<td><em>1</em></td>
<td><em>0</em></td>
<td><em>0</em></td>
<td><em>0 + 0 = 0</em></td>
<td><em>0</em></td>
</tr>
<tr>
<td>2.</td>
<td><em>0</em></td>
<td><em>1</em></td>
<td><em>0</em></td>
<td><em>1 + 0 = 1</em></td>
<td><em>0</em></td>
</tr>
<tr>
<td>3.</td>
<td><em>1</em></td>
<td><em>0</em></td>
<td><em>1</em></td>
<td><em>0 + 1 = 1</em></td>
<td><em>1</em></td>
</tr>
<tr>
<td>4.</td>
<td><em>1</em></td>
<td><em>1</em></td>
<td><em>0</em></td>
<td><em>1 + 0 = 1</em></td>
<td><em>0</em></td>
</tr>
<tr>
<td>5.</td>
<td><em>0</em></td>
<td><em>1</em></td>
<td><em>1</em></td>
<td><em>1 + 1 = 0</em> (mod 2)</td>
<td><em>1</em></td>
</tr>
<tr>
<td>6.</td>
<td><em>0</em></td>
<td><em>0</em></td>
<td><em>1</em></td>
<td><em>0 + 1 = 1</em></td>
<td><em>1</em></td>
</tr>
</tbody>
</table>
<p>In the next step, we would insert the feedback bit (1) at the leftmost position and shift the other bits to the right,
which results in the state (1, 0, 0), which is exactly the state we started with.
From now on, the table rows would just repeat forever.</p>
<p>This means that this LFSR has a period of 7 (because it repeats after 7 steps).
That is the maximal period length you can archieve with a 3-bit LFSR.</p>
<p>Other 3-bit LFSRs that have a <em>shorter</em> period length exist, e.g. if you have an LFSR where the feedback bit is calculated as <em>x = s<sub>0</sub></em> it will already repeats after 3 steps, no 3-bit LFSR will have a longer period.</p>
<p>Hence, we now know how to generate a "maximal-length pseudo random bit sequence using a linear feedback shift register" just like Serato has.
Let's check how that can be used to solve the problem at hand.</p>
<h4>LFSR output as timecode signal</h4>
<p>In each step of the LFSR example above, we get exactly one output bit (the rightmost bit of the LFSR that is "pushed out").
The exampe has six steps, thus we also get 6 output bits:</p>
<div class="highlight"><pre><span></span><code><span class="mf">0</span><span class="w"> </span><span class="mf">0</span><span class="w"> </span><span class="mf">1</span><span class="w"> </span><span class="mf">0</span><span class="w"> </span><span class="mf">1</span><span class="w"> </span><span class="mf">1</span>
</code></pre></div>
<p>Since the LFSR has an internal state of 3 bits, we can use 3 subsequent bits for encoding positions, like this:</p>
<table>
<thead>
<tr>
<th>Position</th>
<th>Bit Sequence</th>
</tr>
</thead>
<tbody>
<tr>
<td>0</td>
<td><code>001___</code></td>
</tr>
<tr>
<td>1</td>
<td><code>_010__</code></td>
</tr>
<tr>
<td>2</td>
<td><code>__101_</code></td>
</tr>
<tr>
<td>3</td>
<td><code>___011</code></td>
</tr>
</tbody>
</table>
<p>The corresponding timecode signal would look like this:</p>
<p><img alt="LFSR bitstream with 3 bits per position" src="https://homepage.ruhr-uni-bochum.de/jan.holthuis/images/timecode-3-bit-lfsr-bitstream.svg"></p>
<p>We can now observe an interesting property of our 3-bit-LFSR-based output signal:
<em>Any</em> sequence of 3 bits in the signal is <em>unique</em>.</p>
<p>This is great and exactly what we want, because after reading 3 bits we now know unambiguously which one of the 4 positions we're at.</p>
<h4>How big does the LFSR need to be?</h4>
<p>The example above is obviously a toy example.
In real life, we want to a lot more than 4 different positions.</p>
<p>The Serato Control CD has a play time of approximately 16 minutes 20 seconds, or 16 * 60 + 20 = 980 seconds.
At a timecode frequency of 1000 Hz (cycles per second), we have 980,000 cycles.
Every cycle encodes one bit, therefore we need an LFSR with an output length of at least 980,000 bits before it starts to repeat.</p>
<p>At most, an n-bit LFSR can output 2<sup>n - 1</sup> bits before it starts to repeat.
An LFSR with such a period size is called a maximal-length LFSR, and we already mentioned that Serato states that it uses a such an LFSR on the packaging of its timecode media.</p>
<p>The smallest possible LFSR that can output at least 980,000 bits without starting to repeat needs at least 20 bits of state.
A 19-bit LFSR is too small, because 2<sup>19</sup> - 1 = 524,288 is less than 980,000, but a 20-bit LFSR can output up to 1048575 bits before it starts to repeat.
We don't want to make the LFSR larger than absolutely necessary, because the larger the register, the more bits we need to read before we can detect a position after a needle drop.
Hence, increasing the LFSR's size also increases the latency.</p>
<h4>A Real Life Example</h4>
<p>Let's have a look at a real-life DVS that relies on an LFSR for position detection.
We already know that Serato's timecodes uses an LFSR, but what does it look like exactly?</p>
<p>Fortunately, we can find out by using the <a href="https://en.wikipedia.org/wiki/Berlekamp%E2%80%93Massey_algorithm">Berlekamp-Massey algorithm</a>.
It takes the bit sequence decoded from Serato's timecode, and finds the shortest LFSR that produces that output.</p>
<p>For the Serato Timecode CD bit sequence, the algorithm finds the following LFSR:</p>
<p><img alt="Serato Timecode CD LFSR" src="https://homepage.ruhr-uni-bochum.de/jan.holthuis/images/serato-cd-lfsr.svg"></p>
<p>We also have to find the correct seed (initial bit state for the LFSR), but this is trivial:</p>
<p>We know the first 20 bits of the timecode (e.g. by simply looking at the waveform) and we also know that this bit sequence is unique.
First, we selecting a random non-zero bit sequence as initial LFSR state and then step through the LFSR states while comparing the output of the LFSR with these first 20 bits of timecode bit sequence.
As soon as we see these 20 bits, we just go back 20 steps to the LFSR state <em>before</em> the first bit of that sequence appeared in the output.
This is the seed.</p>
<p>A DVS could now generate a lookup table (LUT) that maps each LFSR state to a position.
When reading bits from the timecode, it can then perform a simple lookup to get the corresponding position.</p>
<h2>Increasing the signal frequency</h2>
<p>Another way to reduce the latency is to increase the signal frequency.
Serato uses a signal frequency of 1000 Hz, Final Scratch uses 1200 Hz and Traktor Scratch MK2 even uses 2000 Hz.
This means that if you play the record at its original tempo, Serato will read 1000 bits in a single second, Final Scratch will read 1200 bits and Traktor Scratch MK2 can read 2000 bits.
Sound like a good way to reduce the latency, right?</p>
<p>Unfortunately, a higher signal frequency also comes with a cost:
The signal's maximum frequency must not exceed half the sampling frequency, otherwise <em>signal folding</em> will occur and lead to information loss.
If you use an audio interface with a 44100 Hz sampling frequency (or sample rate), the signal's maximum frequency at which is can be sampled losslessly (called Nyquist frequency) is 22050 Hz.</p>
<p>When DJs scratch using DVS, the record is being moved back and forth very fast, so it's <em>not</em> played back at the original speed.
If they move it too fast, signal folding will lead to misdetection of the bits.
With Serato, you can speed up the record up to 22.05 times of the original speed before that happens.
Traktor MK2 only allows a scratching speed of 11.025 times before the Nyquist frequency is reached and signal folding occurs.</p>
<p>Hence, the signal frequency is a tradeoff between latency (how many bits per second) and maximum possible scratching speed.</p>
<h2>Conclusion</h2>
<p>I hope these two blog posts were interesting and helped understanding how DVS works internally.
Of course, this was just a basic introduction, and there's still more to consider, like dealing with noise, dusty vinyl, etc.
And the information read from the timecode vinyl still has to be hooked up to the GUI and audio engine of the DJ software, which is challenging on it's own.</p>
<p>If you want to see an actual DVS implementation, check out Mark Hills' excellent free and open-source <a href="https://xwax.org/">xwax</a> software.
Its timecode decoder is what Mixxx uses internally to provide <a href="https://manual.mixxx.org/en/chapters/vinyl_control.html">vinyl control</a>.</p>
<p>You can also have a look at <a href="https://github.com/Holzhaus/vinylla">vinylla</a>, a toy library that I started to get a better understanding of DVS (and also to learn how to program in <a href="https://www.rust-lang.org/">Rust</a>).</p>
<p>And if you want to help improve the way Mixxx' DVS capabilities or want work on other parts of Mixxx, hit us up on <a href="https://mixxx.zulipchat.com/">Zulip Chat</a>!</p>How Does Timecode Vinyl Actually Work? (Pt. 1)2021-11-21T00:09:09+01:002021-11-21T00:09:09+01:00Jan Holthuistag:homepage.ruhr-uni-bochum.de,2021-11-21:/jan.holthuis/how-does-timecode-vinyl-actually-work-pt-1.html<p><em>This is a mirror of a blog post that was published on the website of Mixxx, an open-source DJ software. You can read the original post over at <a href="https://mixxx.org/news/2021-11-21-dvs-internals-pt1/">mixxx.org</a>.</em></p>
<p>Since the dawn of DJing, spinning vinyl records has never really gone out of fashion.
Even when CDs became popular …</p><p><em>This is a mirror of a blog post that was published on the website of Mixxx, an open-source DJ software. You can read the original post over at <a href="https://mixxx.org/news/2021-11-21-dvs-internals-pt1/">mixxx.org</a>.</em></p>
<p>Since the dawn of DJing, spinning vinyl records has never really gone out of fashion.
Even when CDs became popular in 90s and most music listeners happily phased out their record players, the classical "two turntables and a mixer" setup stayed the epitome of DJing.
In recent years, digital DJ controllers became more popular, but there are still many DJs that stick with their notorious Technics SL-1210s turntables.</p>
<p>There's a reason for that: the tactile feeling of manipulating music with your hands is something that controllers have had trouble archieving.
Motorized controllers like the Rane One try to provide a similar workflow, but it's still to be seen whether they will be able to take significant market share and replace vinyl setups or if they stay a niche product.</p>
<p>On the other hand, going digital has its benefits:
In contrast to the 80s, nowadays many tracks are only released digitally, and a pure vinyl setup makes it impossible to play them.
Jumping to different positions instantly inside a track using hotcues - without picking up, moving and dropping the needle - just isn't possible with vinyl records, and carrying around heavy crates of records isn't the best you can do for your back either.</p>
<p>Digital Vinyl Systems (DVS) aim to provide the best of both worlds by combining the flexibility and power of digital DJing with the tactile control of traditional vinyl DJing.
But how does it actually work?</p>
<h2>The Basics</h2>
<p>Instead of a traditional record that contains music, a "control" record is used.
Such a record contains a special signal that is processed by the computer.
The computer transforms this signal into information about direction ("Is the record playing forwards or backwards?"), pitch ("How fast is the record spinning?") and position information ("At which position of the record is the needle?") and uses it to manipulate the playing track in software.</p>
<p>There are many formats out there:
<a href="http://www.virtualdj.com/buy/controlvinyl.html">VirtualDJ</a>, <a href="https://www.pioneerdj.com/product/features/software/rekordbox-dvs-control-vinyl/">Rekordbox</a>, <a href="https://www.numark.com/product/virtualvinyl">Numark</a>, <a href="https://www.mixvibes.com/cross-dj-4/">MixVibes</a>, <a href="https://www.native-instruments.com/en/products/traktor/digital-vinyl/traktor-scratch-a10/">Traktor</a> and <a href="https://serato.com/dj/pro/expansions/dvs?dvs=dvs-ready">Serato</a> all provide their own solution.
Mixxx (and the underlying <a href="https://xwax.org/">xwax</a> library) <a href="https://manual.mixxx.org/2.3/en/chapters/vinyl_control.html#supported-timecode-media">support the latter 3 formats</a>.</p>
<p>Serato is one of the most popular and robust digital vinyl systems.
The Serato Control CD can be used to achieve the same thing on CDJs instead of turntables.
Since the latter essentially works the same as the vinyl version and can be <a href="https://serato.com/controlcd/downloads">downloaded for free from the Serato Website</a>, I'm going to explain how it works based on that format.</p>
<p>Let's take a look at the control signal (also called "<a href="https://en.wikipedia.org/wiki/Timecode">timecode</a> signal").
If you open the WAV file with an audio editor like Audacity, it looks like this:</p>
<p><img alt="Serato CD Timecode Signal in Audacity" src="https://homepage.ruhr-uni-bochum.de/jan.holthuis/images/timecode-signal.png"></p>
<p>As you can see, the timecode signal is a stereo signal.
The left channel is on top and the right channel is on the bottom.
Both channels look very similar, but the right channel is shifted a little.</p>
<h2>Detecting the Playback Velocity</h2>
<p>Let's focus on the left channel for now.
The wave basically looks similar to a sine wave that you may remember from your high school math classes.
You can divide that signal into "cycles", where each cycle starts from zero and goes into a positive part (above the black line) followed by a negative part (below the black line) and then repeats:</p>
<p><img alt="Cycle of left channel in timecode signal" src="https://homepage.ruhr-uni-bochum.de/jan.holthuis/images/timecode-signal-cycle.png"></p>
<p>Each cycle has the same length and if we count the number of cycles in a single second of audio, there are 1000 of them.
Hence, we now know that the signal has a frequency of 1000 Hz (Hz = 1/s, so it's basically a fancy way saying "per second").</p>
<p>With that information, the DVS system can detect how fast the record is playing by checking how many cycles there are in a second of audio recorded from the turntable.
If there are 2000 cycles per second, it's playing a double speed, at 500 cycles per second the record is playing at half speed, and so on.</p>
<p>But how does the computer know how many cycles there are in a second?
We can't actually wait for a second, because the delay (also called latency) would be very noticeable and make the system unsuitable for scratching.</p>
<h2>Analog vs. Digital Audio</h2>
<p>It's important to know that in contrast to analog audio like vinyl or music cassettes, the computer doesn't actually work with the continuous (smooth) wave that we see in the screenshot.</p>
<p><img alt="Analog Signal" src="https://homepage.ruhr-uni-bochum.de/jan.holthuis/images/signal_analog.svg"></p>
<p>Instead, <a href="https://en.wikipedia.org/wiki/Digital_signal_%28signal_processing%29">digital signals</a> work with measurements (called "samples") that indicate the value of the wave position at specific points in time.
How often these measurements are taken is determined by the sample rate (or sampling rate).
For example, Audio CDs use a sample rate of 44100 Hz (i.e. 44100 measurements per second, or 1 measurement every ~22.68 microseconds).
Such a signal is called "time-discrete":</p>
<p><img alt="Time-Discrete Signal" src="https://homepage.ruhr-uni-bochum.de/jan.holthuis/images/signal_time_discrete.svg"></p>
<p>The reason for this is that a time-continuous signal has infinite resolution and would need infinite memory space to save, which is impossible - and also unnecessary, because as long as you take enough samples per second, you can <a href="https://en.wikipedia.org/wiki/Nyquist%E2%80%93Shannon_sampling_theorem">restore the original signal</a> without any loss of information.</p>
<p>Since computers can't store numbers with a theoretically infinite number of different values, a digital signal is also value-discrete.
This means that the amplitude values are mapped to a finite number of different values.</p>
<p>Let's assume we only have space to store 11 distinct values per sample (-1.0, -0.8, -0.6, -0.4, -0.2, 0, 0.2, 0.4, 0.6, 0.8 and 1.0).
But a sample value may have any value, so if its value would be between two of these values, they have to be rounded (in this case: multiples of 0.2).
This process is called <a href="https://en.wikipedia.org/wiki/Quantization_%28signal_processing%29">Quantization</a> and causes some information loss, which means that it <a href="https://en.wikipedia.org/wiki/Quantization_%28signal_processing%29#Error">adds noise</a> to our signal.
An example for such a quantized, value-discrete version of the analog signal looks like this:</p>
<p><img alt="Value-Discrete Signal" src="https://homepage.ruhr-uni-bochum.de/jan.holthuis/images/signal_value_discrete.svg"></p>
<p>A more realistic example is the Audio CD, which uses 16 bit per sample value.
With 16 bits, you can represent 2<sup>16</sup> = 65536 different values.</p>
<p>A <em>digital</em> signal is both time-discrete and value-discrete:</p>
<p><img alt="Digital Signal" src="https://homepage.ruhr-uni-bochum.de/jan.holthuis/images/signal_digital.svg"></p>
<p>Note that this is just a very basic introduction in how digital signals are represented.
If you're interested in the topic, I recommend watching <a href="https://xiph.org/video/vid2.shtml">Monty Montgomery's "Digital Show & Tell" video</a>, where he demonstrates how digital signals behave in contrast to analog ones and clears up common misconceptions.</p>
<p>You may wonder why the timecode signal in the screenshots looks like a analog signal instead of a "lollypop chart".
<em>Digital</em> vinyl scratch system work with digital signals, not analog ones, right?
And you'd be right, it's just that most audio tools (like Audacity) show a wave instead of individual samples by default.</p>
<h2>Detecting Zero Crossings</h2>
<p>Now that we know the basics of digital audio, we can start thinking about a simple low-latency approach to detect the playback velocity using so-called "zero crossings".</p>
<p>Whenever the wave goes from the positive part to the negative part or vice-versa, it crosses "zero".
For example, in the timecode signal screenshot I used above, we can see 3 zero crossings:</p>
<p><img alt="Zero crossings in timecode signal" src="https://homepage.ruhr-uni-bochum.de/jan.holthuis/images/timecode-signal-samples.png"></p>
<p>Note that we're working with a digital signal.
To illustrate that this signal is time-discrete, each individual sample value is illustrated by a little dot.</p>
<p>To detect these zero crossings reliably, we can't just check if the sample value equals zero, because we cannot assume that a sample measurement is happening at the exact time that the signal is at zero position.</p>
<p>It might be the case the current sample is <em>after</em> the zero position and the previous sample was <em>before</em> the zero position.
Hence, we also compare the previous sample value with the current sample value.
If the current value is positive and the previous value was negative, or if the previous value was negative and the current value is positive, the signal crossed zero.</p>
<h2>Refining Pitch Detection Using Zero Crossings</h2>
<p>Now that we know how to detect zero crossings, we can use them for determining the pitch.</p>
<p>If we look at the timecode signal, we can see that each cycle of the timecode signal has two zero crossings:
At the start of the cycle, when the positive part of the wave starts and in the middle of the cycle when the negative part of the wave starts.</p>
<p>So the equivalent of checking for 1000 cycles per second is to check if there are 2000 zero crossings per second.
Let's say the audio interface uses a sample rate of 44100 Hz, then there should be a zero crossing every 441000 / 2000 = 22.05 samples.</p>
<p>We can now detect the pitch by comparing the expected number of samples between zero crossings with the actual number of samples between them.
This very simple, low latency pitch detection algorithm works because the Serato timecode has a fixed frequency of 1000 Hz, which means that the distance between all zero crossing is fixed.
It would not work with a regular music signal, which has lots of different frequencies added together.</p>
<p>However, we need to use multiple sample distances and calculate the average to make this calculation more accurate, because in reality there are no fractions of samples, only full samples.</p>
<h2>Detecting the Playback Direction</h2>
<p>The DVS system also needs to know if the record is playing forwards or backwards.</p>
<p>To do that, we need to take a look at the right channel.
It's basically the same signal as the left channel, but as we noticed before, it's shifted by a quarter cycle.
This means that whenever there's a negative or positive amplitude peak in the left channel, the right channel is at zero.</p>
<p><img alt="Zero crossing on right channel means there is a peak on the left channel" src="https://homepage.ruhr-uni-bochum.de/jan.holthuis/images/timecode-signal-zero-crossing-right-channel.png"></p>
<p>We can use this property to detect the playback direction.
After the left channel crossed zero, we check if the left and right channel waves are both positive or both negative.
If so, the record is playing forwards, otherwise it's playing backwards.</p>
<p>For the right channels it's the opposite.
After the right channel crossed zero and the left is negative and the right is positive or if the the right is negative and the left is positive, the timecode is playing forwards.
If both waves are positive or both are negative, the record is playing backwards.</p>
<p>Now we have already have the basic building blocks to make a DVS system with a simple relative mode.</p>
<p>For a DVS system that allows you to skip forward and backward in a track by picking up and moving the needle (absolute mode), we need a way to detect the current position in the timecode signal.
I'll explain how that works in the next post.</p>Reversing Serato's GEOB tags2019-07-12T00:00:00+02:002019-07-12T00:00:00+02:00Jan Holthuistag:homepage.ruhr-uni-bochum.de,2019-07-12:/jan.holthuis/reversing-seratos-geob-tags.html<p>The popular DJ application <em>Serato DJ Pro</em> stores a lot data in GEOB-type ID3 tags of MP3 files.
This way, users can share beatgrids, hot cue points, saved loops and more by just copying over an MP3 file.</p>
<p>Unfortunately, the format is not <a href="https://serato.com/forum/discussion/1277101">publicly</a> <a href="https://music.stackexchange.com/questions/53753/documentation-for-the-embedded-cue-point-format-of-traktor-serato-ableton">documented</a>.
Thus, I figured it could …</p><p>The popular DJ application <em>Serato DJ Pro</em> stores a lot data in GEOB-type ID3 tags of MP3 files.
This way, users can share beatgrids, hot cue points, saved loops and more by just copying over an MP3 file.</p>
<p>Unfortunately, the format is not <a href="https://serato.com/forum/discussion/1277101">publicly</a> <a href="https://music.stackexchange.com/questions/53753/documentation-for-the-embedded-cue-point-format-of-traktor-serato-ableton">documented</a>.
Thus, I figured it could be fun to try to learn how these formats work <em>the hard way(tm)</em>.</p>
<h2>Setup and Preparation</h2>
<p>As example file I used <a href="https://soundcloud.com/sundancemusic/pers-phone-retro-funky">Perséphone - Retro Funky (SUNDANCE remix)</a>, which is licensed under the term of the <a href="https://creativecommons.org/licenses/by/3.0/">Creative Commons Attribution 3.0 Unported (CC BY 3.0) license</a>.</p>
<div class="highlight"><pre><span></span><code><span class="c">$ lame </span><span class="nb">-</span><span class="c">b 320 Perséphone\ </span><span class="nb">-</span><span class="c">\ Retro\ Funky\ \(SUNDANCE\ remix\)</span><span class="nt">.</span><span class="c">wav</span>
<span class="c">LAME 3</span><span class="nt">.</span><span class="c">100 64bits (http://lame</span><span class="nt">.</span><span class="c">sf</span><span class="nt">.</span><span class="c">net)</span>
<span class="c">Using polyphase lowpass filter</span><span class="nt">,</span><span class="c"> transition band: 20094 Hz </span><span class="nb">-</span><span class="c"> 20627 Hz</span>
<span class="c">Encoding Perséphone </span><span class="nb">-</span><span class="c"> Retro Funky (SUNDANCE remix)</span><span class="nt">.</span><span class="c">wav</span>
<span class="c"> to Perséphone </span><span class="nb">-</span><span class="c"> Retro Funky (SUNDANCE remix)</span><span class="nt">.</span><span class="c">mp3</span>
<span class="c">Encoding as 44</span><span class="nt">.</span><span class="c">1 kHz j</span><span class="nb">-</span><span class="c">stereo MPEG</span><span class="nb">-</span><span class="c">1 Layer III (4</span><span class="nt">.</span><span class="c">4x) 320 kbps qval=3</span>
<span class="c"> Frame | CPU time/estim | REAL time/estim | play/CPU | ETA</span>
<span class="c">8170/8170 (100%)| 0:03/ 0:03| 0:03/ 0:03| 68</span><span class="nt">.</span><span class="c">457x| 0:00</span>
<span class="nb">-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------</span>
<span class="c">kbps LR MS % long switch short %</span>
<span class="c">320</span><span class="nt">.</span><span class="c">0 99</span><span class="nt">.</span><span class="c">1 0</span><span class="nt">.</span><span class="c">9 87</span><span class="nt">.</span><span class="c">8 6</span><span class="nt">.</span><span class="c">3 5</span><span class="nt">.</span><span class="c">8</span>
<span class="c">Writing LAME Tag</span><span class="nt">...</span><span class="c">done</span>
<span class="c">ReplayGain: </span><span class="nb">-</span><span class="c">10</span><span class="nt">.</span><span class="c">6dB</span>
</code></pre></div>
<p>Initially, there are no tags in the resulting MP3 file:</p>
<div class="highlight"><pre><span></span><code><span class="c">$ eyeD3 </span><span class="nb">-</span><span class="c">v original</span><span class="nt">.</span><span class="c">mp3</span>
<span class="nt">...</span><span class="c">/original</span><span class="nt">.</span><span class="c">mp3 </span><span class="k">[</span><span class="c"> 8</span><span class="nt">.</span><span class="c">14 MB </span><span class="k">]</span>
<span class="nb">-------------------------------------------------------------------------------</span>
<span class="c">Time: 03:33 MPEG1</span><span class="nt">,</span><span class="c"> Layer III </span><span class="k">[</span><span class="c"> 320 kb/s @ 44100 Hz </span><span class="nb">-</span><span class="c"> Joint stereo </span><span class="k">]</span>
<span class="nb">-------------------------------------------------------------------------------</span>
<span class="c">No ID3 v1</span><span class="nt">.</span><span class="c">x/v2</span><span class="nt">.</span><span class="c">x tag found!</span>
</code></pre></div>
<p>Next, I'm starting up a Windows 10 VM with Serato DJ Pro 2.1.1 installed.
For some reason Serato causes my CPU usage to go up to 100%, but that can be ignored since we're not doing anything latency-critical.</p>
<p>To make testing easier, I'll just put the MP3 file into a separate directory on my host system and create a Samba network share to access it from inside the VM.
Serato does not list network shares in the sidebar, but it's still possible to load tracks from network shares into a deck directly by using drag and drop from a file explorer window.</p>
<p>First, let's just load the tack into deck 1, wait until Serato is done with analyzing it and then ejecting it to make sure that ID3 tag changes are written to the file.</p>
<p>We can see that the file has indeed changed by comparing it to the original file:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>sha1sum<span class="w"> </span>original.mp3<span class="w"> </span>analyzed.mp3
521974eb217ef001e6df426d1e2908cd80c39b6b<span class="w"> </span>original.mp3
46e7f16f735758b4d85730b639c908850860537b<span class="w"> </span>analyzed.mp3
$<span class="w"> </span>eyeD3<span class="w"> </span>-v<span class="w"> </span>analyzed.mp3
eyed3.id3.frames:WARNING:<span class="w"> </span>Frame<span class="w"> </span><span class="s1">'RVAD'</span><span class="w"> </span>is<span class="w"> </span>not<span class="w"> </span>yet<span class="w"> </span>supported,<span class="w"> </span>using<span class="w"> </span>raw<span class="w"> </span>Frame<span class="w"> </span>to<span class="w"> </span>parse
.../analyzed.mp3<span class="w"> </span><span class="o">[</span><span class="w"> </span><span class="m">8</span>.16<span class="w"> </span>MB<span class="w"> </span><span class="o">]</span>
-------------------------------------------------------------------------------
Time:<span class="w"> </span><span class="m">03</span>:33<span class="w"> </span>MPEG1,<span class="w"> </span>Layer<span class="w"> </span>III<span class="w"> </span><span class="o">[</span><span class="w"> </span><span class="m">320</span><span class="w"> </span>kb/s<span class="w"> </span>@<span class="w"> </span><span class="m">44100</span><span class="w"> </span>Hz<span class="w"> </span>-<span class="w"> </span>Joint<span class="w"> </span>stereo<span class="w"> </span><span class="o">]</span>
-------------------------------------------------------------------------------
ID3<span class="w"> </span>v2.3:
title:<span class="w"> </span>Perséphone<span class="w"> </span>-<span class="w"> </span>Retro<span class="w"> </span>Funky<span class="w"> </span><span class="o">(</span>SUNDANCE<span class="w"> </span>remix<span class="o">)</span>
artist:
album:
album<span class="w"> </span>artist:<span class="w"> </span>None
track:
BPM:<span class="w"> </span><span class="m">115</span>
GEOB:<span class="w"> </span><span class="o">[</span>Size:<span class="w"> </span><span class="m">3842</span><span class="w"> </span>bytes<span class="o">]</span><span class="w"> </span><span class="o">[</span>Type:<span class="w"> </span>application/octet-stream<span class="o">]</span>
Description:<span class="w"> </span>Serato<span class="w"> </span>Overview
Filename:
GEOB:<span class="w"> </span><span class="o">[</span>Size:<span class="w"> </span><span class="m">2</span><span class="w"> </span>bytes<span class="o">]</span><span class="w"> </span><span class="o">[</span>Type:<span class="w"> </span>application/octet-stream<span class="o">]</span>
Description:<span class="w"> </span>Serato<span class="w"> </span>Analysis
Filename:
GEOB:<span class="w"> </span><span class="o">[</span>Size:<span class="w"> </span><span class="m">22</span><span class="w"> </span>bytes<span class="o">]</span><span class="w"> </span><span class="o">[</span>Type:<span class="w"> </span>application/octet-stream<span class="o">]</span>
Description:<span class="w"> </span>Serato<span class="w"> </span>Autotags
Filename:
GEOB:<span class="w"> </span><span class="o">[</span>Size:<span class="w"> </span><span class="m">318</span><span class="w"> </span>bytes<span class="o">]</span><span class="w"> </span><span class="o">[</span>Type:<span class="w"> </span>application/octet-stream<span class="o">]</span>
Description:<span class="w"> </span>Serato<span class="w"> </span>Markers_
Filename:
GEOB:<span class="w"> </span><span class="o">[</span>Size:<span class="w"> </span><span class="m">470</span><span class="w"> </span>bytes<span class="o">]</span><span class="w"> </span><span class="o">[</span>Type:<span class="w"> </span>application/octet-stream<span class="o">]</span>
Description:<span class="w"> </span>Serato<span class="w"> </span>Markers2
Filename:
GEOB:<span class="w"> </span><span class="o">[</span>Size:<span class="w"> </span><span class="m">15</span><span class="w"> </span>bytes<span class="o">]</span><span class="w"> </span><span class="o">[</span>Type:<span class="w"> </span>application/octet-stream<span class="o">]</span>
Description:<span class="w"> </span>Serato<span class="w"> </span>BeatGrid
Filename:
GEOB:<span class="w"> </span><span class="o">[</span>Size:<span class="w"> </span><span class="m">16401</span><span class="w"> </span>bytes<span class="o">]</span><span class="w"> </span><span class="o">[</span>Type:<span class="w"> </span>application/octet-stream<span class="o">]</span>
Description:<span class="w"> </span>Serato<span class="w"> </span>Offsets_
Filename:
-------------------------------------------------------------------------------
<span class="m">6</span><span class="w"> </span>ID3<span class="w"> </span>Frames:
TIT2<span class="w"> </span><span class="o">(</span><span class="m">52</span><span class="w"> </span>bytes<span class="o">)</span>
TCON<span class="w"> </span><span class="o">(</span><span class="m">11</span><span class="w"> </span>bytes<span class="o">)</span>
TKEY<span class="w"> </span><span class="o">(</span><span class="m">13</span><span class="w"> </span>bytes<span class="o">)</span>
RVAD<span class="w"> </span><span class="o">(</span><span class="m">20</span><span class="w"> </span>bytes<span class="o">)</span>
TBPM<span class="w"> </span><span class="o">(</span><span class="m">14</span><span class="w"> </span>bytes<span class="o">)</span>
GEOB<span class="w"> </span>x<span class="w"> </span><span class="m">7</span><span class="w"> </span><span class="o">(</span><span class="m">21441</span><span class="w"> </span>bytes<span class="o">)</span>
<span class="m">512</span><span class="w"> </span>bytes<span class="w"> </span>unused<span class="w"> </span><span class="o">(</span>padding<span class="o">)</span>
-------------------------------------------------------------------------------
</code></pre></div>
<p>The <code>GEOB</code> tags look interesting, so let's dump them to individual files for further analysis:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>eyeD3<span class="w"> </span>--write-objects<span class="w"> </span>analyzed<span class="w"> </span>analyzed.mp3
eyed3.id3.frames:WARNING:<span class="w"> </span>Frame<span class="w"> </span><span class="s1">'RVAD'</span><span class="w"> </span>is<span class="w"> </span>not<span class="w"> </span>yet<span class="w"> </span>supported,<span class="w"> </span>using<span class="w"> </span>raw<span class="w"> </span>Frame<span class="w"> </span>to<span class="w"> </span>parse
.../analyzed.mp3<span class="w"> </span><span class="o">[</span><span class="w"> </span><span class="m">8</span>.16<span class="w"> </span>MB<span class="w"> </span><span class="o">]</span>
-------------------------------------------------------------------------------
Time:<span class="w"> </span><span class="m">03</span>:33<span class="w"> </span>MPEG1,<span class="w"> </span>Layer<span class="w"> </span>III<span class="w"> </span><span class="o">[</span><span class="w"> </span><span class="m">320</span><span class="w"> </span>kb/s<span class="w"> </span>@<span class="w"> </span><span class="m">44100</span><span class="w"> </span>Hz<span class="w"> </span>-<span class="w"> </span>Joint<span class="w"> </span>stereo<span class="w"> </span><span class="o">]</span>
-------------------------------------------------------------------------------
ID3<span class="w"> </span>v2.3:
title:<span class="w"> </span>Perséphone<span class="w"> </span>-<span class="w"> </span>Retro<span class="w"> </span>Funky<span class="w"> </span><span class="o">(</span>SUNDANCE<span class="w"> </span>remix<span class="o">)</span>
artist:
album:
album<span class="w"> </span>artist:<span class="w"> </span>None
track:
BPM:<span class="w"> </span><span class="m">115</span>
GEOB:<span class="w"> </span><span class="o">[</span>Size:<span class="w"> </span><span class="m">3842</span><span class="w"> </span>bytes<span class="o">]</span><span class="w"> </span><span class="o">[</span>Type:<span class="w"> </span>application/octet-stream<span class="o">]</span>
Description:<span class="w"> </span>Serato<span class="w"> </span>Overview
Filename:
Writing<span class="w"> </span>analyzed/Serato<span class="w"> </span>Overview.octet-stream...
GEOB:<span class="w"> </span><span class="o">[</span>Size:<span class="w"> </span><span class="m">2</span><span class="w"> </span>bytes<span class="o">]</span><span class="w"> </span><span class="o">[</span>Type:<span class="w"> </span>application/octet-stream<span class="o">]</span>
Description:<span class="w"> </span>Serato<span class="w"> </span>Analysis
Filename:
Writing<span class="w"> </span>analyzed/Serato<span class="w"> </span>Analysis.octet-stream...
GEOB:<span class="w"> </span><span class="o">[</span>Size:<span class="w"> </span><span class="m">22</span><span class="w"> </span>bytes<span class="o">]</span><span class="w"> </span><span class="o">[</span>Type:<span class="w"> </span>application/octet-stream<span class="o">]</span>
Description:<span class="w"> </span>Serato<span class="w"> </span>Autotags
Filename:
Writing<span class="w"> </span>analyzed/Serato<span class="w"> </span>Autotags.octet-stream...
GEOB:<span class="w"> </span><span class="o">[</span>Size:<span class="w"> </span><span class="m">318</span><span class="w"> </span>bytes<span class="o">]</span><span class="w"> </span><span class="o">[</span>Type:<span class="w"> </span>application/octet-stream<span class="o">]</span>
Description:<span class="w"> </span>Serato<span class="w"> </span>Markers_
Filename:
Writing<span class="w"> </span>analyzed/Serato<span class="w"> </span>Markers_.octet-stream...
GEOB:<span class="w"> </span><span class="o">[</span>Size:<span class="w"> </span><span class="m">470</span><span class="w"> </span>bytes<span class="o">]</span><span class="w"> </span><span class="o">[</span>Type:<span class="w"> </span>application/octet-stream<span class="o">]</span>
Description:<span class="w"> </span>Serato<span class="w"> </span>Markers2
Filename:
Writing<span class="w"> </span>analyzed/Serato<span class="w"> </span>Markers2.octet-stream...
GEOB:<span class="w"> </span><span class="o">[</span>Size:<span class="w"> </span><span class="m">15</span><span class="w"> </span>bytes<span class="o">]</span><span class="w"> </span><span class="o">[</span>Type:<span class="w"> </span>application/octet-stream<span class="o">]</span>
Description:<span class="w"> </span>Serato<span class="w"> </span>BeatGrid
Filename:
Writing<span class="w"> </span>analyzed/Serato<span class="w"> </span>BeatGrid.octet-stream...
GEOB:<span class="w"> </span><span class="o">[</span>Size:<span class="w"> </span><span class="m">16401</span><span class="w"> </span>bytes<span class="o">]</span><span class="w"> </span><span class="o">[</span>Type:<span class="w"> </span>application/octet-stream<span class="o">]</span>
Description:<span class="w"> </span>Serato<span class="w"> </span>Offsets_
Filename:
Writing<span class="w"> </span>analyzed/Serato<span class="w"> </span>Offsets_.octet-stream...
-------------------------------------------------------------------------------
</code></pre></div>
<h2>A first look</h2>
<p>Now let's have a look at the tag data and see if we can determine the meaning of it.
The <code>Serato Analysis</code> tag only contains 2 bytes, so let's look at that one first.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>hexdump<span class="w"> </span>-C<span class="w"> </span>analyzed/Serato<span class="se">\ </span>Analysis.octet-stream
<span class="m">00000000</span><span class="w"> </span><span class="m">02</span><span class="w"> </span><span class="m">01</span><span class="w"> </span><span class="p">|</span>..<span class="p">|</span>
<span class="m">00000002</span>
</code></pre></div>
<p>It's just a suspicion, but the numbers <code>02 01</code> look like the major/minor part of the version number of Serato that was used for analysis (2.1.1).</p>
<p>The hexdump of next larger tag (in terms of byte length), <code>Serato BeatGrid</code>, is inconclusive at the moment, but <code>Serato Autotags</code> contains recognizable data:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>hexdump<span class="w"> </span>-C<span class="w"> </span>analyzed/Serato<span class="se">\ </span>Autotags.octet-stream
<span class="m">00000000</span><span class="w"> </span><span class="m">01</span><span class="w"> </span><span class="m">01</span><span class="w"> </span><span class="m">31</span><span class="w"> </span><span class="m">31</span><span class="w"> </span><span class="m">35</span><span class="w"> </span>2e<span class="w"> </span><span class="m">30</span><span class="w"> </span><span class="m">30</span><span class="w"> </span><span class="m">00</span><span class="w"> </span>2d<span class="w"> </span><span class="m">33</span><span class="w"> </span>2e<span class="w"> </span><span class="m">32</span><span class="w"> </span><span class="m">35</span><span class="w"> </span><span class="m">37</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="p">|</span>..115.00.-3.257.<span class="p">|</span>
<span class="m">00000010</span><span class="w"> </span><span class="m">30</span><span class="w"> </span>2e<span class="w"> </span><span class="m">30</span><span class="w"> </span><span class="m">30</span><span class="w"> </span><span class="m">30</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="p">|</span><span class="m">0</span>.000.<span class="p">|</span>
<span class="m">00000016</span>
</code></pre></div>
<p>We can easily see that the tag contains the string <code>115.00</code>, which is also the BPM value that is displayed by Serato.
There are two more ASCII strings in there, but I'm not sure what they are for.</p>
<p>Next, let's have a look at hexdump of <code>Serato Markers2</code>:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>hexdump<span class="w"> </span>-C<span class="w"> </span>analyzed/Serato<span class="se">\ </span>Markers2.octet-stream
<span class="m">00000000</span><span class="w"> </span><span class="m">01</span><span class="w"> </span><span class="m">01</span><span class="w"> </span><span class="m">41</span><span class="w"> </span><span class="m">51</span><span class="w"> </span><span class="m">46</span><span class="w"> </span><span class="m">44</span><span class="w"> </span><span class="m">54</span><span class="w"> </span><span class="m">30</span><span class="w"> </span><span class="m">78</span><span class="w"> </span><span class="m">50</span><span class="w"> </span><span class="m">55</span><span class="w"> </span><span class="m">67</span><span class="w"> </span><span class="m">41</span><span class="w"> </span><span class="m">41</span><span class="w"> </span><span class="m">41</span><span class="w"> </span><span class="m">41</span><span class="w"> </span><span class="p">|</span>..AQFDT0xPUgAAAA<span class="p">|</span>
<span class="m">00000010</span><span class="w"> </span><span class="m">41</span><span class="w"> </span><span class="m">45</span><span class="w"> </span><span class="m">41</span><span class="w"> </span><span class="m">50</span><span class="w"> </span>2f<span class="w"> </span>2f<span class="w"> </span>2f<span class="w"> </span><span class="m">30</span><span class="w"> </span>4a<span class="w"> </span><span class="m">51</span><span class="w"> </span><span class="m">54</span><span class="w"> </span><span class="m">55</span><span class="w"> </span><span class="m">78</span><span class="w"> </span><span class="m">50</span><span class="w"> </span><span class="m">51</span><span class="w"> </span><span class="m">30</span><span class="w"> </span><span class="p">|</span>AEAP///0JQTUxPQ0<span class="p">|</span>
<span class="m">00000020</span><span class="w"> </span><span class="m">73</span><span class="w"> </span><span class="m">41</span><span class="w"> </span><span class="m">41</span><span class="w"> </span><span class="m">41</span><span class="w"> </span><span class="m">41</span><span class="w"> </span><span class="m">41</span><span class="w"> </span><span class="m">41</span><span class="w"> </span><span class="m">51</span><span class="w"> </span><span class="m">41</span><span class="w"> </span><span class="m">41</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="p">|</span>sAAAAAAQAA......<span class="p">|</span>
<span class="m">00000030</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="p">|</span>................<span class="p">|</span>
*
000001d0<span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="p">|</span>......<span class="p">|</span>
000001d6
</code></pre></div>
<p>The ASCII data in that file certainly looks like it has been base64-encoded.
Let's verify that that assumption and see what's inside:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>grep<span class="w"> </span>-Poaz<span class="w"> </span><span class="s1">'[\w/]*'</span><span class="w"> </span>analyzed/Serato<span class="se">\ </span>Markers2.octet-stream<span class="w"> </span><span class="p">|</span><span class="w"> </span>tr<span class="w"> </span>-d<span class="w"> </span><span class="s1">'\0'</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>base64<span class="w"> </span>-d<span class="w"> </span><span class="p">|</span><span class="w"> </span>hexdump<span class="w"> </span>-C
<span class="m">00000000</span><span class="w"> </span><span class="m">01</span><span class="w"> </span><span class="m">01</span><span class="w"> </span><span class="m">43</span><span class="w"> </span>4f<span class="w"> </span>4c<span class="w"> </span>4f<span class="w"> </span><span class="m">52</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">04</span><span class="w"> </span><span class="m">00</span><span class="w"> </span>ff<span class="w"> </span>ff<span class="w"> </span>ff<span class="w"> </span><span class="p">|</span>..COLOR.........<span class="p">|</span>
<span class="m">00000010</span><span class="w"> </span><span class="m">42</span><span class="w"> </span><span class="m">50</span><span class="w"> </span>4d<span class="w"> </span>4c<span class="w"> </span>4f<span class="w"> </span><span class="m">43</span><span class="w"> </span>4b<span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">01</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="p">|</span>BPMLOCK.......<span class="p">|</span>
0000001e
</code></pre></div>
<p>The <code>Serato Overview</code> tag is almost 4 KB in size. The ASCII representation that is generated by hexdump -C looks quite familiar:</p>
<p><img alt="Serato Overview" src="https://homepage.ruhr-uni-bochum.de/jan.holthuis/images/serato-overview-hexdump.png"></p>
<p>This tag obviously contains the data for the track overview in Serato.</p>
<p>The remaining tags, <code>Serato Markers_</code> and <code>Serato Offsets_</code> contain a lot of repeating data, but it's uncertain what it means.</p>
<h1>Looking for changes</h1>
<p>Now, let's take our analyzed file and set a hot cue point.
Again, we eject afterwards and use <code>eyeD3</code> to write the data inside the ID3 GEOB tags to a directory:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>mkdir<span class="w"> </span>hotcue-00m00s0-red
$<span class="w"> </span>eyeD3<span class="w"> </span>--write-objects<span class="w"> </span>hotcue-00m00s0-red<span class="w"> </span>hotcue-00m00s0-red.mp3
<output<span class="w"> </span>removed>
$<span class="w"> </span><span class="nb">cd</span><span class="w"> </span>hotcue-00m00s0-red
</code></pre></div>
<p>Next, let's check if anything changed:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="k">for</span><span class="w"> </span>file<span class="w"> </span><span class="k">in</span><span class="w"> </span>*.octet-stream<span class="p">;</span><span class="w"> </span><span class="k">do</span><span class="w"> </span>diff<span class="w"> </span>-q<span class="w"> </span><span class="s2">"</span><span class="nv">$file</span><span class="s2">"</span><span class="w"> </span><span class="s2">"../analyzed/</span><span class="nv">$file</span><span class="s2">"</span><span class="p">;</span><span class="w"> </span><span class="k">done</span>
Files<span class="w"> </span>Serato<span class="w"> </span>Markers2.octet-stream<span class="w"> </span>and<span class="w"> </span>../analyzed/Serato<span class="w"> </span>Markers2.octet-stream<span class="w"> </span>differ
Files<span class="w"> </span>Serato<span class="w"> </span>Markers_.octet-stream<span class="w"> </span>and<span class="w"> </span>../analyzed/Serato<span class="w"> </span>Markers_.octet-stream<span class="w"> </span>differ
</code></pre></div>
<p>Both <code>Serato Markers_</code> and <code>Serato Markers2</code> have been modified.</p>
<p>Let's look at <code>Serato Markers2</code> first.
Before setting the cue point, that file just contained base64-encoded data and some zero bytes.
This is still the case, but the base64-encoded data has grown in size:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>hexdump<span class="w"> </span>-C<span class="w"> </span>Serato<span class="se">\ </span>Markers2.octet-stream
<span class="m">00000000</span><span class="w"> </span><span class="m">01</span><span class="w"> </span><span class="m">01</span><span class="w"> </span><span class="m">41</span><span class="w"> </span><span class="m">51</span><span class="w"> </span><span class="m">46</span><span class="w"> </span><span class="m">44</span><span class="w"> </span><span class="m">54</span><span class="w"> </span><span class="m">30</span><span class="w"> </span><span class="m">78</span><span class="w"> </span><span class="m">50</span><span class="w"> </span><span class="m">55</span><span class="w"> </span><span class="m">67</span><span class="w"> </span><span class="m">41</span><span class="w"> </span><span class="m">41</span><span class="w"> </span><span class="m">41</span><span class="w"> </span><span class="m">41</span><span class="w"> </span><span class="p">|</span>..AQFDT0xPUgAAAA<span class="p">|</span>
<span class="m">00000010</span><span class="w"> </span><span class="m">41</span><span class="w"> </span><span class="m">45</span><span class="w"> </span><span class="m">41</span><span class="w"> </span><span class="m">50</span><span class="w"> </span>2f<span class="w"> </span>2f<span class="w"> </span>2f<span class="w"> </span><span class="m">30</span><span class="w"> </span>4e<span class="w"> </span><span class="m">56</span><span class="w"> </span><span class="m">52</span><span class="w"> </span><span class="m">51</span><span class="w"> </span><span class="m">41</span><span class="w"> </span><span class="m">41</span><span class="w"> </span><span class="m">41</span><span class="w"> </span><span class="m">41</span><span class="w"> </span><span class="p">|</span>AEAP///0NVRQAAAA<span class="p">|</span>
<span class="m">00000020</span><span class="w"> </span><span class="m">41</span><span class="w"> </span>4e<span class="w"> </span><span class="m">41</span><span class="w"> </span><span class="m">41</span><span class="w"> </span><span class="m">41</span><span class="w"> </span><span class="m">41</span><span class="w"> </span><span class="m">41</span><span class="w"> </span><span class="m">41</span><span class="w"> </span><span class="m">41</span><span class="w"> </span><span class="m">41</span><span class="w"> </span><span class="m">41</span><span class="w"> </span>4d<span class="w"> </span><span class="m">77</span><span class="w"> </span><span class="m">41</span><span class="w"> </span><span class="m">41</span><span class="w"> </span><span class="m">41</span><span class="w"> </span><span class="p">|</span>ANAAAAAAAAAMwAAA<span class="p">|</span>
<span class="m">00000030</span><span class="w"> </span><span class="m">41</span><span class="w"> </span><span class="m">41</span><span class="w"> </span><span class="m">41</span><span class="w"> </span><span class="m">45</span><span class="w"> </span>4a<span class="w"> </span><span class="m">51</span><span class="w"> </span><span class="m">54</span><span class="w"> </span><span class="m">55</span><span class="w"> </span><span class="m">78</span><span class="w"> </span><span class="m">50</span><span class="w"> </span><span class="m">51</span><span class="w"> </span><span class="m">30</span><span class="w"> </span><span class="m">73</span><span class="w"> </span><span class="m">41</span><span class="w"> </span><span class="m">41</span><span class="w"> </span><span class="m">41</span><span class="w"> </span><span class="p">|</span>AAAEJQTUxPQ0sAAA<span class="p">|</span>
<span class="m">00000040</span><span class="w"> </span><span class="m">41</span><span class="w"> </span><span class="m">41</span><span class="w"> </span><span class="m">41</span><span class="w"> </span><span class="m">51</span><span class="w"> </span><span class="m">41</span><span class="w"> </span><span class="m">41</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="p">|</span>AAAQAA..........<span class="p">|</span>
<span class="m">00000050</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="p">|</span>................<span class="p">|</span>
*
000001d0<span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="p">|</span>......<span class="p">|</span>
000001d6
$<span class="w"> </span>grep<span class="w"> </span>-Poaz<span class="w"> </span><span class="s1">'[\w/]*'</span><span class="w"> </span>Serato<span class="se">\ </span>Markers2.octet-stream<span class="w"> </span><span class="p">|</span><span class="w"> </span>tr<span class="w"> </span>-d<span class="w"> </span><span class="s1">'\0'</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>base64<span class="w"> </span>-d<span class="w"> </span><span class="p">|</span><span class="w"> </span>hexdump<span class="w"> </span>-C
<span class="m">00000000</span><span class="w"> </span><span class="m">01</span><span class="w"> </span><span class="m">01</span><span class="w"> </span><span class="m">43</span><span class="w"> </span>4f<span class="w"> </span>4c<span class="w"> </span>4f<span class="w"> </span><span class="m">52</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">04</span><span class="w"> </span><span class="m">00</span><span class="w"> </span>ff<span class="w"> </span>ff<span class="w"> </span>ff<span class="w"> </span><span class="p">|</span>..COLOR.........<span class="p">|</span>
<span class="m">00000010</span><span class="w"> </span><span class="m">43</span><span class="w"> </span><span class="m">55</span><span class="w"> </span><span class="m">45</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span>0d<span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span>cc<span class="w"> </span><span class="p">|</span>CUE.............<span class="p">|</span>
<span class="m">00000020</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">42</span><span class="w"> </span><span class="m">50</span><span class="w"> </span>4d<span class="w"> </span>4c<span class="w"> </span>4f<span class="w"> </span><span class="m">43</span><span class="w"> </span>4b<span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="p">|</span>.....BPMLOCK....<span class="p">|</span>
<span class="m">00000030</span><span class="w"> </span><span class="m">01</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="p">|</span>...<span class="p">|</span>
<span class="m">00000033</span>
</code></pre></div>
<p>When we compare it with the previous content, we can see that these 21 bytes have been added:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>grep<span class="w"> </span>-Poaz<span class="w"> </span><span class="s1">'[\w/]*'</span><span class="w"> </span>Serato<span class="se">\ </span>Markers2.octet-stream<span class="w"> </span><span class="p">|</span><span class="w"> </span>tr<span class="w"> </span>-d<span class="w"> </span><span class="s1">'\0'</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>base64<span class="w"> </span>-d<span class="w"> </span><span class="p">|</span><span class="w"> </span>tail<span class="w"> </span>-c<span class="w"> </span>+17<span class="w"> </span><span class="p">|</span><span class="w"> </span>head<span class="w"> </span>-c<span class="w"> </span>-14<span class="w"> </span><span class="p">|</span><span class="w"> </span>hexdump<span class="w"> </span>-C
<span class="m">00000000</span><span class="w"> </span><span class="m">43</span><span class="w"> </span><span class="m">55</span><span class="w"> </span><span class="m">45</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span>0d<span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span>cc<span class="w"> </span><span class="p">|</span>CUE.............<span class="p">|</span>
<span class="m">00000010</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="p">|</span>.....<span class="p">|</span>
<span class="m">00000015</span>
</code></pre></div>
<p>Except for the first 3 bytes that say <code>CUE</code> in ASCII, we don't really know what these values mean though.</p>
<p>To determine what the individual bytes mean, we can change something and check how these are represented in the ID3 tags.
For example, when the cue color is changed to blue, the content of the base64-encoded data changes as well:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>grep<span class="w"> </span>-Poaz<span class="w"> </span><span class="s1">'[\w/]*'</span><span class="w"> </span>hotcue-00m00s0-red/Serato<span class="se">\ </span>Markers2.octet-stream<span class="w"> </span><span class="p">|</span><span class="w"> </span>tr<span class="w"> </span>-d<span class="w"> </span><span class="s1">'\0'</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>base64<span class="w"> </span>-d<span class="w"> </span><span class="p">|</span><span class="w"> </span>tail<span class="w"> </span>-c<span class="w"> </span>+17<span class="w"> </span><span class="p">|</span><span class="w"> </span>head<span class="w"> </span>-c<span class="w"> </span>-14<span class="w"> </span><span class="p">|</span><span class="w"> </span>hexdump<span class="w"> </span>-e<span class="w"> </span><span class="s1">'"%08.8_ax " 21/1 "%02X " "\n"'</span>
<span class="m">00000000</span><span class="w"> </span><span class="m">43</span><span class="w"> </span><span class="m">55</span><span class="w"> </span><span class="m">45</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span>0d<span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span>cc<span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span>
$<span class="w"> </span>grep<span class="w"> </span>-Poaz<span class="w"> </span><span class="s1">'[\w/]*'</span><span class="w"> </span>hotcue-00m00s0-bue/Serato<span class="se">\ </span>Markers2.octet-stream<span class="w"> </span><span class="p">|</span><span class="w"> </span>tr<span class="w"> </span>-d<span class="w"> </span><span class="s1">'\0'</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>base64<span class="w"> </span>-d<span class="w"> </span><span class="p">|</span><span class="w"> </span>tail<span class="w"> </span>-c<span class="w"> </span>+17<span class="w"> </span><span class="p">|</span><span class="w"> </span>head<span class="w"> </span>-c<span class="w"> </span>-14<span class="w"> </span><span class="p">|</span><span class="w"> </span>hexdump<span class="w"> </span>-e<span class="w"> </span><span class="s1">'"%08.8_ax " 21/1 "%02X " "\n"'</span>
<span class="m">00000000</span><span class="w"> </span><span class="m">43</span><span class="w"> </span><span class="m">55</span><span class="w"> </span><span class="m">45</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span>0d<span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span>cc<span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span>
</code></pre></div>
<p>As we can see, the <code>cc</code> byte has moved two bytes to the right, i.e. <code>cc 00 00</code> became <code>00 00 cc</code>.
If that 3-byte value is interpreted as RGB channel values, <code>cc 00 00</code> is indeed red and <code>00 00 cc</code> is blue.</p>
<p>Next, we compare it with a file that 8 cuepoints in different colors.</p>
<p><img alt="8 Hotcues with different colors" src="https://homepage.ruhr-uni-bochum.de/jan.holthuis/images/serato-8-hotcue-colors.png"></p>
<p>The data looks like this:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>grep<span class="w"> </span>-Poaz<span class="w"> </span><span class="s1">'[\w/]*'</span><span class="w"> </span>hotcue-colors/Serato<span class="se">\ </span>Markers2.octet-stream<span class="w"> </span><span class="p">|</span><span class="w"> </span>tr<span class="w"> </span>-d<span class="w"> </span><span class="s1">'\0'</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>base64<span class="w"> </span>-d<span class="w"> </span><span class="p">|</span><span class="w"> </span>tail<span class="w"> </span>-c<span class="w"> </span>+17<span class="w"> </span><span class="p">|</span><span class="w"> </span>head<span class="w"> </span>-c<span class="w"> </span>-14<span class="w"> </span><span class="p">|</span><span class="w"> </span>hexdump<span class="w"> </span>-e<span class="w"> </span><span class="s1">'"%08.8_ax " 21/1 "%02X " "\n"'</span>
<span class="m">00000000</span><span class="w"> </span><span class="m">43</span><span class="w"> </span><span class="m">55</span><span class="w"> </span><span class="m">45</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span>0d<span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span>cc<span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span>
<span class="m">00000015</span><span class="w"> </span><span class="m">43</span><span class="w"> </span><span class="m">55</span><span class="w"> </span><span class="m">45</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span>0d<span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">01</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span>cc<span class="w"> </span><span class="m">88</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span>
0000002a<span class="w"> </span><span class="m">43</span><span class="w"> </span><span class="m">55</span><span class="w"> </span><span class="m">45</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span>0d<span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">02</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span>cc<span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span>
0000003f<span class="w"> </span><span class="m">43</span><span class="w"> </span><span class="m">55</span><span class="w"> </span><span class="m">45</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span>0d<span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">03</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span>cc<span class="w"> </span>cc<span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span>
<span class="m">00000054</span><span class="w"> </span><span class="m">43</span><span class="w"> </span><span class="m">55</span><span class="w"> </span><span class="m">45</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span>0d<span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">04</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span>cc<span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span>
<span class="m">00000069</span><span class="w"> </span><span class="m">43</span><span class="w"> </span><span class="m">55</span><span class="w"> </span><span class="m">45</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span>0d<span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">05</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span>cc<span class="w"> </span><span class="m">00</span><span class="w"> </span>cc<span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span>
0000007e<span class="w"> </span><span class="m">43</span><span class="w"> </span><span class="m">55</span><span class="w"> </span><span class="m">45</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span>0d<span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">06</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span>cc<span class="w"> </span>cc<span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span>
<span class="m">00000093</span><span class="w"> </span><span class="m">43</span><span class="w"> </span><span class="m">55</span><span class="w"> </span><span class="m">45</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span>0d<span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">07</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">88</span><span class="w"> </span><span class="m">00</span><span class="w"> </span>cc<span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span>
</code></pre></div>
<p>As we would expect, there are 8 different entries with 8 different values at the position the color could be stored.</p>
<p>Using <code>gpick</code>, we check if the colors actually match:</p>
<p><img alt="Hotcue Colors in gpick" src="https://homepage.ruhr-uni-bochum.de/jan.holthuis/images/gpick-8-colors.png"></p>
<p>Bingo! These colors look a lot like those displayed in Serato.
However, the colors displayed in Serato are slightly different.
When using a color picker tool (e.g. <code>gpick</code>), we can see that the red displayed in the UI is in fact <code>#c02626</code>:</p>
<p><img alt="Hotcue Colors in Serato with gpick" src="https://homepage.ruhr-uni-bochum.de/jan.holthuis/images/serato-8-hotcue-colors-gpick.png"></p>
<p>However, since the colors and their indices match so well, this can't really be coincidence.
It suppose there's some kind of color scheme or transformation applied to these values.</p>
<h2>Diving deeper into the <code>Serato Markers2</code> internals</h2>
<p>Since color information seem to be easily recognizable, we can use this to check if tracklist colors are stored in the <code>Serato Markers2</code> tag as well.
Hence, let's the tracklist color to green:</p>
<p><img alt="Green tracklist color in Serato" src="https://homepage.ruhr-uni-bochum.de/jan.holthuis/images/serato-tracklist-colorpicker-green.png"></p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>grep<span class="w"> </span>-Poaz<span class="w"> </span><span class="s1">'[\w/]*'</span><span class="w"> </span>tracklist-color/Serato<span class="se">\ </span>Markers2.octet-stream<span class="w"> </span><span class="p">|</span><span class="w"> </span>tr<span class="w"> </span>-d<span class="w"> </span><span class="s1">'\0'</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>base64<span class="w"> </span>-d<span class="w"> </span><span class="p">|</span><span class="w"> </span>hexdump<span class="w"> </span>-C
<span class="m">00000000</span><span class="w"> </span><span class="m">01</span><span class="w"> </span><span class="m">01</span><span class="w"> </span><span class="m">43</span><span class="w"> </span>4f<span class="w"> </span>4c<span class="w"> </span>4f<span class="w"> </span><span class="m">52</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">04</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">99</span><span class="w"> </span>ff<span class="w"> </span><span class="m">99</span><span class="w"> </span><span class="p">|</span>..COLOR.........<span class="p">|</span>
<span class="m">00000010</span><span class="w"> </span><span class="m">42</span><span class="w"> </span><span class="m">50</span><span class="w"> </span>4d<span class="w"> </span>4c<span class="w"> </span>4f<span class="w"> </span><span class="m">43</span><span class="w"> </span>4b<span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">01</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="p">|</span>BPMLOCK.......<span class="p">|</span>
0000001e
</code></pre></div>
<p>Since the color <code>#99FF99</code> is green, the bytes <code>99 ff 99</code> seem to be RGB channel values for the track color.</p>
<p>It makes sense to recapture our findings regarding the <code>Serato Markers2</code> tag and see if we can see a pattern:</p>
<div class="highlight"><pre><span></span><code><span class="mf">1.</span><span class="w"> </span><span class="n">The</span><span class="w"> </span><span class="n">base64</span><span class="o">-</span><span class="n">content</span><span class="w"> </span><span class="k">starts</span><span class="w"> </span><span class="k">with</span><span class="w"> </span><span class="n n-Quoted">`01 01`</span><span class="p">...</span>
<span class="mf">2.</span><span class="w"> </span><span class="p">...</span><span class="w"> </span><span class="n">followed</span><span class="w"> </span><span class="k">by</span><span class="w"> </span><span class="n">multiple</span><span class="w"> </span><span class="n">entries</span><span class="p">,</span><span class="w"> </span><span class="k">where</span><span class="o">:</span>
<span class="w"> </span><span class="mf">1.</span><span class="w"> </span><span class="k">Each</span><span class="w"> </span><span class="n">entry</span><span class="w"> </span><span class="k">starts</span><span class="w"> </span><span class="k">with</span><span class="w"> </span><span class="n">a</span><span class="w"> </span><span class="n">zero</span><span class="o">-</span><span class="k">terminated</span><span class="w"> </span><span class="k">ASCII</span><span class="w"> </span><span class="k">string</span><span class="w"> </span><span class="n">denoting</span><span class="w"> </span><span class="n">the</span><span class="w"> </span><span class="n">entry</span><span class="w"> </span><span class="k">type</span><span class="p">.</span>
<span class="w"> </span><span class="mf">2.</span><span class="w"> </span><span class="n n-Quoted">`COLOR`</span><span class="w"> </span><span class="n">entries</span><span class="w"> </span><span class="n">contain</span><span class="w"> </span><span class="mi">8</span><span class="w"> </span><span class="n">more</span><span class="w"> </span><span class="n">bytes</span><span class="p">.</span>
<span class="w"> </span><span class="mf">3.</span><span class="w"> </span><span class="n n-Quoted">`CUE`</span><span class="w"> </span><span class="n">entries</span><span class="w"> </span><span class="n">contain</span><span class="w"> </span><span class="mi">17</span><span class="w"> </span><span class="n">more</span><span class="w"> </span><span class="n">bytes</span><span class="p">.</span>
<span class="w"> </span><span class="mf">4.</span><span class="w"> </span><span class="n n-Quoted">`BPMLOCK`</span><span class="w"> </span><span class="n">entries</span><span class="w"> </span><span class="n">contain</span><span class="w"> </span><span class="mi">6</span><span class="w"> </span><span class="n">more</span><span class="w"> </span><span class="n">bytes</span><span class="p">.</span>
</code></pre></div>
<p>How would Serato parse this data?
Each entry has a different size, even when we leave out the zero-terminated entry type.
While it might be possible that Serato has a built-in mapping of entry types and their lengths, this would make it impossible to add new entry types in a backward-compatible way.
Older versions would have no way to determine the length and thus can't skip the entry because they do not know where the next entry starts.</p>
<p>Let's look closer at the first few bytes of each entry:</p>
<div class="highlight"><pre><span></span><code><span class="mf">1.</span><span class="w"> </span><span class="n n-Quoted">`COLOR`</span><span class="w"> </span><span class="n">entries</span><span class="w"> </span><span class="n">contain</span><span class="w"> </span><span class="mi">8</span><span class="w"> </span><span class="n">bytes</span><span class="w"> </span><span class="k">and</span><span class="w"> </span><span class="k">start</span><span class="w"> </span><span class="k">with</span><span class="w"> </span><span class="n n-Quoted">`00 00 00 04`</span><span class="p">.</span>
<span class="mf">2.</span><span class="w"> </span><span class="n n-Quoted">`CUE`</span><span class="w"> </span><span class="n">entries</span><span class="w"> </span><span class="n">contain</span><span class="w"> </span><span class="mi">17</span><span class="w"> </span><span class="n">bytes</span><span class="w"> </span><span class="k">and</span><span class="w"> </span><span class="k">start</span><span class="w"> </span><span class="k">with</span><span class="w"> </span><span class="n n-Quoted">`00 00 00 0d`</span><span class="p">.</span>
<span class="mf">3.</span><span class="w"> </span><span class="n n-Quoted">`BPMLOCK`</span><span class="w"> </span><span class="n">entries</span><span class="w"> </span><span class="n">contain</span><span class="w"> </span><span class="mi">6</span><span class="w"> </span><span class="n">bytes</span><span class="w"> </span><span class="k">and</span><span class="w"> </span><span class="k">start</span><span class="w"> </span><span class="k">with</span><span class="w"> </span><span class="n n-Quoted">`00 00 00 01`</span><span class="p">.</span>
</code></pre></div>
<p>If we parse the first 4 bytes as 32-bit integer (little-endian), we can reasonably assume that determine the entry's length (excluding the integer itself).
But wait, this doesn't work for <code>BPMLOCK</code> entries since they contain 2 more bytes, even though according to the length value there should only be 1 byte.</p>
<p>Does this mean that our assumption is wrong?</p>
<p>No, not neccessarily:
Since <code>BPMLOCK</code> is the last entry inside the base64-encoded data, it's possible that the last byte (<code>00</code>) doesn't actually belong to the <code>BPMLOCK</code> entry.
Instead it could be a sentinel value to tell Serato to stop parsing entries.</p>
<h2>Hotcue positions</h2>
<p>Now it's time to decode the hotcue positions.
To archieve this, I created a 5 hotcues with (hopefully) recognizable positions:</p>
<table>
<thead>
<tr>
<th>Hotcue</th>
<th>Position</th>
<th>Comment</th>
</tr>
</thead>
<tbody>
<tr>
<td>1</td>
<td><code>00:00.0</code></td>
<td>Start of the file</td>
</tr>
<tr>
<td>2</td>
<td><code>03:38.4</code></td>
<td>End of the file</td>
</tr>
<tr>
<td>3</td>
<td><code>01:00.0</code></td>
<td></td>
</tr>
<tr>
<td>4</td>
<td><code>00:00.1</code></td>
<td></td>
</tr>
<tr>
<td>5</td>
<td><code>00:01.0</code></td>
<td></td>
</tr>
</tbody>
</table>
<p>Looking at the hexdump, it is apparent that the 5 bytes between the hotcue index and the color contains the position:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>grep<span class="w"> </span>-Poaz<span class="w"> </span><span class="s1">'[\w/]*'</span><span class="w"> </span>hotcue-positions-00m00s0-03m38s4-01m00s0-00m00s1-00m01s0/Serato<span class="se">\ </span>Markers2.octet-stream<span class="w"> </span><span class="p">|</span><span class="w"> </span>tr<span class="w"> </span>-d<span class="w"> </span><span class="s1">'\0'</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>base64<span class="w"> </span>-d<span class="w"> </span><span class="p">|</span><span class="w"> </span>tail<span class="w"> </span>-c<span class="w"> </span>+17<span class="w"> </span><span class="p">|</span><span class="w"> </span>head<span class="w"> </span>-c<span class="w"> </span>-14<span class="w"> </span><span class="p">|</span><span class="w"> </span>hexdump<span class="w"> </span>-e<span class="w"> </span><span class="s1">'"%08.8_ax " 21/1 "%02x " "\n"'</span>
<span class="m">00000000</span><span class="w"> </span><span class="m">43</span><span class="w"> </span><span class="m">55</span><span class="w"> </span><span class="m">45</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span>0d<span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span>cc<span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span>
<span class="m">00000015</span><span class="w"> </span><span class="m">43</span><span class="w"> </span><span class="m">55</span><span class="w"> </span><span class="m">45</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span>0d<span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">01</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">03</span><span class="w"> </span><span class="m">55</span><span class="w"> </span><span class="m">58</span><span class="w"> </span><span class="m">00</span><span class="w"> </span>cc<span class="w"> </span><span class="m">88</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span>
0000002a<span class="w"> </span><span class="m">43</span><span class="w"> </span><span class="m">55</span><span class="w"> </span><span class="m">45</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span>0d<span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">02</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span>ea<span class="w"> </span><span class="m">64</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span>cc<span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span>
0000003f<span class="w"> </span><span class="m">43</span><span class="w"> </span><span class="m">55</span><span class="w"> </span><span class="m">45</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span>0d<span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">03</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span>6c<span class="w"> </span><span class="m">00</span><span class="w"> </span>cc<span class="w"> </span>cc<span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span>
<span class="m">00000054</span><span class="w"> </span><span class="m">43</span><span class="w"> </span><span class="m">55</span><span class="w"> </span><span class="m">45</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span>0d<span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">04</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">03</span><span class="w"> </span>f7<span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span>cc<span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span>
</code></pre></div>
<p>It is likely that the position will be some kind of <code>int</code> or <code>float</code> value.
Since we already found little-endian values, the value will probably have little-endian byte order as well.</p>
<p>Since both types are only 4 bytes long, we do not know which bytes belongs to another field.
However, we can easily check all possibilities and see if we recognize something:</p>
<div class="highlight"><pre><span></span><code><span class="err">$</span> <span class="n">python</span>
<span class="n">Python</span> <span class="mf">3.7.3</span> <span class="p">(</span><span class="n">default</span><span class="p">,</span> <span class="n">Jun</span> <span class="mi">24</span> <span class="mi">2019</span><span class="p">,</span> <span class="mi">04</span><span class="p">:</span><span class="mi">54</span><span class="p">:</span><span class="mi">02</span><span class="p">)</span>
<span class="p">[</span><span class="n">GCC</span> <span class="mf">9.1.0</span><span class="p">]</span> <span class="n">on</span> <span class="n">linux</span>
<span class="n">Type</span> <span class="s2">"help"</span><span class="p">,</span> <span class="s2">"copyright"</span><span class="p">,</span> <span class="s2">"credits"</span> <span class="ow">or</span> <span class="s2">"license"</span> <span class="k">for</span> <span class="n">more</span> <span class="n">information</span><span class="o">.</span>
<span class="o">>>></span> <span class="n">data</span> <span class="o">=</span> <span class="p">[</span>
<span class="o">...</span> <span class="s1">'00 00 00 00 00'</span><span class="p">,</span>
<span class="o">...</span> <span class="s1">'00 03 55 58 00'</span><span class="p">,</span>
<span class="o">...</span> <span class="s1">'00 00 ea 64 00'</span><span class="p">,</span>
<span class="o">...</span> <span class="s1">'00 00 00 6c 00'</span><span class="p">,</span>
<span class="o">...</span> <span class="s1">'00 00 03 f7 00'</span><span class="p">,</span>
<span class="o">...</span> <span class="p">]</span>
<span class="o">>>></span> <span class="kn">import</span> <span class="nn">binascii</span>
<span class="o">>>></span> <span class="n">x</span> <span class="o">=</span> <span class="p">[</span><span class="n">binascii</span><span class="o">.</span><span class="n">unhexlify</span><span class="p">(</span><span class="n">x</span><span class="o">.</span><span class="n">replace</span><span class="p">(</span><span class="s1">' '</span><span class="p">,</span><span class="s1">''</span><span class="p">))</span> <span class="k">for</span> <span class="n">x</span> <span class="ow">in</span> <span class="n">data</span><span class="p">]</span>
<span class="o">>>></span> <span class="n">x</span>
<span class="p">[</span><span class="sa">b</span><span class="s1">'</span><span class="se">\x00\x00\x00\x00\x00</span><span class="s1">'</span><span class="p">,</span> <span class="sa">b</span><span class="s1">'</span><span class="se">\x00\x03</span><span class="s1">UX</span><span class="se">\x00</span><span class="s1">'</span><span class="p">,</span> <span class="sa">b</span><span class="s1">'</span><span class="se">\x00\x00\xea</span><span class="s1">d</span><span class="se">\x00</span><span class="s1">'</span><span class="p">,</span> <span class="sa">b</span><span class="s1">'</span><span class="se">\x00\x00\x00</span><span class="s1">l</span><span class="se">\x00</span><span class="s1">'</span><span class="p">,</span> <span class="sa">b</span><span class="s1">'</span><span class="se">\x00\x00\x03\xf7\x00</span><span class="s1">'</span><span class="p">]</span>
<span class="o">>>></span> <span class="n">bindata</span> <span class="o">=</span> <span class="p">[</span><span class="n">binascii</span><span class="o">.</span><span class="n">unhexlify</span><span class="p">(</span><span class="n">x</span><span class="o">.</span><span class="n">replace</span><span class="p">(</span><span class="s1">' '</span><span class="p">,</span><span class="s1">''</span><span class="p">))</span> <span class="k">for</span> <span class="n">x</span> <span class="ow">in</span> <span class="n">data</span><span class="p">]</span>
<span class="o">>>></span> <span class="kn">import</span> <span class="nn">struct</span>
<span class="o">>>></span> <span class="k">for</span> <span class="n">x</span> <span class="ow">in</span> <span class="n">bindata</span><span class="p">:</span>
<span class="o">...</span> <span class="nb">print</span><span class="p">(</span><span class="s1">'</span><span class="si">%10d</span><span class="s1"> </span><span class="si">%10d</span><span class="s1"> </span><span class="si">%3.10e</span><span class="s1"> </span><span class="si">%3.10e</span><span class="s1">'</span> <span class="o">%</span> <span class="p">(</span><span class="n">struct</span><span class="o">.</span><span class="n">unpack</span><span class="p">(</span><span class="s1">'>xI'</span><span class="p">,</span> <span class="n">x</span><span class="p">)[</span><span class="mi">0</span><span class="p">],</span> <span class="n">struct</span><span class="o">.</span><span class="n">unpack</span><span class="p">(</span><span class="s1">'>Ix'</span><span class="p">,</span> <span class="n">x</span><span class="p">)[</span><span class="mi">0</span><span class="p">],</span> <span class="n">struct</span><span class="o">.</span><span class="n">unpack</span><span class="p">(</span><span class="s1">'>xf'</span><span class="p">,</span> <span class="n">x</span><span class="p">)[</span><span class="mi">0</span><span class="p">],</span> <span class="n">struct</span><span class="o">.</span><span class="n">unpack</span><span class="p">(</span><span class="s1">'>fx'</span><span class="p">,</span> <span class="n">x</span><span class="p">)[</span><span class="mi">0</span><span class="p">]))</span>
<span class="o">...</span>
<span class="mi">0</span> <span class="mi">0</span> <span class="mf">0.0000000000e+00</span> <span class="mf">0.0000000000e+00</span>
<span class="mi">55924736</span> <span class="mi">218456</span> <span class="mf">6.2696093227e-37</span> <span class="mf">3.0612205732e-40</span>
<span class="mi">15361024</span> <span class="mi">60004</span> <span class="mf">2.1525379342e-38</span> <span class="mf">8.4083513053e-41</span>
<span class="mi">27648</span> <span class="mi">108</span> <span class="mf">3.8743099942e-41</span> <span class="mf">1.5134023415e-43</span>
<span class="mi">259840</span> <span class="mi">1015</span> <span class="mf">3.6411339297e-40</span> <span class="mf">1.4223179413e-42</span>
</code></pre></div>
<p>Now let's look at the results while keeping in mind that:
- Hotcue 2 has been placed at 03:38.4 = 218.4 seconds = 218400 milliseconds
- Hotcue 3 has been placed at 1 minute = 60 seconds = 60000 milliseconds.
- Hotcue 4 has been placed at 0.1 seconds = 100 milliseconds.
- Hotcue 5 has been placed at 1 second = 1000 milliseconds.</p>
<p>Column 2 is obviously containing the values we're looking for.
This means that the 4 bytes following the hotcue index value contain the position in milliseconds as little-endian integer.</p>
<h2>What's your name?</h2>
<p>Since it's also possible to assign textual names to hot cue points in Serato, let's check how these are stored.
After preparing a file with 3 named hotcues and dumping its tags, a quick glance at <code>hexdump</code>'s output confirms that the names are also stored in <code>Serato Markers2</code>.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>grep<span class="w"> </span>-Poaz<span class="w"> </span><span class="s1">'[\w/]*'</span><span class="w"> </span>hotcues-with-names/Serato<span class="se">\ </span>Markers2.octet-stream<span class="w"> </span><span class="p">|</span><span class="w"> </span>tr<span class="w"> </span>-d<span class="w"> </span><span class="s1">'\0'</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>base64<span class="w"> </span>-d<span class="w"> </span><span class="p">|</span><span class="w"> </span>hexdump<span class="w"> </span>-C
<span class="m">00000000</span><span class="w"> </span><span class="m">01</span><span class="w"> </span><span class="m">01</span><span class="w"> </span><span class="m">43</span><span class="w"> </span>4f<span class="w"> </span>4c<span class="w"> </span>4f<span class="w"> </span><span class="m">52</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">04</span><span class="w"> </span><span class="m">00</span><span class="w"> </span>ff<span class="w"> </span>ff<span class="w"> </span>ff<span class="w"> </span><span class="p">|</span>..COLOR.........<span class="p">|</span>
<span class="m">00000010</span><span class="w"> </span><span class="m">43</span><span class="w"> </span><span class="m">55</span><span class="w"> </span><span class="m">45</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span>1a<span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span>cc<span class="w"> </span><span class="p">|</span>CUE.............<span class="p">|</span>
<span class="m">00000020</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">48</span><span class="w"> </span><span class="m">65</span><span class="w"> </span>6c<span class="w"> </span>6c<span class="w"> </span>6f<span class="w"> </span>2c<span class="w"> </span><span class="m">20</span><span class="w"> </span><span class="m">57</span><span class="w"> </span>6f<span class="w"> </span><span class="m">72</span><span class="w"> </span>6c<span class="w"> </span><span class="m">64</span><span class="w"> </span><span class="p">|</span>....Hello,<span class="w"> </span>World<span class="p">|</span>
<span class="m">00000030</span><span class="w"> </span><span class="m">21</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">43</span><span class="w"> </span><span class="m">55</span><span class="w"> </span><span class="m">45</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">15</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">01</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">03</span><span class="w"> </span><span class="m">55</span><span class="w"> </span><span class="m">58</span><span class="w"> </span><span class="p">|</span>!.CUE.........UX<span class="p">|</span>
<span class="m">00000040</span><span class="w"> </span><span class="m">00</span><span class="w"> </span>cc<span class="w"> </span><span class="m">88</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span>c3<span class="w"> </span>a4<span class="w"> </span>c3<span class="w"> </span>b6<span class="w"> </span>c3<span class="w"> </span>bc<span class="w"> </span>c3<span class="w"> </span>9f<span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">43</span><span class="w"> </span><span class="p">|</span>...............C<span class="p">|</span>
<span class="m">00000050</span><span class="w"> </span><span class="m">55</span><span class="w"> </span><span class="m">45</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">13</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">02</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span>ea<span class="w"> </span><span class="m">64</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="p">|</span>UE..........d...<span class="p">|</span>
<span class="m">00000060</span><span class="w"> </span>cc<span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span>c3<span class="w"> </span>a9<span class="w"> </span>c3<span class="w"> </span>a8<span class="w"> </span>c3<span class="w"> </span>aa<span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">42</span><span class="w"> </span><span class="m">50</span><span class="w"> </span>4d<span class="w"> </span>4c<span class="w"> </span>4f<span class="w"> </span><span class="m">43</span><span class="w"> </span><span class="p">|</span>..........BPMLOC<span class="p">|</span>
<span class="m">00000070</span><span class="w"> </span>4b<span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">01</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="m">00</span><span class="w"> </span><span class="p">|</span>K.......<span class="p">|</span>
<span class="m">00000078</span>
</code></pre></div>
<p>As we can see in <code>hexdump</code>'s ASCII output, the first hotcue is named <code>Hello, World!</code>.
To check how non-ASCII chars in hotcue names are encoded, hotcue 2 and 3 were named <code>äöüß</code> and <code>éèê</code>.
My guess that <code>UTF-8</code> has been used turned out to be correct.</p>
<h2>Saved Loops</h2>
<p>Using the same approach as above can be used to determine how saved loops are stored.
One problem popped up is that Serato may insert linefeed characters <code>0a</code> into the base64 string.</p>
<p>Also, the base64 string may lead to decoding errors caused by an invalid length.
In that case the last byte can apparently just be ignored.</p>
<h2>What's left to do?</h2>
<p>A lot. Some fields in the <code>Serato Markers2</code> are still unmapped, and I didn't really look at the <code>Serato Offsets_</code> and <code>Serato_Markers_</code> tags.
Maybe some fields could be edited manually to see how Serato reacts to these changes?</p>
<p>I was mainly doing this to add support for importing Serato Hotcues to <a href="https://www.mixxx.org/">Mixxx</a>, an excellent free and open-source DJ application that runs on Windows, Mac and Windows.
Also, I didn't investigate how <a href="https://store.serato.com/us/software/expansion-packs/serato-dj-flip-expansion-pack">Serato Flips</a> are stored <a href="https://bugs.launchpad.net/mixxx/+bug/1768113">since such a functionality is not implemented in Mixxx yet</a>.</p>
<p>Data dumps, example scripts and further information can be found <a href="https://github.com/Holzhaus/serato-tags">at my GitHub repository</a>.
If you want to help completing the documentation of Serato's GEOB data formats, feel free to send me a pull request.</p>Using BurpSuite with qutebrowser2018-06-25T00:00:00+02:002018-06-25T00:00:00+02:00Jan Holthuistag:homepage.ruhr-uni-bochum.de,2018-06-25:/jan.holthuis/using-burpsuite-with-qutebrowser.html<p>Some time ago I switched to
<a href="https://github.com/qutebrowser/qutebrowser">qutebrowser</a>, a keyboard-driven
browser based on QtWebEngine. Thus, I had to adapt my BurpSuite setup for
WebApp pentesting.</p>
<p>When pentesting web applications, a MITM proxy to log HTTP(S) requests is a
necessity. Although open-source <a href="https://mitmproxy.org/">alternatives</a>
<a href="http://www.pappyproxy.com/">exist</a>, PortSwigger's
<a href="https://portswigger.net/burp/">BurpSuite</a> is the de-facto standard in …</p><p>Some time ago I switched to
<a href="https://github.com/qutebrowser/qutebrowser">qutebrowser</a>, a keyboard-driven
browser based on QtWebEngine. Thus, I had to adapt my BurpSuite setup for
WebApp pentesting.</p>
<p>When pentesting web applications, a MITM proxy to log HTTP(S) requests is a
necessity. Although open-source <a href="https://mitmproxy.org/">alternatives</a>
<a href="http://www.pappyproxy.com/">exist</a>, PortSwigger's
<a href="https://portswigger.net/burp/">BurpSuite</a> is the de-facto standard in this
niche.</p>
<h1>Certificate Installation</h1>
<p>To be able to MITM TLS-encrypted connections without certificate errors, you
first need to install Burp's locally generated CA certificate.</p>
<p>Like Chromium and Firefox, qutebrowser checks the user-local NSS
Database at <code>~/.pki/nssdb/</code> for certificates. Using
<a href="https://developer.mozilla.org/en-US/docs/Mozilla/Projects/NSS/Reference/NSS_tools_:_certutil"><code>certutil</code></a>,
you can install the certificate like this:</p>
<div class="highlight"><pre><span></span><code><span class="gp">$ </span>certutil<span class="w"> </span>-d<span class="w"> </span><span class="s2">"sql:</span><span class="nv">$HOME</span><span class="s2">/.pki/nssdb"</span><span class="w"> </span>-A<span class="w"> </span>-i<span class="w"> </span>~/Downloads/cacert.der<span class="w"> </span>-n<span class="w"> </span><span class="s2">"Burp Suite CA"</span><span class="w"> </span>-t<span class="w"> </span>C,,
</code></pre></div>
<h1>Proxy Setup in Qutebrowser</h1>
<p>Next thing you'll need is a proxy setup for qutebrowser. A proxy can
easily be set using:</p>
<div class="highlight"><pre><span></span><code>:set content.proxy http://127.0.0.1:8080/
</code></pre></div>
<p>In order to enable and disable "burp-mode" faster, you can use aliases:</p>
<div class="highlight"><pre><span></span><code>:set aliases '{ "burp": "set content.proxy http://127.0.0.1:8080/", "noburp": "set content.proxy system" }'
</code></pre></div>
<p>Now you can simply type</p>
<div class="highlight"><pre><span></span><code>:burp
</code></pre></div>
<p>to start sending the requests via the proxy. </p>
<p>When you type</p>
<div class="highlight"><pre><span></span><code>:noburp
</code></pre></div>
<p>the browser will use the system proxy again.</p>EFA Departure Monitor on the command line2018-03-05T00:00:00+01:002018-03-05T00:00:00+01:00Jan Holthuistag:homepage.ruhr-uni-bochum.de,2018-03-05:/jan.holthuis/efa-departure-monitor-on-the-command-line.html<p>I just hacked together a small shell script that gets departures from my local
public transportation service. It will list upcoming departures at a stop.
You can find the script <a href="https://gist.github.com/d622714b7b9349c46df42c7e4f934f2a">here</a>.</p>
<h2>Getting Started</h2>
<p>You can either use the stop name:</p>
<div class="highlight"><pre><span></span><code><span class="gp">$ </span>./efa-dm.sh<span class="w"> </span><span class="s2">"Dortmund Hbf"</span>
</code></pre></div>
<p>Or you can use the stop …</p><p>I just hacked together a small shell script that gets departures from my local
public transportation service. It will list upcoming departures at a stop.
You can find the script <a href="https://gist.github.com/d622714b7b9349c46df42c7e4f934f2a">here</a>.</p>
<h2>Getting Started</h2>
<p>You can either use the stop name:</p>
<div class="highlight"><pre><span></span><code><span class="gp">$ </span>./efa-dm.sh<span class="w"> </span><span class="s2">"Dortmund Hbf"</span>
</code></pre></div>
<p>Or you can use the stop ID:</p>
<div class="highlight"><pre><span></span><code><span class="gp">$ </span>./efa-dm.sh<span class="w"> </span><span class="m">20000131</span>
</code></pre></div>
<p>By default, it will simply print upcoming departures as a tab-separated list.</p>
<div class="highlight"><pre><span></span><code><span class="gp">$ </span>./efa-dm.sh<span class="w"> </span>-n<span class="w"> </span><span class="m">3</span><span class="w"> </span><span class="s2">"Essen Rüttenscheider Stern"</span>
<span class="go">1 2018 3 5 16 20 2018 3 5 16 21 101 Essen Helenenstr.</span>
<span class="go">2 2018 3 5 16 21 2018 3 5 16 22 108 Essen Altenessen Bf Schleife</span>
<span class="go">2 2018 3 5 16 22 U11 Essen Messe W.-Süd/Gruga</span>
</code></pre></div>
<table>
<thead>
<tr>
<th>Column</th>
<th>Meaning</th>
</tr>
</thead>
<tbody>
<tr>
<td>1</td>
<td>minutes left until departure</td>
</tr>
<tr>
<td>2</td>
<td>scheduled departure (year)</td>
</tr>
<tr>
<td>3</td>
<td>scheduled departure (month)</td>
</tr>
<tr>
<td>4</td>
<td>scheduled departure (day)</td>
</tr>
<tr>
<td>5</td>
<td>scheduled departure (hour)</td>
</tr>
<tr>
<td>6</td>
<td>scheduled departure (minute)</td>
</tr>
<tr>
<td>6</td>
<td>predicted actual departure (year)</td>
</tr>
<tr>
<td>8</td>
<td>predicted actual departure (month)</td>
</tr>
<tr>
<td>9</td>
<td>predicted actual departure (day)</td>
</tr>
<tr>
<td>10</td>
<td>predicted actual departure (hour)</td>
</tr>
<tr>
<td>11</td>
<td>predicted actual departure (minute)</td>
</tr>
<tr>
<td>12</td>
<td>line name</td>
</tr>
<tr>
<td>13</td>
<td>direction</td>
</tr>
</tbody>
</table>
<p>You can also use the <code>-p</code> flag to get the pretty-printed version:</p>
<div class="highlight"><pre><span></span><code><span class="gp">$ </span>./efa-dm.sh<span class="w"> </span>-p<span class="w"> </span>-n<span class="w"> </span><span class="m">3</span><span class="w"> </span><span class="s2">"Essen Rüttenscheider Stern"</span>
<span class="go">16:20(+1) 101 Essen Helenenstr. in 1 min</span>
<span class="go">16:21 108 Essen Altenessen Bf Schleife in 1 min</span>
<span class="go">16:22 U11 Essen Messe W.-Süd/Gruga in 2 min</span>
</code></pre></div>
<h2>Usage</h2>
<p>Here's a list of all command line options:</p>
<div class="highlight"><pre><span></span><code><span class="gp">$ </span>./efa-dm.sh<span class="w"> </span>-h
<span class="go">Usage: ./efa-dm.sh [-p] [-d] [-a <API_ENDPOINT>] [-n <NUM_DEPARTS>] [-t <TIME_OFFSET>] <STOP_NAME></span>
<span class="go">Options:</span>
<span class="go"> -h Show this help</span>
<span class="go"> -p Petty-printed output (instead of tab-separated values)</span>
<span class="go"> -d Debug mode (output server reply and exit)</span>
<span class="go"> -a <API_ENDPOINT> Use API endpoint at this URL</span>
<span class="go"> -n <NUM_DEPARTS> Limit the number of departures (default: 8)</span>
<span class="go"> -t <TIME_OFFSET> Skip departures in next X minutes (default: 0)</span>
</code></pre></div>
<h2>Services that use EFA</h2>
<p>Here's a list of other public transportation services that also use the
<a href="https://de.wikipedia.org/wiki/Elektronische_Fahrplanauskunft_(Software)"><em>Elektronische Fahrplanauskunft (EFA)</em> system</a>
and thus can also by queried by the script as well:</p>
<ul>
<li>
<p>Verkehrsverbund Rhein-Ruhr (VRR), Germany
<code>http://efa.vrr.de/standard/XSLT_DM_REQUEST</code></p>
</li>
<li>
<p>Verkehrs- und Tarifverbund Stuttgart (VVS), Germany
<code>http://www2.vvs.de/vvs/XSLT_DM_REQUEST</code></p>
</li>
<li>
<p>Münchner Verkehrs- und Tarifverbund (MVV), Germany
<code>http://efa.mvv-muenchen.de/mobile/XSLT_DM_REQUEST</code></p>
</li>
<li>
<p>Nahverkehrsgesellschaft Baden-Württemberg (NVBW), Germany
<code>http://www.efa-bw.de/nvbw/XSLT_DM_REQUEST</code></p>
</li>
<li>
<p>Regional Transportation Authority (RTA) Chicago, USA
<code>http://tripplanner.rtachicago.com/ccg3/XSLT_DM_REQUEST</code></p>
</li>
</ul>
<h2>Departure Monitor in Polybar</h2>
<p>You can easily use this script as a polybar module:</p>
<div class="highlight"><pre><span></span><code><span class="k">[module/efa1]</span>
<span class="na">type</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">custom/script</span>
<span class="na">exec</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">/path/to/efa-dm.sh -p -t 4 -n 1 "Essen Hbf"</span>
<span class="na">format</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s"><label></span>
<span class="c1">; In case this is a bus stop:</span>
<span class="c1">; format = <label></span>
<span class="c1">; For a subway station:</span>
<span class="c1">; format = <label></span>
<span class="na">interval</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">60</span>
</code></pre></div>Fixing WiFi Multicast Flooding in bridged networks2017-09-21T00:00:00+02:002017-09-21T00:00:00+02:00Jan Holthuistag:homepage.ruhr-uni-bochum.de,2017-09-21:/jan.holthuis/fixing-wifi-multicast-flooding-in-bridged-networks.html<p>I'm using <a href="https://www.musicpd.org/">MPD</a> and
<a href="https://www.freedesktop.org/wiki/Software/PulseAudio/Documentation/User/Network/#index3h2">PulseAudio's RTP multicasting</a>
to get a seamless <a href="https://fruit.je/mpd-rtp">multi-room audio</a> experience.</p>
<p>Unfortunately, if you're using a network bridge to connect your wired and
wireless LAN, using multicast RTP might have unintended consequences: All
WiFi clients are flooded with multicast traffic, which can bring down the
entire wireless …</p><p>I'm using <a href="https://www.musicpd.org/">MPD</a> and
<a href="https://www.freedesktop.org/wiki/Software/PulseAudio/Documentation/User/Network/#index3h2">PulseAudio's RTP multicasting</a>
to get a seamless <a href="https://fruit.je/mpd-rtp">multi-room audio</a> experience.</p>
<p>Unfortunately, if you're using a network bridge to connect your wired and
wireless LAN, using multicast RTP might have unintended consequences: All
WiFi clients are flooded with multicast traffic, which can bring down the
entire wireless network.</p>
<blockquote>
<p>When multicast transmission arrives at the receiver's LAN, it is flooded to
every Ethernet switch port unless flooding reduction such as IGMP snooping is
employed (Section 2.7). <a href="https://tools.ietf.org/html/rfc5110#section-2"><em>(RFC 5110, Section 2 "Multicast Routing", page 4)</em></a></p>
</blockquote>
<p>If you don't wanto to set up IGMP snooping, you have two alternatives:
You can either</p>
<ol>
<li>un-bridge Ethernet and WiFi interfaces and switch to a routed approach, or</li>
<li>filter out multicast packets on their way from wired interface to wireless.</li>
</ol>
<p>Since (1) has other implications that I'd rather avoid (e.g. blocking broadcast
traffic, too, so that service autodiscovery won't work anymore), so I chose
the second approach.</p>
<p>This can easily be archieved using <a href="http://ebtables.netfilter.org/">ebtables</a>,
which allow link layer filtering on Linux bridge interfaces.</p>
<p>My router is running <a href="https://openwrt.org/">OpenWRT</a>, which does not with
ebtables by default, so it needs to be installed first:</p>
<div class="highlight"><pre><span></span><code><span class="gp"># </span>opkg<span class="w"> </span>update
<span class="gp"># </span>opkg<span class="w"> </span>install<span class="w"> </span>ebtables
</code></pre></div>
<p>This is how my bridge setup looks like:</p>
<div class="highlight"><pre><span></span><code><span class="gp"># </span>brctl<span class="w"> </span>show
<span class="go">bridge name bridge id STP enabled interfaces</span>
<span class="go">br-lan 7fff.12345678abcd no eth0.1</span>
<span class="go"> wlan0</span>
<span class="go"> wlan1</span>
<span class="go">br-wan 7fff.12345678abcd no eth0.2</span>
</code></pre></div>
<p><code>eth0.1</code>, <code>wlan0</code> and <code>wlan1</code> are bridged. It's a dual band router that has wifi
interfaces for both the 2.4 GHz (<code>wlan0</code>) and the 5 GHz band (<code>wlan1</code>).</p>
<p>Now the filter rules need to be added. One rule for each wifi interface is
necessary:</p>
<div class="highlight"><pre><span></span><code><span class="gp"># </span>ebtables<span class="w"> </span>-A<span class="w"> </span>FORWARD<span class="w"> </span>-o<span class="w"> </span>wlan0<span class="w"> </span>-d<span class="w"> </span>Multicast<span class="w"> </span>-j<span class="w"> </span>DROP
<span class="gp"># </span>ebtables<span class="w"> </span>-A<span class="w"> </span>FORWARD<span class="w"> </span>-o<span class="w"> </span>wlan1<span class="w"> </span>-d<span class="w"> </span>Multicast<span class="w"> </span>-j<span class="w"> </span>DROP
</code></pre></div>
<p>These rules tell ebtables to drop all Multicast packets if their output
device in either <code>wlan0</code> or <code>wlan1</code>.</p>
<p>The effect is immediately noticeable. Before setting up multicast filtering the
wifi interfaces were quite busy:</p>
<p><img alt="WiFi traffic with multicast filtering" src="https://homepage.ruhr-uni-bochum.de/jan.holthuis/images/wifi-multicast-bmon1.png"></p>
<p>Afterwards, there's a lot less going on:</p>
<p><img alt="WiFi traffic without multicast filtering" src="https://homepage.ruhr-uni-bochum.de/jan.holthuis/images/wifi-multicast-bmon2.png"></p>
<p>To make the filtering permanent, simply add the ebtables commands to
<code>/etc/firewall.user</code>.</p>Upgrading iLO 4 on a HPE ProLiant MicroServer from Linux2017-09-20T00:00:00+02:002017-09-20T00:00:00+02:00Jan Holthuistag:homepage.ruhr-uni-bochum.de,2017-09-20:/jan.holthuis/upgrading-ilo-4-on-a-hpe-proliant-microserver-from-linux.html<p>I recently got my hands on a ProLiant MicroServer Gen8 by Hewlett Packard
Enterprise (HPE). As I always do when setting up a server I checked if the
device needs a firmware upgrade.</p>
<p>And indeed it did: It's version of
<a href="https://h20195.www2.hpe.com/v2/GetDocument.aspx?docname=4AA4-5167ENW">Integrated Lights-Out (iLO) 4</a>,
its built-in server provisioning and management …</p><p>I recently got my hands on a ProLiant MicroServer Gen8 by Hewlett Packard
Enterprise (HPE). As I always do when setting up a server I checked if the
device needs a firmware upgrade.</p>
<p>And indeed it did: It's version of
<a href="https://h20195.www2.hpe.com/v2/GetDocument.aspx?docname=4AA4-5167ENW">Integrated Lights-Out (iLO) 4</a>,
its built-in server provisioning and management software, is affected by
<a href="http://h20566.www2.hpe.com/hpsc/doc/public/display?sp4ts.oid=5219984&docId=emr_na-c03693433&docLocale=en_US">CVE-2017-12542</a>,
which is a solid 10.0 on the CVSS 2.0 score chart.</p>
<p>So I decided to update it. Fortunately, the iLO web interface has a page where
firmware upgrades can be uploaded. Since it's in an isolated network, using
the web interface should not pose a security problem.</p>
<p>On the other hand, locating the proper firmware file to upload was not as
easy as it should be. It's Hewlett-Packard, after all.</p>
<p>In case someone else is looking for the iLO 4 *.bin file, here's what I did:</p>
<ol>
<li>Visit the <a href="http://www.hpe.com/support/ilo4">iLO 4 support page</a>, but do
not select <em>OS-Independent</em> (it's not in there). Select
"<em>Red Hat Enterprise Linux 7"</em> instead
(<a href="http://h20565.www2.hpe.com/hpsc/swd/public/readIndex?sp4ts.oid=5228286&swLangOid=8&swEnvOid=4064">direct link</a>)</li>
<li>Open the "<em>Firmware - LOM (Lights-Out Management)</em>" section and download
<code>hp-firmware-ilo4-2.55-1.1.i386.rpm</code>.</li>
<li>To extract the actual firmware file from the RPM, use this command:</li>
</ol>
<div class="highlight"><pre><span></span><code><span class="gp">$ </span>rpm2cpio<span class="w"> </span>hp-firmware-ilo4-2.55-1.1.i386.rpm<span class="w"> </span><span class="p">|</span><span class="w"> </span>bsdtar<span class="w"> </span>-x<span class="w"> </span>-s<span class="s1">'|.*/||'</span><span class="w"> </span>-f<span class="w"> </span>-<span class="w"> </span>./usr/lib/i386-linux-gnu/hp-firmware-ilo4-2.55-1.1/ilo4_255.bin
</code></pre></div>
<p>The resulting file (<code>ìlo4_255.bin</code>) can then be uploaded to the web interface:</p>
<p><img alt="iLO 4 Upgrade Process" src="https://homepage.ruhr-uni-bochum.de/jan.holthuis/images/ilo4-upgrade1.png"></p>
<p>After the upgrade process finishes, you'll be redirected to the brand new
login screen:</p>
<p><img alt="iLO 4 after upgrade" src="https://homepage.ruhr-uni-bochum.de/jan.holthuis/images/ilo4-upgrade2.png"></p>Generating syntax diagrams using the LaTeX rail package2017-05-21T00:00:00+02:002017-05-21T00:00:00+02:00Jan Holthuistag:homepage.ruhr-uni-bochum.de,2017-05-21:/jan.holthuis/generating-syntax-diagrams-using-the-latex-rail-package.html<p>If you ever had the need to add syntax specifications to your document,
you basically have two options: Either write down the syntax in the
<a href="https://en.wikipedia.org/wiki/Backus-Naur_form">Backus-Naur form (BNF)</a> (or
one of its derivatives) or opt for a more graphical approach by adding
<a href="https://en.wikipedia.org/wiki/Railroad_Diagram">"railroad diagrams"</a>. In my
opinon, the latter are …</p><p>If you ever had the need to add syntax specifications to your document,
you basically have two options: Either write down the syntax in the
<a href="https://en.wikipedia.org/wiki/Backus-Naur_form">Backus-Naur form (BNF)</a> (or
one of its derivatives) or opt for a more graphical approach by adding
<a href="https://en.wikipedia.org/wiki/Railroad_Diagram">"railroad diagrams"</a>. In my
opinon, the latter are easier to grasp for less
experienced readers and also look quite nice. </p>
<p>In LaTeX, you can use the <a href="https://www.ctan.org/pkg/rail"><code>rail</code> package</a> to
generate those diagrams from
<a href="https://en.wikipedia.org/wiki/Extended_Backus-Naur_form">EBNF</a>
rules:</p>
<div class="highlight"><pre><span></span><code><span class="k">\begin</span><span class="nb">{</span>rail<span class="nb">}</span>
decl : 'def' identifier '=' ( expression + ';' )
| 'type' identifier '=' type
;
<span class="k">\end</span><span class="nb">{</span>rail<span class="nb">}</span>
</code></pre></div>
<p>This will result in something like this:</p>
<p><img alt="Railroad diagram" src="https://homepage.ruhr-uni-bochum.de/jan.holthuis/images/rail-demo.png"></p>
<p>To archieve this, the package first generates a <code>*.rai</code> file. We then have
to convert the rai file to a <code>*.rao</code> by invoking the accompanying C
program named <code>rail</code>.</p>
<p>However, the <code>rail</code> package is fairly old. It has been written by
Luc Rooijakkers in 1991 (!) and was updated by Klaus Barthelmann until 1998.
Thus, the code is -- at least -- 19 years old and that really shows: Trying
to compile it on modern systems yields a bunch of compilation errors.</p>
<p>Most of the issues stem from missing return types in function declarations
and also missing forward declarations.
I stepped up and fixed these issues, so that it works with a up-to-date
compiler (I tested with <a href="https://gcc.gnu.org/gcc-6/">gcc (GCC) 6.3.1</a> on
<a href="https://www.archlinux.org/">Arch Linux</a>. You can find the result on
<a href="https://github.com/Holzhaus/latex-rail">Github</a>.</p>
<p>I also threw in some <code>Makefile</code> improvements into the mix: You can now use
<a href="https://www.gnu.org/prep/standards/html_node/DESTDIR.html"><code>DESTDIR</code></a> and
<code>PREFIX</code> (defaults to <code>/usr/local</code>) when running <code>make install</code>.</p>
<h2>Installation</h2>
<p>Installation should be fairly straighforward. Here's an example which will
install <code>rail</code> into <code>/usr</code>:</p>
<div class="highlight"><pre><span></span><code><span class="gp">$ </span>curl<span class="w"> </span>-L<span class="w"> </span>https://github.com/Holzhaus/latex-rail/archive/v1.2.1.tar.gz<span class="w"> </span><span class="p">|</span><span class="w"> </span>tar<span class="w"> </span>xzvf<span class="w"> </span>-
<span class="gp">$ </span><span class="nb">cd</span><span class="w"> </span>latex-rail-1.2.1
<span class="gp">$ </span>make
<span class="go">bison -y -dv gram.y</span>
<span class="go">gram.y: warning: 2 reduce/reduce conflicts [-Wconflicts-rr]</span>
<span class="go">cmp -s gram.c y.tab.c || cp y.tab.c gram.c</span>
<span class="go">cmp -s gram.h y.tab.h || cp y.tab.h gram.h</span>
<span class="go">gcc -DYYDEBUG -O -c -o rail.o rail.c</span>
<span class="go">gcc -DYYDEBUG -O -c -o gram.o gram.c</span>
<span class="go">flex -t lex.l > lex.c</span>
<span class="go">gcc -DYYDEBUG -O -c -o lex.o lex.c</span>
<span class="go">gcc -DYYDEBUG -O rail.o gram.o lex.o -o rail</span>
<span class="gp">$ </span>sudo<span class="w"> </span>make<span class="w"> </span><span class="nv">PREFIX</span><span class="o">=</span>/usr<span class="w"> </span>install
<span class="gp">$ </span>sudo<span class="w"> </span>mktexlsr
</code></pre></div>
<p>Please note that installing stuff using <code>sudo make install</code> will circumvent
your package manager and is usually not a good idea. If you're using
Arch Linux you should use the
<a href="https://aur.archlinux.org/packages/latex-rail/">AUR package</a> instead:</p>
<div class="highlight"><pre><span></span><code><span class="gp">$ </span>pacaur<span class="w"> </span>-S<span class="w"> </span>latex-rail
</code></pre></div>
<h2>Manual compilation and Latexmk support</h2>
<p>To generate a document manually, you need to run multiple commands:</p>
<ol>
<li>Run <code>latex mydoc</code>, which will create <code>mydoc.rai</code></li>
<li>Run <code>rail mydoc</code> to generate <code>mydoc.rao</code> from <code>mydoc.rai</code></li>
<li>Run <code>latex mydoc</code> for the final document</li>
</ol>
<p>If you don't want to bother with running LaTeX multiple times, you can
use <a href="https://www.ctan.org/pkg/latexmk">latexmk</a>, a perl script to automate
the document generation.</p>
<p>To make it work with the <code>rail</code> package, you should create a <code>.latexmkrc</code>
in your document folder with this content:</p>
<div class="highlight"><pre><span></span><code><span class="nb">push</span><span class="w"> </span><span class="nv">@file_not_found</span><span class="p">,</span><span class="w"> </span><span class="s">'^Package .* Info: No file (.+) on input line \d+\.'</span><span class="p">;</span>
<span class="n">add_cus_dep</span><span class="p">(</span><span class="s">'rai'</span><span class="p">,</span><span class="w"> </span><span class="s">'rao'</span><span class="p">,</span><span class="w"> </span><span class="mi">0</span><span class="p">,</span><span class="w"> </span><span class="s">'rail'</span><span class="p">);</span>
<span class="k">sub</span><span class="w"> </span><span class="nf">rail</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="k">my</span><span class="w"> </span><span class="p">(</span><span class="nv">$base_name</span><span class="p">,</span><span class="w"> </span><span class="nv">$path</span><span class="p">,</span><span class="w"> </span><span class="nv">$ext</span><span class="p">)</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">fileparse</span><span class="p">(</span><span class="w"> </span><span class="nv">$_</span><span class="p">[</span><span class="mi">0</span><span class="p">],</span><span class="w"> </span><span class="sx">qr/\.[^.\/</span><span class="p">]</span><span class="o">*/</span><span class="w"> </span><span class="p">);</span>
<span class="w"> </span><span class="n">pushd</span><span class="w"> </span><span class="nv">$path</span><span class="p">;</span>
<span class="w"> </span><span class="k">my</span><span class="w"> </span><span class="nv">$return</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nb">system</span><span class="w"> </span><span class="s">"rail $base_name"</span><span class="p">;</span>
<span class="w"> </span><span class="n">popd</span><span class="p">;</span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="nv">$return</span><span class="p">;</span>
<span class="p">}</span>
</code></pre></div>
<p>The first line will add the appropriate RegEx to Latexmk's missing file
detection, the second line will instruct latexmk to run the <code>rail</code> subroutine
with a <code>*.rai</code> file as input and <code>*.rao</code> file as output.</p>
<h2>Alternatives</h2>
<p>I you don't quite like the <code>rail</code> package, you might want to look into
one of these alternative packages:</p>
<ul>
<li><a href="https://www.ctan.org/pkg/syngen"><code>syngen</code></a> by Jens Klöcker</li>
<li><a href="https://www.ctan.org/pkg/syntax-mdw"><code>syntax-mdw</code></a> by Mark Wooding</li>
<li><a href="https://www.ctan.org/pkg/syntax2"><code>syntax</code></a> by Bernd Worsch</li>
</ul>
<p>These also an
<a href="http://www.bottlecaps.de/rr/ui">online tool to generate railroad diagrams</a> if
you don't want to do it in LaTeX.</p>How to create an UEFI-bootable Windows 7 stick from Linux.2016-08-10T00:00:00+02:002016-08-10T00:00:00+02:00Jan Holthuistag:homepage.ruhr-uni-bochum.de,2016-08-10:/jan.holthuis/how-to-create-an-uefi-bootable-windows-7-stick-from-linux.html<p>In case you rely on Windows-only software or want to play a game that isn't on Linux yet, you might want to keep Windows 7 on a second partition. Although I didn't use it for months, I still keep one around just in case.</p>
<p>Unfortunately, it's not really straightforward to …</p><p>In case you rely on Windows-only software or want to play a game that isn't on Linux yet, you might want to keep Windows 7 on a second partition. Although I didn't use it for months, I still keep one around just in case.</p>
<p>Unfortunately, it's not really straightforward to create a UEFI bootable USB installation disk without using Windows. For some strange reason we can't just <code>dd</code> the ISO image to a USB disk. Instead, we need to use the <a href="https://www.microsoft.com/en-us/download/windows-usb-dvd-download-tool">Windows USB/DVD Download Tool</a> which - incidentally - only runs on Windows.</p>
<p>However, there's also a way to do this from Linux:</p>
<p>First, you need to create a GPT partition table with a FAT32 partition on your USB pen drive. Then you simple mount the ISO file and copy the files over. After you've done that, your need to extract the file <code>1/Windows/Boot/EFI/bootmgfw.efi</code> from the <code>install.wim</code> file inside <code>/sources</code> folder on the Windows 7 installation ISO and move the extracted file to <code>/EFI/Boot/bootx64.efi</code> on the pen drive.</p>
<p>You don't have to do this manually: There's a neat little tool called <a href="https://github.com/slacka/WinUSB">WinUSB</a> that can do this for you (it even has a GUI if you want it) and my pull request) that <a href="https://github.com/slacka/WinUSB/pull/59">adds Windows 7 UEFI support</a> support has just been merged.</p>Smartcard authentification in Chromium2015-07-06T00:00:00+02:002015-07-06T00:00:00+02:00Jan Holthuistag:homepage.ruhr-uni-bochum.de,2015-07-06:/jan.holthuis/smartcard-authentification-in-chromium.html<p>My university's website for exam enrollment needs smardcard authentification, but only contains <a href="https://www.flexnow.rub.de/studierende/">instructions</a> how to use it with Mozilla Firefox. If you prefer Chrome/Chromium over Firefox and don't want to keep a Firefox installation around, you can do so - here are the instructions.</p>
<p><em>OT:</em> The Ruhr-University of Bochum (RUB …</p><p>My university's website for exam enrollment needs smardcard authentification, but only contains <a href="https://www.flexnow.rub.de/studierende/">instructions</a> how to use it with Mozilla Firefox. If you prefer Chrome/Chromium over Firefox and don't want to keep a Firefox installation around, you can do so - here are the instructions.</p>
<p><em>OT:</em> The Ruhr-University of Bochum (RUB) uses two different systems for exam enrollment, VSPL and FlexNow. The reason for using two different, incompatible systems surpasses my understanding. I was very happy that my faculty uses FlexNow, which is browser-based, rather than VSPL, which only works via a proprietary, <a href="https://web-rubicon.ruhr-uni-bochum.de/rubicon/download/">Windows-only client</a> (even though the download page for that application used to show a hip student with a MacBook... oh, the irony). Anyway, <a href="https://ecampus.ruhr-uni-bochum.de/portal/">VSPL recently got a web-interface</a>, too, so if you have to use VSPL, you can also use Firefox or Chromium now.</p>
<p>Since I'm using ArchLinux, I'll base my instruction on that, but any other distro also should work fine if you accomodate package names and file paths.</p>
<p>You'll need a CCID-conformant smardcard reader (<a href="http://www.amazon.de/HP-USB-SmartCard-CCID-Keyboard/dp/B00GBSN4QM/">this is the one I am using</a>, but the <a href="http://www.scm-pc-card.de/index.php?lang=de&page=product&function=show_product&product_id=226">"official" reader that the university sells</a> works too).</p>
<p>First, we need the software - apart from chrome that is the CCID-driver, the OpenSC-library and Mozilla's Network Security Services:</p>
<div class="highlight"><pre><span></span><code># pacman -Sy ccid opensc nss
</code></pre></div>
<p>Next, we need to add the PKCS#11 module from OpenSC to the Chromium NSS module. Quit Chromium if necessary and run:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>modutil<span class="w"> </span>-dbdir<span class="w"> </span>sql:.pki/nssdb/<span class="w"> </span>-add<span class="w"> </span><span class="s2">"opensc"</span><span class="w"> </span>-libfile<span class="w"> </span>/usr/lib/pkcs11/opensc-pkcs11.so
</code></pre></div>
<p>You can verify that everything worked by running:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>modutil<span class="w"> </span>-dbdir<span class="w"> </span>sql:.pki/nssdb/<span class="w"> </span>-list
</code></pre></div>
<p>It should print a listing of PKCS#11 modules. If your smardcard driver is attached, one of the slots of the <code>opensc</code> module shows the name of your smardcard reader. If you also plugged in your smardcard, the according token tells you what smardcard has been detected.</p>
<div class="highlight"><pre><span></span><code><span class="n">Listing</span><span class="w"> </span><span class="n">of</span><span class="w"> </span><span class="n">PKCS</span><span class="w"> </span><span class="c1">#11 Modules</span>
<span class="o">-----------------------------------------------------------</span>
<span class="w"> </span><span class="mf">1.</span><span class="w"> </span><span class="n">NSS</span><span class="w"> </span><span class="n">Internal</span><span class="w"> </span><span class="n">PKCS</span><span class="w"> </span><span class="c1">#11 Module</span>
<span class="w"> </span><span class="n">slots</span><span class="p">:</span><span class="w"> </span><span class="mi">2</span><span class="w"> </span><span class="n">slots</span><span class="w"> </span><span class="n">attached</span>
<span class="w"> </span><span class="n">status</span><span class="p">:</span><span class="w"> </span><span class="n">loaded</span>
<span class="w"> </span><span class="n">slot</span><span class="p">:</span><span class="w"> </span><span class="n">NSS</span><span class="w"> </span><span class="n">Internal</span><span class="w"> </span><span class="n">Cryptographic</span><span class="w"> </span><span class="n">Services</span>
<span class="w"> </span><span class="n">token</span><span class="p">:</span><span class="w"> </span><span class="n">NSS</span><span class="w"> </span><span class="n">Generic</span><span class="w"> </span><span class="n">Crypto</span><span class="w"> </span><span class="n">Services</span>
<span class="w"> </span><span class="n">slot</span><span class="p">:</span><span class="w"> </span><span class="n">NSS</span><span class="w"> </span><span class="n">User</span><span class="w"> </span><span class="n">Private</span><span class="w"> </span><span class="n">Key</span><span class="w"> </span><span class="ow">and</span><span class="w"> </span><span class="n">Certificate</span><span class="w"> </span><span class="n">Services</span>
<span class="w"> </span><span class="n">token</span><span class="p">:</span><span class="w"> </span><span class="n">NSS</span><span class="w"> </span><span class="n">Certificate</span><span class="w"> </span><span class="n">DB</span>
<span class="w"> </span><span class="mf">2.</span><span class="w"> </span><span class="n">opensc</span>
<span class="w"> </span><span class="n">library</span><span class="w"> </span><span class="n">name</span><span class="p">:</span><span class="w"> </span><span class="o">/</span><span class="n">usr</span><span class="o">/</span><span class="n">lib</span><span class="o">/</span><span class="n">opensc</span><span class="o">-</span><span class="n">pkcs11</span><span class="o">.</span><span class="n">so</span>
<span class="w"> </span><span class="n">slots</span><span class="p">:</span><span class="w"> </span><span class="mi">2</span><span class="w"> </span><span class="n">slots</span><span class="w"> </span><span class="n">attached</span>
<span class="w"> </span><span class="n">status</span><span class="p">:</span><span class="w"> </span><span class="n">loaded</span>
<span class="w"> </span><span class="n">slot</span><span class="p">:</span><span class="w"> </span><span class="n">Virtual</span><span class="w"> </span><span class="n">hotplug</span><span class="w"> </span><span class="n">slot</span>
<span class="w"> </span><span class="n">token</span><span class="p">:</span><span class="w"> </span>
<span class="w"> </span><span class="n">slot</span><span class="p">:</span><span class="w"> </span><span class="n">Hewlett</span><span class="o">-</span><span class="n">Packard</span><span class="w"> </span><span class="n">Company</span><span class="w"> </span><span class="n">HP</span><span class="w"> </span><span class="n">USB</span><span class="w"> </span><span class="n">CCID</span><span class="w"> </span><span class="n">Smartcard</span><span class="w"> </span><span class="n">Keyboard</span><span class="w"> </span><span class="p">[</span><span class="n">HP</span><span class="w"> </span><span class="n">USB</span><span class="w"> </span><span class="n">C</span>
<span class="w"> </span><span class="n">token</span><span class="p">:</span><span class="w"> </span><span class="n">Student</span><span class="w"> </span><span class="n">Card</span><span class="w"> </span><span class="p">(</span><span class="n">User</span><span class="w"> </span><span class="n">Pin</span><span class="p">)</span>
<span class="o">-----------------------------------------------------------</span>
</code></pre></div>
<p>That's it.</p>
<p>If you now start Chromium and click "Manage certificates" button in the HTTPS/SSL section of Chromiums settings (Settings -> Show advanced settings...), you should see your smartcard certificate in the "Your certificates" tab.</p>
<p>You can now log into FlexNow (or VSPL), de-register all you exams and chill.</p>ODROID-C1 issues finally fixed2015-06-11T00:00:00+02:002015-06-11T00:00:00+02:00Jan Holthuistag:homepage.ruhr-uni-bochum.de,2015-06-11:/jan.holthuis/odroid-c1-issues-finally-fixed.html<p>When I decided to <a href="https://homepage.ruhr-uni-bochum.de/jan.holthuis/odroid-c1-post-installation-tips.html">replace my beloved Raspberry Pi media center with something more powerful</a>, I didn't expect so many issues. I purchased <a href="http://www.hardkernel.com/main/products/prdt_info.php?g_code=G141578608433">Hardkernel's ODROID-C1</a> and was rather excited. The <a href="http://kodi.tv">Kodi Entertainment Center</a> worked much smoother than before (especially since I compiled it to use the Framebuffer instead of X11 …</p><p>When I decided to <a href="https://homepage.ruhr-uni-bochum.de/jan.holthuis/odroid-c1-post-installation-tips.html">replace my beloved Raspberry Pi media center with something more powerful</a>, I didn't expect so many issues. I purchased <a href="http://www.hardkernel.com/main/products/prdt_info.php?g_code=G141578608433">Hardkernel's ODROID-C1</a> and was rather excited. The <a href="http://kodi.tv">Kodi Entertainment Center</a> worked much smoother than before (especially since I compiled it to use the Framebuffer instead of X11). I'm even able to play a lot of retro games (not all though) at full speed via <a href="https://github.com/Holzhaus/odroid-c1-PKGBUILDs">emulators</a> for Sony Playstation, Sega Genesis, Dreamcast and Nintendo's SNES, NES and N64 consoles.</p>
<p>But after some time, I figured there were still some problems that I were unable to fix on my own.</p>
<p>This super-annoying issue for example: Occasionally, the audio would drop for 1 or 2 seconds - sometimes only once or twice during a 2 hour movie, sometimes every 2 minutes. I was not the only one with this issue <a href="http://forum.odroid.com/viewtopic.php?f=117&t=9134">when I reported it</a> at the start of February. But it took quite some time, until <a href="https://github.com/hardkernel/linux/commit/6b290b18721983f5a221aa13bedd58843a50eabb">this was fixed</a>. The main problem here: The hardware vendor, <a href="http://www.amlogic.com/">Amlogic</a>, writes really crappy code. I had a look at their kernel sources and their code quality is so poor that I wonder why this stuff even works most of the time. At least, Amlogic seems to embrace Open Source Software (<a href="http://linux-sunxi.org/GPL_Violations">which can't be said about some other vendors</a>) and <a href="http://forum.odroid.com/viewtopic.php?f=111&t=11043#p89763">sent a patch to Hardkernel</a>.</p>
<p>Another issue was fixed too. The CEC implementation on the C1 is kinda broken (hardware-wise), so you're unable to control Kodi with your TV remote. You still can do this via the built-in IR receiver, but since my C1 sits inside a closed cupboard, this was not an option for me. You can fix the CEC-Issue yourself <a href="http://forum.odroid.com/viewtopic.php?f=111&t=7540">with some soldering</a>, but fortunately, some devices even work without any soldering. All you need is to install RTC battery and hope for the best. In my case that worked.</p>
<p>Finally, after more than 4 month, I can finally use this little device as a fully-featured media center.</p>Working around UEFI madness on a HP ProBook 6465b2015-04-09T00:00:00+02:002015-04-09T00:00:00+02:00Jan Holthuistag:homepage.ruhr-uni-bochum.de,2015-04-09:/jan.holthuis/working-around-uefi-madness-on-a-hp-probook-6465b.html<p>It's past 6 o'clock in the morning and - while writing this blog post - I'm wondering why manufacturers seem unable to implement stuff properly. If I had known what pain UEFI would cause me, I would have sticked to my good ol' BIOS boot setup. </p>
<p>But first things first: Since I …</p><p>It's past 6 o'clock in the morning and - while writing this blog post - I'm wondering why manufacturers seem unable to implement stuff properly. If I had known what pain UEFI would cause me, I would have sticked to my good ol' BIOS boot setup. </p>
<p>But first things first: Since I wanted to play some games with my friends, I decided to boot the fallow Windows 7 partition that I had set up for this very purpose. I'm using ArchLinux almost exclusively, thus I never booted that partition after installation.</p>
<p>So Windows was booting... All of a sudden, the infamous Blue Screen of Death appeared and my Laptop rebooted. "Shoot!", I thought. I still cannot think of any valid reason why this BSOD occured in the first place. How did Windows managed break down without even being booted? Anyway, after various failed attempts to fix it (including Windows Startup repair, safe mode and recovery console), I eventually gave up and faced the inevitable: I had to reinstall Windows.</p>
<p>After struggling with the Windows installer (which first refused to install to my GPT-partitioned hard disk because for some reason it always started in legacy BIOS mode), I finally succeeded. Unfortunately, Microsoft - as the ultimate authority of enlightenment - is destined to enforce their commandments, including "I am Windows thy OS, Thou shalt have no other OS before me". Therefore, it just overwrote my existing boot manager <em>rEFInd</em> and replaced it with it's own one, so that I was unable to boot into Linux.</p>
<p>Since I was using UEFI (mainly because I was curious, it <a href="https://www.linux.com/learn/tutorials/588498-uefi-secure-boot-big-hassle-questionable-benefit">doesn't really make sense from a security standpoint</a>), I was actually able to reboot and quickly press the F9 key on my HP ProBook 6465b to select a boot device. One of the options was to select an EFI file to boot, which at least enabled me to boot Linux at all, but than can only be a temporary solution.</p>
<p>Now the real pain began: First, I tried to get back rEFInd by using Windows' <code>bcdedit</code>. I <a href="http://www.rodsbooks.com/refind/getting.html">downloaded it</a> and followed the steps <a href="http://www.rodsbooks.com/refind/installing.html#windows">here</a>, but to no avail:</p>
<div class="highlight"><pre><span></span><code><span class="n">C</span><span class="p">:</span>\<span class="o">></span><span class="w"> </span><span class="n">mountvol</span><span class="w"> </span><span class="n">S</span><span class="p">:</span><span class="w"> </span><span class="o">/</span><span class="n">S</span>
<span class="n">C</span><span class="p">:</span>\<span class="o">></span><span class="w"> </span><span class="n">cd</span><span class="w"> </span>\<span class="n">Users</span>\<span class="n">jan</span>\<span class="n">Downloads</span>\<span class="n">refind</span><span class="o">-</span><span class="n">bin</span><span class="o">-</span><span class="mf">0.8</span><span class="o">.</span><span class="mi">7</span>
<span class="n">C</span><span class="p">:</span>\<span class="o">></span><span class="w"> </span><span class="n">xcopy</span><span class="w"> </span><span class="o">/</span><span class="n">E</span><span class="w"> </span><span class="n">Users</span>\<span class="n">jan</span>\<span class="n">Downloads</span>\<span class="n">refind</span><span class="o">-</span><span class="n">bin</span><span class="o">-</span><span class="mf">0.8</span><span class="o">.</span><span class="mi">7</span>\<span class="n">refind</span><span class="w"> </span><span class="n">S</span><span class="p">:</span>\<span class="n">EFI</span>\<span class="n">refind</span>\
<span class="n">C</span><span class="p">:</span>\<span class="o">></span><span class="w"> </span><span class="n">S</span><span class="p">:</span>
<span class="n">S</span><span class="p">:</span>\<span class="o">></span><span class="w"> </span><span class="n">cd</span><span class="w"> </span><span class="n">EFI</span>\<span class="n">refind</span>
<span class="n">S</span><span class="p">:</span>\<span class="n">EFI</span>\<span class="n">refind</span><span class="o">></span><span class="w"> </span><span class="n">rename</span><span class="w"> </span><span class="n">refind</span><span class="o">.</span><span class="n">conf</span><span class="o">-</span><span class="n">sample</span><span class="w"> </span><span class="n">refind</span><span class="o">.</span><span class="n">conf</span>
<span class="n">S</span><span class="p">:</span>\<span class="n">EFI</span>\<span class="n">refind</span><span class="o">></span><span class="w"> </span><span class="n">bcdedit</span><span class="w"> </span><span class="o">/</span><span class="n">set</span><span class="w"> </span><span class="p">{</span><span class="n">bootmgr</span><span class="p">}</span><span class="w"> </span><span class="n">path</span><span class="w"> </span>\<span class="n">EFI</span>\<span class="n">refind</span>\<span class="n">refind_x64</span><span class="o">.</span><span class="n">efi</span>
<span class="n">S</span><span class="p">:</span>\<span class="n">EFI</span>\<span class="n">refind</span><span class="o">></span><span class="w"> </span><span class="n">bcdedit</span><span class="w"> </span><span class="o">/</span><span class="n">set</span><span class="w"> </span><span class="p">{</span><span class="n">bootmgr</span><span class="p">}</span><span class="w"> </span><span class="n">description</span><span class="w"> </span><span class="s2">"rEFInd Boot Manager"</span>
</code></pre></div>
<p>It kinda worked, and there was no error message, but Windows still kept booting instead of rEFInd.</p>
<p>So, I rebooted into Linux to restore my boot manager via <code>efibootmgr</code>. But that still didn't work, because the new boot entry that I added immediately disappeared.</p>
<div class="highlight"><pre><span></span><code># efibootmgr -c -d /dev/sda -p 1 -L "rEFInd" -l /EFI/refind/refind_x64.efi
</code></pre></div>
<p>Next step: Compiling the <a href="https://aur.archlinux.org/packages/uefi-shell-svn/">EFI Shell v2</a> in order to use <code>bcfg</code> to edit my boot entries. After compiling, rebooting, pressing F9 and selecting the <code>shellx64.efi</code> file, the next disappointment awaited me:</p>
<div class="highlight"><pre><span></span><code>ASSERT_EFI_ERROR (Status = Not Found)
</code></pre></div>
<p>After some googling, I found the <a href="http://sourceforge.net/p/edk2/mailman/message/28699596/">cause of this</a>: The UEFI implementation my ProBook uses is too old to be compatible with the <em>EFI Shell v2</em>. Using <em>EFI Shell v1</em> was no option either, since it lacks the <code>bcfg</code> tool that was the reason to boot the shell in first place.</p>
<p>Now I was a bit baffled to say the least.</p>
<p>After massive googling, I finally found the single most ridiculous way I can think of to get my beloved rEFInd setup back: <em>**drum roll**</em></p>
<p>Some very bad UEFI implemtations apparently expect the <code>*.efi</code> file to boot at <code>EFI/Microsoft/Boot/bootmgfw.efi</code>. So I mounted my EFI System Partion at <code>/esp</code>:</p>
<div class="highlight"><pre><span></span><code># mkdir /esp
# mount -t vfat /dev/sda1 /esp
</code></pre></div>
<p>Then I moved my Windows <code>*.efi</code> files to a newly created folder, moved my refind to <code>EFI/Microsoft/Boot/bootmgfw.efi</code> and renamed <code>refind_x64.efi</code> to <code>bootmgfw.efi</code>:</p>
<div class="highlight"><pre><span></span><code># cd /esp/EFI/Microsoft
# mkdir Windows
# mv Boot Windows
# mv ../refind Boot
# mv Boot/refind_x64.efi Boot/bootmgfw.efi
</code></pre></div>
<p>And - to my surprise - it actually worked. Simple as that. Wow. I just needed to adapt my <code>refind.conf</code> to the new path of the Windows bootloader (since I disabled to autodetection of efi files in favor of manual entries), but that's it. I finally finished getting my usual boot setup back.</p>
<p>I still don't know who I should condemn more: Microsoft for still replacing the boot manager without even asking or HP for putting such a crappy UEFI implementation into my Laptop. But one thing I know: Next time I reinstall my PC, I'll choose Legacy BIOS instead of UEFI.</p>ODROID-C1 post-installation tips2015-02-13T00:00:00+01:002015-02-13T00:00:00+01:00Jan Holthuistag:homepage.ruhr-uni-bochum.de,2015-02-13:/jan.holthuis/odroid-c1-post-installation-tips.html<p>Some weeks ago I decided you replace my old and rusty Raspberry Pi (serving as a media center) with something faster (and less likely to suddenly lose ethernet connection). I stumbled over the dirt-cheap ODROID-C1, which outruns the Raspberry Pi (and even the brand-new Raspberry Pi 2) thanks to its …</p><p>Some weeks ago I decided you replace my old and rusty Raspberry Pi (serving as a media center) with something faster (and less likely to suddenly lose ethernet connection). I stumbled over the dirt-cheap ODROID-C1, which outruns the Raspberry Pi (and even the brand-new Raspberry Pi 2) thanks to its Amlogic S805 ARM Quad Core Cortex™-A5 processor and Dual Core Mali-450 GPU. And it's also supported by <a href="http://archlinuxarm.org/platforms/armv7/amlogic/odroid-c1">ArchLinuxARM</a>.</p>
<p>If you've also purchased that little device, here are some solutions to problems I encountered upon first boot (after setting up basic stuff like timezone, hostname, locale, etc.), so I hope that these tips help you, too.</p>
<h1>Blacklisting the Dallas 1-Wire interace</h1>
<p>In case you're not going to use the 1-wire interface, you might want to blacklist the kernel module. Otherwise, it'll keep scanning all day long for attached sensors. Create <code>/etc/modprobe.d/blacklist-w1.conf</code> and insert:</p>
<div class="highlight"><pre><span></span><code><span class="p">#</span><span class="w"> </span><span class="n">Disable</span><span class="w"> </span><span class="n">the</span><span class="w"> </span><span class="mh">1</span><span class="o">-</span><span class="kt">wire</span><span class="w"> </span><span class="n">interface</span>
<span class="n">blacklist</span><span class="w"> </span><span class="n">w1_gpio</span>
</code></pre></div>
<h1>Enabling the Hardware Random Number Generator</h1>
<p>Use the hardware RNG instead of <code>haveged</code>. You need to install <code>rng-tools</code> and make sure that <code>/etc/conf.d/rngd</code> contains:</p>
<div class="highlight"><pre><span></span><code><span class="nv">RNGD_OPTS</span><span class="o">=</span><span class="s2">"-o /dev/random -r /dev/hwrng"</span>
</code></pre></div>
<p>Then, you can enable the rngd service via <code>systemd</code>:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>sudo<span class="w"> </span>systemctl<span class="w"> </span>stop<span class="w"> </span>haveged
$<span class="w"> </span>sudo<span class="w"> </span>systemctl<span class="w"> </span>start<span class="w"> </span>rngd
</code></pre></div>
<p>To keep using the hardware RNG after a reboot:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>sudo<span class="w"> </span>systemctl<span class="w"> </span>disable<span class="w"> </span>haveged
$<span class="w"> </span>sudo<span class="w"> </span>systemctl<span class="w"> </span><span class="nb">enable</span><span class="w"> </span>rngd
</code></pre></div>
<h1>HDMI-related stuff</h1>
<h2>Getting video working</h2>
<p>If you've configured everything correctly, but you still get no image on your display, chances are that you bought a non-standard HDMI cable that misses the group pin. Either go buy another cable (it's rumoured that the AmazonBasics HDMI-to-Micro-HDMI cable works) or short the shell of the Micro-HDMI with the (grounded) Micro-USB-shell. I used a paperclip to do this. Have a look at <a href="http://forum.odroid.com/viewtopic.php?f=117&t=8894#p69549">this post</a> in the ODROID forum.</p>
<h2>Getting audio working</h2>
<p>Install <code>pulseaudio</code>. You need to edit Pulseaudio's <code>default.pa</code>, so either edit <code>/etc/pulse/default.pa</code> in-place or copy it to <code>~/.config/pulse/default.pa</code> of the respective user.</p>
<p>Search for these lines:</p>
<div class="highlight"><pre><span></span><code><span class="c1">#load-module module-alsa-sink</span>
<span class="c1">#load-module module-alsa-source device=hw:1,0</span>
</code></pre></div>
<p>And replace them with:</p>
<div class="highlight"><pre><span></span><code><span class="nb">load</span><span class="o">-</span><span class="n">module</span><span class="w"> </span><span class="n">module</span><span class="o">-</span><span class="n">alsa</span><span class="o">-</span><span class="n">sink</span>
<span class="nb">load</span><span class="o">-</span><span class="n">module</span><span class="w"> </span><span class="n">module</span><span class="o">-</span><span class="n">alsa</span><span class="o">-</span><span class="n">source</span><span class="w"> </span><span class="n">device</span><span class="o">=</span><span class="n">hw</span><span class="p">:</span><span class="mi">0</span><span class="p">,</span><span class="mi">1</span>
</code></pre></div>
<p><strong>Important:</strong> Make sure that it's <code>hw:0,1</code> and not <code>hw:1,0</code>!</p>
<p>Also, do not forget that your user need to be part of the <code>audio</code> group to use the sound device.</p>
<p>Now you can check if everything worked by running:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>paplay<span class="w"> </span>/usr/share/sounds/alsa/Front_Center.wav<span class="w"> </span>
</code></pre></div>
<h2>Getting CEC working</h2>
<p>Unfortunately, that's a hardware issue. The guys at Hardkernel are working on it, so chill. You'll probably need to RMA your device or do some soldering yourself, though. Check out <a href="http://forum.odroid.com/viewtopic.php?f=111&t=7540">this thread</a> in the ODROID forum.</p>Adblock on your router2014-12-25T00:00:00+01:002014-12-25T00:00:00+01:00Jan Holthuistag:homepage.ruhr-uni-bochum.de,2014-12-25:/jan.holthuis/adblock-on-your-router.html<p>A lot of people already use Adblockers in their desktop browser, but what if you're using your phone, tablet or an alternative browser? Getting rid of annoying ads is simple: Block them with your router.</p>
<p>The idea is simple: when the DNS server on your router receives queries, we'll sort …</p><p>A lot of people already use Adblockers in their desktop browser, but what if you're using your phone, tablet or an alternative browser? Getting rid of annoying ads is simple: Block them with your router.</p>
<p>The idea is simple: when the DNS server on your router receives queries, we'll sort out queries that ask for the <a href="http://en.wikipedia.org/wiki/List_of_DNS_record_types">A Resource Records</a> of ad servers and return <code>127.0.0.1</code> instead of the ad server's IP address.</p>
<p>There are loads of scripts out there to accomplish something like that. IMHO, some are too complex (like changing the <em>FORWARD</em> chain in iptables), some do stuff I don't like (e.g. manipulating stuff in <code>/etc/</code>) and some even pose a security threat, because they download config files and use them without checking their content.</p>
<p>This is why I wrote such a script myself. What It does:</p>
<ul>
<li>download a list of ad servers in dnsmasq config file format from <a href="http://pgl.yoyo.org/adservers/">yoyo.org</a></li>
<li>check if the file only contains <code>address</code> statements that assign the IP address <code>127.0.0.1</code> to some domain name (with <code>grep</code>)</li>
<li>save the file as <code>/tmp/dnsmasq.d/adblock.conf</code></li>
<li>restart dnsmasq</li>
</ul>
<p>You need at least OpenWRT <a href="https://dev.openwrt.org/changeset/39312/">revision 39312</a> to use it.</p>
<p>Enough talk, let's get going:
Simply SSH to your router, download <a href="https://gist.github.com/Holzhaus/ed4ac1675a57f11c3057">this script</a>:</p>
<div class="highlight"><pre><span></span><code><span class="c1"># mkdir -p /opt/bin</span>
<span class="c1"># curl "https://gist.githubusercontent.com/Holzhaus/ed4ac1675a57f11c3057/raw/6a2b59ce046ad6da5f9eac48db925f0afb292a00/adblockupdater.sh" > /opt/bin/adblockupdater.sh</span>
<span class="c1"># chmod +x /opt/bin/adblockupdater.sh</span>
<span class="c1"># /opt/bin/adblockupdater.sh</span>
</code></pre></div>
<p>To update the server list on a regular basis, add the line <code>0 0 */1 * * /opt/bin/adblockupdater.sh</code> to your crontab by typing <code>crontab -e</code>.</p>
<p>Congrats, you now have adblock on your router!</p>
<p><em>Note:</em> If you can't download the script, because you're getting <code>curl: (51) Cert verify failed: BADCERT_NOT_TRUSTED</code>, you're probably missing CA certificates. Run:</p>
<div class="highlight"><pre><span></span><code><span class="c1"># opkg update</span>
<span class="c1"># opkg install ca-certificates</span>
</code></pre></div>
<p>Also check if <code>export SSL_CERT_DIR=/etc/ssl/certs</code> is in your <code>/etc/profile</code> and add it, if neccessary. Run <code>source /etc/profile</code> and retry.</p>Ogg videos on RPi/Kodi2014-12-17T00:00:00+01:002014-12-17T00:00:00+01:00Jan Holthuistag:homepage.ruhr-uni-bochum.de,2014-12-17:/jan.holthuis/ogg-videos-on-rpikodi.html<p>I recently wanted to play an OGG video on my <a href="http://www.raspberrypi.org/">Raspberry Pi</a> with XBMC (which has been <a href="http://kodi.tv/introducing-kodi-14/">renamed to Kodi recently</a>). Unfortunately, it didn't work out of the box. This is how I solved the problems.</p>
<h1>Problem 1: Ogg video file not recognized.</h1>
<p>If you try to open your file …</p><p>I recently wanted to play an OGG video on my <a href="http://www.raspberrypi.org/">Raspberry Pi</a> with XBMC (which has been <a href="http://kodi.tv/introducing-kodi-14/">renamed to Kodi recently</a>). Unfortunately, it didn't work out of the box. This is how I solved the problems.</p>
<h1>Problem 1: Ogg video file not recognized.</h1>
<p>If you try to open your file by using the file browser in Kodi's video section, it won't be displayed. This is because Kodi assumes that files with the <code>*.ogg</code> extension are audio files.</p>
<p>Solving this is easy: Simply rename <code>somefile.ogg</code> to <code>somefile.ogv</code> (<code>somefile.ogm</code> is also possible). For the sake of completeness: Your Ogg audio files can have <code>*.ogg</code> or <code>*.oga</code> as file extension.</p>
<p>Although you can also change the way Kodi detects files and make <code>*.ogg</code> a video file extension, I do not recommend this. That extension is much more common with audio files and renaming Ogg videos to <code>*.ogv</code> makes audio and video files distinguishable by just reading the filename.</p>
<p>Now you can see the Ogg video file in Kodi's file browser. You hit enter and .... have another problem.</p>
<h1>Problem 2: Ogg video file playback is audio-only</h1>
<p>Now the file actually plays and you can hear the sound, but there's no video. This is probably because the RPi's GPU does not know how to decode the video. GPU accelerated software codecs are only available from <code>start_x.elf</code>.</p>
<p>Open <code>/etc/profile/</code> and append <code>/opt/vc/bin</code> to your <code>$PATH</code>:</p>
<div class="highlight"><pre><span></span><code># Set our default path
PATH="/usr/local/sbin:/usr/local/bin:/usr/bin:/opt/vc/bin"
</code></pre></div>
<p>Then add this to your <code>/boot/config.txt</code>:</p>
<div class="highlight"><pre><span></span><code>start_file=start_x.elf
fixup_file=fixup_x.dat
</code></pre></div>
<p>That causes the RPi to load testing versions of the GPU firmware. These enable <em>potentially unstable/not-fully-tested/hacky functionality - currently, using these files instead of the usual fixup.dat/start.elf will cause extra video codecs to become available.</em><a href="http://elinux.org/RPi_Software#Overview">[1]</a></p>
<p>Note that you should set your GPU memory to at least 128 MB:</p>
<div class="highlight"><pre><span></span><code><span class="nv">gpu_mem_256</span><span class="o">=</span><span class="mi">128</span><span class="w"> </span>#<span class="w"> </span><span class="k">If</span><span class="w"> </span><span class="nv">you</span><span class="w"> </span><span class="nv">have</span><span class="w"> </span><span class="mi">256</span><span class="w"> </span><span class="nv">MB</span><span class="w"> </span><span class="nv">RAM</span><span class="w"> </span><span class="ss">(</span><span class="nv">Model</span><span class="w"> </span><span class="nv">A</span><span class="ss">)</span>
<span class="nv">gpu_mem_512</span><span class="o">=</span><span class="mi">128</span><span class="w"> </span>#<span class="w"> </span><span class="k">If</span><span class="w"> </span><span class="nv">you</span><span class="w"> </span><span class="nv">have</span><span class="w"> </span><span class="mi">512</span><span class="w"> </span><span class="nv">MB</span><span class="w"> </span><span class="nv">RAM</span><span class="w"> </span><span class="ss">(</span><span class="nv">Model</span><span class="w"> </span><span class="nv">B</span><span class="ss">)</span>
</code></pre></div>
<p>Et voilà. Kodi is now able to play Ogg video files (hopefully)!</p>Welcome2014-05-15T00:00:00+02:002014-05-15T00:00:00+02:00Jan Holthuistag:homepage.ruhr-uni-bochum.de,2014-05-15:/jan.holthuis/welcome.html<p>Voilà, there is my new website. Do not expect regular updates.</p>