Capturing Cafe Oto. The technical challenges behind the musical ambition (Part 2)

Cafe Oto has won Joining the Dots funding and support to help them achieve their ambition to make their progressive music programme more widely available. Last time around  digital producer James Dunn explained how he and his colleagues figured out the technicals needed to store and release every note performed at the venue. This time they’ve looked at some of the issues they’ve had with the upload of recordings from the Raspberry Pi to their Amazon S3 cloud server – and solutions they’ve found….

An early uploading problem
One issue was that the aws command line utility provided by Amazon would fail to upload some files due to some TCP errors. We solved this problem by installing wondershaper to limit the bandwidth. We also switched distributions from  Arch Linux to Raspian which contains this software in its repositories and is generally better supported.

Data corruption on our SD cards
Another issue we had was the corruption of data on the SD cards. This would happen periodically and was caused by the insertion of our external USB hard drive which caused a voltage drop in the Pi’s on-board power supply, thereby causing a hard reboot and corrupting data on the card. After re-imaging the cards several times, they finally became unusable and could no longer boot the image.

Strangely the SD cards still appear to work fine when used on a Macbook Pro, but I assume this is because the Mac recognises the bad sectors and automatically avoids writing to them. This is not possible to do when re-imaging a card as it is a block-for-block transfer of the backup image.

The solution was to move the boot partition to an external USB hard drive as detailed on the Raspberry Pi forums here. We also required a new SD card to hold the initial boot information and we had a 1GB card lying around so we used this to install a minimal Moebius distribution and then moved the root partition to the external drive. We had to use the GUID method for booting from the external drive, as we can’t guarantee the external drives will have the same device name on boot. We also needed to add a powered USB hub to prevent our other external USB hard drive causing the power outs/reboots during hotplugging.

Success! And looking ahead….
As of late August, the Pi is up and running again and is now emailing daily logs of the uploaded files with the URLs to the download links. Success!

Further development is planned, and involves an automated backup of important system files via rsync and regular imaging of the drive. There also needs to be some logic implemented to handle the plugging of multiple hard drives to determine which drive should be prioritised for upload.

Additionally, the JoeCo BBR has trouble recording to the 2TB drive when it is nearing 50% capacity, so a longer term task is to set up a RAID array to store the recordings locally and automatically move them to the RAID array from a temporary smaller drive when it is switched between the BBR and the Pi.

We hope this is useful to anyone else attempting anything similar! If you have any queries just get in touch.