Things have been getting a bit, well, unreliable at the hospital radio studio of late. We've been plagued with power outages, and things don't recover well.
Mainly the old Ubuntu 16 box, which supplies shoutca.st with the MP3 stream.
It usually comes back on OK, but half the time fails to run the darkice service on start-up. No amount of fiddling the cron job that starts the service has helped.
What's needed is a system that checks and monitors itself, and attempts to repair what's broken.
Here's what I've got in mind.
Two Raspberry Pi's, one transmitting, one receiving. The receiver will be monitored to ensure that the stream is up.
The Pi's audio will be fed into and internet-equipped Arduino of some description, which will do the monitoring for us.
It'll work something like this:
- If the received audio is ok, do nothing. All is well.
- If the received audio is missing, check there's audio coming in to the transmitter. If there isn't then there's nothing we can do except to flash a "something's up" LED to get someone to fix it.
- If the received audio is not present, but there's audio to the transmitter, check there's internet access; If there isn't there's little we can do, except for flash a "something's up" LED and wait for it to come back. If there is, then we need to take some action:
- Shut down and restart the receiver, just in case it's crashed. If that doesn't work, then ...
- Shut down and restart the transmitter, just in case that's crashed.
OK, so there are a good few guides on getting internet radio receivers to run on a headless Pi. I've tried a couple, and have found one that is very tolerant of dodgy internet connections, it's called mplayer, it's a bit processor hungry for what it is, but runs on a model A+ Pi I have. It comes back if the internet goes missing as soon as it recovers, and starts reliably on boot. I did this a while back, and it's been sat on the workshop shelf for ages, reliably receiving :)
It's dead easy to install..
apt-get install mplayer
add a cron job...
crontab -e
and add the following line...
@reboot sleep 29; nohup mplayer -loop 0 http://188.165.192.5:8525/stream 2>&1 > /dev/null
(replacing the IP address and mount point with your own, unless you want to listen to Cotswold hospital radio!)
You may need to adjust the output volume.. Just type
alsamixer
adjust as required, then type
sudo alsactl store
to store the setting.
Now we need to add a bit of Python script in to safely reboot the monitor receiver in the event of a failure, and also add a shutdown function, so we can safely shut our unit down, without risking corruption of the memory card.
We are going to use the Pi's GPIO pins 18 and 23. Pulling 18 low will instigate a safe shutdown. Pin 23 will force a shutdown and restart.
I've copied the code from
here, but duplicated it here as well, just in case it ever goes missing.
First off SSH into the Pi...
Once logged in, type
mkdir Scripts
cd Scripts
touch shutdown_pi.py
sudo nano shutdown_pi.py
Then cut and paste this code into the editor.
#!/bin/python
# Simple script for shutting down the raspberry Pi at the press of a button.
# by Inderpreet Singh https://www.element14.com/community/docs/DOC-78055/l/adding-a-shutdown-button-to-the-raspberry-pi-b
import RPi.GPIO as GPIO
import time
import os
# Use the Broadcom SOC Pin numbers
# Setup the Pin with Internal pullups enabled and PIN in reading mode.
GPIO.setmode(GPIO.BCM)
GPIO.setup(18, GPIO.IN, pull_up_down = GPIO.PUD_UP)
GPIO.setup(23, GPIO.IN, pull_up_down = GPIO.PUD_UP)
# Our function on what to do when the button is pressed
def Shutdown(channel):
os.system("sudo shutdown -h now")
def Restart(channel):
os.system("sudo shutdown -r now")
# Add our function to execute when the button pressed event happens
GPIO.add_event_detect(18, GPIO.FALLING, callback = Shutdown, bouncetime = 2000)
GPIO.add_event_detect(23, GPIO.FALLING, callback = Restart, bouncetime = 2000)
# Now wait!
while 1:
time.sleep(1)
Press CTRL + X and yes to save and exit
now type
sudo nano /etc/rc.local
and add this line just above the last exit 0
python /home/pi/Scripts/shutdown_pi.py &
now safely shutdown the pi
sudo shutdown now
wait for a minute and remove the power.
Remove the memory card and make an image of it and keep that safe, just in case...
Job done.
The transmitter is a new Raspberry Pi 3, and I followed the instructions
here to load up darkice. I don't need to run icecast, as this is provided to us by remotely by shoutca.st, I just need to send the stream to them. I also ignored the warning about not installing the latest version of Raspbian. Seems to work great, never fails to start reliably, and seems to be tolerant of errant internet connections. You'll need an external USB sound card, as the pi's inbuilt card can't record.
Also, add in the shutdown and reboot python script in the same manner as before.
Image the memory card just in case!
Now our incoming audio is balanced, so we need a balanced line receiver, a buffer for the received audio monitoring, and whilst we're there, a couple of little drivers for some VU meters (because meters are cool), which we can also use to feed our Arduino monitor.
As an aside, I've now made the jump to Kicad, as since Autodesk have purchased Eagle, they no longer wish to honour my original Cadsoft licence, without incurring unwanted expense.
A circuit is conjured up ...
A board designed...
And modelled ...
And turned into reality using toner transfer.
Loving Kicad...
U1 and U2 form the balanced to unbalanced audio receiver, and it's output level is adjusted by RV3. This then feeds the transmitting Pi's external sound card. There's also a buffer (U2A) which feeds a small monitoring amp. ICU2B is a variable gain precision rectifier, the output of which is used to drive the VU meter, and also feed one of the Arduino's A to D converters.
The same circuit is also duplicated and used to buffer the audio from our receiving Pi to drive the monitoring amp, and to drive a VU meter and the Arduino.
There's a regulated +/- 15V supply. As it turns out I had a suitable switched mode supply, with +/- 12V and 5V outputs, so I utilised that, and just bypassed the regulators.
An Arduino nano was pressed into service, to do the monitoring and control.
Received rectified monitor audio is passed from the analogue PCB from the VU drive circuit to A0 , rectified transmitted audio is received likewise, and fed to A3. A front panel mounted LED is connected to Digital pin 3, to serve as our error indicator. I'll also make the Arduino Nano's USB interface available on the rear panel to facilitate debugging/ updating as required.
An Ethernet module (EN28J60) is connected to the Arduino's SPI interface, and is powered from a 3.3V regulator derived from the main +5V supply. The EN28J60 consumes around 120mA, and that's more than the available 3.3V supply on board the nano. The EN28J60 module is provided with network access from the same hub that distributes the network to the Raspberry Pi's.
Two additional outputs (D5 & D6) on the Arduino are used to pull a GPIO pin low on the respective Pi to initiate a graceful reboot, and D7 is diode "anded" to shutdown the two Pi's. A push button is connected to D2 to shut down the system.
If it all goes really badly wrong, I've also added two reset buttons on the front panel to manually hard reset the Raspberry Pi's (and arduino, via a diode "and").
D4 is coupled back to the arduino's reset pin, to reset the controller completely. This is coded to happen if we fail to obtain a DHCP lease more than 5 times.
The VU illumination is connected to D8 via a BC547 transistor. This is used to flash the VU illumination during shutdown to give the user some feedback that shutdown is in progress.
In the event of failure of the monitor audio, the transmit audio is checked.
If the transmit audio has failed, the error LED on the front panel will flash 3 times. The monitor circuit will repeat these tests until audio is resumed.
If the monitor audio has failed, and transmit audio is present, the controller checks internet connectivity. If the internet connectivity is not present, attempts are made to restore DHCP. If this process fails, the fault is presumed to be external, and the error led will flash 5 times, followed by illuminating permanently. If the internet does not recover after 5 DHCP renew attempts, the controller will restart and repeat the process.
If the monitor audio has failed, TX audio present, and internet OK, two possibilities remain. Either the monitor receiver computer has crashed, or the TX computer has crashed. Firstly the controller will shut down and restart the monitor computer. If this fails , the TX computer is restarted.
A board is designed, but I've made it up on perfboard...
Power is supplied to the whole rack from a switched mode supply, recovered from some redundant equipment... It supplies +/-12V and 5V. Excellent.
Putting things together and running some tests, and there's an irritating 4KHz whistle on the audio, both transmitted and on the in built monitor amp... nasty
Scoping up the outputs from the power supply shows there's a 4KHz tone superimposed on our 5V rail. It's about 170mV, and enough to cause the annoying whistle. Tests show this isn't down to a failure in the supply (I suspected caps, but changing them out made no discernible improvement)
A simple C-L-C filter is quickly made up from some junque parts, and greatly improves the situation.
Much improved.
And the finished product, ready for installation once this damn virus goes away...
