We’d like to apologize for the outage the last few days. Initially, we discovered the server hosting the station was down late on the 20th, and a support ticket was placed with the ISP hosting our server. We spoke to the ISP’s support people early on the 21st, and was told that apparently our server had no boot files, and the likely fix was to reinstall the operating system and restore everything from BACKUP?????? yikes!! Some heated discussion was held in BNA HQ and it was determined that apparently some cleanup of old boot files on the server had inadvertantly deleted the old boot/kernel files AND the currently needed one, and of course, one of the server updates automatically done in the afternoon of the 20th, rebooted the server. Yikes!! Can you say “Shot myself in the foot”??? It was decided to immediately reinstall the operating system and start restoring the station and website from backups.. (backups?? fingers crossed!!!) .. The backup of the actual shoutcast music library was restored and we got the station back up “on the air”.. So we thought.. We placed a notice on the website indicating that the stream was back up. We then discovered that the shoutcast stream would randomly disconnect/drop during our testing.  Finally one of the team noticed something odd.. the server only had 128MB of ram….. yikes!!  Since we pay for 512MB of ram for the server, we immediately contacted the ISP who, after a bit of checking, had egg on its face..  They upped the server back to its 512MB of ram, and we then were able to complete the restore of the website to pre-problem configuration…  Aint Backups great!!!!!

Regards,BNA Staff

The BestNewAge server administrator noticed the other day that we had maxxed out our 32 listener slots for the very first time. What this means is that anyone else who tried to connect for some of our great music was turned away, until one of the 32 listening users disconnected. As a listener, aren’t you glad we upgraded to 32 user slots? At the moment this post is being written, there are 16 listeners, with an average listen time of 11 hours and 30 minutes. Some Shoutcast stations limit the listen time, but we don’t, and never will..

Regards,
BestNewAge Staff

We’ve upgraded the station from a Shoutcast 32 listener slot 64kbps configuration to a 32 listener slot 128kbps configuration. Since the host we’re running the station on gives us a 1 TB/month (terabyte/month) data allowance, and we weren’t even coming close to hitting it, we decided to give the listeners a significantly better listening experience. We hope you enjoy the change..

Regards
BestNewage Staff

Its appearing that the move fixed the rebooting problem. Its been nearly four days since we made the move and there have been NO reboots.  Non-techies can tune out here, as rest of this post is a techy description of what was done.  The station had been running for over two years on a 512mb OpenVZ virtualized server, and had been very stable up until about the middle of September. We decided that since there was a problem with the OpenVPN link we use to manage the server, we’d go ahead and backup/reinstall the server operating system (Ubuntu 10.04LTS) and install the latest version of the Shoutcast system on the newly rebuilt server. This went fairly well, and we learned quite a bit about XML and PHP in getting the “Now Playing” script to work under the new Shoutcast version. About a week after completion of the rebuild/upgrade, we started noticing the station down, and the underlying server recently rebooted. Checks of the server logs were done with nothing amiss found, then we contacted the VPS vendors tech support, who reported they could find nothing wrong on the host, which, in this case was running sixteen other 512mb OpenVZ “servers”.  We finally decided that the problem was probably due to one or more of these fifteen other servers doing *something* on their virtual machine that used more resources than they were allocated/paying for. One of the downsides to OpenVZ is just this kind of problem. Tracking it down by us was impossible, and the vendor has no incentive to spend the time needed to find and fix the problem. We then decided that we needed to migrate the station to a Xen-based virtual server. The current vendor also had a Xen-based offering, but their pricing was double what the OpenVZ offering was, and so a search for another vendor was launched. We found a reference to a vendor with a 512mb Xen offering which was about 15% less than the 512mb OpenVZ offering at our current vendor. This new vendor also had been in business since 2006, and had a very good reputation in the industry. We then decided to move the station to this vendor. The move was done with no downtime (other than the incessant reboots on the old host) on October 25. With the move we also upgraded the OS on the station from Ubuntu 10.04LTS to Ubuntu 12.04LTS.

Further info as it happens!!

Best Regards,

BestNewAge Management

We believe we’ve solved the random system reboots and concurrent station outages. The station had been running on a 512MB Ubuntu virtual server. We moved it to a different underlying type of virtual server, one which isolates the other virtual servers better from each other. For the techies among our listeners, we previously were on an OpenVZ-based virtual server, and we’ve moved to a Xen-based virtual server. We believe this will take care of the random reboots. Thanks for your patience..

BestNewAge Management

We’re very sorry about the frequent outages. The problem is the virtual server that the station runs on has decided to reboot itself frequently. We’re currently looking into WHY the Shoutcast software doesn’t restart on these reboots. Since the station does not restart after a reboot without one of the station staff noticing it, and manually logging on to the Linux server that the station runs on, and restarting the Shoutcast services. This means we may not notice an outage for a few hours. Please bear with us… 🙂

BestNewAge Management