Building an EPG for Home Media Systems

Why I Wrote This Post

At some point in 2017, a couple of different things came together for me that changed my entire approach to television and movie content. On one side, I was happily ripping all of my DVD and BluRay content from physical media and onto a home media system which would allow me to play whatever I wanted, in any room, without having to go find discs and put them into a DVD player. On the other side, I became aware that 99% of the television content that was of any actual interest to me was available via over-the-air (OTA) broadcast signals that I could receive using an antenna. I also learned I could go just a bit further with this by adding digital tuner devices that connected to my home network and could be accessed from any media device on my network to play any channel.

It has been ten years or so since I started down the path of creating a fully-integrated home media system to combine live broadcasts, recorded broadcast content, and television and movie content extracted from physical discs. While the majority of the hardware and software that I use today is the same as I used in the beginning, how I have everything integrated and configured has changed along the way. My hope in writing this is to help others that want to cut the cord with their satellite or cable television provider to understand some of the main things they need to know and have but also to document the very specific way that I have evolved the creation of my Electronic Program Guide (EPG) that is used on my home media system software.

Buckle up… There’s a lot to get through here.

What is an EPG?

EPG stands for Electronic Program Guide and it’s a digital schedule of various programs that will be available on different broadcast channels at different times over the coming days. Want to know what’s on a particular channel right now? Want to know what time the football game starts on Sunday? Want to know if this week’s episode of Law and Order: SVU is new or a repeat? All of those questions can be answered by checking the EPG.

These guides are available in a variety of forms, from many places, some for a free and some not. There is also a basic guide that is broadcast as part of the live OTA broadcast content as well. If you are building any form of system that is going to access live broadcast content and you want to what the upcoming schedule of programming is, you will need to have access to some form of EPG.

Some setups, like using a Tablo or SiliconDust HD HomeRun, will have an EPG baked right into their app. Your major networks and their subchannels will tend to be pretty accurately shown although some of the channels that carry a lot of paid programming or similar might not have very accurate details in guides that are part of the apps. For these setups, you get a “really good” overall setup that requires no extra effort, but you also don’t have any way to fix things that aren’t right except maybe let the company know that a channel isn’t showing accurate details about programming – they may be able to correct it.

For more configurable and controllable setups like integrating Live TV and PVR/DVR functions into a media server, you will have more flexibility and more options for choosing and even modifying a guide. This is really the edge case scenario that led me to write this post.

How It Started

When I first got started with a home media server, I chose Plex and opted to integrate Live TV and DVR functions via a networked tuner from SiliconDust (HD HomeRun). I chose the EPG that Plex had integrated into their software, and things were up and running pretty quickly.

It worked great. Until it didn’t. I lost EPG data, no channels were showing what programs were coming up, none of my recordings were showing any upcoming episodes, etc. It basically just sort of ‘died’ overnight and I had no idea why. A lot of digging around forum posts from Plex and some other resources and what I found was that, apparently, Plex had discontinued the use of a provider in my area for EPG info and chose to go with a different provider. Since the exact format of content is provider-specific, none of my existing configuration was any good so I had to tear out the entire Live TV configuration and start again (Plex does not allow modifying the EPG in any way once it’s set).

It was frustrating, but I did it, and moved on. I kept watch of the guide to locate the shows that needed to have new recording options set up and tagged all of them again. Eventually, everything was back to where it was previously and things were good again. And once again, it worked great. Until it didn’t.

Yep, Plex did it again. I think this situation actually occurred three or four times before I got sick of it and found an EPG “scraper” that could collect channel information for me and produce my own personal XML file of information.

By this time, my recording calendar had a lot more on it. Rebuilding the whole setup again from scratch was much more ‘painful’ to do but once it was done, I could drop an XML into the specified location any time I wanted from any source I wanted and Plex would read it in.

This was a much better overall solution for me until one day it broke again due to the primary site that was hosting the channel guide content being taken over by a different entity and the entire domain being changed along with all of the specific URLs where the channel details were found. Ugh. What a mess.

How It’s Going

Some sharp folks made changes to the EPG scraper fairly quickly to support the new site and things came back easily at that point. Nothing on Plex needed to be touched. At all. I just needed a new XML file for it to read and that was now being produced again. Over the next few months there were a couple more “breaks” in the process that required fixes to the scraper, but it’s been running very well since overall.

I have seen a new batch of channels get added in my local area but I am NOT seeing any data for those channels from the scraper. Since I wanted to have some content show up for those channels, I decided to build a TVHeadend server specifically for its ability to collect transmitted channel information that I could extract and create a custom piece of content to add into my XML file. Once that system was built, it only took a couple of scripts to get the channel info periodically, format it to insert into the XML file, insert it, and then drop the final file into the correct location.

The bigger news here is that not only did I have to do absolutely nothing to Plex to begin using this new data, but the same is true for Emby as well. Both systems point to the same locations for the same file and they both read it in without any issue. Emby gets the advantage because they support using their EPG alongside my XML file at the same time while Plex is one or the other only, never both. In other words – if my XML file goes offline again, Emby will continue to hum right along because their content will still be integrated. Plex, however, will be dead in the water.

How Did I Get Here / Why Am I Here

I wanted to be able to share some details of exactly what I built, how I built it, and how it runs so that others may benefit from it as well. It’s definitely a bit more complex than not, but it’s also very flexible and quite reliable for me.

Plex and Emby both have a specific drive mounted off of one of my file servers. They both point to a specific file on that drive that contains the XML formatted information they both need. That part is reasonably straight-forward. Creating that file is a little less so. I’ll break it down into the main components.

  • Zap2Xml: This is a Docker container that I have to pull and then modify by replacing the default script with a specific script that I keep on a local drive (the script in the container is no longer supported and they’ve changed to some different components that I don’t use).
  • TVHeadend: I have this configured to scrape basic guide information from what is sent out with the programming itself to create a sort of “EPG Lite” for all of the channels my HD HomeRun tuners are configured to receive.
  • TVHeadend “pull” script: I run a bash script twice per day to download the guide data from TVHeadend into a text file. This script targets a short list of specific channels that I can not get guide data elsewhere.
  • “Merge” script: This parses the guide data from TVHeadend, reformats the data to match what Zap2It creates, then merges the two together into a single file.
  • “Fine tune” script: This parses the final file and replaces URL information to point to a specific location to be able to pull images for channels and shows to display.

Some of the scripting I wrote myself, some I leaned on some of the AI code bots to build some baseline pieces before I heavily modified them for my own use.

Zap2Xml

Zap2Xml was created as a way to scrape programming information from the web site Zap2It and create an XML-based guide file. Many people used either the scripts themselves or a flavor of this container (there are a number of them out there) for a long time until Zap2It was purchased by Gracenote and the site was “rewritten” (the base URL changed and that changed everything else about it as well). The Zap2It web site allowed one to create an account, log in with their username and password, and see a list of programming information assocaited with their local area. They could tag channels and such as favorites to see just those or they could view the whole listing. Zap2Xml leveraged the user credentials (and the account) to access a ‘preferred’ program listing, downloaded the information, and then formatted what it needed to keep in an XML file that adhered to standards for an EPG.

When Gracenote took over the web site, the base URL was changed and the contents of the site was folded into a larger site owned and operated by Gracenote. All of the scripts and containers stopped being able to scrape the information. Eventually, someone figured out that all of the accounts (and usernames / passwords) still worked on the Gracenote site as long as you accessed the correct area. Then some folks figured out that there some components of the user agent string that needed to be modified for an automated routine to work on the new site along with the new URL pattern. It seems that only a little of this ever made it into the actual container updates, though, which is at least partially why my process involves the original container combined with a custom script replacement.

The specific container I use comes from GitHub and is located here: jef/zap2xml. If you read the notes on that page, you’ll see that the “historical” version that uses Perl scripts is not (fully/well) maintained. This is the cause of the gap I have… I pull and deploy the container using a username, password, a few variables to set timeout settings and such and then specify the output file to be written to a mounted location by the underlying Docker host. Once it’s up and running, I attach to the running container to launch a shell and copy my customized zap2xml.pl file into the /opt directory and I restart the container. This implements my modified zap2xml.pl script that has been edited with the latest changes to fully support the Gracenote site and I have 95% of the content being scraped twice per day.

TVHeadend and “Pull” / Merge / Format Script

This is another Docker container that I deployed solely to tie in my HD HomeRun tuners and collect program information via the broadcast. I’m using the OTA PSIP grabber module to collect all program guide data twice per day. I created a user in the TVHeadend system that has access to the guide info and set a password on that account. I have the EPG Grabber set to collect program data from -all- channels even though I only need data from a few at this point because this will allow me to quickly and easily switch over to using everything from this guide if Zap2Xml has an issue at any point.

With all of this up and running, I execute a python script that pulls everything together, edits / formats it, and then places the resultant file where my media systems read it from. The script starts with a curl command that connects to TVHeadend and collects program data for seven specific channels (I can’t get guide data for these channels from Gracenote at this time) and writes it to a file. It then formats the data to align with what is required for an XML guide file and writes it out blended together with the original XML content that was downloaded from Gracenote.

Once that final file is written, I do one global search and replace on everything, changing any remaining instances of “zap2it.tmsimg” to “zpmc.tmsimg” so that URLs that point to channel or show image files are correct and accessible. This allows my media systems to correctly load these images for the graphical portion of the Guide / What’s On content in the clients.

It takes a bit of extra effort, but everything is fully automated to do all of the heavy lifting for me and I end up with a reasonably well-detailed program guide for my media systems that shows me everything I need to see. And, the DVR functions can trigger correctly to record content when it airs!

What Comes Next

The next steps for me are to continue to monitor my setup, watch for timeouts or other errors that indicate the guide data isn’t pulling, and keep the overall system operating as needed. If I find that the Zap2It container tool stops working again because of another major URL rewrite by Gracenote, then maybe I’ll move to a paid subscription offering for a guide. No matter how it turns out, though, I will continue to use the XML format simply because I don’t have to make any changes to me media systems directly – as long as I continue to provide a file that they can read, it “should” continue to work as I want it to.

What does your setup look like? Let me know in the comments!

Leave a Reply

Your email address will not be published. Required fields are marked *