Understanding PSIP Data as part of Digital OTA ATSC TV

[videobruce]

I use to have that MyHD-120 tuner card.
That allowed the option of NOT using channel mapping. I assume the current MyHD-130 will allow the same thing.

[stampeder]

Who makes that one? That's a valuable option. With my 2 pcHDTV cards I can address the channels without PSIP remapping, but I have to do that outside MythTV so I don't bother.

[stampeder]

Let's be clear about what "locking" is - when an ATSC tuner has all the PSIP data (arranged in tables) it needs to identify the station by its name, call sign, frequency, original channel, remapped channel, etc. etc. it can then process the AV data stream because it now knows where to look. It begins sampling the data stream looking for headers that identify the data packets so that they can be processed in order. If the data stream has less than "n" numbers of data errors along with a high Signal to Noise Ratio your ATSC tuner will "lock" onto the station and you will have ATSC TV in all its audio and video glory.

If you are scanning frequencies and a channel shows up with high signal strength, that's great but its not a lock until the above takes place.

[kirok1701]

Stampeder, is PSIP such a beast because of the inherent complexities of digital broadcasting, or because the specification was overengineered? I had a look at a few sections of the spec a while back, and it does seem rather complicated, but I'm unsure whether that level of complexity is merely par for the course.

[stampeder]

kirok1701, as far as PSIP's complexity, one of the problems is that if you affect the tables of certain data you might unfortunately affect the tables of one or more other types of data (similar to the way a relational database works). Also while some of the data tables are mandatory, some have been decreed as voluntary, so figuring out if there are any dependencies between them is part of the process.

Accuracy of the PSIP data is very important. If it is out of whack with other stations, it might be causing the consumer's ATSC tuner to keep resetting its internal clock and/or other data, or causing overlap of remapped virtual channels, as we saw with the effect of CIII-DT 2.1 accidentally stepping on WGRZ-DT's own 2.1 remap. Getting the data right is critical.

Familiarity is another factor. There are no standardized tools for implementing PSIP across different brands of gear, so there isn't a large group of PSIP practitioners out there who can come in as hired guns and do the work on contract. PSIP tools are vendor-based, as in the case of Global and CBC using Tandberg.

Private products like PSIP Pro showed up late in the U.S. digital cutover process to have much effect, but industry people I've showed this product's web site to were very interested: http://www.dtvinnovations.com/Overview.htm The big caution about any such tools as PSIP Pro is that they still require the data to be previously understood, collated, and implemented according to plan.

[kirok1701]

So basically it boils down to a combination of lack of implementation experience (both for Global in particular and in general with Canadian broadcasters), the absence of an adequate, mature toolchain for the protocol, and the spectacularity of failures when such critically sensitive data are wrong or misconfigured? Right. Thanks, Stampeder; very helpful.

Offhand I share your sentiment (from the Global PSIP thread): I hope the good folks at Global can get it figured out good and proper so that they can kick back and all can revel in the fruits of their labours. I've often complained privately about CanWest taking so long to begin digital broadcasts, but I'm far from inflexible: now that broadcasts have begun I am most appreciative. It's unfortunate that they've had such growing pains: hopefully technologies such as PSIP Pro will be helpful if (when, hopefully!) they choose to launch other digital broadcast endeavours.

[danbcman]

Really clear info:

http://splicer.foxpico.com/Manuals/ATSC_TECH_25.pdf

[WSYRTV9]

Originally Posted by stampeder

The Master Control centre in Calgary pipes the signal to the local digital stations over fibre land lines, who receive it on a Tandberg Transport Stream Processor box that they use to locally process the signal (overlays, etc.). That Tandberg box requires PSIP to be added using Tandberg's own proprietary tool. The finished product signal is fed into the local ATSC exciter for transmission.

We have been doing DTV for about five years with mainly Tandberg equipment... this seems like a remarkable amount of difficulty.

The TT6120 family of transport stream processor is a flexible device: it has a slot for an optional input interface, and a slot for an optional output interface. Its native format is ASI, and has ASI input and outputs. So generically, it takes whatever format you are presenting and converts it to ASI, does whatever packet and table manipulation is necessary, produces an ASI output, then converts that processed ASI to whatever optional output format might be present.

In a typical application (for example, going from our Syracuse NY regional hub to WHAM-DT in Rochester) we have a 6120 with a DS3 output that sends their feed out to the fiber; at their studio there's a second 6120 with a DS3 input card and a SMPTE 310 output that feeds the microwave link; at the transmitter there's a third 6120 with SMPTE 310 input and SMPTE 310 output that buffers and de-jitters the feed before it hits the transmitter. At any rate, we've been using them to ship DTV all over the state, through DS3 links as well as ATM networks.

We do all the MPEG encoding and PSIP insertion at the central facility... and there's no reason you necessarily need to use Tandberg equipment ahead of the processor. We use Triveni Guidebuilder PSIP generators for all of the stations, and most stations have a Logic Innovations multiplexer to combine all of the encoders with the PSIP.

We have always added PSIP prior to the 6120s, so it's never been necessary for us to get into unusual configurations... the processors just sit there and work with default settings. The only time we have an issue is when the PSIP generator decides to spit out something crazy, and then we reboot the processors to flush out any invalid PSIP that they might be buffering. Happily, that's pretty rare.

One problem we have observed with some Tandberg equipment: some of their encoders have an optional remultiplexer card that you can use to combine external PSIP with that encoder's feed. We've found that the remux card won't run properly unless the encoder is set to program 3... so it's much simpler to use an external multiplexer like the Logic Innovations that lets you re-map the packets from each encoder to their proper addresses.

By way of background, the program number essentially forms the base for the packet ID's used for each component of a program stream. Each stream uses a block of 16 addresses (PIDs) -- so program 2 ranges from PID 32 (decimal) to PID 47; program 3 ranges from PID 48 to 63; program 4 ranges from 64 to 79... and so on. The PIDs corresponding to programs 0 and 1 have always been reserved... and program 2 became "off limits" several years ago. It's a bit confusing because some people equate the virtual channel number with the program number... but they're different. For WHAM-DT, virtual channel 13-1 is actually program 3; virtual channel 13-2 is program 4.

Anyway, if you have several encoders that you need to combine into a single DTV channel, you need to either have them producing data at the correct PIDs in the first place, or your multiplexer needs to be smart enough to remap them: for instance, if both encoders are producing packets corresponding to program 3, the mux translates the feed from one of them up to program 4 PIDs. And then it wraps in the feed from the PSIP generator.

Someone had made an observation about not using up bandwith adding PSIP at the main feed point... that seems rather odd, since PSIP occupies a minimal amount of bandwidth compared to the audio and video. We find it much simpler to create the complete feed -- encoded programs and PSIP -- in one spot, and transmit the whole package together. For one thing, it lets you use standard processing and test equipment at every point, and lets you use an ordinary receiver/decoder to check your stream along the transmission path...

This is pretty convoluted stuff, and I'm really tired this morning -- so this is probably about as clear as mud...

-- Jeff

[danbcman]

Jeff as you say:

We have been doing DTV for about five years

In my understanding there inlies the trouble for many here it is really new to the staff that have been given the task of this new system to them I see it like writing with the other hand knowing what you want and getting is a learning process but I am sure one day it will be old hat for them too. I learned lots from you info and I am glad you took the time to share it.

[Schmerm]

How important is it that the first program start at Program #3, and what are #0, #1 and #2 reserved for? I looked through all the Toronto stations and saw that City-DT was using Program 2.

[stampeder]

One of the Vancouver stations uses Program 8... maybe its just arbitrary? I haven't looked it up.

[WSYRTV9]

Originally Posted by stampeder

One of the Vancouver stations uses Program 8... maybe its just arbitrary? I haven't looked it up.

Within reason, it is fairly arbitrary. Program 0, which doesn't actually exist, would equate to PIDs 0-15... since the Program Association Table (PAT), which lists all of the programs present in the stream, is fixed at PID 0, putting something down there isn't an option. I don't know why programs 1 and 2 are taboo, unless it's to maintain compatibility with non-broadcast systems.

In simplest terms, a receiver looks first at PID 0 -- the PAT -- to find out what programs exist. Taking WHAM-DT as an example, the PAT will tell the receiver that there's a something at program 3 and program 4. Each program in turn consists of a block of 16 PID addresses, the first of which is the Program Map Table (PMT), which defines the parameters for that particular program. For program 3, the PMT is PID 48; for program 4, it's PID 64. Typically, the next PID up from the PMT (49 for program 3, 65 for program 4) contains video; four up from the PMT (52 for program 3, 68 for program 4) is the main audio. You can specify a PID in a program's block for secondary audio, or teletext, or any number of other applications.

At the upper end of the PID range -- PID 7,424 through 7,427 -- are the PSIP tables that more neatly describe the services, tables for program guides and so forth. Since PIDs are allocated in blocks of 16, that leaves room for over 450 separate program streams. Seems like overkill, but there's no reason I can see why you can't assign whatever program numbers are convenient.

I suppose if you have really large facility that feeds a number of different stations different combinations of similar content, you could have an encoder for each service, each set to a unique program number. For instance (I'm making this up):

• Program 3: CBC HD English Eastern Time
• Program 4: CBC HD French Eastern Time
• Program 5: CBC HD English Central Time
• Program 6: CBC HD French Central Time
  
• Program 21: Weather, English
• Program 22: Weather, French
• Program 23: Montreal local, French
• Program 24: Ottawa local, English
• Program 25: Ottawa local, French

A given station would simply multiplex the several programs they want to carry with the PSIP appropriate to that one station. So a mythical DTV channel 26 for English programming in Ottawa might consist of 26-1, which would be program 3, 26-2, which would be program 21, and 26-3, which would be program 24. I have no idea if that's how it's actually being done, but it would certainly be an easy-to-implement model. Please excuse my south-of-the-border ignorance...

This is definitely a steep learning curve... and please don't take my earlier comment as any sort of criticism. My surprise was mainly that Tandberg wasn't more helpful in resolving the initial configuration issues. In my experience, they're great on the phone... but the equipment manuals are a really difficult read, partly because much of their equipment isn't specific to DTV, so you have to wade through a lot of options and settings that don't apply.

-- Jeff

[99gecko]

I have checked out how the local broadcasters are doing now since the latest DST change, and summarized the results with my previous observations. Previously I did not record the status of guide info, but based on recent observations I have started recording this as well. In the interest of tracking how the various stations are doing with fixing and maintaining their PSIP data over time I created a small spreadsheet to record the observations, download available here:

http://www.mediafire.com/?sharekey=b...1f53590efcaab0

These observations were made using a Samsung T-451. Not all tuners will get identical results to me.

Notes for the 04-NOV-2008 data set:

• OMNI 1- DT and CIII-DT were both off by -4.5 hours (interesting...),
• OMNI 2-DT was off by +26.5 hours. CITY-DT was off by -30 minutes.
• WUTV-DT was off by only 5 minutes so I considered that a pass.
• WNED-DT had Guide data, but it's time was offset (not sure by how much).
• CKXT-DT had guide data, but the PSIP data for the current show was not populated properly - when I selected [INFO], data for the current show was not there, but when I selected the [GUIDE], the data for the current show was present - I recorded this as a pass.

I will try to maintain this spreadsheet going forward. If other users wish to provide me with status info for stations I missed in the GTA (i.e. Hamilton etc.) I will include it. As well, if members from other areas across Canada want me to include data for their OTA stations I will be happy to do so. Please PM me with the data.

cheers.

[El Gran Chico]

Just curious, what is the purpose of the ds_status, ds_day_of_month, and ds_hour fields on the STT if the timezone and dst information is kept on the user side?

If the system_time minus the amount of leap seconds in gps_utc_offset is correct, can't the user device use its timezone/dst to calculate the time at the user?

We can't determine the time at the broadcaster since the timezone is not specified in STT... Or am I missing something?

[El Gran Chico]

^^^ I was able to answer my own question above ^^^

Not all DTV receivers have a Daylight Saving Time user setting, instead letting the broadcaster make this determination for them. Times such as system_time in STT and event_start_time in EIT all come in UTC so DTV receivers do need to know what time zone they are in if they are going to use anything with a time field.

ds_day_of_month and ds_hour are useful to navigate around DST transitions since EITs in the stream on the Saturday night need to reflect the post-change start times for Sunday's listings.