Hope this helps someone having Dolby 5.1 streaming issues.
I record OTA TV using EyeTV and then compress the programs to stream from iTunes to my new AppleTV. My issue was first What to use as a preset to have 720P video and Dolby 5.1 audio. I tried the 720P setting in Turbo.264 HD software and had excellent picture quality but only stereo sound. I had recorded a program with Dolby 5.1 so I checked the EyeTV file and sure enough the audio was 5.1.
I tried many hardware things but nothing fixed the issue. I the looked at the compress version from the Turbo.264 HD software in QuickTime 7 player and check the "movie properties" under the Window menu. I found two audio tracks. a stereo ACC track and a Dolby 5.1 AC3 track. I deleted the stereo track and resaved the compressed version as a self-contained movie. I added it to my iTunes library and when streamed plays the Dolby 5.1 audio. So for those having trouble with videos you have compressed for Apple TV 2, it seems that when two audio tracks are present Apple TV defaults to the stereo track or perhaps my MacBook Pro is triggering that since it does not have an optical out connected to the audio jack. There you go for what it is worth.