Wednesday, June 18, 2008

How to download webradio and webtv, RealMedia (RTSP) or Windows Media (MMS) streams easily - a comprehensive ripping tutorial

Now that it's once again time for P1 Sommar (check out my talker schedule!) for me it also means it's time to freshen up the streaming media dumping skills. I happen to enjoy my webradio with music left in rather than the crippled music-less media files Swedish Radio / SVT provide. Below I in three (an optional fourth) steps show that there is little practical difference between providing something through a streaming media protocol or providing it for download - it is pretty easy to download and save it for later anyway.

The case is pretty simple - when you listen to webradio or webtv, the browser window commonly embeds a player which is given the URL of a "playlist" provided over regular HTTP. The playlist however contains the URL of the actual media stream, commonly "hidden" to only be accessible using a streaming media protocol like RTSP or MMS. Media providers think (or hope, or assume, it doesn't really matter) that players able to receive these protocols will not allow the user to save the media stream to be replayed later and thus are more relaxed about letting licensed content be broadcasted through streaming media protocols. Fortunately using an open-source player like MPlayer or a download manager like FlashGet, also media streams can be saved like a charm.

The gray areas of radio ripping is discussed over at copyriot.se (in Swedish) and also don't miss out my list of great SR and SVT shows with all URLs conveniently in one place. Curiously this is not a simple case of piracy - the DMCA would surely be quick to decide, but there are plenty of arguable fair use purposes for stream ripping such as saving the content for replay later, bringing it with you on a portable player or converting it into a suitable format. The case is especially strong for content which is anyway broadcasted over analog radio or TV and as such are anyway freely available. However, IANAL, do this on your own risk and don't break any applicable laws.

  1. Open the source code of the page or frame the regular player is embedded in. Get the playlist SRC by looking for the player object parameter "PARAM NAME="SRC" VALUE="...". However, what comes after "VALUE=" commonly does not start with "http://" and thus is a relative URL, see the next point for where to find the base location to be able to download the playlist. Update: You can also try right-clicking on the Real Player embedded in the page and selecting "Copy clip URL" instead of even touching the source code




    If the playlist SRC above was a relative URL, get the base location from the location displayed in the frame or page info:



  2. Add the location and the playlist name together to download the playlist from that URL. The playlist then contains the stream URL at a line saying "src=":
    $ wget -O "030702 Mustafa Can.ram" "http://www.sr.se/webbradio/include/CreatePlaylist.asp?Type=db&Id=1115295&broadcastDate=&IsBlock=0&AudioFormat=1&AvailableAudioFormats=1"
    $ grep src "030702 Mustafa Can.ram"
    <audio src="rtsp://lyssna-rm.sr.se/isidor/p1/2008/04/47_webb_rh_sthlm_mustafa_cans_sommar_2_jul_p1-063383a05_376ff8c2_r2.rm"/>

  3. Download the stream with a program capable of receiving the protocol at hand (see wikipedia lists for rtsp:// and mms://, though I know for a fact some download programs like FlashGet for Windows or CocoaJT for Mac (in French ?!) and others listed here are also very capable of that). Below I use MPlayer, which works the same both for Linux, Windows and Mac OS X (except under Windows the binary is mplayer.exe, and under Mac OS X you should use /Applications/MPlayer OS X 2.app/Contents/Resources/mplayer.app/Contents/MacOS/mplayer ):

    $ mplayer -dumpfile "030702 Mustafa Can.rm" -dumpstream "rtsp://lyssna-rm.sr.se/isidor/p1/2008/04/47_webb_rh_sthlm_mustafa_cans_sommar_2_jul_p1-063383a05_376ff8c2_r2.rm"

  4. If you want to, edit the audio with Audacity, convert it to mp3 with lame or even iTunes. Video you can edit with JayCut and convert with FFmpeg or any from the jungle of mediocre video conversion tools there are out there, like Handbrake for Mac OS X or Videora iPod Converter for Windows.

    A note about file types, with RealMedia the playlist is a .ram file and the audio or video contained in .ra, .rv, .rm or .rmvb streams. With Windows Media, the playlist file is commonly an .asx or .asf file containing a .wmv or .wma stream. Also, in an .asx file, the stream address is found by the href parameter. I have not looked currently looked into ripping other formats such as Quicktime or Flash video.


And friends, remember sharing is caring!

(Update: I found out that my MPlayer OS X is both a bit outdated and while it has RTSP-streaming support, it doesn't seem to have the RealMedia codec. So to be able to decode the .rm to .wav on my mac before I clean it up in Audacity and export to wav, I found out the monster package of codes FFmpeg (it's Mac gui sister program is the somewhat less reliable FFmpegX) of course does it excellently like this:
$ ffmpeg -i foo.rm foo.wav
I also checked out the blogger exertia who seems to like Videolan/VLC, but while it's better maintained than MPlayer OS X, it didn't help me this time)

8 comments:

Oberg said...

A comprehensive guide indeed! I've been looking for something like this for a while now, and I'm sure so have others. Thanks!

(PS: your link to your favourite shows on SR/SVT is broken; there's a
there that messes it up :P)

Carl-Johan Sveningsson said...

(Why didn't I receive an email about your comment?)

Thank you, and you're very welcome!

I'd like to think myself it's pretty good and compact, but so link to it as well so that more can find it then! :-)

Osynlige Mannen said...

Very helpful indeed! What surprised me was how easy it's done, once you know how it's done. Right click and there you are. Now I understand all the stories about teen-agers hacking military systems. Sure, those kids were clever but the defenses weren't high either. Thanks a bunch, now I can download the stuff the kids wanted.

Carl-Johan Sveningsson said...

@O.M. That's very cool! Yes, it is very simple stuff, but the technicalities can confuse many anyway. I could update this post with my scripts now by the way, and I can also leech SVT Play, and proxy to seem like I am still in Sweden (or whatever)

My motivation for documenting this is not primarily to break (stupid) licenses, but to indicate how easy the annoying measures are to circumvent and thus that material should be provided in the easiest and best way possible. Or I will help people do it themselves...

Carl-Johan Sveningsson said...

Just wanted to mention that I fully intend to update this post, now that svt play has switched to rtmp streaming of its flash video, I think also a nicely crafted VLC commandline is the simplest way to process things, but that remains to see

Emanuel said...

Can anyone tell me if it's possible somehow to rip a svt stream that due to legal restrictions can't be played outside of Sweden? (as in the case of true blood, konflikt etc.) Can I still get access to the video URL somehow?

niz said...
This comment has been removed by a blog administrator.
Sildenafil Citrate said...
This comment has been removed by a blog administrator.