OK found the answer. Looks like Silverlight only supports utf-8/utf-16 out of the box, so you have to write your own encoding support. (generate with http://www.hardcodet.net/2010/03/silverlight-text-encoding-class-generator)
The corrected code would be something like:
HtmlWeb web = new HtmlWeb(); //CustomEncoding is "iso-8859-2" CustomEncoding enc = new CustomEncoding(); web.LoadCompleted += new EventHandler<HtmlDocumentLoadCompleted>(_DownLoadCompleted); web.LoadAsync(baseAddress, enc);