University of Washington, Department of Electrical EngineeringEE 595A: Information Theory, Winter 2004 |
Topics will include entropy, mutual information, asymptotic
equipartition properties, data compression to the entropy limit
(source coding), communication at the channel capacity limit (channel
coding theorem), coding theory (including ECC and other modern codes),
method of types, differential entropy, maximum entropy,
rate-distortion theory, alternating minimization, and information
geometry in general. Additional topics will include information
theory as it is applicable to pattern recognition, natural language
processing, computer science and complexity, biological science, and
communications.
Prerequisites: basic probability, statistics, and random processes
(e.g., EE505 or a Stat 5xx class or consent of the
instructor). Knowledge of matlab. The course is open to students in
all UW departments.
Course Format: Four hours of lecture (MW 3:30-5:20
EE1-
042)
per week.
Final exam 2:30-4:20 p.m. Thursday, Mar. 21, 2002
Texts: We will use two texts including:
Course overview in PS and PDF formats, which gives more information (such as grading policy, other interesting texts, etc.).
(Feb 9th) The figure from today's lecture is online
(Jan 5th) Please fill out the following survey (PS or PDF) if you haven't already. .
Homework 2, due Wednesday, Feb 4th, in class. PS or PDF (solutions in PS and PDF)
Homework 3, due Friday, Feb 27th, at 4:30pm. PS or PDF (solutions in PS and PDF)
Homework 4, due Wednesday, March 10th, in class. PS or PDF (solutions in PS and PDF)
Course news group (it is listed as ee595, but this is our course news group, the last time the course was offered, it was ee595).
IEEE Information Society Home Page
IEEE Electronic Library and access to online version IEEE Transactions on Information Theory for UW locations.
Arithmetic coding for data compression, an article written in 1987 that has become the standard introduction on this topic.
Arithmetic coding revisited Written by the same authors as the previous article, this one written in 1998 provides an update to issues surrounding arithmetic coding.
Data compression, a survey of the field to 1987.
Bayesian networks for lossless dataset compression. Can Bayesian networks be used for compression? Read here and find out.
Modeling for text compression, a good survey on text compression. The same authors later wrote a book on the subject with the same title.
High Quality Document Image Compression with DjVu. by Léon Bottou, Patrick Haffner, Paul G. Howard, Patrice Simard, Yoshua Bengio, and Yann Le Cun, describes the DjVu method in detail.
Is Huffman coding dead?, with a title like this, you have to take a look.
The Miraculous Universal Distribution, Ming Li's introductory essay about Kolmogorov complexity and randomness. Also see Paul Vitanyi's extended randomness paper.
Quotations by Laplace about Kolmogorov complexity, but 100 years earlier.
David MacKay's online IT course