The real problem is trusting yourself because you trust yourself.
What it means to 'trust' PA is to say "PA can prove only things that are true". It is impossible within PA to prove that, but it is not inherently impossible to prove that outside PA. I can believe that "PA can prove only things which are true" and use that as one of my premises without much trouble, because it is true. (If it isn't, then PA+1 can prove anything)
The premise "I can prove only things which are true." is what is falsified by Lob's theorem- and it is falsified in the general case. What Lob's theorem doesn't prove is "No system can prove only things which are true."
In reference to ethics and meta-ethics, PA makes no claims. What we need is a system that has proven as many things as possible, while proving no false things. In order to identify that, we need to figure out how to falsify a meta-ethical, ethical, or moral statement.
Today's post, You Provably Can't Trust Yourself was originally published on 19 August 2008. A summary (taken from the LW wiki):
Lob's theorem provides, by analogy, a nice explanation for why you really can't trust yourself. Don't trust thoughts because you think them, trust them because they were generated by trustworthy rules.
Discuss the post here (rather than in the comments to the original post).
This post is part of the Rerunning the Sequences series, where we'll be going through Eliezer Yudkowsky's old posts in order so that people who are interested can (re-)read and discuss them. The previous post was The Cartoon Guide to Löb's Theorem, and you can use the sequence_reruns tag or rss feed to follow the rest of the series.
Sequence reruns are a community-driven effort. You can participate by re-reading the sequence post, discussing it here, posting the next day's sequence reruns post, or summarizing forthcoming articles on the wiki. Go here for more details, or to have meta discussions about the Rerunning the Sequences series.