Two passages in the Philosophical Investigations have generated more philosophical commentary than almost anything else written in the 20th century. Both are arguments against a picture of language and meaning that most people find intuitively compelling.
The private language argument (sections 243-315) targets the following picture: I have inner experiences (sensations, feelings, thoughts), and I can give these experiences names that refer to them. The meaning of these names is fixed by my private mental act of attending to the experience and labeling it. This private vocabulary could, in principle, be a language only I could understand, because only I have access to the experiences it refers to.
Wittgenstein argues that this is incoherent. A genuine language requires criteria of correctness: the possibility of distinguishing between using a word correctly and merely thinking you are using it correctly. But in a purely private language, there is no such distinction. If I name a sensation "S" by privately attending to it, and then the next day I apply "S" to a new experience, I might be using it correctly (the new sensation resembles the original) or I might simply be misremembering what the original was like. There is no independent check. But then there is no genuine rule, no genuine meaning, no genuine language. "Whatever is going to seem right to me is right. And that only means that here we can't talk about 'right'." A rule that can always be interpreted to fit any case is not a rule at all.
The conclusion is not that inner experiences don't exist. It is that their language is not private: the words we use for sensations get their meaning from public criteria, from behavior, from the shared form of life in which pain-behavior, comfort-giving, and medical treatment are embedded. Language about the inner is still a public language.
The rule-following considerations (sections 138-242, and developed by Saul Kripke in his influential but controversial 1982 reading) raise a related puzzle. When you follow a rule, what determines that your continuation of the rule is correct? You learned to add by working through examples. But any finite set of examples is consistent with infinitely many rules. When you say "2 + 2 = 4" and then "1000 + 2 = 1002," what makes the second application correct rather than some other continuation? There is no fact about your past mental states that determines the answer, because any mental state is also interpretable in multiple ways.
Wittgenstein's answer: rule-following is not grounded in a private mental act or a logical necessity that you consult. It is grounded in practice and agreement: in the shared form of life in which the practice of mathematics is embedded. "Following a rule" is a practice, not a private performance. This is why a single person cannot follow a rule just once, in isolation from any practice or community: "to think one is obeying a rule is not to obey a rule. Hence it is not possible to obey a rule 'privately': otherwise thinking one was obeying a rule would be the same thing as obeying it."
These arguments have enormous implications beyond philosophy of language. For philosophy of mind: mental states cannot be fully characterized in purely private, inner terms. For mathematics: mathematical necessity is not Platonic nor purely logical but is grounded in human practices of calculation. For ethics: moral concepts get their meaning from human practices and forms of life, not from a private moral faculty tracking mind-independent moral facts.