The person in the middle, and both of them. The aisle passenger can lean into the aisle and the window passenger can lean into the wall. The middle passenger has nowhere to go, so the two armrests are theirs. I never had a passenger argue the point once it was explained, and most of them looked relieved that somebody had a rule.

The armrest is also a boundary. Whatever seat you are in, nothing of yours crosses it, and that includes your knees. If you are in the aisle or window seat and would like a turn on the armrest, ask. The middle-seat passenger will usually share, because they were asked. That is the whole difference.