Thanks all for your replies! The only thing is now I feel like I need to keep the 70-200 4L IS and buy a 135L and a 70-300L ...
And that's leaving out the Tamron 70-300 and the 1200 ;D Lots of good points have been made for and against the various options, so plenty for me to think about.
To respond to some of the ideas people have raised ...
As much as I'd like to shoot birds and other wildlife, for now I'm resigned to not having anything longer than 200 (or perhaps 300 I suppose) in my kit at the moment. The 100-400 IS II sounds great, and obviously there are other options around, but for the time being I don't think a lens like that is what I'm looking for as a travel zoom for hiking/backpacking and I don't think I'd use it enough to justify spending the money. Over the last year or two I've been generally trying to reduce the amount of gear I have and make more use of what I have, than add more gear.
Marsu - I certainly understand what you mean about having both the f/4 and f/2.8 70-200s being too much duplication. It does feel a bit like that sometimes, but the difference in weight is significant enough it does make a real difference to me. The f/4's biggest drawback as a travel lens is, in my opinion, the fact it is still quite long.
Regarding the 70-300L - nice photo Georgecpappas! - I am going on a four day photography course just after Easter and I know one of the other people who will be there has a 70-300L. If I do nothing else, I'll try to spend a while playing with it then. At this point I'm still having trouble getting excited about the idea though. Compared with the 70-200 4, I get 100mm extra range and a lens which is easier to pack (since it's shorter), but at the expense of almost 50% more weight and a variable aperture which gets progressively slower at longer focal lengths (OK, it's only 1 stop at most, but still). The fact gregorywood bought a 70-300L to replace his 70-200 4L but is having trouble parting with the 70-200 isn't encouraging me either. Hhmmm.
I did think about the 200 2.8L prime, but I'd been leaning towards the 135L because I thought it added more to my kit overall - faster aperture, and should be a little easier to hand-hold noting neither has IS. I might give it a bit more thought.
I'd also thought about the option of a 70-300 non-L too. I decided against though after reading reviews. I don't think any are weather sealed and perhaps not even that robust, plus it seemed like they were all pretty ordinary optically ...?