Interesting. It sounds like it might be swapping. You can actually check on what is going on by installing the sysstat package if it is not already installed and then run a command like this:
$ sar -o transcoding_activity_report 5
This will log all available system activity data at five second intervals, indefinitely. Once that starts, begin transcoding/streaming one of the smooth/non-buffering media streams and let it run for a minute or so to establish an activity baseline. Then, stop that and start one of the choppy/buffering ones and let it go for a minute or two, long enough to span several of buffering behavior cycles. Then you can hit Ctrl+C to interrupt sar and review all the data with:
$ sar -f transcoding_activity_report -A
Or to just look at the paging behavior in particular with:
$ sar -f transcoding_activity_report -B
I might be wrong about the cause of the behavior (paging) but the -A report should have some outliers and patterns that identify a subsystem or parameter experiencing activity correlated to the buffering which can help narrow down the source of the problem.
Let us know what you find out!