[Portaudio] Write to Disk Performance - what is your opinion?

[Ross Bencina]

On 2/03/2012 6:33 PM, Peter Shaw wrote:
> What do you think is better: writte every recorded buffer to disk, or
> buffer the encoded buffer and write this larger buffer to disk an
> once? (Maybe all 30 seconds?)

I see it this way:

It's always a write-behind scheme. PA is filling a ring buffer at 
audio-rate, then maybe you're encoding it, then you're writing it to 
disk. Because you don't want block on i/o operations, the disk access 
should be asynchronous to encoding and encoding could be asynchronous to 
capture (although you could also do synchronous encoding in the PA 
callback if you're only encoding one stream at a time).

- capture must never get corrupted by temporary delays in encoding and 
disk io

- encoding happens as fast as possible

- disk access happens as fast as possible.

but because disk access (and maybe encoding?) don't have predictable 
timing behavior, you have one or more buffers (perhaps ring buffers) 
that allow each stage to fall behind the previous stage up to the limit 
of the buffering.

For maximum reliability you want enough buffering to soak up all 
anticipated system timing delays (disk latency, CPU/thred scheduling 
issues that might delay encoding etc, slower encoding on a fully loaded 
system etc).

In this view, I don't think your question makes complete sense. You will 
need buffering. Maybe 30 seconds is a bit extreme, but it doesn't sound 
unreasonable to me, for a super-reliable system. But at the same time 
you should always write to disk as soon as possible -- you might want to 
use a disc write size that is optimised for I/O (e.g. I think SATA ioops 
have a certain buffer size maybe), I use 64Kb as the default disk write 
size for example.. but this is independent of the amount of in-RAM 
buffering that can soak up temporary delays/outages in disk writing.

So to me, the real question is: what's the maximum required fill level 
for a disk write-behind buffer. You'd need to know things like what will 
happen to your disk throughput if the system goes on a page thrashing 
frenzy.

If you can afford the ram, 30 seconds sounds fine to me. But try to keep 
the buffer empty, not full. I would write every encoded frame, or 
perhaps every 64 kb, whatever works best. Maybe there are also bit 
torrent considerations.

Ross.