Promise chain with subtle asynchronous misordering

357

June 30, 2017, at 12:37 PM

A promise chain is useful for not over-writing a stream in node (see Bergi's comment). But here's an example that in subtle asynch circumstances produces mis-ordered writes on the write stream:

var fs = require( 'fs' );
function WTest() {
  this.writeStream = fs.createWriteStream( 'junkTest.txt' );
  this.readyToWrite = Promise.resolve( true );
};
WTest.prototype.writeNext = function writeNext( text, callNum ) {
  this.readyToWrite = this.readyToWrite
    .then( status => writeToStream( text, this.writeStream, status, callNum ) );
}
var wTest = new WTest();
for ( var i = 0; i < 10; i++ ) wTest.writeNext( 
  `${i}, some line of text that might be xxxxxxxxxxxxxxxxxxxxxxxx long \n`, i );
function writeToStream( text, writableStream, readyToWrite, callNum ) {
  return new Promise( ( resolve ) => {
    if ( readyToWrite ) resolve( writableStream.write( text ) );
    else writableStream.once( 'drain', () => resolve( writableStream.write( text ) ) );
  } );
}

If you run it as is in node 7.8, it works. But in complicated asynch circumstances, it fails, e.g. one line is written out of order. What's asynchronously dangerous about the above code? Can you create an asynch example where it fails?

My instance of failure includes multiple async file reads with the above code serving a logger function. I've verified that the calls to writeNext are arriving in the expected order by writing them to the console immediately after the call to the writeNext function. As mentioned in a comment, I've written with both (both direct to the same file), with the direct "just write" write (-) immediately after the promises write (=). The line sequence in the output file is 1-,1=,2-,3-,4-,5-,6-,7-,8-,2=,7=,3=,4=,5=,6=,9-,9=,10-,10=

Big picture wisdom from Bergi: "The only way to prevent over-writing is to stop generating new chunks." So the above is only of technical interest. Creating a promise chain worsens the memory cost of buffering.

Answer 1

The line sequence in the output file is
1-,1=,2-,3-,4-,5-,6-,7-,8-,2=,7=,3=,4=,5=,6=,9-,9=,10-,10=

The code from your question won't do that when there's only a single WTest instance and nothing else writes into that file.

What's asynchronously dangerous about the above code? Can you create an asynch example where it fails?

The only problem I see is that you might miss a drain event since you only start listening when there's another chunk to write instead of immediately after you got false, but that would lead to a forever-hanging stream and not to lines written out of order.

Better:

function WTest(filename) {
  this.writeStream = fs.createWriteStream(filename);
  this.readyToWrite = Promise.resolve();
}
WTest.prototype.writeNext = function writeNext(text, callNum) {
  this.readyToWrite = this.readyToWrite.then(() => {
    const status = writeStream.write(text);
    if (!status)
      return new Promise(resolve => this.writeStream.once('drain', resolve));
  });
};