stream.push() after EOF at readableAddChunk

[ankiit]

I'm facing following error with readable-stream 2.2.9

Error: stream.push() after EOF at readableAddChunk (_stream_readable.js:156:17) at RedisRStream.Readable.push (_stream_readable.js:134:10)

What can be possible reasons causing this behavior?

[ankiit]

It's closing the stream after 16kB. There is some logical error after Add Range support commit.

[jeffbski]

@ankiit Thanks for reporting this. Do you know of a way that we can reproduce this so I can create a test and then a fix for it?

[thomas-lee]

I have encountered the similar problems but the data itself has no pattern.
Some is working properly with a very large data.

at emitOne (events.js:96:13)
    at RedisRStream.emit (events.js:188:7)
    at readableAddChunk (_stream_readable.js:157:14)
    at RedisRStream.Readable.push (_stream_readable.js:134:10)
    at Command.callback (/Users/thomas1lee/Documents/git/node_modules/redis-rstream/lib/redis-rstream.js:56:14)
    at normal_reply (/Users/thomas1lee/Documents/git/node_modules/redis/index.js:662:21)
    at RedisClient.return_reply (/Users/thomas1lee/Documents/git/node_modules/redis/index.js:722:9)
    at JavascriptReplyParser.returnReply (/Users/thomas1lee/Documents/git/node_modules/redis/index.js:146:18)
    at JavascriptReplyParser.run (/Users/thomas1lee/Documents/git/node_modules/redis/node_modules/redis-parser/lib/javascript.js:137:18)
    at JavascriptReplyParser.execute (/Users/thomas1lee/Documents/git/node_modules/redis/node_modules/redis-parser/lib/javascript.js:112:10)
    at Socket.<anonymous> (/Users/thomas1lee/Documents/git/node_modules/redis/index.js:223:27)
    at emitOne (events.js:96:13)
    at Socket.emit (events.js:188:7)
    at readableAddChunk (_stream_readable.js:176:18)
    at Socket.Readable.push (_stream_readable.js:134:10)
    at TCP.onread (net.js:547:20)

[rdy]

I'm definitely running into the same error, the content is getting cut off at exactly 16384. It appears you cannot stream values back from redis that exceed this size, the error kills the stream.

To reproduce this, just SET a key with a value larger than 16384 bytes. You'll see if you read it back you'll never get the complete stream it will cut off at 16384.

[tarekrached]

I think this issue cropped up between redis-rstream 0.1.2 and 0.1.3. 0.1.2 is fine for me, but I'm able to repro with redis-rstream 0.1.3.

I'm seeing this same error on some large values - for me it's truncating at around 81k. I have multi-MB values that I'm able to stream fine, so I'm not sure what's going on.

events.js:182
      throw er; // Unhandled 'error' event
      ^

Error: stream.push() after EOF
    at readableAddChunk (_stream_readable.js:243:30)
    at RedisRStream.Readable.push (_stream_readable.js:211:10)
    at Command.callback (.../node_modules/redis-rstream/lib/redis-rstream.js:56:14)
    at normal_reply (.../node_modules/redis/index.js:726:21)
    at RedisClient.return_reply (.../node_modules/redis/index.js:824:9)

This shows up on redis 2.8.23 and 3.2.4, both on AWS ElastiCache.

This may be the underlying cause of dvonlehman/express-request-proxy#24, which is how I encountered it.

[Kasher]

@tarekrached , I was able to reproduce it in 0.1.2
That's the stack:

uncaughtException: Error: stream.push() after EOF
	at readableAddChunk (_stream_readable.js:133:17)
	at RedisRStream.Readable.push (_stream_readable.js:111:10)
	at Command.callback (/build/node_modules/redis-rstream/lib/redis-rstream.js:46:14)
	at normal_reply (/build/node_modules/redis/index.js:714:21)
	at RedisClient.return_reply (/build/node_modules/redis/index.js:816:9)
	at JavascriptRedisParser.Parser.returnReply (/build/node_modules/redis/index.js:188:18)
	at JavascriptRedisParser.execute (/build/node_modules/redis/node_modules/redis-parser/lib/parser.js:574:12)
	at Socket.&<anonymous> (/build/node_modules/redis/index.js:267:27)
	at emitOne (events.js:77:13)
	at Socket.emit (events.js:169:7)
	at readableAddChunk (_stream_readable.js:153:18)
	at Socket.Readable.push (_stream_readable.js:111:10)
	at TCP.onread (net.js:540:20) "}	

Was anyone able to fix this issue?

[rdy]

I'm locked down to the older version which still works correctly. But this module is probably outdated with redis adding proper stream API http://antirez.com/news/114.

[baldurh]

@rdy The new Stream data structure is unrelated to Node streams.

[farhumaglass]

This PR should solve the issue, or at least not bring the whole process down due to uncaughtException
#5

[ikappas]

@jeffbski Is there any update on this? I also hit this issue on heroku:

Error: stream.push() after EOF
    at readableAddChunk (_stream_readable.js:240:30)
    at RedisRStream.Readable.push (_stream_readable.js:208:10)
    at Command.callback (/app/node_modules/redis-rstream/lib/redis-rstream.js:56:14)
    at normal_reply (/app/node_modules/redis/index.js:726:21)
    at RedisClient.return_reply (/app/node_modules/redis/index.js:824:9)
    at JavascriptRedisParser.returnReply (/app/node_modules/redis/index.js:192:18)
    at JavascriptRedisParser.execute (/app/node_modules/redis-parser/lib/parser.js:553:10)
    at Socket.<anonymous> (/app/node_modules/redis/index.js:274:27)
    at emitOne (events.js:116:13)
    at Socket.emit (events.js:211:7)Exception

[ikappas]

I seem to have solved my issue.

When calling redis.createClient I had to add the detect_buffers: true option.

According to the redis-rstream readme under usage:

...Be sure to enable an option in your redis client to return Buffers for the data, like detect_buffers: true so that binary data will be read properly...

[farhumaglass]

@ikappas not sure this will resolve your issue as I'm using the detect_buffers: true option, and it still occurs randomly.

[jeffbski]

Is this still an issue for anyone? If so could you give me a test and any ideas on how I might implement a fix if you believe you know how. Thanks.

[danilaplee]

I'am getting this issue every time when using docker node:14.16.1-slim,
thought i don't get this issue on my local env windows 10 node 14
"redis-rstream": "^0.1.3"

Error [ERR_STREAM_PUSH_AFTER_EOF]: stream.push() after EOF
    at readableAddChunk (internal/streams/readable.js:272:30)
    at RedisRStream.Readable.push (internal/streams/readable.js:223:10)
    at /usr/app/node_modules/redis-rstream/lib/redis-rstream.js:56:14
    at tryCatcher (/usr/app/node_modules/standard-as-callback/built/utils.js:12:23)
    at /usr/app/node_modules/standard-as-callback/built/index.js:22:53
    at processTicksAndRejections (internal/process/task_queues.js:93:5)

import * as Redis from "ioredis";
const redis = new Redis(config.redis);
const redisRStream = require("redis-rstream");

router.get("/files/:key", async (req, res) => {
  const cKey = "prefix"+ req.params.key;
  redis.exists(cKey, (err, exists) => {
    if (err) {
      console.error("==== redis_error =====", err);
      return res.status(500).send({ error: "redis_error" });
    }

    if (exists) return redisRStream(redis, cKey).pipe(res);

    return res.status(404).send({ error: "not_found" });
  });
});

if I downgrade to "redis-rstream": "0.1.2" the issue happens 1 out 20 times