Implemented first version of the asReader interface on IString

[DavyLandman]

This implements the addition discussed in #71

I've added some tests, hope they are enough.

[github-actions]

Test Results

     99 files  ±0       99 suites  ±0   5m 29s ⏱️ -4s
242 304 tests +2  242 303 ✅ +2  1 💤 ±0  0 ❌ ±0 
726 999 runs  +6  726 996 ✅ +6  3 💤 ±0  0 ❌ ±0 

Results for commit f7d591e. ± Comparison against base commit 38556b4.

♻️ This comment has been updated with latest results.

[jurgenvinju]

The tests are very good because many random values will be used.

[jurgenvinju]

Nice to have progress on this! Thanks. I think this demonstrates exactly what is needed.

The reader for indented values requires rethinking without the use of the IntStream. The reader should at least be able to stream entire lines, up too and including the newline character. But larger buffers are preferable. One typical use case here is large json objects serialized as strings with indentation. So if we can cram several lines into buffers up to 8k that will surely help with server response times.

With respect to tests; I'm pretty sure the random string generator does not play with indent yet, or with deep concatenation of long strings. To test this better we could either generate some examples ourselves, or extend the default random generator for strings..

[jurgenvinju]

Could use some comments to explain that read is allowed to return fewer chars than Len.

[DavyLandman]

Ah okay, I didn't want to repeat the manual of read, but indeed, at least 1 char is enough.

[jurgenvinju]

I think this is correct, although I see we need to work hard with the surrogate pairs. Maybe If we would always read until len - 1 unless the last character is a newline, and the int iterator always provides codepoints, there are no further special cases.

[jurgenvinju]

But before we merge we should find an implemention that can say least stream line by line. I suspect that the current experiment with the int iterator will slow down the stream to the point that it doesn't make sense to stream anymore.

[DavyLandman]

Well, we need to do work with surrogate pairs since every int the iterator returns can be either 1 or 2 chars. I initially had some code that would write 2 chars if there was room, but that just lead to duplication of logic with the queedLowSurrogate field. So I rewrote that to be a bit less wordy.

[jurgenvinju]

To make indented write easier i distributed indentedWrite methods over all implementation classes. Possibly an indentedRead method would help as well in this case. For larger buffers than 1.

[jurgenvinju]

Here we allocate a reader at every level in the tree and every node in every level. For the writer this design didn't hold up because of the enormous amounts of concat nodes in some applications. Typically we have at least as many concat nodes as lines in the output.

So let's have a look at this and also use the benchmarks in the main method so we can fine-tune it. Typically a recursion is ok for the concat nodes and maximally log(n) allocations (a spine of reader objects that moves over the tree).

[DavyLandman]

The problem is that we have to keep state, the writer was easier as it's push, while the reader is pull. I tried to do it in a similar way as the writer, but could not in a reasonable timeframe figure it out. We need co-routintes ;)

[jurgenvinju]

This algorithm is what we ended up with to avoid allocation of iterator or stream objects for every node in the balanced binary tree:

@Override
        public OfInt iterator() {
            return new OfInt() {
                final Deque<AbstractString> todo = new ArrayDeque<>(depth);
                OfInt currentLeaf = leftmostLeafIterator(todo, LazyConcatString.this);

                @Override
                public boolean hasNext() {
                    return currentLeaf.hasNext(); /* || !todo.isEmpty() is unnecessary due to post-condition of nextInt() */
                }

                @Override
                public int nextInt() {
                    int next = currentLeaf.nextInt();

                    if (!currentLeaf.hasNext() && !todo.isEmpty()) {
                        // now we back track to the previous node we went left from,
                        // take the right branch and continue with its first leaf:
                        currentLeaf = leftmostLeafIterator(todo, todo.pop());
                    }

                    assert currentLeaf.hasNext() || todo.isEmpty();
                    return next;
                }
            };
        }

If we turn this into an Iterator<AbstractString> where each AbstractString is guaranteed to be a leaf node, then that would work. Only have to check what the indented nodes do in this context.

[DavyLandman]

@jurgenvinju the new implementation uses the exact same style as the int iterators.

[jurgenvinju]

This is fast and useful. Hope we can apply it in the webservers to improve visuals and other features like salix diffs.