sftp-server 'ls' command takes too long to execute

[Main-Tomi]

Version

2.14.0

Bug description

When the account directory file has a thousand tickets (all small files), it will be very slow when in ‘ls’ directory, taking about 30-40 seconds.

Actual behavior

When there are more and more files, the time for ‘ls’ is also very long. In actual production environments, as long as the number of files exceeds 3000, the more time will be spent.

Expected behavior

1, Hope to refactor the relevant methods to improve the speed of 'ls' files
2, Is there a relevant API that can modify the value of the segmented upload buffer? Its default value now is 16 * 1024.

Relevant log output

No response

Other information

No response

[Main-Tomi]

Through testing, it was found that the slow loading of file directories is due to the following method（class is: org.apache.sshd.sftp.server.AbstractSftpSubsystemHelper）：
The first place：Map<String, ?> attrs = resolveFileAttributes(f, SftpConstants.SSH_FILEXFER_ATTR_ALL, !followLinks, options);, it takes 6-7 milliseconds each time.
Second place: String longName = getLongName(f, shortName, options));, it takes 5-6 milliseconds each time.

protected void writeDirEntry(
            int id, DirectoryHandle dir, Map<String, Path> entries, Buffer buffer,
            int index, Path f, String shortName, LinkOption... options)
            throws IOException {
        boolean followLinks = resolvePathResolutionFollowLinks(SftpConstants.SSH_FXP_READDIR, "", f);
        Map<String, ?> attrs = resolveFileAttributes(
                f, SftpConstants.SSH_FILEXFER_ATTR_ALL, !followLinks, options);
        entries.put(shortName, f);

        SftpFileSystemAccessor accessor = getFileSystemAccessor();
        ServerSession session = getServerSession();
        accessor.putRemoteFileName(this, f, buffer, shortName, true);

        int version = getVersion();
        if (version == SftpConstants.SFTP_V3) {
            String longName = getLongName(f, shortName, options);
            accessor.putRemoteFileName(this, f, buffer, longName, false);

            if (log.isTraceEnabled()) {
                log.trace("writeDirEntry({} id={})[{}] - {} [{}]: {}", session, id, index, shortName, longName, attrs);
            }
        } else {
            if (log.isTraceEnabled()) {
                log.trace("writeDirEntry({} id={})[{}] - {}: {}", session, id, index, shortName, attrs);
            }
        }

        writeAttrs(buffer, attrs);
    }

[tomaswolf]

You wrote in your PR that the change brings a speed improvement of a factor 2. Above you stated 1000 files taking 30-40 seconds, so now we'd be at 15-20.

Property SftpModuleProperties.MAX_READDIR_DATA_SIZE can be set on the session to change the buffer size for ls.

What's the client? Also MINA sshd? Or something else?

[Main-Tomi]

Hi @tomaswolf,

Yes, the speed of 'ls' has increased by half now. Would it be possible to release a temporary version first, please?

SftpModuleProperties.MAX_READDIR_DATA_SIZE can change the buffer size for 'ls', but can not increase speed.

We don't know which client the customer is using, so we may need to consult