Downloads on queue for hours

[DasHammett]

Downloading in aMule works for few MB (around 20MB) and then goes "on queue" for hours and hours. Resetting aMule restores the download, but just for another 20MB.

I have another machine with eMule and I noticed that same behaviour when I upload to amule clients. My upload will cut off at around the same size sent.

This issue has been discussed in the forum, but I see no messages if this is being worked on:
http://forum.amule.org/index.php?topic=71860.0

At this current state, aMule is useless, I am not sure if this is happening in all OSs, I have tried in Linux only. Let me know what details you need

Thanks for having a look

[Stoatwblr]

This is more or less the same symptom of what I'm seeing in #270

Note that I reduced the buffering there which "kind of helped" but didn't solve the problerm

It's some kind of interaction between amule stopping to write things out and emule not liking the pauses when running at high speeds (happens with emule in wine or native windows)

Are you using any kind of sysctl tcp tweaks for fast resume, etc?

[DasHammett]

Hi,
No, I am not using any tweaks for fast resume.

Thanks

[rsu611]

On Manjaro (Arch Based) this happens even with version 2.3.2.

[demanuel]

On Arch as well.

Maybe all of these issues are related:

#300

#315

[rsu611]

Found this

[demanuel]

Link above translated with deepl.com:

In the newly released version of CN, the problem that aMule v2.3.2 and later versions are prone to interruptions in downloads due to 'Wrong header' underlying errors has been effectively mitigated. This article is to record the process of solving this problem and to discuss my thoughts on some program development rules.

I. Review of the problem

About some time after the early release of aMule v2.3.2, I noticed an increase in the number of bad blocks when downloading, which I thought was caused by a relatively bad modification. But later, by analyzing the verbose log, I found that it was probably caused by the official aMule client.

An example of the log is as follows.

2/1/2021 4:07:55 PM: Client TCP socket: Error: Wrong header; [email protected] (xxx.xxx.xx.xxx) 'http://www.aMule. org' (aMule v2.3.2,Downloading/None/None)

This is a relatively low-level error where the application cannot find a legitimate data header when processing TCP data streams and thus cannot continue. This problem was carried over to the daily build version of aMule v2.4.0 and the newly released aMule v2.3.3 this year.

The initial eMule CN mitigation method

If the download was only interrupted, it would be fine, but since the data transfer error occurred before the interruption, the download data was incorrect and bad blocks appeared. To deal with such bad blocks, eMule CN added the ability to drop some previously downloaded data from this problematic client once such a serious underlying error occurs, thus effectively mitigating the bad blocks caused by this problem.

Third, the ultimate mitigation solution - rewrite the download bandwidth allocation algorithm

Since March this year, I have been rewriting the download bandwidth allocation algorithm of eMule CN with std::list, and thanks to the flexibility of std::list, I am able to change and test various methods more freely. After a period of debugging and stabilization of the speed control features, it was discovered by chance that a 'normal' download client (i.e. not aMule v2.3.2 and later) may also generate a similar 'Wrong header' error when testing some of the extreme modes.

This finding gives some insight that although such errors occur after the introduction of the boost::asio library for socket handling in aMule v2.3.2 and should be attributed to the client, it is probably possible to mitigate them locally with algorithmic adjustments.

The next task is to find the algorithm pattern that can make this problem can be greatly mitigated. This is a very delicate task. After many attempts, we finally found an algorithm pattern where one of the parameters can cause the problem to worsen, which means that a better parameter can be found by tuning the parameter in the opposite direction; there are also other aspects to match the algorithm tuning, and finally we got an existing algorithm that can be tested to avoid most of the possible problems when downloading It is tested to avoid most of the possible 'Wrong header' errors when downloading.

IV. Some thoughts arising

According to the general rules developers follow, it is impossible to spend time and effort on local client development to mitigate the problems caused by the other client, especially since aMule clients do not account for a very high percentage.

So is this work worth it? eMule Community's maintainer fox88 thinks not!

But from another point of view, isn't it because of this problem in aMule v2.3.2 and later that the error triggering of aMule can be used as an indicator to measure the error tolerance and robustness of the download distribution algorithm?

At least, I used to be scared when I saw aMule v2.3.2 and later clients starting to download, but now I can safely watch the download data from these clients flowing in, which brings such a change of mind, and for me, the development effort is worth it.

What do you think?

[Ganton]

The problem still happens.

More cases can be found on https://forum.amule.org/index.php?topic=73928.0

[Anuskuss]

Okay scratch that. In between football matches I decided to compile v2.3.3 the traditional way and found out that the bug does not happen. So it's not a regression by the code itself but probably from a dependency, most likely wxWidgets. Debian compiles with libwxgtk3.2-dev instead of v2.8 which was used back when aMule development was active.

This is the command I used:

docker run -it -v /tmp:/tmp debian bash -c '
apt update; \
apt -y install curl libgtk2.0-dev libcrypto++-dev; \
curl -L http://prdownloads.sourceforge.net/wxwindows/wxWidgets-2.8.12.tar.gz | tar xzf -; \
cd wxWidgets-2.8.12; \
./configure CFLAGS=-std=c99 CXXFLAGS=-std=c++98 --prefix=$PWD/wxprefix --enable-unicode --enable-optimise --disable-shared; \
make; \
make install; \
curl -L http://repo.or.cz/w/amule.git/snapshot/10b4d56b786a0989e354f9d29610d6f7f4868e28.tar.gz | tar xzf -; \
cd amule-*; \
sed -i "/#include <vector>/a#include <exception>" src/libs/common/MuleDebug.cpp; \
./configure --disable-debug --enable-optimize --disable-static --disable-nls --disable-monolithic --enable-amule-daemon --disable-amulecmd --disable-webserver --disable-amule-gui --disable-cas --disable-wxcas --disable-ed2k --disable-alc --disable-alcc --with-wx-prefix=$PWD/../wxprefix; \
make; \
mv src/amuled /tmp'

This will give you a clean and working v2.3.3 daemon. If you're too lazy to do that and trust a random stranger on GitHub you can also download my build: amuled-wx2.zip
Note that you'll still need libcrypto++=8.

I also noticed that the bug doesn't just show itself by getting downloads stuck but also just being very slow. You can tell the difference almost immediately if you download something popular (e.g. debian-12.5.0-amd64-DVD-1.iso) and looking at the download speed: If it's ~100KB/s you got a bad version, if it's >1MB/s you got a good version.

Maybe the current Debian maintainer wants to take a look at this and revert the decision to (patch aMule and) use a shared wxWidgets? @sandrotosi

Edit: I just couldn't leave it alone and decided to mess with compiling again and I think I figured it out: It's Boost! Compiling with --with-boost gives you a bad executable regardless of wxWidgets version (no-Boost: amuled-wx3.zip). I will probably still run with statically linked wxWidgets v2.8 as that's most likely the most tested path but at least the Debian maintainer now has to only remove a single argument to build a good package.

[arximboldi]

@Anuskuss thank you for this research, you're wonderful! I was stuck with this on (#372 #300) and actually just gave up and moved to eMule via Wine. I may give this a try some time and maybe PR https://github.com/NixOS/nixpkgs/ to disable Boost there...

To be clear... I doubt the bug is in Boost itself, but probably it is in how aMule uses Boost. Either way, great find!