Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Expire only boosts #40

Open
lapineige opened this issue Jan 5, 2019 · 41 comments
Open

Expire only boosts #40

lapineige opened this issue Jan 5, 2019 · 41 comments

Comments

@lapineige
Copy link

Hello,

Is it possible to add an option to expire only our boosted toots, instead of expiring all statuses ?

Thanks

@kensanata
Copy link
Owner

kensanata commented Jan 5, 2019 via email

@BartG95
Copy link
Contributor

BartG95 commented Jan 5, 2019

@lapineige what exactly do you mean by boosted toots?
Do you mean:
a) Your statuses, that are boosted by others (i.e. you only want to retain your statuses are not boosted);
b) Statuses from others, that you boosted (i.e. you want your account to show only your toots)

Option A seems difficult to implement, but option B should be relatively easy. Essentially, you need a version with these lines removed:
https://github.com/kensanata/mastodon-backup/blob/036df62e97277f21672033e27bc21740e0230119/mastodon_archive/expire.py#L44-L45
For the time being, you can make a clone of the repo, remove these lines and install the package locally with pip install -e . (replace the dot with the path to your local repo, or just cd into the repo.)

@BartG95
Copy link
Contributor

BartG95 commented Jan 5, 2019

If you want to make a patch, you will need to add an option here:
https://github.com/kensanata/mastodon-backup/blob/43deaa2a53e14c6e4f2d653fe71647d61c33d87f/mastodon_archive/__init__.py#L153
Then, modify the delete function to take an extra boolean here:
https://github.com/kensanata/mastodon-backup/blob/036df62e97277f21672033e27bc21740e0230119/mastodon_archive/expire.py#L36
Then, transform this else-statement into an elif-statement:
https://github.com/kensanata/mastodon-backup/blob/036df62e97277f21672033e27bc21740e0230119/mastodon_archive/expire.py#L44
Then, modify the call to the delete function here:
https://github.com/kensanata/mastodon-backup/blob/036df62e97277f21672033e27bc21740e0230119/mastodon_archive/expire.py#L108
Feel free to write a patch ;)

@kensanata kensanata changed the title [Feature Request] Expire only boosts Expire only boosts Jan 5, 2019
@lapineige
Copy link
Author

lapineige commented Jan 6, 2019

@lapineige what exactly do you mean by boosted toots?
Do you mean:
a) Your statuses, that are boosted by others (i.e. you only want to retain your statuses are not boosted);
b) Statuses from others, that you boosted (i.e. you want your account to show only your toots)

I mean option b :)

I don't have the time right now, but I'll have a look.
Thanks for your help, I appreciate it :)

@LeonardoFurtado
Copy link

i need it, maybe i'll implement.

@kensanata
Copy link
Owner

kensanata commented Jul 2, 2019 via email

@lapineige
Copy link
Author

Hello, any update on this ? :)

@kensanata
Copy link
Owner

kensanata commented Sep 15, 2019 via email

@lapineige
Copy link
Author

I'm actually wondering if it expires boost right now… I've searched my old post, and the classic toots seems to be deleted, while boosts are still there.
@kensanata does Mastodon-backup expire boosts ?

@kensanata
Copy link
Owner

kensanata commented Feb 9, 2020 via email

@lapineige
Copy link
Author

How can I check this ?

@lapineige
Copy link
Author

Another person reported to me that her boost were not expired.
2 questions:

  • can it expire boost that has not been archived ?
  • how can I check if the boosts are taken into account when I expire my statuses ?

@kensanata
Copy link
Owner

What happens when you run mastodon-archive expire --older-than 8 --collection favourites --confirm [email protected], for example? What sort of output do you see?

@lapineige
Copy link
Author

lapineige commented Feb 10, 2020

It's just expiring toots… normal output, Loading archive then expiring.

Note that there is an issue when expiring favorites:

  • without --pace:

Traceback (most recent call last):
File "/home/name/.local/bin/mastodon-archive", line 10, in
sys.exit(main())
File "/home/name/.local/lib/python3.7/site-packages/mastodon_archive/init.py", line 293, in main
args.command(args)
File "/home/name/.local/lib/python3.7/site-packages/mastodon_archive/expire.py", line 104, in expire
mastodon.ratelimit_method = 'wait'
UnboundLocalError: local variable 'mastodon' referenced before assignment

  • with --pace, it doesn't take it into account and goes full speed.

@kensanata
Copy link
Owner

kensanata commented Feb 11, 2020 via email

@Mellerin
Copy link
Contributor

Hi!
Looks like i'm the other one who have troubles with the same issue.
I ran mastodon-backup with several options, it built my archive, deleted my toots, my favorites, downloaded my media files. All i can see in my profile now are still the toots from other people that i boosted.

mastodon-archive report --all --include-boosts [email protected]
Considering the entire archive
Statuses:             21909
Boosts:                3296
Media:                  556

Top 10 hashtags:
#pouetradio(276) #mastodon(97) #tonfilmenpetit(80) #photo(65)
#photography(54) #tootradio(41) #photographie(40) #nsfw(40)
#mercredifiction(39) #question(35)

Favourites:           10797
Boosts:                   0
Media:                 2048

No more statuses on my account (when connected on the web ui of my instance), but only the 3K+ boosts reported here.
Maybe I just don't run the appropriate command? I test the one you suggested earlier mastodon-archive expire --older-than 4 --collection favourites --confirm [email protected]
but nothing new, even if it ran smoothly, even if it expired 10197 something but I don't know what are those 10197 since it doesn't match any numbers reported above (i guess it's some of the favourites, but not all)/
I also tried mastodon-archive expire --older than 4 with and without all the --collection but it just now says "no statuses/favourites/mentions are older than x weeks".

If I understand it clearly, your program doesn't erase all my stuff with the same command, need to run it with appropriate command (not a bad thing). So :
1- build an archive mastodon-archive archive --with-mentions myaccount
2- delete regular statuses (toots) :
mastodon-archive expire --older-than 8 --collection statuses --confirm myaccount
3- delete favorited :
mastodon-archive expire --older-than 8 --collection favourites --confirm myaccount
4- delete notifications :
mastodon-archive expire --older-than 8 --collection mentions --delete-other-notifications --confirm myaccount

Basically all that you kindly suggest in your example setup : https://github.com/kensanata/mastodon-backup#example-setup

extra-step :
5- mastodon-archive expire --older-than 8 --confirmed myaccount (still expires some stuff)

6- Still have all the boosts! ^_^! (statuses from other people that i boosted)

Sorry, I'm not a programmer and I just don't understand what BartG95 suggested to do earlier :/
#40 (comment)
I tried some modifications without any results but maybe i didn't ran it the good way (not a dev!)

Besides that, thanks for your program and your help, have a good day ^_^

@lapineige
Copy link
Author

I’m interested in seeing whether you specify the collection – if it’s expiring toots it is not expecting favorites and vice verse.

Well I specify --collection favourites

@kensanata
Copy link
Owner

Hm. Let's see… This is the code that does the deleting. Apparently you need to specify the statuses collection in order to "unreblog" (unboost) a status.

def delete(mastodon, collection, status):
    """
    Delete toot, unfavour favourite, or dismiss notification and mark
    it as deleted. The "record not found" error is handled elsewhere.
    """
    if collection == 'statuses':
        if status["reblog"]:
            mastodon.status_unreblog(status["id"])
        else:
            mastodon.status_delete(status["id"])
    elif collection == 'favourites':
        mastodon.status_unfavourite(status["id"])
    elif collection == 'mentions':
        mastodon.notifications_dismiss(status["id"])
    status["deleted"] = True

So, either the status isn't marked as "reblog" on the archive, or the code isn't finding it, or the status isn't in the archive in the first place.

So, looking for a status in the JSON file with "reblogged": false...

I'm opening my text editor, loading the JSON file, and looking for this string…

Finding this one:

    {
      "id": 100148697908936006,
      "created_at": "2018-06-04T21:30:56.281000+00:00",
...
      "url": "https://octodon.social/@kensanata/100148697908936006",
...
      "reblogged": false,
...
      "reblog": {
        "id": 100148655205233270,
        "created_at": "2018-06-04T21:19:55+00:00",
...
        "url": "https://trunk.mad-scientist.club/@algernon/100148654631327735",
...
        "reblogged": true,
...

Let's check! I'm searching for https://trunk.mad-scientist.club/@algernon/100148654631327735 in my Mastodon session, and finding it. I'm checking the "boost" icon and it's off, so I haven't boosted it.

So, it seems to be working.

I think I'd need this kind of investigation from one of you.

For example:

  1. visit your profile and scroll down until you find a boosted toot that should have been unboosted
  2. right-click on the timestamp and "Copy Link Location" (or whatever your browser says)
  3. open your JSON archive in a text editor
  4. search for the URL you copied
  5. did you find it?
  6. as you can see in the example above, your status status (in the example above the one with id 100148697908936006) contains the boosted status in the reblog section (in the example above the one with id 100148655205233270), is this reblog status marked with "reblogged": true?
  7. when you visit the URL of the boosted status, does it look boosted?
  8. what version of Mastodon is your instance running? (see the very bottom of the /about/more link, e.g. I'm checking https://octodon.social/about/more)
  9. what version of Mastodon.py are you running? (something like locate "Mastodon.py"|grep dist-info\$ might help?)
  10. what version of Mastodon Archive are you running?

@Mellerin
Copy link
Contributor

Mellerin commented Feb 23, 2020

Hello and thanks for your reply ^_^
Firstly, versions :
mastodon-archive v1.3.0
mastodon.py v1.5.0 (the one that was installed while installing mastodon-archive)
mastodon instance v3.0.1 (mamot.fr)
So I visited my profile, scroll down and find many that should have been unboosted, like these ones (it's an image which should be deleted in 180 days)
Regarding the date I'm sure i ran all mastodon-backup commands multiples times on these dates, so they shouldn't appear as boosted here but as you can see in the picture, they are still boosted.
I take the one from Unpied copy its link and search for it in my archive :

    {
      "id": 102999751926001273,
      "created_at": "2019-10-21T09:51:28.130000+00:00",
...
      "language": null,
      "uri": "https://mamot.fr/users/EllerinPrv/statuses/102999751926001273/activity",
      "url": "https://mamot.fr/users/EllerinPrv/statuses/102999751926001273/activity",
      "replies_count": 0,
...
      "reblogged": false,
...
      "reblog": {
        "id": 102999749014241944,
        "created_at": "2019-10-21T09:50:32+00:00",
...
        "uri": "https://birdsite.link/users/Unpied/statuses/102999748288513511",
        "url": "https://birdsite.link/@Unpied/102999748288513511",
        "replies_count": 0,
        "reblogs_count": 5,
        "favourites_count": 2,
        "favourited": false,
        "reblogged": true,
...

So even if i visit this particular toot alone, it appears as boosted.
If i understand the reblog section says i boosted it but the first "reblogged": false before that says it is not boosted (un-boosted) anymore right now?
So, maybe mastodon-archive seems to work as expected but maybe there's something odd when sending unboost/unreblog command towards the instance which doesn't execute the unreblog command or doesn't receive it?
Last time i ran the expire command i didn't see any errors in the output. Is there a way to have a more detailed report from the program in order to see if for example the mastodon instance ignore the unreblog command?
Anyway, again, thanks for your time 🙇‍♀️

@kensanata
Copy link
Owner

If i understand the reblog section says i boosted it but the first
"reblogged": false before that says it is not boosted (un-boosted)
anymore right now?

I'm not sure that's how you read it. I read it like this: you did
something, so there's a status associated with it. The thing you did was
boost (reblog) another status (which is now marked as reblogged true).
That is, the outer status (your status) is never marked as reblogged.
The status that you're boosting (the one that's is part of the the
reblog attribute) is always marked as reblogged true). In short, the
data I'm seeing says: you reblogged this status.

maybe mastodon-archive seems to work as expected but maybe there's
something odd when sending unboost/unreblog command towards the
instance which doesn't execute the unreblog command or doesn't receive
it?

That is of course always possible but not a very satisfying answer. I
wonder what's going on.

Last time i ran the expire command i didn't see any errors in the
output. Is there a way to have a more detailed report from the program
in order to see if for example the mastodon instance ignore the
unreblog command? Anyway, again, thanks for your time 🙇‍♀

The code also says the following:

    def matches(status):
        created = datetime.strptime(status["created_at"][0:10], "%Y-%m-%d")
        deleted = "deleted" in status and status["deleted"] == True
        pinned = "pinned" in status and status["pinned"] == True
        return created < cutoff and not deleted and not pinned

    statuses = list(filter(matches, data[collection]))

That is, we're not doing anything to statuses that are marked as
deleted. So I went back to my JSON file and checked status
100148697908936006 and indeed, at the very end:

    {
      "id": 100148697908936006,
...
      "reblog": {
        "id": 100148655205233270,
...
        "reblogged": true,
...
        "account": {
          "id": 23975,
          "username": "algernon",
...
        },
...
      },
      "application": null,
      "account": {
        "id": 12540,
        "username": "kensanata",
...
      "deleted": true
    },

That is, my own "reblogging action" (id 100148697908936006) is marked as
deleted.

I wonder, however. Perhaps my code is confusing the two ids? What
happens is that it goes through the list of statuses, finds my
"reblogging action" (id 100148697908936006) and tries to delete it.

def delete(mastodon, collection, status):
    """
    Delete toot, unfavour favourite, or dismiss notification and mark
    it as deleted. The "record not found" error is handled elsewhere.
    """
    if collection == 'statuses':
        if status["reblog"]:
            mastodon.status_unreblog(status["id"])
        else:
            mastodon.status_delete(status["id"])
    elif collection == 'favourites':
        mastodon.status_unfavourite(status["id"])
    elif collection == 'mentions':
        mastodon.notifications_dismiss(status["id"])
    status["deleted"] = True

As you can see, the delete code then checks whether the status contains
a "reblog" and if it does, it unreblogs the id of the status – but not
the id of the reblogged status. That is, 100148697908936006 is
unreblogged, not 100148655205233270.

So now I'm wondering: perhaps the code should be changed to the
following?

def delete(mastodon, collection, status):
    """
    Delete toot, unfavour favourite, or dismiss notification and mark
    it as deleted. The "record not found" error is handled elsewhere.
    """
    if collection == 'statuses':
        if status["reblog"]:
            mastodon.status_unreblog(status["reblog"]["id"]) # ← THIS!
        else:
            mastodon.status_delete(status["id"])
    elif collection == 'favourites':
        mastodon.status_unfavourite(status["id"])
    elif collection == 'mentions':
        mastodon.notifications_dismiss(status["id"])
    status["deleted"] = True

Can you make this change to your copy and give it a try?

(Sadly, I will be leaving for a trip, soon, and won't be able to get
back to this for a few weeks.)

@Mellerin
Copy link
Contributor

Hello,
Ok, i admit all these kind of statuses and their mechanics are a bit confusing me ^_^!
So, in fact if I boost someone else's toot, it creates my own toot status id which is marked as a reblog but not reblogged, then the reblog section refer to the original status (toot) which is now marked as reblogged=true.
Well, sadly, at the very end of this (my) status id, I also find a "deleted": true (just before the begining of another { "id" : ...
I will check your modification and let you know. I messed with the little Raspberrry Pi which drives your program so i need to re-install it before. From my side this boost/reblog is not a hurry, so please enjoy you trip (well, as fas as it's a trip you can enjoy).
Thank you o/

@kensanata
Copy link
Owner

I'm back from my trip. Did it work as intended, @Mellerin?

@Mellerin
Copy link
Contributor

Mellerin commented Mar 22, 2020

Hello and wb o/ I hope everything's going well for you during these troubled times.

Well, it worked i suppose, it just lets some boosted toots and the very end of my own timeline. I didn't investigate yet exactly why except that the (few) statuses that remain go to a "the page you are looking for isn't here" on the mastodon web interface, maybe deleted statuses but looks like it's not a problem with mastodon-archive ^_^
So, back to mastodo-archive, what i did :

  • create a new venv : python3 -m venv mastobackup-dev
  • activate and go in it : source mastobackup-dev/bin/activate
  • git clone mastodon-archive in it : (mastobackup-dev) me@host:~ $ git clone https://github.com/kensanata/mastodon-backup.git
  • edit the file : ./mastodon-backup/mastodon-archive/expire.py and change the code as stated previously :
if collection == 'statuses':
        if status["reblog"]:
            mastodon.status_unreblog(status["reblog"]["id"])
  • now installing the modified local repo : $ pip3 install -e ./mastodon-backup
  • create a new dir for archive and go in : mkdir tootback and cd ./tootback
  • archive new archive : ~/tootback $ mastodon-archive archive [email protected]
  • token validation
  • delete statuses : ~/tootback $ mastodon-archive expire --older-than 3 --confirm [email protected]
  • token validation again (maybe i missed something here for it to ask again, the *secret files exist after that)
  • expiring ## / 3684
  • OK \o/

I detailed all the things i've done, not sure all the steps are necessary neither the right ones to do ^_^!
So it looks like the little code modification does the job. Thank you 👍

Little edit : btw I'm not sure it answers exactly what @lapineige asked at the very beginning ?
Done a PR, please ignore it if you want ^_^

Mellerin added a commit to Mellerin/mastodon-backup that referenced this issue Mar 22, 2020
Add ["reblog"] status type to the unreblog function because just ["id"] let some boosted statuses and does not unreblog them.
See kensanata#40
@kensanata
Copy link
Owner

If this is good enough for you, @lapineige, please close this issue. Otherwise I'm going to assume that you still want an option to expire only boosts without expiring your own toots.

@lapineige
Copy link
Author

Sorry I read the whole conversation but had no time to react/test it myself.

If I understand well the changes, now I need to make my archive again (to create the "reblog" category) and then remove toots just like before and it will include all reblogs ?

Well an option would be cool, but maybe I can make the PR myself :)

@kensanata
Copy link
Owner

kensanata commented Mar 23, 2020 via email

@lapineige
Copy link
Author

Ok thanks. I guess I need to wait for the next release ?

@kensanata
Copy link
Owner

kensanata commented Mar 23, 2020 via email

@lapineige
Copy link
Author

Finally I don't think I'll try a non-release version.
Do you have an idea about when the next release can be done ?

@kensanata
Copy link
Owner

I made a release 1.3.1.

@lapineige
Copy link
Author

Yeah 🎉 Thanks a lot !

So I tried it… I was planning to delete old boosts, on data that was already "deleted" (but not the boosts).
But it says that no status is older than [X] weeks…

Could it be that it tried to delete them, marked them as "deleted", and then it won't try again because it is supposed to already be removed ?

@kensanata
Copy link
Owner

Hm, could be. What do you think we should do, now? Perhaps we could write a command that goes through all the boosts and marks them as not deleted?

@lapineige
Copy link
Author

That's a possibility indeed 🤔
@Mellerin do you think you could (and want to) do it, or should I try to ?

@kensanata
Copy link
Owner

kensanata commented Jul 14, 2020

I am super confused when I read through this thread. 😅

I wrote a new subcommand, fix-boosts which does what I proposed. I run it as follows:

mastodon-archive fix-boosts [email protected]

If you don't confirm, it prints how many changes it would have made, and twenty examples. To verify, you need to paste these URLs into the search box of your Mastodon account so that you can see whether they are still boosted. This is important: if you just click on the links and look at them in your browser, they will always look unboosted.

Let me know if that fixes it for your?

I would like to avoid making a release for this. I'd love for you to test it from the git repository:

git clone https://github.com/kensanata/mastodon-backup.git
cd mastodon-backup
pip3 install --upgrade .

That should get you commit bc891f7.

Once you do confirm (make a copy of your archive first, just in case!) it makes the changes and then the next time you run mastodon-archive expire it should try to expire all those boosts once again.

@lapineige
Copy link
Author

Coming back to this issue after a while… I am a bit lost 😅

I will give it a try 🙂.
Could you please merge current commits to this version ? I don't know if that necessary to actually try it, but I would prefer to use the most up-to-date version just in case in creates any compatibility issue or such 😅

@kensanata
Copy link
Owner

This has been integrated into mastodon-archive for a long time time, now. Two years ago was the last commit referring to fix-boosts.

@lapineige
Copy link
Author

Ok sorry I didn't catch that change. Thank you :)

@lapineige
Copy link
Author

So the fix-boosts command does give a list of still boosted toots. Great !

But how can I expire them ?
And what if I want to expire only boosts older than a specific date ?

Thank you

@lapineige
Copy link
Author

lapineige commented Jul 24, 2022

Ok I think I figured out : I need to run the fix-boosts command (with --confirmed option) to mark them as non expired, then run the expire command.

But that means I can't expire only boosts.
Would that be a possible addition ?
As the command can list boosts only, I guess it would be possible to reuse that list in the expire command… I guess it would be a new --collection boosts argument ?
Ready that code https://github.com/kensanata/mastodon-backup/blob/main/mastodon_archive/fix.py#L39= I understand that this new option code would "just" need a condition like this if status["reblog"] : [command to expire it]. Am I right ?

@kensanata
Copy link
Owner

Hm, indeed, there currently is no option to delete just boosts. I think you are right. In expire.py lines 38–54:

def delete(mastodon, collection, status):
    """
    Delete toot or unfavour favourite and mark it as deleted. The
    "record not found" error is handled elsewhere. Mentions cannot be
    dismissed because mastodon.notifications_dismiss requires a
    notification id, not a status id.
    """
    if collection == 'statuses':
        if status["reblog"]:
            mastodon.status_unreblog(status["reblog"]["id"])
            status["deleted"] = True
        else:
            mastodon.status_delete(status["id"])
            status["deleted"] = True
    elif collection == 'favourites':
        mastodon.status_unfavourite(status["id"])
        status["deleted"] = True
    elif collection == 'boosts' and status["reblog"]:
        mastodon.status_unreblog(status["reblog"]["id"])
        status["deleted"] = True

This assumes we don't want to change the existing functionality for the "statuses" collection. I'm not sure what would be best, here.

For the command line option in __init__.py, lines 186–189:

    parser_content.add_argument("--collection", dest='collection',
                                choices=['statuses', 'favourites', 'mentions', 'boosts'],
                                default='statuses',
                                help='delete statuses, unfavour favourites, clear mention notifications, or only boosts')

Now there's still a problem: boosts are stored as statuses "containing" the boosted status in the "reblog" attribute. If we use "--collection=boosts" then the code in expire.py will try to load the collection which doesn't exist. It's actually the "statuses" collection we need to load. Yuck.

Something needs to change in expire.py line 95:

        statuses = list(filter(matches, data[if collection == "boosts" then "statuses" else collection]))

All of the code examples untested. 😢

I'm leaving on a trip for the next few days so there's going to be no answer for a while. Give it a try, perhaps with a test account. 😅

We could of course just add a different option, --boosts-only, which only works for --collection=statuses. 🤔

@lapineige
Copy link
Author

All of the code examples untested. cry

I'm leaving on a trip for the next few days so there's going to be no answer for a while. Give it a try, perhaps with a test account. sweat_smile

Wow that's great !
Thanks a lot for your help !

I think I'll try some dry-run first :D

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

5 participants