Release

[jacobtomlinson]

I'm currently blocked on needing some unreleased changes so would really appreciate a release soon.

@jcrist I know your time is limited so if there's anything I can do to help please let me know!

[rsignell-usgs]

Particularly looking forward to seeing #353 in a new release!
Would be great to have that functionality in the next Qhub release (which uses dask gateway), due out Mar 29!
Plus it would be cool to have version 1.0.0 😃

[jacobtomlinson]

Plus it would be cool to have version 1.0.0

I expect dask-gateway will follow the other Dask projects and adopt CalVer.

[rsignell-usgs]

@jacobtomlinson , good point! I guess that should take the pressure off a new release then! 😺

[TomAugspurger]

Jim's probably a bit busy these days (check his Twitter 😄).

But Jim, when you're back, I'd be happy to help out with release stuff. I think the first time it'd be helpful to pair on the release so that we can figure out the processes together.

[jacobtomlinson]

I think the first time it'd be helpful to pair on the release so that we can figure out the processes together.

This is also what has put me off doing a release here.

I'd also be keen to help out.

[consideRatio]

I'd also be happy to be onboarded to contribute to this process and in general contribute to helping dask-gateway stay alive - especially with anything related to the k8s Helm chart.

[AndreaGiardini]

Same here! Happy to help

[droctothorpe]

+1

[adamdrozdz]

+1

[ghost]

+1

[kyprifog]

Noticed this stalled a bit, @jacobtomlinson anything I can do to push it over the finish line?

[AndreaGiardini]

Hey everyone! Any update on this? I am willing to help

[TomAugspurger]

Thanks for the offer. There's some work in #408 fixing up the CI and failing tests. I'm not sure if @aktech could use any help there.

Other than that, we should have all the help we need. We just need to find a time where Jim can do some knowledge transfer to the other Dask maintainers about the release process.

[consideRatio]

If someone has time to do some code review, it would make me very happy but could probably also help nudge us towards a release :)

• Augment dask-gateway releases to gh-pages hosted Helm repo helm-chart#208
• images: some refactoring and version bumps for arm64 compatible Dockerfiles #423
• Helm chart: add values.schema.yaml and associated maint. scripts #429
• Helm chart: update to traefik v2.5.x #431
• Migrate already released Helm charts to the Dask org's Helm repo (gh-pages branch of dask/helm-charts) #436 (links to related PRs to review)

[consideRatio]

I really appreciate dask-gateway and think it has such a great impact and synergy with JupyterHub that I also care for as an open source project. I'd be very happy to be enabled to better help dask-gateway thrive.

I'd love to jump in video calls, do practical tests of the default branch, put in work regarding CI, helm, k8s, and documentation where needed, and more. I'm not experienced in golang though but in general a happy learner.

Cheers!

[choldgraf]

Hey all - I don't know the technology enough to develop, but I am happy to help improve the developer documentation around releases if it helps to spread the load across the community.

Could we use this release process as an opportunity to add some steps to release in the developer docs (https://gateway.dask.org/develop.html#) so that we reduce the bus factor on that workflow a bit? If we can get a simple set of steps to take, I am happy to make a PR to the docs or something

[TomAugspurger]

@jcrist this came up on the pangeo call today. Do you have any availability over the next couple weeks to month to push out a release?

If you aren't able to, then I'll start to work through the steps in #296. I likely won't have time until after Thanksgiving though. I'll post here with places where Jim is a sole-owner on some resource.

• https://pypi.org/project/dask-gateway/ (fix at https://pypi.org/manage/project/dask-gateway/collaboration/)
• https://pypi.org/project/dask-gateway-server/ (fix at https://pypi.org/manage/project/dask-gateway-server/collaboration/) (EDIT: added by @consideRatio)
• DockerHub (https://hub.docker.com/orgs/daskgateway)

[jacobtomlinson]

I'm still keen to sync up while this is being done to spread the knowledge.

[TomAugspurger]

Sounds good. I plan to work on this Friday the 3rd. I'm busy 8:00 - 9:00 Central, but am free afterwards. I might be on an hour or two earlier if I get a chance. @jacobtomlinson is there sometime on Friday that you would want to sync up?

I'll be taking notes (using #296 as a starting point) and will automate things where possible.

@jcrist are you able to add another maintainer on PyPI for dask-gateaway (https://pypi.org/manage/project/dask-gateway/collaboration/) and dask-gateway-server (https://pypi.org/manage/project/dask-gateway-server/collaboration/)

Another possible single-point-of-failure is uploading docker images to DockerHub (https://hub.docker.com/r/daskgateway/dask-gateway-server). I think https://hub.docker.com/orgs/daskgateway would be the right link to add people (I'm tomaugspurger on DockerHub). We could also explore publishing those images to GitHub container registry, so that we can re-use GitHub's permissions.

[consideRatio]

I think https://hub.docker.com/orgs/daskgateway would be the right link to add people (I'm tomaugspurger on DockerHub).

Unless you pay for dockerhub at this point, you can't add maintainers as we could before. They have limited their free services to an unusable state. The jupyterhub team is moving to use quay.io in general whenever we rework things at this point.

I suggest the dask org registers an organization on quay.io right away unless you haven't done that.

[jacobtomlinson]

There is a discussion around moving away from Docker Hub in dask/dask-docker/issues/177. I've used Quay in the past and GitHub has a registry too. @consideRatio I'd be keen to know if you could share the decision process behind selecting Quay?

@TomAugspurger I'm afraid I'm on PTO Friday-Monday. But would be glad to sync up Tuesday.

[martindurant]

I would love to be in the sync as an observer, if possible.

[consideRatio]

@jacobtomlinson I responded in dask/dask-docker/issues/177!

I can't join efforts on thursday/friday though, and should avoid scheduling additional things on monday/tuesday as I have an important deadline 7th (tuesday). I'd love to be part helping this project come to life again though, so please feel free to involve me in any way I can contribute!

[jcrist]

Apologies for dropping the ball here for so long. This started as a "I don't have time, but I'd love to get to this eventually" thing (new baby and all), and as the community pressure increased my mental well-being around this issue decreased. I really wanted to help, but even when I had free time I could not get myself to even respond here, much less do any work. Since I'm not longer paid to work on this project, this is all a free time thing, and doing high pressure OSS as a hobby clearly isn't sustainable for me.

That said, knowing how the release process goes I should definitely help out here. It'd be best if we could pair on this @TomAugspurger (and others) whenever you're free too. I'm free after 12 on Friday, as well as most afternoons next week.

That said, there is at least one blocking issue (#414) that needs to be resolved before the next release. This shouldn't be a complicated fix, but it does need to be resolved. We should also verify the helm chart works with the latest distributed/dask releases (the tests are good but not sufficient).

As far as release locations, now would be a good time to switch off of dockerhub. No thoughts on a good alternative, quay.io seems fine, github has a thing too now I think? No strong thoughts.

[consideRatio]

Thanks @jcrist for sharing this, I can really empathize with your situation. I recognize especially the "even respond" part, I find it is often the first step to get hooked into doing more work or having a bad feeling about not doing more work, so it can be a big deal to respond.

I hope your response above felt like a starting point for relaxing any built up tension. I would really like you to have a good feeling about where this project goes going onwards and that you can stay in touch with it in a low pressure kind of way.

---

We should also verify the helm chart works with the latest distributed/dask releases (the tests are good but not sufficient).

I can and will test that on wednesday 8th december! On 8th december and onwards, I'll be able to make it to any meetings or similar about dask-gateway as well.

[TomAugspurger]

No worries Jim! I really, really don't want to add more pressure to an already high-pressure situation.

I'll plan to work on #414 / do a bit of pre-release prep Thursday / Friday morning. Then maybe we sync up on https://whereby.com/dask-dev around 1:30 or 2:00 on Friday? It'd be good to chat about stuff, even if it's not release related :)

[jcrist]

Thanks Tom & Erik! That sounds good to me, lets shoot for 1:30 central on Friday.

[martindurant]

Would appreciate an invite or ping nearer the time, else I'm really likely to miss the time slot :)

[choldgraf]

Just want to show some love for @jcrist. Thanks for being awesome - open source maintenance is super hard and we all empathize :-)

[jacobtomlinson]

Totally agree with @choldgraf, thanks for driving this project to this point @jcrist! Sorry that this has become a source of stress, I totally empathise. Also totally agree with @consideRatio that hopefully, this is an opportunity for us to help you remove that stress.

Also thank you for openly talking about how it has affected your mental health. It's a big deal opening up about that topic so thank you for setting a great example. Seeing all the responses here really gives me good feelings about how supportive the Dask community is of each other.

[TomAugspurger]

Short update here: The only outstanding work items for the release are merging the changelog (#467) and possibly an update to the docker images from @jcrist (#468).

We're blocked on upload to PyPI right now (waiting for help with some account issues). Once we get that sorted out, hopefully in the next week or two, we'll be able to release.

In the meantime, I'll do a bit of work on release automation, including migrating to GitHub container registry (#466) and building / uploading the sdist and wheels from GitHub actions using cibuildwheel. Merging those will likely wait until just after the release.

[jacobtomlinson]

@TomAugspurger did the PyPI issues get resolved?

[TomAugspurger]

I'm not sure. @jcrist, do you know?

[mmccarty]

@jcrist Would it be possible to add another owner to PyPi so they can get it resolved? I'm also happy organize a call to coordinate, divide up tasks, and share knowledge among those expressing interest on this issue.

[martindurant]

ping to other interested people in this thread, incase you didn't get a notification: @consideRatio @TomAugspurger @jacobtomlinson @choldgraf

@jcrist , we are happy to take over and I think we have enough manpower to follow through.

[mmccarty]

@jcrist @consideRatio @TomAugspurger @jacobtomlinson @choldgraf @martindurant I made a doodle poll for times next week to get together to coordinate on a release.

[TomAugspurger]

The blocker from last time was uploading to PyPI. I don't know if Jim has had a chance to get his credentials sorted out, but IMO it's not worth scheduling something till we're able to do that.

[mmccarty]

Jim mentioned yesterday at the maintainers sync that he got that sorted out.

[consideRatio]

I've outlined action points and requests for input in #484 regarding setting up automation for publishing release artifacts. My vision is to contribute making that happen given agreement and permissions and then do some manual end-to-end tests.

[consideRatio]

I'm opened #543 to track some remaining action points towards a release, I'll go ahead and close this issue in favor of it with concrete action points!

[choldgraf]

Congratulations everyone on getting this one in! I really like the collaborative way that this issue played out, thanks everybody :-)

[consideRatio]

A release has been made! PyPI package and Helm chart is available but the conda-forge packages remains to be released still.

If someone has experience with conda-forge feedstocks it can become helpful as at least I personally have a lacking experience and know that we will need to make some changes. See conda-forge/dask-gateway-feedstock#22 (comment) for details!