Hook up running make test with testport. #355

mat813 · 2015-10-16T14:59:41Z

No description provided.

bdrewery · 2015-10-16T15:01:00Z

Thanks for working on this.

Note 'test' needs to be handled by the '-k' mechanism, implied by '-cat', so the error from 'test' doesn't immediately stop but is just noted as a failure in the end. - So plist/qa testing is still done.

mat813 · 2015-10-16T15:04:22Z

It's going to make the patch a bit bigger 😄

mat813 · 2015-10-16T15:20:41Z

Is that something like this that you had in mind ?

bdrewery · 2015-10-16T15:22:54Z

Yup, looks familiar (I don't have time to dive into this now).

Do we want 'test' to always run?

Note also that DEVELOPER is not passed down unless we allow it. Grep for DEVELOPER in common.sh and you'll see you need to do similar where the test framework needs it. We also need to fetch TEST_DEPENDS in list_deps

bdrewery · 2015-10-16T15:23:19Z

While I don't have a lot of time on this now, I'll merge what looks right and it can be put into -devel for wider testing.

mat813 · 2015-10-16T15:25:07Z

test is a no-op if not enabled in the port's Makefile, so I would say yes.

I'll have a look at the rest later.

AMDmi3 · 2015-10-16T15:29:01Z

src/share/poudriere/common.sh

@@ -2090,7 +2090,7 @@ _real_build_port() {
 		# Don't need to install if only making packages and not
 		# testing.
 		[ -n "${PORTTESTING}" ] && \
-		    install_order="${install_order} install-mtree install"
+		    install_order="${install_order} test install-mtree install"


test runs after install

Well, they don't depend on install, they depend on stage, so, I'm not certain it's true.

Oh. Stage and install are messed up and install has different meaning from inside and outside the port. I assume the later is meant here, so the code is correct.

Well, maybe tests need to run after the port has been really installed, and not just staged, I don't know, I'm just saying this is coherent with what bsd.port.mk does :-)

mat813 · 2015-10-27T10:52:14Z

Ok, now, it also builds TEST_DEPENDS.

mat813 · 2015-10-30T15:06:55Z

So, there is another problem, that I'm not quite certain on how to fix, test depends are sometime circular, today, I got a:

[00:00:35] ====>> Cleaning the build queue
tsort: cycle in data
tsort: py27-py-1.4.30
tsort: py27-pytest-2.7.1
tsort: cycle in data
tsort: p5-Test-Warnings-0.021
tsort: p5-Test-Kwalitee-1.22

I'm not exactly sure on how to fix that.

mat813 · 2015-11-10T12:13:24Z

ping ?

bdrewery · 2015-11-11T01:01:11Z

You said there was a circular dependency problem. Is that fixed?

Regards,
Bryan Drewery

On Nov 10, 2015, at 04:13, Mathieu Arnold [email protected] wrote:

ping ?

—
Reply to this email directly or view it on GitHub.

AMDmi3 · 2015-12-01T13:51:37Z

Can we maybe only enable tests which have no TEST_DEPENDS as a first step?

that's only used when building the package.

DragonSA · 2016-02-24T17:15:57Z

Given that the make test target depends on make stage I would submit that any circular dependencies are a problem with the port and not a feature that needs to be supported by poudriere. My motivation for that is if one where to do make test and the port isn't installed it would (kinda) fail from ports. (Also, how common is the circular dependency? If minor cannot one error out and have something available than nothing at all)

bdrewery · 2016-06-09T01:04:35Z

I think what we need to do to fix the cyclic dependency problem is add separate items into the queue. One to build the package, and another to test it. The test one will install the test depends (extract it again), install itself, then handle running the tests. Granted I haven't looked at this in depth but I think this may ultimately be the need. #259 would be needed first.

mat813 · 2016-06-09T02:53:56Z

If poudriere could detect that adding TEST_DEPENDS doesn't add a cyclic dependency, it could run make test at the same time as the rest, and only postpone make test if there is a cyclic dependency.

bdrewery · 2017-03-02T17:22:18Z

I'm going to make testport into a two-pass thing here. Phase 1. Nothing changes from today. Phase 2. Build dependency tree considering TEST_DEPENDS, then run the tests.

bulk will have to wait until later. I at least want some kind of testing support in here, at the very least for testport.

bdrewery · 2017-06-09T13:30:04Z

A fix for #259 is nearly ready to merge. I will try to get port testing into 3.2.

bdrewery · 2018-01-30T18:57:04Z

The queue needs to be smarter. Rather than everything in the queue being implicitly "build foo" it needs to be explicit. So "build foo deps bar" and "test foo deps foo baz". This avoids the double pass since the 1 queue can deal with it without reworking all of bulk/testport to deal with 2 passes.

moufjenkins · 2018-01-30T18:57:05Z

Can one of the admins verify this patch?

bdrewery · 2018-01-30T20:51:53Z

Ok the whole idea of testing separately is killed because 'make test' depends on 'make stage'. So it must happen at the same time or I need to tar the workdir. So it seems the simplest solution is going to be to drop testing of anything that has a circular dependency. Not sure if tsort will allow that or not.

mandree · 2020-04-27T09:40:31Z

ping

bdrewery · 2021-09-08T05:26:02Z

The queue needs to be smarter. Rather than everything in the queue being implicitly "build foo" it needs to be explicit. So "build foo deps bar" and "test foo deps foo baz". This avoids the double pass since the 1 queue can deal with it without reworking all of bulk/testport to deal with 2 passes.

I have this working in a local branch named generic-queue-job_type-nodir. I just need to add in the part of saving the wrkdir and take the build_port changes here as a base for a new test_port that picks up from the tarred wrkdir.

And for some ports we can likely combine the build and test jobs into 1 without the wrkdir tarring.

ocochard · 2022-01-07T16:57:59Z

ping

This is needed to support a 2nd queue for TEST_DEPENDS and smarter job dependencies. Issue #355

bdrewery · 2024-02-27T15:48:01Z

The queue work is done. The feature can be implemented now.

mat813 force-pushed the hook_tests branch 2 times, most recently from d29848c to 74e1e40 Compare October 16, 2015 15:20

AMDmi3 reviewed Oct 16, 2015
View reviewed changes

mat813 force-pushed the hook_tests branch from 7972d9b to 74e1e40 Compare October 19, 2015 13:45

mat813 added 4 commits February 1, 2016 17:25

Hook up running make test with testport.

807aa6c

Work with -k.

230be08

Don't append PACKAGES definition to the jail make.conf, remove the one

a3a5644

that's only used when building the package.

When testing, build test-depends too, and install the depends as root.

ca19a3c

mat813 mentioned this pull request Feb 23, 2016

Add support for make test to testport #377

Closed

grembo mentioned this pull request Mar 30, 2016

Run test target on testport and bulk -t #382

Closed

mat813 force-pushed the hook_tests branch from 017be2e to ca19a3c Compare April 11, 2016 12:40

bdrewery mentioned this pull request Feb 9, 2017

invoke TEST_TARGET for testport #435

Closed

bdrewery added this to the 3.2 milestone Jun 9, 2017

bdrewery mentioned this pull request Jun 9, 2017

regression-test target support #291

Closed

bdrewery mentioned this pull request Jun 9, 2017

feature request: make test #280

Closed

bdrewery added the Ports Workflow label Jun 14, 2017

bdrewery modified the milestones: 3.2, 3.2.1 Nov 10, 2017

bdrewery modified the milestones: 3.2.1, 3.2.2, 3.2.3, 3.2.4 Nov 27, 2017

bdrewery modified the milestones: 3.2.4, 3.3.0 Dec 14, 2017

bdrewery mentioned this pull request Mar 16, 2018

[feature] Testing and QA: regression-test & TEST_DEPENDS support #55

Closed

bdrewery mentioned this pull request Apr 2, 2018

poudriere does not build TEST_DEPENDS #609

Closed

bdrewery modified the milestones: 3.3.0, 3.4.0 Feb 26, 2019

bdrewery mentioned this pull request Oct 11, 2019

testport: automated tests with *installed* package? #724

Open

bdrewery self-assigned this Sep 8, 2021

bdrewery added Build queue Popular request labels Nov 8, 2022

bdrewery mentioned this pull request Dec 16, 2022

Incremental build changes #822

Closed

bdrewery modified the milestones: 3.4.0, 3.5.0 Feb 19, 2024

bdrewery added a commit that referenced this pull request Feb 27, 2024

pkgqueue: Rework to carry along a "type" of queued job.

48f74bf

This is needed to support a 2nd queue for TEST_DEPENDS and smarter job dependencies. Issue #355

bdrewery mentioned this pull request Mar 6, 2024

TEST_DEPENDS aren't taken from packages #1131

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Hook up running make test with testport. #355

Hook up running make test with testport. #355

mat813 commented Oct 16, 2015

bdrewery commented Oct 16, 2015

mat813 commented Oct 16, 2015

mat813 commented Oct 16, 2015

bdrewery commented Oct 16, 2015

bdrewery commented Oct 16, 2015

mat813 commented Oct 16, 2015

AMDmi3 Oct 16, 2015

mat813 Oct 19, 2015

AMDmi3 Oct 19, 2015

mat813 Oct 19, 2015

mat813 commented Oct 27, 2015

mat813 commented Oct 30, 2015

mat813 commented Nov 10, 2015

bdrewery commented Nov 11, 2015

AMDmi3 commented Dec 1, 2015

DragonSA commented Feb 24, 2016

bdrewery commented Jun 9, 2016

mat813 commented Jun 9, 2016

bdrewery commented Mar 2, 2017

bdrewery commented Jun 9, 2017 •

edited

Loading

bdrewery commented Jan 30, 2018

moufjenkins commented Jan 30, 2018

bdrewery commented Jan 30, 2018

mandree commented Apr 27, 2020

bdrewery commented Sep 8, 2021

ocochard commented Jan 7, 2022

bdrewery commented Feb 27, 2024

Hook up running make test with testport. #355

Are you sure you want to change the base?

Hook up running make test with testport. #355

Conversation

mat813 commented Oct 16, 2015

bdrewery commented Oct 16, 2015

mat813 commented Oct 16, 2015

mat813 commented Oct 16, 2015

bdrewery commented Oct 16, 2015

bdrewery commented Oct 16, 2015

mat813 commented Oct 16, 2015

AMDmi3 Oct 16, 2015

Choose a reason for hiding this comment

mat813 Oct 19, 2015

Choose a reason for hiding this comment

AMDmi3 Oct 19, 2015

Choose a reason for hiding this comment

mat813 Oct 19, 2015

Choose a reason for hiding this comment

mat813 commented Oct 27, 2015

mat813 commented Oct 30, 2015

mat813 commented Nov 10, 2015

bdrewery commented Nov 11, 2015

AMDmi3 commented Dec 1, 2015

DragonSA commented Feb 24, 2016

bdrewery commented Jun 9, 2016

mat813 commented Jun 9, 2016

bdrewery commented Mar 2, 2017

bdrewery commented Jun 9, 2017 • edited Loading

bdrewery commented Jan 30, 2018

moufjenkins commented Jan 30, 2018

bdrewery commented Jan 30, 2018

mandree commented Apr 27, 2020

bdrewery commented Sep 8, 2021

ocochard commented Jan 7, 2022

bdrewery commented Feb 27, 2024

bdrewery commented Jun 9, 2017 •

edited

Loading