static vs shared library #95

gdsjaar · 2015-11-12T14:21:21Z

Is it possible to force a static library even if the BUILD_SHARED_LIBS is enabled? I see that the CMake ADD_LIBRARY command has a STATIC option, but I don't see that listed as a valid option in the TRIBITS_ADD_LIBRARY command.

When I add STATIC to my TRIBITS_ADD_LIBRARY options, it doesn't complain, but it doesn't seem to force a static library.

Thanks,
..Greg

bartlettroscoe · 2015-11-12T18:01:40Z

Greg, that is a good point. I guess TRIBITS_ADD_LIBRARY() should support all of the options passed the raw cmake command ADD_LIBRARY which includes (from CMake 2.8.11 documentation):

add_library(<name> [STATIC | SHARED | MODULE]
              [EXCLUDE_FROM_ALL]
              source1 source2 ... sourceN)

(we have to stick with CMake 2.8.11 until we upgrade the minimum CMake version again)

At this point, I would not add the IMPORTED or GLOBAL options because that involves a very different use case. I have yet to figure out how to cleanly import pre-built libraries into TriBITS and also generate Config.cmake and Makefile.export. files. I will need to figure that out as part of the large upcoming refactoring in #63.

To effectively test this, I am going to need to add the ability to unit test TRIBITS_ADD_LIBRARY() (which is far overdue).

How urgent/important is this for you?

gdsjaar · 2015-11-12T18:05:52Z

Currently with SEACAS, I cannot build everything with ENABLE_SHARED=ON since there is an SVDI library that won't work correctly as shared in its current configuration. However, another piece of seacas (exodus.py) requires the shared exodus library, so currently, I require people to do two builds - one with ENABLE_SHARED=OFF and then another to rebuild exodus with ENABLE_SHARED=ON.

As for urgency/importance, it is important, but I can live with the current state if I need to.

Thanks,
..Greg

"A supercomputer is a device for turning compute-bound problems into I/O-bound problems"

From: "Roscoe A. Bartlett" <[email protected]mailto:[email protected]>
Reply-To: TriBITSPub/TriBITS <[email protected]mailto:[email protected]>
Date: Thursday, November 12, 2015 at 1:01 PM
To: TriBITSPub/TriBITS <[email protected]mailto:[email protected]>
Cc: "Sjaardema, Gregory D" <[email protected]mailto:[email protected]>
Subject: [EXTERNAL] Re: [TriBITS] static vs shared library (#95)

Greg, that is a good point. I guess TRIBITS_ADD_LIBRARY() should support all of the options passed the raw cmake command ADD_LIBRARY which includes (from CMake 2.8.11 documentation):

add_library( [STATIC | SHARED | MODULE]
[EXCLUDE_FROM_ALL]
source1 source2 ... sourceN)

(we have to stick with CMake 2.8.11 until we upgrade the minimum CMake version again)

At this point, I would not add the IMPORTED or GLOBAL options because that involves a very different use case. I have yet to figure out how to cleanly import pre-built libraries into TriBITS and also generate Config.cmake and Makefile.export. files. I will need to figure that out as part of the large upcoming refactoring in #63 #63.

To effectively test this, I am going to need to add the ability to unit test TRIBITS_ADD_LIBRARY() (which is far overdue).

How urgent/important is this for you?

Reply to this email directly or view it on GitHubhttps://github.com//issues/95#issuecomment-156185126.

bartlettroscoe · 2015-11-12T18:07:02Z

Let me try to get to this in the next week or so along with other work. I will add this to my todo list.

gdsjaar · 2015-11-12T18:50:35Z

Thanks.

..Greg

"A supercomputer is a device for turning compute-bound problems into I/O-bound problems"

From: "Roscoe A. Bartlett" <[email protected]mailto:[email protected]>
Reply-To: TriBITSPub/TriBITS <[email protected]mailto:[email protected]>
Date: Thursday, November 12, 2015 at 1:07 PM
To: TriBITSPub/TriBITS <[email protected]mailto:[email protected]>
Cc: "Sjaardema, Gregory D" <[email protected]mailto:[email protected]>
Subject: [EXTERNAL] Re: [TriBITS] static vs shared library (#95)

Let me try to get to this in the next week or so along with other work. I will add this to my todo list.

Reply to this email directly or view it on GitHubhttps://github.com//issues/95#issuecomment-156186445.

gdsjaar · 2015-11-12T18:52:16Z

I have verified that this capability would enable SEACAS to build with ENABLE_SHARED=ON with a few of the libraries being forced to static archive libs, so I would definitely use the capability.

..Greg

"A supercomputer is a device for turning compute-bound problems into I/O-bound problems"

From: "Sjaardema, Gregory D" <[email protected]mailto:[email protected]>
Date: Thursday, November 12, 2015 at 1:50 PM
To: TriBITSPub/TriBITS <[email protected]mailto:[email protected]>, TriBITSPub/TriBITS <[email protected]mailto:[email protected]>
Subject: Re: [EXTERNAL] Re: [TriBITS] static vs shared library (#95)

Thanks.

..Greg

"A supercomputer is a device for turning compute-bound problems into I/O-bound problems"

From: "Roscoe A. Bartlett" <[email protected]mailto:[email protected]>
Reply-To: TriBITSPub/TriBITS <[email protected]mailto:[email protected]>
Date: Thursday, November 12, 2015 at 1:07 PM
To: TriBITSPub/TriBITS <[email protected]mailto:[email protected]>
Cc: "Sjaardema, Gregory D" <[email protected]mailto:[email protected]>
Subject: [EXTERNAL] Re: [TriBITS] static vs shared library (#95)

Let me try to get to this in the next week or so along with other work. I will add this to my todo list.

Reply to this email directly or view it on GitHubhttps://github.com//issues/95#issuecomment-156186445.

bartlettroscoe · 2015-11-12T19:24:22Z

Okay, so I looked at the implementation for TRIBITS_ADD_LIBRARY() and adding unit tests would be pretty hard since it calls a lot of raw global CMake commands that would need to be mocked out. That will likely be worth the effort when I do the refactoring #63. But for now, I think the easiest and most solid testing to is to create an example/test package TribitsMixStaticSharedLibsPkg where I will use combinations TRIBITS_ADD_LIBRARY([SHARED|STATIC] ...) with different combinations of BUILD_SHARED_LIBS=ON|OFF and grep to make sure the right libraries are built in each case. This testing will need to be restricted to Linux machines where I know the generated library files (e.g. 'lib.so' and 'lib.a'). Those will be more expensive tests because you actually have to run the compiler but it will also be a nice validation that CMake does the right thing in these cases.

I will try to get to this soon.

bartlettroscoe · 2015-11-12T19:44:24Z

Note that it does not look like you can set the STATIC vs. SHARED property after the fact with SET_TARGET_PROPERTIES() looking at the list of target properties for CMake 2.8.11. Therefore, these have to be passed in to the call ADD_LIBRARY() and therefore must be passed through TRIBITS_ADD_LIBRARY().

bartlettroscoe · 2015-12-23T19:10:44Z

@gdsjaar, I am working on this now. But I was wondering what your exact use case is for this. Are these libraries used to link into exectuables but not used in downstream packages? The issues is that if you create a static lib then try to link that to a downstream shared lib, this is not portable in general.

gdsjaar · 2015-12-23T19:21:17Z

The libraries are all used with SEACAS. Specifically, they are the

../lib/libsvdi_cdr.a
../lib/libfastqlib.a
../lib/libsvdi_cgi.a
../lib/libblotlib.a

Blotlib and fastqlib call routines which are in svdi_cgi and svdi_cdr. There are no shareable libraries which depend on these static libraries.

TRIBITS_ADD_EXECUTABLE(blot_x11 NOEXEPREFIX NOEXESUFFIX
  LINKER_LANGUAGE Fortran
  SOURCES ${CMAKE_CURRENT_SOURCE_DIR}/blot.f
    ${CMAKE_CURRENT_SOURCE_DIR}/cgi/x11_cgisx11.c
  COMM serial mpi)

TRIBITS_ADD_EXECUTABLE(blot_xcps NOEXEPREFIX NOEXESUFFIX
  LINKER_LANGUAGE Fortran
  SOURCES ${CMAKE_CURRENT_SOURCE_DIR}/blot.f
    ${CMAKE_CURRENT_SOURCE_DIR}/cgi/vdx11cps.F
    ${CMAKE_CURRENT_SOURCE_DIR}/cgi/vdicps_dual.f
 COMM serial mpi)

TRIBITS_ADD_EXECUTABLE(blot_cps NOEXEPREFIX NOEXESUFFIX
  LINKER_LANGUAGE Fortran
  SOURCES ${CMAKE_CURRENT_SOURCE_DIR}/blot.f
    ${CMAKE_CURRENT_SOURCE_DIR}/cgi/vdicps.f
 COMM serial mpi)

The x11_cgisx11.c and vdx11cps.F and vdicps_dual.f and vdicps.f each resolve the same symbols in the libsvdi_cgi.a and libsvdi_cdr.a. If the svdi_cgi and svdi_cdr are shareable libraries, then there are unresolved symbols up until they are linked into the executable which doesn't work well with cmake.

This is from a very old SVDI (Sandia Virtual Device Interface) library which is used to provide different graphics drivers for blot and fastq.

..Greg

"A supercomputer is a device for turning compute-bound problems into I/O-bound problems"

From: "Roscoe A. Bartlett" <[email protected]mailto:[email protected]>
Reply-To: TriBITSPub/TriBITS <[email protected]mailto:[email protected]>
Date: Wednesday, December 23, 2015 at 2:10 PM
To: TriBITSPub/TriBITS <[email protected]mailto:[email protected]>
Cc: "Sjaardema, Gregory D" <[email protected]mailto:[email protected]>
Subject: [EXTERNAL] Re: [TriBITS] static vs shared library (#95)

@gdsjaarhttps://github.com/gdsjaar, I am working on this now. But I was wondering what your exact use case is for this. Are these libraries used to link into exectuables but not used in downstream packages? The issues is that if you create a static lib then try to link that to a downstream shared lib, this is not portable in general.

Reply to this email directly or view it on GitHubhttps://github.com//issues/95#issuecomment-166973007.

bartlettroscoe · 2015-12-23T19:46:59Z

@gdsjaar, thanks for the explanation. Is the issue that you want to build these sources into a static library once and then reuse then in the various exectuables instead of having to build them again and again into different executabes? What is the real problem?

The concern that I have right now is that TriBITS would add static libraries to the ${PACKAGE_NAME}_LIBRARIES list and they would get linked into libraries in downstream packages and I don't think we want that. For example, of some downstream package listed a dependency on all of SEACAS, they would get all of the SEACAS libraries linked into their downstream (shared) libraries and that would be bad. That is something we can address but we need to know.

It would be good if we could talk briefly over the phone so that I can fully understand what the requirements are.

Let's try to talk over the phone briefly if possible. I will send you a private email.

I added the example/test package MixedSharedStaticLibs to demonstrate/test the new STATIC and SHARED TRIBITS_ADD_LIBRARY() arguments. See MixedSharedStaticLibs/README for more details. I also added the printing of BUILD_SHARED_LIBS so you can see it in the basic configure output. I added a check for this in the test. Build/Test Cases Summary Enabled Packages: TriBITS Enabled all Forward Packages 0) MPI_DEBUG => passed: passed=202,notpassed=0 (1.12 min) 1) SERIAL_RELEASE => passed: passed=202,notpassed=0 (0.92 min) Other local commits for this build/test group: ff1267d

bartlettroscoe · 2015-12-23T23:29:14Z

Pushed commit 73b3827 to github. You can see updated documentation showing the new STATIC and SHARED options here.

I shapshotted this to Trilinos and pushed the following commits to the Trilinos github/master branch:

5b06652 "Merge branch 'tribits_github_snapshot'"
Author: Roscoe A. Bartlett <[email protected]>
Date:   Wed Dec 23 17:54:14 2015 -0500 (54 seconds ago)

24b7a7a "Automatic snapshot commit from tribits at 73b3827"
Author: Roscoe A. Bartlett <[email protected]>
Date:   Wed Dec 23 17:53:27 2015 -0500 (22 minutes ago)

@gdsjaar, I think this is resolved, Please go ahead and snapshot TriBITS to your GitHub repo for seacas with as described here. You can then snapshot seacas into Trilinos using:

$ cd Trilinos/packages/seacas/
$ ../../cmake/tribits/python_utils/snapshot-dir.py --orig-dir=<some-base-dir>/seacas/

(the final / is very important!). Not sure what your workflow is w.r.t. SIERRA so perhaps you have to snapshot to SIERIRA first and then Brent snapshots from SIERRA to Trilinos?

Anyway, let me know if you have an issue with this but you should be able to just use this TriBITS version and not touch your github version of seacas. I am therefore closing as complete.

P.S. As for info on linking between shared and static libs and why you can't link a static lib into a downstream shared lib, see here.

bartlettroscoe mentioned this issue Nov 12, 2015

TRIBITS_ADD_EXECUTABLE missing ADD_EXECUTABLE properties #76

Open

bartlettroscoe closed this as completed Dec 23, 2015

bartlettroscoe added the type: enhancement label Dec 23, 2015

bartlettroscoe added this to the 6_deployed milestone Dec 23, 2015

bartlettroscoe self-assigned this Dec 23, 2015

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

static vs shared library #95

static vs shared library #95

gdsjaar commented Nov 12, 2015

bartlettroscoe commented Nov 12, 2015

gdsjaar commented Nov 12, 2015

bartlettroscoe commented Nov 12, 2015

gdsjaar commented Nov 12, 2015

gdsjaar commented Nov 12, 2015

bartlettroscoe commented Nov 12, 2015

bartlettroscoe commented Nov 12, 2015

bartlettroscoe commented Dec 23, 2015

gdsjaar commented Dec 23, 2015

bartlettroscoe commented Dec 23, 2015

bartlettroscoe commented Dec 23, 2015

static vs shared library #95

static vs shared library #95

Comments

gdsjaar commented Nov 12, 2015

bartlettroscoe commented Nov 12, 2015

gdsjaar commented Nov 12, 2015

bartlettroscoe commented Nov 12, 2015

gdsjaar commented Nov 12, 2015

..Greg

gdsjaar commented Nov 12, 2015

..Greg

bartlettroscoe commented Nov 12, 2015

bartlettroscoe commented Nov 12, 2015

bartlettroscoe commented Dec 23, 2015

gdsjaar commented Dec 23, 2015

..Greg

bartlettroscoe commented Dec 23, 2015

bartlettroscoe commented Dec 23, 2015