Merge pull request #1 from cognizant-ai-labs/first-release

First release

.gitignore

@@ -127,3 +127,6 @@ dmypy.json
 
 # Pyre type checker
 .pyre/
+
+# JSON
+*.json

LICENSE.txt

@@ -0,0 +1,224 @@
+License
+The following license governs the use of Act-Bench Datasets in academic
+and educational environments. Commercial use requires a commercial license from
+Cognizant Technology Solutions Corp, www.cognizant.com.
+
+ACADEMIC PUBLIC LICENSE
+Written by Andras Varga (license text is in public domain)
+Adapted by Karl Mutch (license text is in public domain)
+Adapted by Garrett Bingham (license text is in public domain)
+
+Preamble
+This license contains the terms and conditions of using Act-Bench Datasets
+in noncommercial settings: at academic institutions for teaching and research
+use, and at non-profit research organizations. You will find that this license
+provides noncommercial users of Act-Bench Datasets with rights that are
+similar to the well-known GNU General Public License, yet it retains the
+possibility for Act-Bench Datasets authors to financially support the
+development by selling commercial licenses. In fact, if you intend to use
+Act-Bench Datasets in a “for-profit” environment, where research is
+conducted to develop or enhance a product, is used in a commercial service
+offering, or when a commercial company uses Act-Bench Datasets to
+participate in a research project (for example government-funded or EU-funded
+research projects), then you need to obtain a commercial license for
+Act-Bench Datasets. In that case, please contact the Author to inquire
+about commercial licenses.
+
+What are the rights given to noncommercial users? Similarly to GPL, you have the
+right to use the software, to distribute copies, to receive source code, to
+ehange the software and distribute your modifications or the modified software.
+Also similarly to the GPL, if you distribute verbatim or modified copies of
+this software, they must be distributed under this license.
+
+By modeling the GPL, this license guarantees that you’re safe when using
+Act-Bench Datasets in your work, for teaching or research. This license
+guarantees that Act-Bench Datasets will remain available free of charge
+for nonprofit use. You can modify Act-Bench Datasets to your purposes,
+and you can also share your modifications. Even in the unlikely case of the
+authors abandoning Act-Bench Datasets entirely, this license permits
+anyone to continue developing it from the last release, and to create further
+releases under this license.
+
+We believe that the combination of noncommercial open-source and commercial
+licensing will be beneficial for the whole user community, because income from
+commercial licenses will enable faster development and a higher level of
+software quality, while further enjoying the informal, open communication and
+collaboration channels of open source development.
+
+The precise terms and conditions for using, copying, distribution and
+modification follow.
+
+TERMS AND CONDITIONS FOR USE, COPYING, DISTRIBUTION AND MODIFICATION
+
+Definitions
+“Program” means a copy of Act-Bench Datasets, which is said to be
+distributed under this Academic Public License.
+
+“Work based on the Program” means either the Program or any derivative work
+under copyright law: that is to say, a work containing the Program or a portion
+of it, either verbatim or with modifications and/or translated into another
+language. (Hereinafter, translation is included without limitation in the term
+“modification”.)
+
+“Using the Program” means any act of creating executables that contain or
+directly use libraries that are part of the Program, running any of the tools
+that are part of the Program, or creating works based on the Program.
+
+Each licensee is addressed as “you”.
+
+§1. Permission is hereby granted to use the Program free of charge for any
+noncommercial purpose, including teaching and research at universities, colleges
+and other educational institutions, research at non-profit research
+institutions, and personal non-profit purposes. For using the Program for
+commercial purposes, including but not restricted to consulting activities,
+design of commercial hardware or software networking products, and a commercial
+entity participating in research projects, you have to contact the Author for an
+appropriate license. Permission is also granted to use the Program for a
+reasonably limited period of time for the purpose of evaluating its usefulness
+for a particular purpose.
+
+§2. You may copy and distribute verbatim copies of the Program’s source code as
+you receive it, in any medium, provided that you conspicuously and appropriately
+publish on each copy an appropriate copyright notice and disclaimer of warranty;
+keep intact all the notices that refer to this License and to the absence of any
+warranty; and give any other recipients of the Program a copy of this License
+along with the Program.
+
+§3. You may modify your copy or copies of the Program or any portion of it, thus
+forming a work based on the Program, and copy and distribute such modifications
+or work under the terms of Section 2 above, provided that you also meet all of
+these conditions:
+
+a) You must cause the modified files to carry prominent notices stating that
+you changed the files and the date of any change.
+
+b) You must cause any work that you distribute or publish, that in whole or in
+part contains or is derived from the Program or any part thereof, to be
+licensed as a whole at no charge to all third parties under the terms of this
+License.
+
+These requirements apply to the modified work as a whole. If identifiable
+sections of that work are not derived from the Program, and can be reasonably
+considered independent and separate works in themselves, then this License, and
+its terms, do not apply to those sections when you distribute them as separate
+works. But when you distribute the same sections as part of a whole which is a
+work based on the Program, the distribution of the whole must be on the terms of
+this License, whose regulations for other licensees extend to the entire whole,
+and thus to each and every part regardless of who wrote it. (If the same,
+independent sections are distributed as part of a package that is otherwise
+reliant on, or is based on the Program, then the distribution of the whole
+package, including but not restricted to the independent section, must be on the
+unmodified terms of this License, regadless of who the author of the included
+sections was.)
+
+Thus, it is not the intent of this section to claim rights or contest your
+rights to work written entirely by you; rather, the intent is to exercise the
+right to control the distribution of derivative or collective works based or
+reliant on the Program.
+
+In addition, mere aggregation of another work not based on the Program with the
+Program (or with a work based on the Program) on a volume of storage or
+distribution medium does not bring the other work under the scope of this
+License.
+
+§4. You may copy and distribute the Program (or a work based on it, under
+Section 3) in object code or executable form under the terms of Sections 2 and 3
+above provided that you also do one of the following:
+
+a) Accompany it with the complete corresponding machine-readable source code,
+which must be distributed under the terms of Sections 2 and 3 above on a medium
+customarily used for software interchange; or,
+
+b) Accompany it with a written offer, valid for at least three years, to give
+any third party, for a charge no more than your cost of physically performing
+source distribution, a complete machine-readable copy of the corresponding
+source code, to be distributed under the terms of Sections 2 and 3 above on a
+medium customarily used for software interchange; or,
+
+c) Accompany it with the information you received as to the offer to distribute
+corresponding source code. (This alternative is allowed only for noncommercial
+distribution and only if you received the program in object code or executable
+form with such an offer, in accord with Subsection b) above.)
+
+The source code for a work means the preferred form of the work for making
+modifications to it. For an executable work, complete source code means all the
+source code for all modules it contains, plus any associated interface
+definition files, plus the scripts used to control compilation and installation
+of the executable. However, as a special exception, the source code distributed
+need not include anything that is normally distributed (in either source or
+binary form) with the major components (compiler, kernel, and so on) of the
+operating system on which the executable runs, unless that component itself
+accompanies the executable.
+
+If distribution of executable or object code is made by offering access to copy
+from a designated place, then offering equivalent access to copy the source code
+from the same place counts as distribution of the source code, even though third
+parties are not compelled to copy the source along with the object code.
+
+§5. You may not copy, modify, sublicense, or distribute the Program except as
+expressly provided under this License. Any attempt otherwise to copy, modify,
+sublicense or distribute the Program is void, and will automatically terminate
+your rights under this License. However, parties who have received copies, or
+rights, from you under this License will not have their licenses terminated so
+long as such parties remain in full compliance.
+
+§6. You are not required to accept this License, since you have not signed it.
+Nothing else grants you permission to modify or distribute the Program or its
+derivative works; law prohibits these actions if you do not accept this License.
+Therefore, by modifying or distributing the Program (or any work based on the
+Program), you indicate your acceptance of this License and all its terms and
+conditions for copying, distributing or modifying the Program or works based on
+it, to do so.
+
+§7. Each time you redistribute the Program (or any work based on the Program),
+the recipient automatically receives a license from the original licensor to
+copy, distribute or modify the Program subject to these terms and conditions.
+You may not impose any further restrictions on the recipients’ exercise of the
+rights granted herein. You are not responsible for enforcing compliance by third
+parties to this License.
+
+§8. If, as a consequence of a court judgment or allegation of patent
+infringement or for any other reason (not limited to patent issues), conditions
+are imposed on you (whether by court order, agreement or otherwise) that
+contradict the conditions of this License, they do not excuse you from the
+conditions of this License. If you cannot distribute so as to satisfy
+simultaneously your obligations under this License and any other pertinent
+obligations, then as a consequence you may not distribute the Program at all.
+For example, if a patent license would not permit royalty-free redistribution of
+the Program by all those who receive copies directly or indirectly through you,
+then the only way you could satisfy both it and this License would be to refrain
+entirely from distribution of the Program.
+
+If any portion of this section is held invalid or unenforceable under any
+particular circumstance, the balance of the section is intended to apply and the
+section as a whole is intended to apply in other circumstances.
+
+§9. If the distribution and/or use of the Program are restricted in certain
+countries either by patents or by copyrighted interfaces, the original copyright
+holder who places the Program under this License may add an explicit
+geographical distribution limitation excluding those countries, so that
+distribution is permitted only in or among countries not thus excluded. In such
+case, this License incorporates the limitation as if written in the body of this
+License.
+
+NO WARRANTY
+
+§10. BECAUSE THE PROGRAM IS LICENSED FREE OF CHARGE, THERE IS NO WARRANTY FOR
+THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN OTHERWISE
+STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM
+“AS IS” WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING,
+BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A
+PARTICULAR PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE
+PROGRAM IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF
+ALL NECESSARY SERVICING, REPAIR OR CORRECTION.
+
+§11. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED ON IN WRITING WILL
+ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY AND/OR REDISTRIBUTE THE
+PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY GENERAL,
+SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE USE OR INABILITY
+TO USE THE PROGRAM INCLUDING BUT NOT LIMITED TO LOSS OF DATA OR DATA BEING
+RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD PARTIES OR A FAILURE OF
+THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS), EVEN IF SUCH HOLDER OR OTHER
+PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.
+
+END OF TERMS AND CONDITIONS

README.md

@@ -1,2 +1,60 @@
 # act-bench
-Activation function benchmark datasets Act-Bench-CNN, Act-Bench-ResNet, and Act-Bench-ViT
+Activation function benchmark datasets `Act-Bench-CNN`, `Act-Bench-ResNet`, and `Act-Bench-ViT`
+
+---
+
+## Getting Started
+
+Clone the repository and download the datasets.
+
+```bash
+git clone https://github.com/cognizant-ai-labs/act-bench.git
+cd act-bench
+bash download-datasets.sh
+```
+
+In total, the datasets are about 381 MB in size.
+
+---
+
+## Loading the Datasets
+
+Load the datasets with the `json` module.
+
+```python
+import json
+
+with open('datasets/act-bench-cnn.json', 'r') as f:
+    act_bench_cnn = json.load(f)
+
+with open('datasets/act-bench-resnet.json', 'r') as f:
+    act_bench_resnet = json.load(f)
+
+with open('datasets/act-bench-vit.json', 'r') as f:
+    act_bench_vit = json.load(f)
+```
+
+---
+
+## Exploring the Datasets
+
+Each dataset is a dictionary, where the keys are activation function names and the values are themselves dictionaries containing information about that activation function.  For example:
+
+```python
+>>> for k, v in act_bench_cnn.items():
+...   print(k)
+...   print(v.keys())
+...   break
+... 
+min(square(x),selu(x))
+dict_keys(['Train Accuracy', 'Validation Accuracy', 'Train Loss', 'Validation Loss', 'Test Accuracy', 'Test Loss', 'Equivalent Activation Functions', 'Fisher log eigenvalue CDF', 'Runtime (s)'])
+>>> 
+```
+
+The train, validation, and test accuracies and losses are the median of three independent runs for `Act-Bench-CNN` and `Act-Bench-ResNet`, and the result of a single run for `Act-Bench-ViT`.  The same is true of the runtime, which is reported in seconds.  
+
+If desired, the activation function name can be used to instantiate the activation function as a TensorFlow `Layer` using the `ActivationFunction` class in the `aquasurf` package at https://github.com/cognizant-ai-labs/aquasurf.
+
+The `Equivalent Activation Functions` key contains a list of activation function names that are functionally equivalent to the activation function.  For example, `'min(selu(x),square(x))'` is equivalent to `'min(square(x),selu(x))'`.
+
+Finally, `Fisher log eigenvalue CDF` contains a feature vector encoding the CDF of the log-scaled eigenvalues of the Fisher Information Matrix associated with the weights at each layer of the network.

download-datasets.sh

@@ -0,0 +1,15 @@
+#!/bin/bash
+
+# make a datasets directory
+mkdir -p datasets
+
+# move into the datasets directory
+cd datasets
+
+# download the datasets
+wget https://act-bench.s3.us-west-2.amazonaws.com/act-bench-cnn.json
+wget https://act-bench.s3.us-west-2.amazonaws.com/act-bench-resnet.json
+wget https://act-bench.s3.us-west-2.amazonaws.com/act-bench-vit.json
+
+# move back to the root directory
+cd ..