Skip to content

Commit

Permalink
Merge branch 'develop'
Browse files Browse the repository at this point in the history
  • Loading branch information
rockdaboot committed Aug 15, 2014
2 parents cdb8cd1 + 80f8849 commit 3d8c70d
Show file tree
Hide file tree
Showing 49 changed files with 3,213 additions and 1,009 deletions.
2 changes: 1 addition & 1 deletion .travis.yml
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ compiler:
- gcc
- clang
# Change this to your needs
script: ./autogen.sh && ./configure --enable-gtk-doc && make -j4 && make check -j4 && make distcheck -j4
script: ./autogen.sh && ./configure --enable-gtk-doc && make -j4 && make check -j4 && make distcheck
before_install:
- sudo apt-get -qq update
- sudo apt-get -q install autoconf automake autopoint libtool gtk-doc-tools gettext flex liblzma5 liblzma-dev libidn2-0 libidn2-0-dev libunistring0 libunistring-dev
26 changes: 0 additions & 26 deletions ChangeLog
Original file line number Diff line number Diff line change
@@ -1,26 +0,0 @@
2014-02-03 gettextize <[email protected]>

* m4/gettext.m4: Upgrade to gettext-0.18.3.
* m4/iconv.m4: Upgrade to gettext-0.18.3.
* m4/lib-ld.m4: Upgrade to gettext-0.18.3.
* m4/lib-link.m4: Upgrade to gettext-0.18.3.
* m4/lib-prefix.m4: Upgrade to gettext-0.18.3.
* m4/nls.m4: Upgrade to gettext-0.18.3.
* m4/po.m4: Upgrade to gettext-0.18.3.
* m4/progtest.m4: Upgrade to gettext-0.18.3.
* configure.ac (AM_GNU_GETTEXT_VERSION): Bump to 0.18.3.

2013-01-02 gettextize <[email protected]>

* m4/gettext.m4: Upgrade to gettext-0.18.1.
* m4/iconv.m4: Upgrade to gettext-0.18.1.
* m4/lib-ld.m4: Upgrade to gettext-0.18.1.
* m4/lib-link.m4: Upgrade to gettext-0.18.1.
* m4/lib-prefix.m4: Upgrade to gettext-0.18.1.
* m4/nls.m4: Upgrade to gettext-0.18.1.
* m4/po.m4: Upgrade to gettext-0.18.1.
* m4/progtest.m4: Upgrade to gettext-0.18.1.
* Makefile.am (SUBDIRS): Add po.
* configure.ac (AC_CONFIG_FILES): Add po/Makefile.in.
(AM_GNU_GETTEXT_VERSION): Bump to 0.18.1.

16 changes: 15 additions & 1 deletion NEWS
Original file line number Diff line number Diff line change
@@ -1,3 +1,17 @@
15.08.2014
New release v0.1.6
Check cookie domains via libpsl against the Public Sufix List
Fixed -p/--page-requisites
Fixed HTML parsing of uppercase attributes (fixes issue #21)
Added --accept, --reject, --ignore-case
Added -k/--convert-links and -K/--backup-converted

08.03.2014
Fixed sigfault in cookie handling, introduced in v0.1.5.

02.03.2014
Added --gnutls-options to directly set a GnuTLS priority string for secure connections.

26.02.2014
New release v0.1.5

Expand All @@ -6,7 +20,7 @@

17.02.2014
Fixed a race condition that could lead to sigfaults.
Fixed waiting for threads a program exit.
Fixed waiting for threads at program exit.

16.02.2014
Made HTTPS test work with GnuTLS 2.12.
Expand Down
15 changes: 6 additions & 9 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,9 @@ Mget has already many features that go beyond what Wget provides.<br>

An incomplete list of implemented features:

- support for RFC 6266 compliant Content-Disposition
- Use [libpsl](https://github.com/rockdaboot/libpsl) for cookie domain checking (using Public Suffix List)
- Support link conversion (-k/--convert-links and -K/--backup-converted)
- Support for RFC 6266 compliant Content-Disposition
- RFC 6797 HSTS (HTTP Strict Transport Security)
- Support for bzip2 Content-Encoding / Accept-Encoding compression type
- New Year gimmick: added support for XZ Content-Encoding / Accept-Encoding compression type
Expand All @@ -72,7 +74,6 @@ sitemap index files.
- proxy support
- cookies (session/non-session), detection of supercookies via Mozilla Public Suffix List
(use the new option --cookie-suffixes <filename>, better: put it into ~/.mgetrc)
- respect cookie public suffix list (http://publicsuffix.org/list/)
- recursive download of websites with or without spanning hosts
- download of single web pages / resources
- zlib/gzip compressed HTTP/HTTPS downloads (gzip, deflate)
Expand Down Expand Up @@ -103,8 +104,6 @@ The mid-range goal is to come as close to Wget, that Wget's units test work for

Some ideas of what could be done next (but contact me before you start bigger changes!):

- use [gtk-doc-tools](http://developer.gnome.org/gtk-doc-manual/unstable/settingup.html.en) for documentation.<br>
I want the docs stay with the code: already tested Doxygen, but the man page support seems broken/orphaned.
- WARC support
- read credentials from secure wallets (e.g. kwallet, firefox, http://sourceforge.net/projects/passwordsafe/)
[Chromium HSTS domain list](http://src.chromium.org/viewvc/chrome/trunk/src/net/http/transport_security_state_static.json)
Expand Down Expand Up @@ -135,8 +134,9 @@ The following packages are needed to build Mget:
* libgnutls >= 2.10.0
* libidn2 >= 0.9 + libunistring >= 0.9.3 (libidn >= 1.25 if you don't have libidn2)
* flex >= 2.5.35
* libpsl >= 0.5.0

The versions are recommended, but older versions are supposed to work.
The versions are recommended, but older versions may also work.


Building from git
Expand Down Expand Up @@ -180,7 +180,4 @@ There is no own documentation yet, but Mget aims to be Wget1.14 compatible.
prints the usage and the current set of integrated options.
For more info, see the man pages of Wget.

My idea is to use the free Serna WYSIWYG/WYMIWYG editor for documentation.
It creates docbook format which can be converted into texinfo format.
And it opens the documentation process to almost any volunteers without
texinfo knowledge.
The Mget library API documentation has been started.
109 changes: 58 additions & 51 deletions configure.ac
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ CFLAGS=$CFLAGS
LDFLAGS=$LDFLAGS

AC_PREREQ([2.67])
AC_INIT([mget], [0.1.5], [[email protected]], [mget], [https://github.com/rockdaboot/mget])
AC_INIT([mget], [0.1.6], [[email protected]], [mget], [https://github.com/rockdaboot/mget])
AM_INIT_AUTOMAKE

CFLAGS=$CFLAGS
Expand Down Expand Up @@ -39,10 +39,11 @@ AC_CANONICAL_HOST

# If CFLAGS isn't defined and using gcc, set CFLAGS to something reasonable.
# Otherwise, just prevent autoconf from molesting CFLAGS.
AC_PROG_CC
#AC_PROG_CC
AC_PROG_CC_STDC

if test "x$CFLAGS" = "x" ; then
if test "x$GCC" = "xyes" ; then
if test "x$CFLAGS" = "x"; then
if test "x$GCC" = "xyes"; then
CFLAGS="-g -std=gnu99 -pedantic -fPIC\
-Wall -Wstrict-prototypes -Wold-style-definition -Wmissing-prototypes\
-Wwrite-strings -Wformat=2 -Wformat -Wformat-security\
Expand All @@ -52,14 +53,14 @@ if test "x$CFLAGS" = "x" ; then
else
CFLAGS="$CFLAGS -fPIC"
fi
if test "x$LDFLAGS" = "x" ; then
if test "x$GCC" = "xyes" ; then
if test "x$LDFLAGS" = "x"; then
if test "x$GCC" = "xyes"; then
# LDFLAGS="-fPIE -pie -Wl,-z,relro,--as-needed"
LDFLAGS="-fPIE -pie -Wl,--as-needed -ffunction-sections"
fi
fi
# Append EXTRA_CFLAGS to CFLAGS, if defined.
if test "x$EXTRA_CFLAGS" != "x" ; then
if test "x$EXTRA_CFLAGS" != "x"; then
CFLAGS="$CFLAGS $EXTRA_CFLAGS"
fi

Expand All @@ -80,7 +81,7 @@ AM_PROG_CC_C_O
# Require C99 support
#
AC_PROG_CC_C99
if test "$ac_cv_prog_cc_c99" = "no" ; then
if test "$ac_cv_prog_cc_c99" = "no"; then
AC_ERROR([Compiler does not support C99. Aborting.])
fi

Expand Down Expand Up @@ -126,12 +127,10 @@ AS_IF([ test "$enable_man" != no ], [


# Check for valgrind
ac_enable_valgrind=no
AC_ARG_ENABLE(valgrind-tests,
AS_HELP_STRING([--enable-valgrind-tests], [enable using Valgrind for tests]),
[ac_enable_valgrind=$enableval], [ac_enable_valgrind=no])

if test "${ac_enable_valgrind}" = "yes" ; then
if test "${ac_enable_valgrind}" != "no" ; then
AC_CHECK_PROG(HAVE_VALGRIND, valgrind, yes, no)
if test "$HAVE_VALGRIND" = "yes" ; then
VALGRIND_ENVIRONMENT="valgrind --error-exitcode=1 --leak-check=yes --show-reachable=yes --track-origins=yes"
Expand All @@ -147,8 +146,7 @@ fi
#AC_ARG_WITH(gnutls, AS_HELP_STRING([--with-gnutls[=PATH]], [compile with SSL support]), , with_gnutls="no")
#if test $with_gnutls != "no"
#then
# if test $with_gnutls != "yes"
# then
# if test $with_gnutls != "yes"; then
# export PKG_CONFIG_PATH="$with_gnutls:$with_gnutls/pkgconfig:$with_gnutls/lib/pkgconfig:$PKG_CONFIG_PATH"
# GNUTLS_BIN_DIR="$with_gnutls/bin/"
# AC_SUBST(GNUTLS_BIN_DIR)
Expand All @@ -160,74 +158,82 @@ fi
# echo "GNUTLS_BIN_DIR=$GNUTLS_BIN_DIR"
#fi

# Check for needed libraries
AC_SEARCH_LIBS(clock_gettime, rt)
AC_SEARCH_LIBS(pthread_create, pthread, [AC_DEFINE([HAVE_LIBPTHREAD], [1], [Use pthreads])])

AC_ARG_WITH(gnutls, AS_HELP_STRING([--without-gnutls], [disable GNUTLS SSL support]), with_gnutls=$withval, with_gnutls=yes)
if test $with_gnutls != "no"
then
AC_CHECK_LIB(gnutls, gnutls_global_init, [with_gnutls=yes; AC_SUBST(GNUTLS_LIBS, "-lgnutls") AC_DEFINE([WITH_GNUTLS], [1], [Use gnutls])], [with_gnutls=no; AC_MSG_WARN(*** GNUTLS was not found. You will not be able to use HTTPS. Fallback to libnettle for Metalink checksumming.)])
AC_CHECK_LIB(gnutls, gnutls_hash, [with_gnutls_hash=yes])
if test $with_gnutls != "no"; then
AC_SEARCH_LIBS(gnutls_global_init, gnutls,
[with_gnutls=yes; AC_DEFINE([WITH_GNUTLS], [1], [Use gnutls])],
[with_gnutls=no; AC_MSG_WARN(*** GNUTLS was not found. You will not be able to use HTTPS. Fallback to libnettle for Metalink checksumming.)])
AC_SEARCH_LIBS(gnutls_hash, gnutls, [with_gnutls_hash=yes])
fi
AM_CONDITIONAL([WITH_GNUTLS], [test $with_gnutls = "yes"])

#AC_DEFINE([WITH_SSL], [1], [Use SSL])]
#AM_CONDITIONAL([WITH_SSL], [test $with_gnutls = "yes"])


if test $with_gnutls != "yes" || test x$with_gnutls_hash != "xyes"
then
AC_CHECK_LIB(nettle, nettle_md5_init, [with_libnettle=yes; AC_SUBST(NETTLE_LIBS, "-lnettle") AC_DEFINE([WITH_LIBNETTLE], [1], [Use libnettle])], [with_libnettle=no; AC_MSG_WARN(*** LIBNETTLE was not found. Metalink checksumming is not available.)])
if test $with_gnutls != "yes" || test x$with_gnutls_hash != "xyes"; then
AC_SEARCH_LIBS(nettle_md5_init, nettle,
[with_libnettle=yes; AC_DEFINE([WITH_LIBNETTLE], [1], [Use libnettle])],
[with_libnettle=no; AC_MSG_WARN(*** LIBNETTLE was not found. Metalink checksumming is not available.)])
fi
AM_CONDITIONAL([WITH_LIBNETTLE], [test x$with_libnettle = "xyes"])

AC_ARG_WITH(libpsl, AS_HELP_STRING([--without-libpsl], [disable support for libpsl cookie checking]), with_libpsl=$withval, with_libpsl=yes)
if test $with_libpsl != "no"; then
AC_SEARCH_LIBS(psl_builtin, psl,
[with_libpsl=yes; AC_DEFINE([WITH_LIBPSL], [1], [PSL support enabled])],
[with_libpsl=no; AC_MSG_WARN(*** libpsl was not found. Fallback to builtin cookie checking.)])
fi
AM_CONDITIONAL([WITH_LIBPSL], [test $with_libpsl = "yes"])

AC_ARG_WITH(zlib, AS_HELP_STRING([--without-zlib], [disable gzip compression support]), with_zlib=$withval, with_zlib=yes)
if test $with_zlib != "no"
then
AC_CHECK_LIB(z, deflate, [with_zlib=yes; AC_SUBST(ZLIB_LIBS, "-lz") AC_DEFINE([WITH_ZLIB], [1], [Use zlib])], [with_zlib=no; AC_MSG_WARN(*** ZLIB was not found. You will not be able to use gzip (de)compression)])
if test $with_zlib != "no"; then
AC_SEARCH_LIBS(deflate, z,
[with_zlib=yes; AC_DEFINE([WITH_ZLIB], [1], [Use zlib])],
[with_zlib=no; AC_MSG_WARN(*** ZLIB was not found. You will not be able to use gzip (de)compression)])
fi
AM_CONDITIONAL([WITH_ZLIB], [test $with_zlib = "yes"])

AC_ARG_WITH(lzma, AS_HELP_STRING([--without-lzma], [disable LZMA compression support]), with_lzma=$withval, with_lzma=yes)
if test $with_lzma != "no"
then
AC_CHECK_LIB(lzma, lzma_code, [with_lzma=yes; AC_SUBST(LZMA_LIBS, "-llzma") AC_DEFINE([WITH_LZMA], [1], [Use liblzma])], [with_lzma=no; AC_MSG_WARN(*** liblzma was not found. You will not be able to use LZMA (de)compression)])
if test $with_lzma != "no"; then
AC_SEARCH_LIBS(lzma_code, lzma,
[with_lzma=yes; AC_DEFINE([WITH_LZMA], [1], [Use liblzma])],
[with_lzma=no; AC_MSG_WARN(*** liblzma was not found. You will not be able to use LZMA (de)compression)])
fi
AM_CONDITIONAL([WITH_LZMA], [test $with_lzma = "yes"])

AC_ARG_WITH(bz2, AS_HELP_STRING([--without-bzip2], [disable bzip2 compression support]), with_bzip2=$withval, with_bzip2=yes)
if test $with_bzip2 != "no"
then
AC_CHECK_LIB(bz2, BZ2_bzDecompress, [with_bzip2=yes; AC_SUBST(BZIP2_LIBS, "-lbz2") AC_DEFINE([WITH_BZIP2], [1], [Use libbz2])], [with_bzip2=no; AC_MSG_WARN(*** libbz2 was not found. You will not be able to use bzip2 (de)compression)])
if test $with_bzip2 != "no"; then
AC_SEARCH_LIBS(BZ2_bzDecompress, bz2,
[with_bzip2=yes; AC_DEFINE([WITH_BZIP2], [1], [Use liblzma])],
[with_bzip2=no; AC_MSG_WARN(*** libbz2 was not found. You will not be able to use BZIP2 (de)compression)])
fi
AM_CONDITIONAL([WITH_BZIP2], [test $with_bzip2 = "yes"])

AC_ARG_WITH(libidn2, AS_HELP_STRING([--without-libidn2], [disable IDN2 support]), with_libidn2=$withval, with_libidn2=yes)
if test $with_libidn2 != "no"
then
AC_CHECK_LIB(idn2, idn2_lookup_u8, [with_libidn2=yes; AC_SUBST(IDN_LIBS, "-lidn2") AC_DEFINE([WITH_LIBIDN2], [1], [Use libidn2])], [with_libidn2=no; AC_MSG_WARN(*** LIBIDN2 was not found. You will not be able to use IDN2008 support)])
if test x$with_libidn2 = "xyes"
then
AC_CHECK_LIB(unistring, u8_tolower, [with_libunistring=yes; AC_SUBST(IDN_LIBS, "-lidn2 -lunistring") AC_DEFINE([WITH_LIBUNISTRING], [1], [Use libunistring])], [with_libunistring=no; AC_MSG_WARN(*** LIBUNISTRING was not found. You will not be able to use IDN2008 on uppercase domains)])
if test $with_libidn2 != "no"; then
AC_SEARCH_LIBS(idn2_lookup_u8, idn2,
[with_libidn2=yes; AC_DEFINE([WITH_LIBIDN2], [1], [Use libidn2]) IDNA_INFO="IDNA 2008 (libidn2)"],
[with_libidn2=no; AC_MSG_WARN(*** LIBIDN2 was not found. You will not be able to use IDN2008 support)])
if test $with_libidn2 = "yes"; then
AC_SEARCH_LIBS(u8_tolower, unistring,
[with_libunistring=yes; AC_DEFINE([WITH_LIBUNISTRING], [1], [Use libunistring]) IDNA_INFO="IDNA 2008 (libidn2+libunistring)"],
[with_libunistring=no; AC_MSG_WARN(*** LIBUNISTRING was not found. You will not be able to use IDN2008 on uppercase domains)])
fi
fi
AM_CONDITIONAL([WITH_LIBUNISTRING], [test x$with_libunistring = "xyes"])
AM_CONDITIONAL([WITH_LIBIDN2], [test $with_libidn2 = "yes"])

if test x$with_libidn2 != "xyes"
then
if test x$with_libidn2 != "xyes"; then
AC_ARG_WITH(libidn, AS_HELP_STRING([--without-libidn], [disable IDN support]), with_libidn=$withval, with_libidn=yes)
if test x$with_libidn != "xno"
then
AC_CHECK_LIB(idn, idna_to_ascii_8z, [with_libidn=yes; AC_SUBST(IDN_LIBS, "-lidn") AC_DEFINE([WITH_LIBIDN], [1], [Use libidn])], [with_libidn=no; AC_MSG_WARN(*** LIBIDN was not found. You will not be able to use IDN support)])
if test $with_libidn != "no"; then
AC_SEARCH_LIBS(idna_to_ascii_8z, idn,
[with_libidn=yes; AC_DEFINE([WITH_LIBIDN], [1], [Use libidn]) IDNA_INFO="IDNA 2003 (libidn)"],
[with_libidn=no; AC_MSG_WARN(*** LIBIDN was not found. You will not be able to use IDN support)])
fi
fi
AM_CONDITIONAL([WITH_LIBIDN], [test x$with_libidn = "xyes"])

# Checks for libraries.
AC_CHECK_LIB([pthread], [pthread_attr_init])
AC_CHECK_LIB([rt], [clock_gettime])

AC_SUBST(LIBS, "$GNUTLS_LIBS $ZLIB_LIBS $LZMA_LIBS $BZIP2_LIBS $IDN_LIBS $NETTLE_LIBS $LIBS")

# Checks for header files.
AC_CHECK_HEADERS([\
fcntl.h inttypes.h libintl.h locale.h netdb.h netinet/in.h stddef.h stdlib.h string.h\
Expand Down Expand Up @@ -287,6 +293,7 @@ AC_MSG_NOTICE([Summary of build options:
GZIP compression: $with_zlib
BZIP2 compression: $with_bzip2
LZMA compression: $with_lzma
IDN support: $IDN_LIBS
IDNA support: $IDNA_INFO
PSL support: $with_libpsl
Tests: ${TESTS_INFO}
])
2 changes: 1 addition & 1 deletion data/Makefile.am
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
filesdir = $(datadir)/@PACKAGE@
files_DATA = public_suffixes.txt
files_DATA = effective_tld_names.dat
EXTRA_DIST = $(files_DATA)
Loading

0 comments on commit 3d8c70d

Please sign in to comment.