Make a Release

[snejad-quantopian]

Our last Zipline release was 1.3 and released on Jul 17, 2018. That is quite a long time ago but we are definitely planning on releasing a new version of the project. There are a few issues that we're actively working on and are planning to address before we do so.

[snejad-quantopian]

@ssanderson

[ssanderson]

A couple issues that we'd probably like to address before releasing:

• Remove Implicit Dependency on Benchmarks and Treasury Returns #2627
• Support for Newer Python Versions #2616
• We're in the process of trying to migrate a bunch of our packaging infrastructure to conda forge. See notes in Migrate CI to GitHub Actions #2637.

Are there other important issues that should block releasing a new version?

[carstenf]

@ssanderson
Would it as well include a possibility to use fundamental data?
There are some ideas already on this GitHub.

[ssanderson]

@carstenf I'm not sure what it would look like for Zipline to support fundamental data in a first class way.

On Quantopian we support two fundamental data providers, Morningstar and FactSet, but their data structure (and even what they count as "fundamental" data) is quite different. It's not clear to me how we could allow Zipline to support general fundamental data in a way that wouldn't be tied relatively tightly to a specific (and likely non-free) vendor's data.

[carstenf]

@ssanderson

I agree, it would not a plug-and-solution, but the same is true with your regular ingest process. If one wants to use a different data provider, one needs to change the code slightly.

If found two solution were you could start, and just maybe, add a example how to use Quandl data. The examples uses this as well.
(Morningstar and FactSet might be a bit expensive for starting...)

I could use these, but maybe after one of your updates, things will not work.
It would be better using something which is supported by zipline.

Thanks
Carsten

Examples:
http://alphacompiler.com/blog/6/

https://github.com/calmitchell617/Springbok

[cemal95]

Does anyone have example or know how to make a Pipeline locally on Zipline?

[tdrobbin]

@ssanderson perhaps not fundamental data specifically, but if you could add a utility or documentation on incorporating general data into a DataSet to be used by factors and pipelines that would be incredibly helpful.

Currently it's pretty easy to add custom pricing data for running backtests by adding somthing like this in extensions:

register(
    'new_bundle', 
    csvdir_equities(
        ['daily'],
        '/path',
    ),
    calendar_name=name
    start_session=start_session,
    end_session=end_session
)

then running: zipline ingest

I'm not entirely sure what the APIs would look like but would it be possible to mirror this type behavior for loading arbitrary csv data to be used as datasets in factors and pipelines. perhaps something like this could also go in the extensions file:

class FundamentalData(DataSet):
    price_to_earnings = Column(float64_dtype)
    price_to_book = Column(float64_dtype)
    ...

register_dataset(
    FundamentalData,
    csvdir_equities(
        ['daily'],
        '/path/',
    ),
    # maybe just explicitly map the names of the columns in the csv to the Column names
    # defined in the DataSet in case they differ
    columns={
        'price_to_earnings': 'price_to_earnings',
        'price_to_book': 'price_to_book',
        ...
    }
)

You might then need to run a similar ingest type command or zipline could just read the extension and then load the data dynamically from the csv file at backtest runtime.

is something like this possible? or does functionality like this already exist?

@cemal95 have you looked here: https://github.com/quantopian/zipline/blob/master/zipline/examples/momentum_pipeline.py

[billmoling]

#2434
the hard coded NYSE in data bundle. if there could be a fix, or a new document to address this would be great

[ksyme99]

While I would appreciate the fundamental data enhancement, genuinely would be good to get a new stable release out of whatever features are currently done, ASAP, with the focus being on bugs and supporting newer versions of pandas/etc. I have used zipline in-house for 18 months or so, but are exploring other options now as the number of workarounds and issues we are having is growing, and while supposedly supported by Quantopian, the lack of releases in almost 2 years doesn't fill us with confidence any more.

(We ingest our own price data from an external (paid for) provider - writing the custom data bundle in general works fine, apart from performance issues. We have a separate database of PiT fundamentals and just look that up when needed for the given backtest day. Don't use any of the factor stuff yet, mainly as can't load in custom data for creating them).

[suitablyquantified]

It's difficult to tell whether there will be much more development going on. I sure hope so.

There are still many new people trying Zipline and running into significant roadblocks due to the 1.3.0 release having issues that have been resolved in the latest code. You can see the new issues coming in.

However, the majority of employees were furloughed in late April and others in May (including Scott Sanderson who appears to be leading the commits):
https://permadeath.com/Moving-on-from-Quantopian/
https://www.partiallyapplied.com/blog/moving-on/

It's a pity because there were a bunch of commits and really good work done in April.

It would be nice to get an opinion from any current or ex-Quantopians of:

1. "where things are realistically currently at" with regards to a release of Zipline. Please tell us how it is (and if you want to do it anonymously or via a third party that's fine too).
2. Likelihood of Quantopian issuing a new release in the next few months

Of course, open source has a way of working through such difficulties through forks and/or project ownership transfer,, but the folks at Quantopian have done so much good work to date it'd be great to hear from them.

[ksyme99]

However, the majority of employees were furloughed in late April and others in May (including Scott Sanderson who appears to be leading the commits):

I did not realise that had happened, obviously hope people who have been impacted are OK, and hope to get an update from Quantopian people when they can on the impact on zipline.

[weiguang-zz]

fundamental support +1

[richafrank]

Hi all,

As mentioned above, Quantopian is a much smaller team now, though we hope to continue making (slower) progress.

Our first goal is to cut a zipline 1.4.0 release that includes the benchmark/treasury curve downloading improvements, python 3.6 CI/packages, and all the international tooling - basically everything from the past 2 years (changelog and highlights in progress).

Our next goal is for a subsequent release that improves the packaging/installation situation, including moving to conda-forge and supporting newer pythons.

I believe the benchmark downloading and packaging issues are the two biggest, based on the discussions here in GitHub, and improving our packaging/CI will also speed up our ability to fix/test/confirm issues and pull in PRs.

[richafrank]

Released https://github.com/quantopian/zipline/releases/tag/1.4.0