Process tags in [source]:[asset] format where source is MUTF_CA

[bradsteels]

Getting the following error messages (for mutual fund files). It seems to update ETFs (XIC.TO) No clue how to rectify. Is there an easy way to get Google finance data instead?

Unable to get information for TD Canadian Bond Index - e (MUTF_CA:TDB909)
Unable to get information for Mawer New Canada (MUTF_CA:MAW107)
Unable to get information for Chou RRSP (MUTF_CA:CHO102)
Unable to get information for Chou Asia (MUTF_CA:CHO300)
Unable to get information for CIBC Canadian Bond Index (MUTF_CA:CIB503)
Traceback (most recent call last):
File "investoscope_update.py", line 203, in
main()
File "investoscope_update.py", line 192, in main
csv_text = get_quote(item)
File "investoscope_update.py", line 140, in get_quote
return get_yahoo_quote(item)
File "investoscope_update.py", line 134, in get_yahoo_quote
return download_yahoo_csv(item, yahoo_crumb, yahoo_cookie)
File "investoscope_update.py", line 117, in download_yahoo_csv
return tidy_yahoo_csv(csv_data.decode('utf-8'))
File "investoscope_update.py", line 93, in tidy_yahoo_csv
row.pop(5)
IndexError: pop index out of range

[Rod-Persky]

At present I've only scraped data from the Yahoo Finance site. However, when you added the symbol MUTF_CA:CIB503 you've found a bug due to the assumption that the stock codes exists 👍

@bradsteels I'll look into how to extract data from the google finance pages, although it will be some time before I'll have time to do so.

[charlietomo]

I had the same issue IndexError: pop index out of range. However the script was NOT detailing which tickers it was having an issue with. I suspect your error is not related to the 5 tickers you have listed - the script isn't getting data for them, but it should complete otherwise. The error is with another ticker.

I traced it down to the underlying data on Yahoo - and some instruments do not have the same data as shares / the same data the script is looking for.
If you look at VOD.L (https://finance.yahoo.com/quote/VOD.L/history?p=VOD.L) as an example of a share that works. If you go through all your shares that are set to Yahoo as a quote feed, if any don't look like the above link then change them to a manual feed, run the script again and see how you get on.

I went through 150+ pages, removed approx 10, then my error message went away - phew!

@Rod-Persky it would be great if the script could output which instrument it is having an error on - would save a lot of manual effort :-) Thanks for your work with this script.

[Rod-Persky]

I know this bug, but it isn't really to do with the yahoo data source 👍.

The reason it occurs is due to the assumption that the user added a ticker to their library using the search function, where the user selects yahoo as the data source and therefore the code should be formatted in the way yahoo expects. MUTF_CA:CIB503 is not a yahoo code (it's a google code) however it's likely that the user changed the quote source to yahoo. This app expects that the user uses the search function and only therefore the tickers are properly formatted yahoo formatted codes.

There are two options really:

1. Ignore the error, issue a warning "error occurred when accessing data" and move right along (user friendly)
2. Fix the scraper to ensure that the ticker code is correctly formatted and keep the warning (as it's not actually a bug)

I'll take action 1 for now and then look into fixing the script. The error probably will not help much.

[bradsteels]

Thanks for the update! Meanwhile the script is updating my ETFs that are listed in Yahoo and I'm certainly grateful for the time it saves.

[Rod-Persky]

@bradsteels I should still do something regarding this 👍

[charlietomo]

Regarding this it would be great if the script listed out the instrument that was having issues - rather than just failing cryptically "IndexError: pop index out of range".

[Rod-Persky]

Hi @charlietomo, I should check the status of the scripts, but I think I did update the program to give better feedback last time you mentioned this 😎. 1. Were you able to update since then? 2. Would it be a good idea to add a mechanism to check for the latest release 3. Should i simply tag some pushes as a release Cheers!

[charlietomo]

Hi @Rod-Persky actually I've just upgraded to the app and it worked well - and I can't replicate my previous errors, so that is great.

1. Yes I have updated.
2. A mechanism to check for release would be good, if you are planning on regular updates :-)
3. I'm not a big Github user so am not sure on the repercussions of this, but being clear on releases would be helpful - and having release notes. There is a massive jump (for the better) e.g. between script and app.

[Rod-Persky]

+1 release notes and some update mechanism would be fantastic additions. I will look into how this may be implemented in the future.

[Rod-Persky]

@charlietomo I added a 'release' that can simply be downloaded... that should address point number 3

[Rod-Persky]

• A mechanism to check for release
• Having release notes
• Show the symbol for which error occur (e.g. pop index out of range)
• Mutual funds e.g. MUTF_CA

[Rod-Persky]

@bradsteels do you have a website that you usually get this info from?

[bradsteels]

i used to get it from Google Finance using Google Docs spreadsheet but recently they stopped updating the mutual fund price histories, so I don't know where to get it now, unfortunately.

[Rod-Persky]

@bradsteels does any provider on the internet show an interactive chart of these mutual funds? If so it should be fairly easy to load it 👍 ... unless:

1. It requires a login
2. It runs on an asp.net server
3. It has some very special state based stuff (ewwwww)

[bradsteels]

it seems that google and yahoo finance do show daily fund prices but do not show historical or chart data now for funds such as CHO102 that I am interested in. i can find historical charts at www.fundlibrary.com without a special login but it may be an asp.net unfortunately.

[Rod-Persky]

Fundlibrary uses non-interactive charts (static images) and asp.net which means getting the data involves pain and image processing haha

[charlietomo]

I understand the app uses either the historical page or the interactive chart. I'm looking at some funds (e.g. GB0007716078.L) and the latest trade on the chart is correct - but there is no history (just zero). The historical data tab shows, somewhat randomly, just over a week in June(!). Investoscope Updater pulls the out of date June data - not the up to date latest trade.

For me, it would be great if we could pull that last trade price, and update Investoscope - understanding that there is no history. My current solution is to have these as manual entries, which get updated manually after I run the updater. Anything I can do to force using the last value from the chart?

[Rod-Persky]

The data seems really sporadic. Is the value meant to be around 2165 or 20? check the historical quote here

[charlietomo]

The data seems really sporadic. Is the value meant to be around 2165 or 20? check the historical quote here

Yes the history is very sporadic. I haven't found a good source for historic.
2,165 GBp (pence) or 21.65 GBP (pound) is correct. That historical link showing 20 is in GBP (not GBp as says on Yahoo page...).