calculate prices from csv

[scientes]

This pr does 4 Things:

1. updates the year to 2021
2. All values are now stored as str in db to prevent floating point bugs in the future (there is no decimal in sqlite)
3. Tablenames are now Alphabeticaly sorted ETH/BNB -> BNB/ETH and converted to prevent two tables being created for the same pair
4. If there are two timestamps from the same platform the price between them is calculated when one is a buy operation and the other is a sell operation ([Binance] Calculate price from transaction statement #50)

[provinzio]

Hey @scientes,

good idea to group the tables of opposite coin pairs. I am currently unsure, if this is a good idea. e.g. BTC/USD is like 60.000$ but USD/BTC would be 0.00001666666 which is unnecessary small.

Is it enough to save the decimals as string for that? Please check this explicitly.

[Griffsano]

Hey, I think the inversion of prices will get problematic for coins that are listed for 0 EUR (e.g. airdrops of some new tokens). Would it be possible to search the database for the inverse coin pair if the original cannot be found, before creating a new table?

[provinzio]

Hey, I think the inversion of prices will get problematic for coins that are listed for 0 EUR (e.g. airdrops of some new tokens). Would it be possible to search the database for the inverse coin pair if the original cannot be found, before creating a new table?

If the coin pair ABC/XYZ is valued at 0 Eur, the coin pair XYZ/ABC should be 0 Eur as well. Therefore creating the reciprocal requies to except the case when the pair is valued at 0 Eur. The reciprocal function in misc.py does exactly that. I mentioned it in the code review.

Furthermore, this PR sorts the coin pair alphabetically to form the table name. Therefore, each coin pair plus his "inverse coin pair" will have exactly one table. There wouldn't be the need to search for another table, as there is only this one.

[Griffsano]

You're right, it should also work with this function, basically overwriting inf by 0. Of course, if the inverse price can always be computed, the alphabetic sorting works and we don't have to search for the inverse coin pair in the database.

[scientes]

Hey @scientes,

good idea to group the tables of opposite coin pairs. I am currently unsure, if this is a good idea. e.g. BTC/USD is like 60.000$ but USD/BTC would be 0.00001666666 which is unnecessary small.

Is it enough to save the decimals as string for that? Please check this explicitly.

The default precision of decimal is 28
also small test:

>>> Decimal(1)/(Decimal(1) /Decimal(123456789123456789))
Decimal('123456789123456789.0000000000')
>>> Decimal(1)/(Decimal(1) /Decimal(123456789123456789.1))
Decimal('123456789123456784.0000000000')
>>> Decimal(1)/(Decimal(1) /Decimal(12345.1))
Decimal('12345.10000000000036379788071')

the max of string length in sqllite is 2^31-1 bytes which i think is more than enough

I'm not super sure how to fix that properly, i mean it's still more accurate than fetching the 1 minute average. The easiest would be just to up the precision on decimal. or we could make two columns in the db one for the price and one for if its inverted relative to the table name and we do not invert the price at all

[provinzio]

The precision should be high enough as it is right now. At least I don't know of a coin pair with such a huge factor. Binance shows the prices with 8 decimals places (at least in the GUI).

[scientes]

@provinzio I added the migrations to the check_db function. but apparently its not used anywhere. is that the correct place for the migration or should i make an extra function for them?

[provinzio]

I added some docstrings and comments and worked on an first idea of a patch system. Unfortunatly, I wasn't able to finish my work and am still unsure whether this is the best approach. I think that we are currently missing a central database class which handles the interaction between the code and our database system. We might want to factor out the database functions from price_data.py and some functions from patch_database.py and move them to a new database.py file. What do you think of that?

The patches in patch_database.py aren't finished, but my ground work should give an idea of what is missing. We need a create patches, so that (my) old databases with unsorted tablenames and float prices are converted to the new format with sorted tablenames and string prices.

Can you work on it?

[provinzio]

This PR definitly got quite out of scope quickly, but you two, @Griffsano and @scientes, handled it really good, so that we can now calculate the prices directly from CSV files and have an database versioning system!

While reviewing, I noticed, that we got the database type of the price column wrong. "STR" is not supported by sqllite. Please check your databases for the correct type, which should be at least VARCHAR(255). Otherwhise your prices will be stored with a maximal precision of 1e-9.

I added the functionality, that prices from CSV overwrite already existing prices in the database.

Again, good job you two!