Bitcoin-data manipulaton and plotting in python

 

plot.png - https://i0.wp.com/raw.githubusercontent.com/Altoidnerd/bitcoin-price/master/plot.png?resize=840%2C474&ssl=1
I wrote a python mini-api to manipulate and visualize the price of bitcoin historically using the coinbase version 1 api.

bitcoin-price (python module)

Get bitcoin price data from the coinbase API and parse it into an easily manipulated form.

>>> from fast_dump_v22 import *
>>> get_page(1)

Will return all the data on page 1 of coinbase’s bitcoin price API (this is the latest data).

You can almost always turn on optional debug statements

>>> get_first_N(3, show=True)
... fetching first 3

You can get all the price data that coinbase has

>>> get_all(show=True)
... getting page 1
... getting page 2
... getting page 3
... getting page 4
... getting page 5
... getting page 6
... getting page 7
... getting page 8
... getting page 9
... etc ...

All the ‘get_*’ functions return a price_data string, which is interlaced timestamps and prices littered with newlines and commas. You can print them to see what is going on more clearly:

>>> print(get_page(1))
2015-09-11T06:44:04-05:00,241.14
2015-09-11T06:34:04-05:00,240.8
2015-09-11T06:24:04-05:00,240.75
2015-09-11T06:14:05-05:00,240.68
2015-09-11T06:04:04-05:00,240.83
2015-09-11T05:54:05-05:00,240.92
2015-09-11T05:44:04-05:00,240.64
2015-09-11T04:34:04-05:00,241.27
2015-09-11T04:24:04-05:00,240.73
...

Turn on the optional show switch for printing large vectors

>>> prices(get_page(11), show=True)
... returning 11000 prices in specified range ...
['239.9',
'239.9',
'239.4',
'239.77',
'239.33',
'239.99',
'239.81',
'240.28',
'240.4',

You can use prices(data)[k] and timestamps()[j] to return the kth price in data, or the jth timestamp in data.

>>> data = get_page(1)
>>> prices(data)[4]
'241.2'
>>> prices(data, index=4)
'241.2'

are two equivalent ways of returning only the 4th price in the requested range (in this case, page 1). This also works for timestamps.

>>> timestamps(get_page(1)+get_page(2))[1166] == timestamps(get_first_N(2), index=1166)
True

This shows the expressiveness of this module. In general:

>>> prices(get_page(2)) == parse(get_page(2))[0]
True

prices() and timestamps() are just functions that return a parsed() object having a specific index, or indices.

>>> parse(get_page(1)+get_page(2)+get_page(3))[0] == prices(get_first_N(3))
True
>>> parse(get_page(2)+get_page(3))[0] == prices(get_range(2,3))
True

The parse() function is there to manually control the outputs instead of just getting prices, or timestamps

>>> x = parse(get_page(1))
>>> x[0][0]
'241.2'
>>> x[0][1]
'241.14'
>>> x[1][1]
'2015-09-11T04:34:04-07:00'

As you can see, parse(price_data)[0][k] returns the kth price in the list. Indices [1][k] return the kth timestamp.

The parse() function takes care of some weird edge cases:

>>> get_first_N(3) == get_page(1)+get_page(2)+get_page(3)
False
>>> parse(get_first_N(3)) == parse(get_page(1)+get_page(2)+get_page(3))
True
>>> x = get_page(1)
>>> y = get_range(2,7)
>>> prices(get_first_N(7)) == prices(x+y)
True

In general,

OPERATOR( get_page(1) + get_page(2) + ... + get_page(k) ) == OPERATOR(get_first_N(k))

where OPERATOR is parsed(), prices(), or timestamps(). We also know prices() can obviously display and return ranges of values. When returning large vectors, you can verify their length by setting show=True. The “show” parameter is optional for all get_* functions and provides some information about the operation being performed.

>>> print( prices(get_first_N(11), show=True) )
... returning 11000 prices in specified range...

since each page is a thousand pairs of values (timestamp, price).

>>> len(prices(get_page(2))
1000
>>> prices(get_page(2)) # returns a long list
['230.11',
'229.64',
'230.04',
'229.71',
'229.69',
'229.74',
'229.92',
'229.43',
'229.43',
'229.41',
...

fast_dump_v1* are older versions that are somewhat different. They are designed to store the fetched data in the .data directory. This in v2*, this was abandoned in favor of stdout redirection.
// 2015 altodinerd.com

Getting started with Bitcoin the “right way” – encrypted wallets, bitcoin core, and linux

A brief introduction to getting started with bitcoin on a linux distribution

For the impatient: Skip to step 1

Bitcoin is one of the most interesting technologies we have seen emerge in the past ten years. And it confuses everyone who sets out to use it.  This post is my definitive guide to beginning your journey with bitcoin the right way.

tuxbig

To me, the right way includes, but is not limited to:

  • Running bitcoin on a linux machine.  My reasons for this are too many to really list here, but suffice it to say that since linux is widely regarded as the most secure operating system available, and the usage of bitcoin involves being your own bank, using bitcoin on linux is the best way to ensure your coins stay safe.
  • Using the bitcoin-core wallet software.  There are other wallets available.  You actually don’t even need a wallet software to own bitcoins or even spend bitcoins.  However, if you want to understand how bitcoin works under the hood – which you should, because you are going to be your own bank, you should use this software.
  • Using the command line to do bitcoin things. Actually, using both the GUI and the command line is the way to go.  This tutorial (this is part 1 of 2) will be a command-line-centric description because afterall, I’m trying to show you the right way to do things.

But that’s just, like, your opinion man.

It is. This is my blog.

You shouldn’t necessarily trust me or anyone – bitcoin isn’t about trust. But if you google “altoidnerd”, you’ll find that I have been around the bitcoin eco-system for quite some time. That’s all I can tell you.

Screen Shot 2015-09-10 at 8.31.28 PM

There are just so many wallets available – why bitcoin-core?  Because I have been using it for years, I have a very good method of keeping my coins safe, and that is what I am sharing with you today.  Use it- it is THE wallet.

But altoidnerd, I don’t want to run linux. I like windows / OSX

I am a linux and an OSX user.  So please understand the scope of my knowledge includes these operating systems. If you want to use OSX, most of this tutorial will make sense with minor changes perhaps.

Screen Shot 2015-09-10 at 8.32.59 PM

If you want to use windows, that’s ok, I’m not going to judge you. Bitcoin-core is available for windows as well. Please understand however, I am not a windows user, so I will not be able to give you “the definitive bitcoin windows guide”. You can still read on, however, since you can apply much of this tutorial to usage of bitcoin-qt’s debug window instead of the command line.

Step 1: Download and install bitcoin-core

Navigate to bitcoin.org’s official download page and select the bitcoin-core distribution for your operating system. And do choose linux (tgz).  Do that because you should be running linux.  But if you aren’t, even though you should be, choose the distribution for your operating system. Extract and install the client.






The “right way”

To really take control of things, and understand what is happening, you’re going to want to run bitcoin software from the command line. Though it isn’t really necessary to download the thing from the command line, I’m going to describe that here because this tutorial is the first installment of a start to finish command line approach. For a sneak peak of what I mean, see an earlier post where I described how to use bitcoin-core to generate QR codes without having to trust shady ass websites.

Go to your home directory and create a new directory just for bitcoin.  This is not a necessary step, but once again, this is my “right way” tutorial and I will explain reasons for this later on.  Enter the directory, and download the tarball with the “wget” command.  The extract the tarball like so:

    cd ~

    mkdir bitcoin

    cd bitcoin

    wget 'https://bitcoin.org/bin/bitcoin-core-0.11.0/bitcoin-0.11.0-linux64.tar.gz'

    tar -xzvf bitcoin-0.11.0-linux64.tar.gz

bitcointerm1

 

Sweet.  Now change directory to where the binaries are,

    cd bitcoin-0.11.0/bin/

and launch “bitcoin-qt” (qt means its the GUI version).

./bitcoin-qt

It’s going to show you the dialogue box below and ask you if you’d like to use the default directory. Do it. Click “OK”.

 

default

 

You’ll see a friendly startup screen if you’ve done things right!

bitcointerm2

 




 

Step 2: Encrypt your wallet, and wait a few days

Bitcoin-core is called a “full-node” implementation, which means the first time you start it on your machine, it’s going to download every single bitcoin transaction that has ever taken place.  Wait until the blockchain is synced to start making transactions.

In the mean time, you need to set a passphrase for your wallet.  This is super important.  In the upper left corner of bitcoin-qt, go to settings -> Encrypt Wallet, and set a very strong passphrase.

encrypt

This will make it impossible for anyone to send bitcoins from your wallet without entering your password:

PASSWORD

Just make sure you:

    • Make your password extremely long. I don’t know if there is a length limit in the bitcoin-core code or not. Regardless, seriously push your personal boundaries on what you think a safe password is. Make it super duper long, and impossible to guess. Wise men have discussed password strength to a great extent. Pick a strong password. 20+ characters. Go for 30. No 40. Just make it long, and keep it an absolute secret.
    • Never, ever lose or forget this password. If you lose or forgot your password, your coins will be unspendable forever. Neither you, nor anybody on earth, nor God himself will be able to spend your coins. Your coins will be effectively lost.





Picking a strong passphrase makes security so, so easy

Once you encrypt your wallet, you can copy and store wallet.dat everywhere (I explain wallet.dat in great detail in “Step 3”). I literally have copies of my main wallet.dat on 6 or 7 computers, because come hell or highwater, I will always be able to find a copy of my wallet.dat file. I suggest you do the same. Here is why strong encryption and insane redundancy works:

  • If your password is ridiculously long, you can be very lazy and downright cavelier about where you keep your wallet.dat, because even if someone finds the wallet file, they wont be able to do a damn thing with it. They won’t be able to spend your coins without your passphrase.
  • This means that you can make copies of wallet.dat and store them like literally everywhere, on all your machines, three times over. In the cloud. On your mom’s computer. Hell, theoretically speaking, you can post your encrypted wallet.dat on the internet and just about everyone will have a copy, and be unable to spend your coins, because they don’t know your password. Someone remind me to put some coins into an address, and post the encrypted wallet online to prove my point.

    Bottom Line

    If you encrypt your wallet like a boss, you never forget your password, you never tell anyone your password, and you copy the file to everywhere you possibly can imagine, you will never lose your coins. And nobody will be able to steal them from you. Seriously, nobody. It is computationally impossible.

    Step 3: Understand what you just did on your computer

    Let’s break it down this way. Here are some things that just happened when you launched bitcoin-qt for the first time:

    • A hidden directory ~/.bitcoin was created. It contains among other things the single most important thing ever: a file called wallet.dat. This file is critical. ~/.bitcoin/wallet.dat contains all of your private keys. wallet.dat is life. When you see in headlines in the news like “Frustrated gentleman quite upset having lost a usb jump drive containing 100,000 BTC“, it means he doesn’t have a copy of wallet.dat, so neither he, nor anyone on earth, nor God himself can spend his coins anymore.
    • Your computer started downloading the blockchain. The blockchain is a record of every bitcoin transaction ever to have taken place. This will take a few days. As of today, the blockchain is about 42 gigabytes. Make sure you have space for it.

    Do not delete your wallet.dat file. Just don’t. If you want to start a new wallet, instead of deleting wallet.dat, just rename it to something else, like wallet.dat.old, restart bitcoin-qt, and it will create a new wallet for you. Keep your wallets people. Keep them good.

    wallet

    Notice how I have a file called wallet.dat.default? That’s because I never delete a wallet. When I installed bitcoin on this computer, I moved the wallet.dat that stores my coins into ~/.bitcoin/, but first I renamed the existing file so I can keep it. Because, why not? I’m telling you – don’t delete a file called wallet.dat, because mistakes happen, and its better to have hundreds of files named wallet.dat.* than it is to lose your coins.

In my next post, I will describe some of the features of the bitcoin-qt debug window, bitcoind and its helper bitcoin-cli.  You can do interesting things, like import new keys, or convert keys to qr-codes.




If you have ay questions, please feel free to comment or contact me.
If you liked this post, you can donate bitcoins to me here: 12gKRdrz7yy7erg5apUvSRGemypTUvBRuJ
Learn about the author here:
http://www.phys.ufl.edu/~majewski

How import a bitcoin address private key into breadwallet (iOS) with a QR code using bitcoin-core for Ubuntu / Linux

On encoding private keys from Bitcoin-core as QR codes for use in paper wallets, other wallet software, etc …

Note: Breadwallet will not import the private key upon scanning the qr code. It will offer to transfer the funds in that address to your breadwallet keyring.

Breadwallet, an awesome open source iOS bitcoin SPV client, allows you to import a private key for use within breadwallet.  So, if you’re a user of bitcoin-core, and want to be able to spend some of your funds you have there with breadwallet, you’ll need to create a QR code of your private key to scan it with breadwallet.

Install qrencode

This tutorial uses qrencode, an open source string to QR converter. The source can be found here: https://github.com/fukuchi/libqrencode/

Alternatively, you can install qrencode on ubuntu/debian through repositories

sudo apt-get install qrencode

Then, as of bitcoin-core version 0.11.0 you’ll need to run bitcoind rather than the GUI bitcoin-qt. If you’re using an old version of bitcoin-core that does not have the bitcoin-cli tool, you can get the latest version of bitcoin-core and extract it without having to verify the blockchain again or mess with wallet.dat – your ~/.bitcoin directory will remain unchanged.

Get the newest version of Bitcoin-core


wget 'https://bitcoin.org/bin/bitcoin-core-0.11.0/bitcoin-0.11.0-linux64.tar.gz'
tar -xzvf bitcoin-0.11.0-linux64.tar.gz
cd bitcoin-0.11.0/bin

You should see the executables for the bitcoin-core suite here.
image1



Run bitcoind and create your QR

Be sure you quit any versions of bitoin-qt or bitcoind that are already running, then start the bitcoin daemon

./bitcoind &

The & operator makes sure bitcoind detaches from your terminal so you can keep going.

Now you are ready to get a QR code for your desired address. Is your address is “addr”, you can generate a QR code of your private key without ever showing your private key on screen in plain text. Just do:

./bitcoin-cli dumpprivkey "addr" | qrencode -o ~/Desktop/Key.png

Or better yet, if you want to create a new address and QR-encode its private key,

addr=$(./bitcoin-cli getnewaddress)
./bitcoin-cli dumpprivkey $addr| qrencode -o
~/Desktop/$addr.privkey.png
echo "private key dumped for: "$addr

The QR code should appear on your desktop. Below is a screen shot of me doing an example where I create a new address, and get a QR for its private key.  Enjoy!

image0



Note that if your wallet is password protected (which it really should be), you’ll need to first do

./bitcoin-cli walletpassphrase "your_password" 100

where the argument 100 means the wallet will allow all actions (like spending, or dumping keys) for 100 seconds. That should be plenty of time.




Bitcoin midterm slump and log linear price prediction model

Perhaps you have noticed bitcoin has taken a dive in recent weeks.  The question on everybody’s mind must then be “gosh, is this normal? From a historical context, is bitcoin OK?”

If we disregard all real world facts (which show the bitcoin network IS healthy, and the transaction malleability doesn’t represent any fundamental failure, much less a new risk to the network) and if we just stick to data, and math… my humble opinion would be “this behavior is nothing out of the ordinary for bitcoin;  we are seeing a price fluctuation very typical of historical data.”

I have constructed a log plot and fitted to two parameters, as I have done before, just to demonstrate that since bitcoin grows in powers of 10, (seemingly large) contemporary fluctuations may look alarming. This rests on the observation I and perhaps a few others have made, that the bitcoin price follows a log linear growth model, on average, according to what we may expect from the diffusion equation.

In the following chart, the blue dots represent base 10 logarithm of the price data since September 13, 2011. The orange line is the best fit line, allowing the price on day (1) to float and be fitted.  The vertical axis is the base ten log of the price, so you can get the price meant for vertical value “3” by computing 10^3 = 1,000 (all in USD).

fit PNG

You might compare this plot with the one I made months ago.

You can use my sheets to perform other types of fits if you wish to choose a particular value for the price on a particular day, or if feeling frisky, try to extract periodic trends.  

By taking the exponential of the log fit, we can “predict the bitcoin price into the future.”  Note: This does not really work.  But it’s interesting.  I decided to take the fit out to 1 year from today – what does the log linear fit say the price will be then, on February 20, 2015? For this we must look to day 1258 (that’s 893 + 365)…and the result is about 5 grand (the prediction is $4,943.89

future white

Just for fun, here is the log of the price with all axes blacked out and no trendline so you can see the raw (log) data for what it is.

logplotfree

Here is the raw price data in similar fashion, without compression via the logarithm:

nolog

I grabbed the daily weighted price from bitcoincharts.com for selections “all data” and “auto” for resolution.  That resulted in a vector of 893 numbers – the bitcoin average price in dollars each day since September 13, 2011.   I had to erase a handful of “infinities” that came out of the raw data at bitcoincharts; there were perhaps ten such cells for which I manually put in reasonable values.  All of the infinities happened within the first 100 days of the series.

In general, these plots cannot be taken too seriously, although they have given me insight in the past.  If you have any questions, please feel free to contact me.  Cheerio.

Sheet and raw data:

data, pastebin http://pastebin.com/YntGdH9t

Note: the following link is to directly download my excel workbook. This link is not for the paranoid.  

future predict

Use solver; on sheet 2, minimize cell I4 by changing cells G2 and H2

Here is a github repo for doing stuff with discrete lists in mathematica Here’s the github repository: https://github.com/Altoidnerd/Spectra

———————————————————

You can follow my guide here also on how to extract frequencies from the data.  I use the more powerful Mathematica software for fourier analysis, but Excel can take fourier transforms as long as the length of the vector is a power of two.  So add zeros at the end.

Donations: 13xdMqkaVKkHKT3ZZx5ikAvQUEkzqpDkDb

ALTOIDNERD 2014

 

The Bitcoin Price Model – Large Time Calculations of the Bitcoin Price

exp

logplot

Just for a toy model, to start us off, I used excel to insist the log data is strictly linear, implying the price follows a perfect exponential curve.  We observe bitcoin hits $1000 on precisely….ok lets not get carried away.  The Price cannot be a pure exponential function of time – that seems to violate some economic conservation laws.  But these are still fun plots.  The source is an excel sheet which you can find here https://drive.google.com/folderview?id=0B2HU2oGcAN_SbDRHZmFsWldSX0E&usp=sharing

A somewhat sophistocated model for the price of bitcoin based on diffusion

In a previous post I suggested that the log linear behavior of bitcoin was a trend that will continue and proposed a model by which this would operate.  Here is a gogle drive containing the mathematica code I wrote to investigate the matter.  These equations do support the long term log linear behavior and produce price curves which seem to match the behavior we have seen from bitcoin.

https://drive.google.com/folderview?id=0B2HU2oGcAN_SSllnYkd0VThTcW8&usp=sharing

You will be able to manipulate a parameter controlling the effective bitcoin supply.

The rules I used to derive the formulas were:

1) the Bitcoin demand D(t) is proportional to the probability that somebody on earth has learned of Bitcoins existence.  This function is an exponential in t for small times

D(t) ~ exp[-t / to]

However the probability question has vastly distinct long time behavior.  The function that really represents the long term demand of bitcoin over time is closely related to the hyperbolic tangent 

600px-Logistic-curve.svg

This family of functions has the form

D(t) = 1 / ( 1 + exp[- t / t0] )

2) The Bitcoin supply is known in the short term to be proportional to t, but since the number of bitcoins generated per unit time halves every fixed amount of years, dictated by the protocol, I said we should model the supply as

S(t) = t * 2^(-t / t’)

Where t’ is in general a parameter that can be adjusted to dictate the effective supply of bitcoin as a function of time.

This incredibly simplistic model was in fact successful at generating the family of curves that the bitcoin price and its log seem to belong to.

Perhaps in the future more sophisticated models for the supply curve can be employed, since it seems clear the supply curve is responsible for the volatility.  See tables beneath:

Anyone who has mathematica ready to can run this code.  An animation can be generated as well as a dynamic plot you may manipulate.  A simple table of outputs for the price and log of the price is given in the above google drive as the supply parameter is swept.  You can observe the supply in the model has a strong influence on the price – many orders of magnitude in fact.  Below are some excerpts from the PDF and interactive mathematica program provided in the google drive, as well as the source code.

NULL

1

2

3

4

Demand[t_] := 1/(1 + Exp[-t/tp])
 Supply[t_] := t*2^(-t/th)
Slider[Dynamic[tp], {0, 29, .1}, ImageSize -> 1300], Dynamic[tp],
 "= propogation time tp" ,
 Slider[Dynamic[th], {0, 4, .1}], ImageSize -> 1300, Dynamic[th],
 "= effective bitcoin supply"}
Dynamic[
 Plot[Demand[t]/Supply[t], {t, .0000001, 5}, ImageSize -> 500]]
 Dynamic[
 Plot[Log[Demand[t]/Supply[t]], {t, .0000001, 5}, ImageSize -> 500]]
 {Slider[Dynamic[tp], {0, 29, .1}, ImageSize -> 1300], Dynamic[tp],
 "= propogation time tp", Slider[Dynamic[th], {0, 2, .001}],
 ImageSize -> 1300, Dynamic[th], "= effective bitcoin supply"}
{Dynamic[
 Plot[Demand[t]/Supply[t], {t, .0000001, 5}, ImageSize -> 500]],
 Dynamic[Plot[Log[Demand[t]/Supply[t]], {t, .0000001, 5},
 ImageSize -> 500]]}*)

——
Donate Bitcoins
Want more of this?  I have a donate button.    Invest wisely and safely!
Donate Bitcoins