How long does it take a new Bitcoin full node to sync to the blockchain in 2016? Does libsecp256k1 speed up the sync?

Bitcoin-Core version 12.0.0 shipped with a new heavily optimized elliptic curve math library. Prior to the release, there were whispers that the new library would significantly speed up the blockchain sync – a sync which was taking me 12-24 hours to complete. Excited for the potential speed up, I decided to benchmark the sync and talk about it on reddit live. Needless to say I was amazed at the difference. Thanks to the core developers for this!

How long does it take a new Bitcoin full node to sync to the blockchain? Does the recently merged libsecp256k1 speed up the sync?

687474703a2f2f692e696d6775722e636f6d2f6f62634c6a44442e706e67

These are good questions, so I wrote a small bash script that gives us some data about the sync time of the new release candidate, whose new optimized library for elliptic curve operations has been rumored to give up to a “700%” increase in the speed of verifying transactions. Disclaimer: it’s not wonderfully modular or well thought out; I did it quickly just to see how the performance was.

So let’s benchmark bitcoin-core with the new libsecp256k1 to see this speedup ourselves. Here is a script for making a log and getting a birds eye view of the sync process. Spoilers: it was really, really fast.

Motivation

This is the kind of thing I always thought about doing, but never did. It’s a simple thing to do, however it just takes time to write up a useful albeit simple script to give you the data you want. Not only that, but previously, when syncs were taking days to complete, this sort of test would eat up my machine.

Since I have been involved with bitcoin, I have known the user base to be (I love you guys) …well, prone to getting carried about with the fear of an impending doom of the BTC — especially if the inevitable doom will be brought upon by some physical limitations of the client, its capacity, and scalability.

In fact, back in 2012-2013, the total size of the blockchain was really all the rage, and the buzzword of the times was “blockchain pruning”. This involved lot of media focus on this problem, as well as efforts by developers and theorists alike to put forth a concerted effort to find a method for reduction of the very scary, problematic growth rate of the block chain!

That’s right everyone was talking about the block chain being too large on disk already, and growing too fast. How bloody ironic, eh.  I won’t say it.  Don’t even say it!

Over time I have built up a thick skin for these sorts of mobs of fear and uncertainty.  I just keep using my coins and really tend to take the doomsday scenarios with less than a grain of salt, because I have seen bitcoin overcome obstacle after obstacle, so many times that it becomes almost illogical to be anything but calm through it all, and feel as though we are in good hands with the devlopers (we are).

But node sync time has been a subtle concern to me for many months – not because I read it on www.scary-bitcoin-news.ninja, but because I noticed this sync time change over a long period of time and become a real hassle to me and running my node as a casual.

In early 2013, it took “pretty damn long” to bootstrap a new node. But I was able to run a full node that year, all year, on a mid level laptop, that I would wipe and re sync every few months.  But by the end of 2014 I was no longer running the laptop node and noticed long sync times on my main high performance monster of a desktop. I felt it took so much longer in fact, that I considered not even running the node, which would have really sucked for me, and put a big damper in the way I got used to used bitcoin. By the end of 2015 I was just trying to avoid a fresh node sync at all costs, which is hard as a man who re-installs his OS every few months. The sync would, on either of my computers, take between 36 and 96 hours for me, depending on how uninterrupted I could run the process and the state of my internet service, and other variables.




It dawned on me at some point that if I was dreading a sync, I couldn’t have been the only one. And so I did consider whether node sync time was a real deterrent to running a node.

However I decided to go for it because I thought if the figures I heard are true, and this library gives a significant speedup, it might require less than a day. And when I inquired in the IRC channel, I was told “could be as fast as 4 hours.” Hot damn – this I had to so. So I finally did it.  Result: Sp happy!  It finished in a blazing 4 hrs, 4 minutes, and 4 seconds!  Sweet victory for node operators and bitcoin as a whole!

This did truly make me very happy as it served my purposes and the community as a whole in no small part.  I was stoked, and to such a degree that had to spread the joy around the community, encouraging others to take note of the speedy sync time by benching it themselves.  I made a reddit post, and put the code on github, along with the data from my 4 hour full bootstrapping of bitcoin core.

Well anyway, I enjoyed this little treat from the devs, and hope y’all do as well.  Thanks guys! Long live the core!




Bitcoin-data manipulaton and plotting in python

 

plot.png - https://i0.wp.com/raw.githubusercontent.com/Altoidnerd/bitcoin-price/master/plot.png?resize=840%2C474&ssl=1
I wrote a python mini-api to manipulate and visualize the price of bitcoin historically using the coinbase version 1 api.

bitcoin-price (python module)

Get bitcoin price data from the coinbase API and parse it into an easily manipulated form.

>>> from fast_dump_v22 import *
>>> get_page(1)

Will return all the data on page 1 of coinbase’s bitcoin price API (this is the latest data).

You can almost always turn on optional debug statements

>>> get_first_N(3, show=True)
... fetching first 3

You can get all the price data that coinbase has

>>> get_all(show=True)
... getting page 1
... getting page 2
... getting page 3
... getting page 4
... getting page 5
... getting page 6
... getting page 7
... getting page 8
... getting page 9
... etc ...

All the ‘get_*’ functions return a price_data string, which is interlaced timestamps and prices littered with newlines and commas. You can print them to see what is going on more clearly:

>>> print(get_page(1))
2015-09-11T06:44:04-05:00,241.14
2015-09-11T06:34:04-05:00,240.8
2015-09-11T06:24:04-05:00,240.75
2015-09-11T06:14:05-05:00,240.68
2015-09-11T06:04:04-05:00,240.83
2015-09-11T05:54:05-05:00,240.92
2015-09-11T05:44:04-05:00,240.64
2015-09-11T04:34:04-05:00,241.27
2015-09-11T04:24:04-05:00,240.73
...

Turn on the optional show switch for printing large vectors

>>> prices(get_page(11), show=True)
... returning 11000 prices in specified range ...
['239.9',
'239.9',
'239.4',
'239.77',
'239.33',
'239.99',
'239.81',
'240.28',
'240.4',

You can use prices(data)[k] and timestamps()[j] to return the kth price in data, or the jth timestamp in data.

>>> data = get_page(1)
>>> prices(data)[4]
'241.2'
>>> prices(data, index=4)
'241.2'

are two equivalent ways of returning only the 4th price in the requested range (in this case, page 1). This also works for timestamps.

>>> timestamps(get_page(1)+get_page(2))[1166] == timestamps(get_first_N(2), index=1166)
True

This shows the expressiveness of this module. In general:

>>> prices(get_page(2)) == parse(get_page(2))[0]
True

prices() and timestamps() are just functions that return a parsed() object having a specific index, or indices.

>>> parse(get_page(1)+get_page(2)+get_page(3))[0] == prices(get_first_N(3))
True
>>> parse(get_page(2)+get_page(3))[0] == prices(get_range(2,3))
True

The parse() function is there to manually control the outputs instead of just getting prices, or timestamps

>>> x = parse(get_page(1))
>>> x[0][0]
'241.2'
>>> x[0][1]
'241.14'
>>> x[1][1]
'2015-09-11T04:34:04-07:00'

As you can see, parse(price_data)[0][k] returns the kth price in the list. Indices [1][k] return the kth timestamp.

The parse() function takes care of some weird edge cases:

>>> get_first_N(3) == get_page(1)+get_page(2)+get_page(3)
False
>>> parse(get_first_N(3)) == parse(get_page(1)+get_page(2)+get_page(3))
True
>>> x = get_page(1)
>>> y = get_range(2,7)
>>> prices(get_first_N(7)) == prices(x+y)
True

In general,

OPERATOR( get_page(1) + get_page(2) + ... + get_page(k) ) == OPERATOR(get_first_N(k))

where OPERATOR is parsed(), prices(), or timestamps(). We also know prices() can obviously display and return ranges of values. When returning large vectors, you can verify their length by setting show=True. The “show” parameter is optional for all get_* functions and provides some information about the operation being performed.

>>> print( prices(get_first_N(11), show=True) )
... returning 11000 prices in specified range...

since each page is a thousand pairs of values (timestamp, price).

>>> len(prices(get_page(2))
1000
>>> prices(get_page(2)) # returns a long list
['230.11',
'229.64',
'230.04',
'229.71',
'229.69',
'229.74',
'229.92',
'229.43',
'229.43',
'229.41',
...

fast_dump_v1* are older versions that are somewhat different. They are designed to store the fetched data in the .data directory. This in v2*, this was abandoned in favor of stdout redirection.
// 2015 altodinerd.com

Getting started with Bitcoin the “right way” – encrypted wallets, bitcoin core, and linux

A brief introduction to getting started with bitcoin on a linux distribution

For the impatient: Skip to step 1

Bitcoin is one of the most interesting technologies we have seen emerge in the past ten years. And it confuses everyone who sets out to use it.  This post is my definitive guide to beginning your journey with bitcoin the right way.

tuxbig

To me, the right way includes, but is not limited to:

  • Running bitcoin on a linux machine.  My reasons for this are too many to really list here, but suffice it to say that since linux is widely regarded as the most secure operating system available, and the usage of bitcoin involves being your own bank, using bitcoin on linux is the best way to ensure your coins stay safe.
  • Using the bitcoin-core wallet software.  There are other wallets available.  You actually don’t even need a wallet software to own bitcoins or even spend bitcoins.  However, if you want to understand how bitcoin works under the hood – which you should, because you are going to be your own bank, you should use this software.
  • Using the command line to do bitcoin things. Actually, using both the GUI and the command line is the way to go.  This tutorial (this is part 1 of 2) will be a command-line-centric description because afterall, I’m trying to show you the right way to do things.

But that’s just, like, your opinion man.

It is. This is my blog.

You shouldn’t necessarily trust me or anyone – bitcoin isn’t about trust. But if you google “altoidnerd”, you’ll find that I have been around the bitcoin eco-system for quite some time. That’s all I can tell you.

Screen Shot 2015-09-10 at 8.31.28 PM

There are just so many wallets available – why bitcoin-core?  Because I have been using it for years, I have a very good method of keeping my coins safe, and that is what I am sharing with you today.  Use it- it is THE wallet.

But altoidnerd, I don’t want to run linux. I like windows / OSX

I am a linux and an OSX user.  So please understand the scope of my knowledge includes these operating systems. If you want to use OSX, most of this tutorial will make sense with minor changes perhaps.

Screen Shot 2015-09-10 at 8.32.59 PM

If you want to use windows, that’s ok, I’m not going to judge you. Bitcoin-core is available for windows as well. Please understand however, I am not a windows user, so I will not be able to give you “the definitive bitcoin windows guide”. You can still read on, however, since you can apply much of this tutorial to usage of bitcoin-qt’s debug window instead of the command line.

Step 1: Download and install bitcoin-core

Navigate to bitcoin.org’s official download page and select the bitcoin-core distribution for your operating system. And do choose linux (tgz).  Do that because you should be running linux.  But if you aren’t, even though you should be, choose the distribution for your operating system. Extract and install the client.






The “right way”

To really take control of things, and understand what is happening, you’re going to want to run bitcoin software from the command line. Though it isn’t really necessary to download the thing from the command line, I’m going to describe that here because this tutorial is the first installment of a start to finish command line approach. For a sneak peak of what I mean, see an earlier post where I described how to use bitcoin-core to generate QR codes without having to trust shady ass websites.

Go to your home directory and create a new directory just for bitcoin.  This is not a necessary step, but once again, this is my “right way” tutorial and I will explain reasons for this later on.  Enter the directory, and download the tarball with the “wget” command.  The extract the tarball like so:

    cd ~

    mkdir bitcoin

    cd bitcoin

    wget 'https://bitcoin.org/bin/bitcoin-core-0.11.0/bitcoin-0.11.0-linux64.tar.gz'

    tar -xzvf bitcoin-0.11.0-linux64.tar.gz

bitcointerm1

 

Sweet.  Now change directory to where the binaries are,

    cd bitcoin-0.11.0/bin/

and launch “bitcoin-qt” (qt means its the GUI version).

./bitcoin-qt

It’s going to show you the dialogue box below and ask you if you’d like to use the default directory. Do it. Click “OK”.

 

default

 

You’ll see a friendly startup screen if you’ve done things right!

bitcointerm2

 




 

Step 2: Encrypt your wallet, and wait a few days

Bitcoin-core is called a “full-node” implementation, which means the first time you start it on your machine, it’s going to download every single bitcoin transaction that has ever taken place.  Wait until the blockchain is synced to start making transactions.

In the mean time, you need to set a passphrase for your wallet.  This is super important.  In the upper left corner of bitcoin-qt, go to settings -> Encrypt Wallet, and set a very strong passphrase.

encrypt

This will make it impossible for anyone to send bitcoins from your wallet without entering your password:

PASSWORD

Just make sure you:

    • Make your password extremely long. I don’t know if there is a length limit in the bitcoin-core code or not. Regardless, seriously push your personal boundaries on what you think a safe password is. Make it super duper long, and impossible to guess. Wise men have discussed password strength to a great extent. Pick a strong password. 20+ characters. Go for 30. No 40. Just make it long, and keep it an absolute secret.
    • Never, ever lose or forget this password. If you lose or forgot your password, your coins will be unspendable forever. Neither you, nor anybody on earth, nor God himself will be able to spend your coins. Your coins will be effectively lost.





Picking a strong passphrase makes security so, so easy

Once you encrypt your wallet, you can copy and store wallet.dat everywhere (I explain wallet.dat in great detail in “Step 3”). I literally have copies of my main wallet.dat on 6 or 7 computers, because come hell or highwater, I will always be able to find a copy of my wallet.dat file. I suggest you do the same. Here is why strong encryption and insane redundancy works:

  • If your password is ridiculously long, you can be very lazy and downright cavelier about where you keep your wallet.dat, because even if someone finds the wallet file, they wont be able to do a damn thing with it. They won’t be able to spend your coins without your passphrase.
  • This means that you can make copies of wallet.dat and store them like literally everywhere, on all your machines, three times over. In the cloud. On your mom’s computer. Hell, theoretically speaking, you can post your encrypted wallet.dat on the internet and just about everyone will have a copy, and be unable to spend your coins, because they don’t know your password. Someone remind me to put some coins into an address, and post the encrypted wallet online to prove my point.

    Bottom Line

    If you encrypt your wallet like a boss, you never forget your password, you never tell anyone your password, and you copy the file to everywhere you possibly can imagine, you will never lose your coins. And nobody will be able to steal them from you. Seriously, nobody. It is computationally impossible.

    Step 3: Understand what you just did on your computer

    Let’s break it down this way. Here are some things that just happened when you launched bitcoin-qt for the first time:

    • A hidden directory ~/.bitcoin was created. It contains among other things the single most important thing ever: a file called wallet.dat. This file is critical. ~/.bitcoin/wallet.dat contains all of your private keys. wallet.dat is life. When you see in headlines in the news like “Frustrated gentleman quite upset having lost a usb jump drive containing 100,000 BTC“, it means he doesn’t have a copy of wallet.dat, so neither he, nor anyone on earth, nor God himself can spend his coins anymore.
    • Your computer started downloading the blockchain. The blockchain is a record of every bitcoin transaction ever to have taken place. This will take a few days. As of today, the blockchain is about 42 gigabytes. Make sure you have space for it.

    Do not delete your wallet.dat file. Just don’t. If you want to start a new wallet, instead of deleting wallet.dat, just rename it to something else, like wallet.dat.old, restart bitcoin-qt, and it will create a new wallet for you. Keep your wallets people. Keep them good.

    wallet

    Notice how I have a file called wallet.dat.default? That’s because I never delete a wallet. When I installed bitcoin on this computer, I moved the wallet.dat that stores my coins into ~/.bitcoin/, but first I renamed the existing file so I can keep it. Because, why not? I’m telling you – don’t delete a file called wallet.dat, because mistakes happen, and its better to have hundreds of files named wallet.dat.* than it is to lose your coins.

In my next post, I will describe some of the features of the bitcoin-qt debug window, bitcoind and its helper bitcoin-cli.  You can do interesting things, like import new keys, or convert keys to qr-codes.




If you have ay questions, please feel free to comment or contact me.
If you liked this post, you can donate bitcoins to me here: 12gKRdrz7yy7erg5apUvSRGemypTUvBRuJ
Learn about the author here:
http://www.phys.ufl.edu/~majewski

How import a bitcoin address private key into breadwallet (iOS) with a QR code using bitcoin-core for Ubuntu / Linux

On encoding private keys from Bitcoin-core as QR codes for use in paper wallets, other wallet software, etc …

Note: Breadwallet will not import the private key upon scanning the qr code. It will offer to transfer the funds in that address to your breadwallet keyring.

Breadwallet, an awesome open source iOS bitcoin SPV client, allows you to import a private key for use within breadwallet.  So, if you’re a user of bitcoin-core, and want to be able to spend some of your funds you have there with breadwallet, you’ll need to create a QR code of your private key to scan it with breadwallet.

Install qrencode

This tutorial uses qrencode, an open source string to QR converter. The source can be found here: https://github.com/fukuchi/libqrencode/

Alternatively, you can install qrencode on ubuntu/debian through repositories

sudo apt-get install qrencode

Then, as of bitcoin-core version 0.11.0 you’ll need to run bitcoind rather than the GUI bitcoin-qt. If you’re using an old version of bitcoin-core that does not have the bitcoin-cli tool, you can get the latest version of bitcoin-core and extract it without having to verify the blockchain again or mess with wallet.dat – your ~/.bitcoin directory will remain unchanged.

Get the newest version of Bitcoin-core


wget 'https://bitcoin.org/bin/bitcoin-core-0.11.0/bitcoin-0.11.0-linux64.tar.gz'
tar -xzvf bitcoin-0.11.0-linux64.tar.gz
cd bitcoin-0.11.0/bin

You should see the executables for the bitcoin-core suite here.
image1



Run bitcoind and create your QR

Be sure you quit any versions of bitoin-qt or bitcoind that are already running, then start the bitcoin daemon

./bitcoind &

The & operator makes sure bitcoind detaches from your terminal so you can keep going.

Now you are ready to get a QR code for your desired address. Is your address is “addr”, you can generate a QR code of your private key without ever showing your private key on screen in plain text. Just do:

./bitcoin-cli dumpprivkey "addr" | qrencode -o ~/Desktop/Key.png

Or better yet, if you want to create a new address and QR-encode its private key,

addr=$(./bitcoin-cli getnewaddress)
./bitcoin-cli dumpprivkey $addr| qrencode -o
~/Desktop/$addr.privkey.png
echo "private key dumped for: "$addr

The QR code should appear on your desktop. Below is a screen shot of me doing an example where I create a new address, and get a QR for its private key.  Enjoy!

image0



Note that if your wallet is password protected (which it really should be), you’ll need to first do

./bitcoin-cli walletpassphrase "your_password" 100

where the argument 100 means the wallet will allow all actions (like spending, or dumping keys) for 100 seconds. That should be plenty of time.




It may be time to buy some LTC again. Here’s why. The golden rules for altcoin trading.

Problem with image? See it on imgur here https://i2.wp.com/i.imgur.com/6ibi1z1.png?resize=840%2C428
Litecoin price in BTC on the btc-e exchange, for as long as bitcoinwisdom.com can remember. We are at about all time lows.

This blog post is a glorified (with some images and minor changes) copy pasta of my post on /r/cryptomarkets.

LTC – how low can it go? It might be time to take a shot with LTC for a potential bitcoin profit.

Problem with image?  Go to https://i2.wp.com/i.imgur.com/RL7HENT.png?resize=840%2C431
Past performance is no indication of future…oh whatever. Let’s get some LTC. 1) it’s old 2) its inflation rate is very slow 3) it’s not bitcoin 4) historically speaking, this is an OK price.  5) LTC enjoys special status in the cryptocurrency world as a secondary market, and should always be watched out of the corner of one’s greedy eye.

I have done this with LTC a few times – the best of which was last November. Priced in BTC, it is now near parity with DRK and below .01 … which isn’t far from the price I think I paid for a bunch of LTC right before the boom in November, when I started writing about trading in /r/cryptomarkets with “the golden rules for alt trading.”

One thing that makes LTC different from other scrypt alts is its very slow ditribution – it’s just as slow as bitcoin. To me, that makes it still attractive as a swing trade, because big investors are probably thinking about it. What do y’all think?

BTW, just looked. I decided to buy lots off LTC last year at .008, and it’s not far from this mark. I wrote the golden rules to this sub from a 5-star hotel (I don’t stay in those) thanks to that trade. Good times…

Sad grep I – V

~$ sad grep poetry of the internet


I

# facebook | grep brains

#

 

 

II

# cat ./telecom| egrep (options|competit.?.?.?.?.?)

#

# dmesg

comcast

#

 

 

III

~$ ls

~$

~$ pwd

home/fridge/

~$ ls -laR | grep food

~$

 

 

IV

# cat ./reddit/r/bitcoin/* | grep criminals

Display all 169,236 possibilities? (y or n)


 

 

V

~$ ./github/ | grep working code

OSX

~$

 

 

 

 

When should I sell my Bitcoin Mining hardware? Bitcoin Mining Hardware Resale Value vs Projected Return

It’s interesting to look back at these “old” numbers from November 2013!

20140605-062924.jpg

Avoid the 5 Common Mistakes of Altcoin Traders

heh.

Altcoin cryptocurrency trading strategy based on market capitalization distributuion

I grab the data from coinmarketcap.com and look at plots of the logarithm of the vector of market caps. It looks kinda like this.

TANH
http://i.imgur.com/TFwe5lD.png [1]

Then I zoom in on sections and look for deviation from linearity, kinda like this.

LINBE
http://i.imgur.com/w8wWnKb.png [2]

I believe the distribution in the first pic (with log scale) to be the inverse hyperbolic tangent

http://mathworld.wolfram.com/InverseHyperbolicTangent.html[3]

Anyway I look for points that are low and buy them hoping their BTC price rises. Gambling yes, but at least it’s math.

Discussion on reddit:

Altcoin trading strategy based on market cap distribution from BitcoinMarkets

ALTOIDNERD 2014

Donations: 17NA1jYg5u6ejboArdM7HW4MwSa6cWfnEd