Jump to content
Bitcoin Foundation
George Lloyd

Mike Hearn: Underfunding is Leaving Bitcoin Development in Crisis

Recommended Posts

George Lloyd    13

Hi,

 

I have just read the recent CoinDesk article and would like to comment. If I may. This seems like the least likely place to get shot down by trolls.

 

I am an experienced 30-something developer but I am not an open-source community developer, I just know how to code. I know C and other languages and I have experience in small multi-dev environments using source control. I consider Debian a "friend". But I have absolutely zero experience working in the public eye and with FOSS stuff (e.g. github). I have looked a couple of times at helping out with the core Bitcoin development, everything nowadays seems to be in Java or Python both of which I am not capable enough to contribute with.. C I can do and C++ is not alien to me. But I don't know where I can start to help. I need some direction.

 

The problem is, the strict protocol / etiquette when working on such projects and the tight-knit developers. 10 minutes on the developer's lists and you are scared off if you aren't an experienced FOSS type of person with oodles of experience and time to spare. I have a serious day job, I didn't code in the 80's, I don't currently feel I have a place that I can slot into. I would like to. That is the only reason I am posting this.

 

So, my question, what can I do to help? Are there some small coding tasks I can start with? Is there a Bugzilla type list I can work from? Where should I start? I'm hardly going to jump into deep core protocol work and I don't have QT GUI experience but as I say, I can code, I understand security, efficiency, overheads and code quality. I would just like to offer a couple of hours a week to help out. I'm not after coins nor glory nor PR. I just want to assist for the love of Bitcoin. Can anyone offer something constructive for me to follow up on? Or maybe even the foundation finds a project manager to source and manage guys like me, who just want to help without the politics.. code monkeys if you will.

 

George.

  • Upvote 4

Share this post


Link to post
Share on other sites
Brian Goss    554

I'm a hobby level programmer, but, from periodically following the dev list (which I don't always understand) I feel like testing is very needed and very non-controversial.

 

I've also donated a small sum to tip4commit for bitcoin (http://tip4commit.com/projects/2). Unfortunately, not many have donated, although I'm told by Wladimir that the tips are a nice gesture, it's not really a viable funding model.

 

I really like Mike's lighthouse idea, and perhaps that code is worth contributing to as well.

Share this post


Link to post
Share on other sites
Luke-Jr    87

Unit tests are a great way to get started: the project desperately needs them, and it helps the new developer learn pretty much any (potentially every) part of the codebase.

  • Upvote 4

Share this post


Link to post
Share on other sites

Unit tests are a great way to get started: the project desperately needs them, and it helps the new developer learn pretty much any (potentially every) part of the codebase.

 

This issue, which I opened, is now closed. There is a bitcoin development thread related to a part of it, which is still in somewhat active discussion on the bitcoin-development mailing list. The issue was opened originally with an eye to support of bitcoin development, and also covered how to incentivize individuals' running of full nodes regardless of whether or not the individual running a full node would be mining, thus emphasizing decentralizing and strength of the network while encouraging voluntary distribution of personal surplus for both development as well as anything else which one might be interested in supporting. Anyone is welcome to draw from the ideas there to open a more specific or focused issue or alternately, you could bring it to a new discussion in the bitcoin development, or develop your own pull request should you be so inclined.

Share this post


Link to post
Share on other sites

Below is a quote from the Bitcoin Core Github

Testing and code review is the bottleneck for development; we get more pull requests than we can review and test on short notice. Please be patient and help out by testing other people's pull requests, and remember this is a security-critical project where any mistake might cost people lots of money.

 

If you have extra time you might want to get up to speed on the how the core works and then I'm sure you'll be welcomed with open arms when reviewing other people's code. Just have a look at the open issues and pull requests on Github.

 

If you want to get an idea of the top priority for devs. at the moment #bitcoin-dev on freenode (IRC) is a great place to find out what other devs are working on and might need help with.

Share this post


Link to post
Share on other sites
Mike Hearn    459

Finding small tasks to get people involved is difficult, especially on Bitcoin Core. Everyone loves to say "unit testing" mostly because nobody else wants to work on it, which isn't great. If you dig in you'll quickly find that the code lacks basic unit testing infrastructure: every time I try to write some myself I come away frustrated. It'd require a lot of work and refactoring to make Core really testable; it was written by a guy (Satoshi) who had apparently been bypassed by the unit testing religion. So that'd be a huge and useful contribution but not a very exciting one, I'm afraid.

 

Right now big chunks of the system are tested by Matt's pull tester, which is written in Java.

 

Other tasks that need doing are mostly large and/or run into what I can only describe as political/philosophical issues around design.

  • Upvote 3

Share this post


Link to post
Share on other sites
John Wnuk    70

Seems that a part of the Bitcoin Foundation income should be used to fund development and test. Without some kind of central point such as the Bitcoin Foundation, will always be difficult to justify cost when there may be benefit to all - but no business-like profit. The Bitcoin Foundation should also be going to the exchanges to get them to start donating some % of profits for core development and test. I'm sure companies such as Coinbase, CampBX, Bitstamp and others would understand why this contribution would be important. Did not use the word "Research" here as that will most likely be part of work done on new crypto-protocols and -coins.

Share this post


Link to post
Share on other sites

btw, what are those large tasks no one is working on and how are they critical?

I think it would be a lot easier to raise some money if that is clearly commuicated.

 

For example, right now I would only support developers focusing on scalabity, stability, privacy and safer storage and spending. Once the currency itself is sufficently successful, finding developers and funding for the extra features Mike mentions will likely be a lot easier. To that end I think it is also critical to keep the development of the most used wallets (wild guess: Core, Multibit, Armory, Schildbach) well funded.

Share this post


Link to post
Share on other sites
Mike Hearn    459

The foundation does fund development! To the tune of three full time guys, that's critical, because otherwise the maintainers would probably be employed doing something else.

 

Things that could be done but which aren't really making much progress: proper resource scheduling for DoS resistance (our current anti-DoS strategy is not excellent) - this is also a big scalability issue, chain pruning, vending high quality IP address lists in getaddr so SPV nodes don't have to rely on DNS seeds so much, link-level encryption (maybe), a new transaction version that fixes malleability and allows low trust calculation of floating tx fees, improvements to Script such as new signature types, better unit testing infrastructure, better monitoring and metrics infrastructure .... and that's just Bitcoin Core consensus related code, it doesn't include any wallet features. And then there's miner decentralisation stuff too. So there's absolutely lots to do.

  • Upvote 2

Share this post


Link to post
Share on other sites
Dan Plante    147

The foundation does fund development! To the tune of three full time guys, that's critical, because otherwise the maintainers would probably be employed doing something else.

 

Things that could be done but which aren't really making much progress: proper resource scheduling for DoS resistance (our current anti-DoS strategy is not excellent) - this is also a big scalability issue, chain pruning, vending high quality IP address lists in getaddr so SPV nodes don't have to rely on DNS seeds so much, link-level encryption (maybe), a new transaction version that fixes malleability and allows low trust calculation of floating tx fees, improvements to Script such as new signature types, better unit testing infrastructure, better monitoring and metrics infrastructure .... and that's just Bitcoin Core consensus related code, it doesn't include any wallet features. And then there's miner decentralisation stuff too. So there's absolutely lots to do.

 

 

Michael, the core Bitcoin value proposition - "no central control" - has been breached several times in the last two years, most recently by GHash.IO. Nothing decisive has been done about this in those two years.

 

I am not a core developer, nor a VC firm or Bitcoin business owner or investor. I don't care about any of that, that is their bailywick and they spend their capital and take the risk. They come and go, they live and die. I am simply a Bitcoin buyer / saver / spender. My focus is on the soundness of Bitcoin as a medium of exchange and a store of value. Period. Bitcoin will live or die based on how well it responds to the rank-and-file's desire for a sound money.

 

With all due respect, we do not care what you or the Foundation think or believe. If we lose confidence in the soundness of Bitcoin as money, we will divest and $/BTC will tend to zero. That's just how it goes. There has never been a single exception to this in recorded history, and the reason for that should be obvious.

 

I suggest you redirect your efforts to more fundamental issues.

Share this post


Link to post
Share on other sites
George Lloyd    13

Thanks for all your comments guys, if I've done nothing else I've started what seems to be a productive conversation on here. The answer (for me) seems to be to concentrate on the GitHub issues and look out for small low priority items that I can maybe sort out leaving the big guns an extra bit of free time to sort out the pressing issues. That way I can become familiar with the code over time without being a counter-productive needy person.

  • Upvote 1

Share this post


Link to post
Share on other sites
Brad Wheeler    296

Michael, the core Bitcoin value proposition - "no central control" - has been breached several times in the last two years, most recently by GHash.IO. Nothing decisive has been done about this in those two years.

 

...

 

I suggest you redirect your efforts to more fundamental issues.

 

To be fair, Mike did point out mining decentralization. It's also stated in the Risk Management study. This problem is a tricky one that requires concerted effort and is not likely to be solved by a single person or someone just starting to get their feet wet with the project. Your concerns _are_ valid, but the "solution" is one that involves the broader community.

Share this post


Link to post
Share on other sites
David R Allen    318

Why not take Jim Harper's paper and identify where we are with the real threats, in real time?

 

I propose a simple graph over time with clear progress or failure on the confidence issues that make Bitcoin work or bottom out.

 

Tomorrow's crash or rocket ride, or continued levelling could be a good time to start.

 

Oh the Joy of Bitcoin!

  • Upvote 1

Share this post


Link to post
Share on other sites
Dan Plante    147

It isn't really that tricky, honestly. Adding one (or more) optional proofs-of-work (or proofs of memory, proofs-of-resource, etc) is quite straight forward and would only require an addition to the protocol, not a fundamental change to it (that pools/miners would not accept anyway).

 

I've devised a protocol addition that I think might devolve control back to the rank-and-file's PCs and laptops, while still being inclusive of the ASIC-centered miners and the time-sensitive capital deployed by them. But I have had scant minutes per day to research this problem, and the GitHub site's UI seems ridiculously Byzantine to me. I am not a coder, I am a hardware guy. I don't have the time to decode that site's format.

 

The demands on my time by my employer are rather insane, and I suspect that if I detailed it you probably woud not believe me. But I do try to contribute regardless. I try to focus my energies on the desperate and clueess moms & pops. I try to speak with their voice. I am not a troll Brad.

Share this post


Link to post
Share on other sites
Brad Wheeler    296

It isn't really that tricky, honestly. Adding one (or more) optional proofs-of-work (or proofs of memory, proofs-of-resource, etc) is quite straight forward and would only require an addition to the protocol, not a fundamental change to it (that pools/miners would not accept anyway).

 

I've devised a protocol addition that I think might devolve control back to the rank-and-file's PCs and laptops, while still being inclusive of the ASIC-centered miners and the time-sensitive capital deployed by them. But I have had scant minutes per day to research this problem, and the GitHub site's UI seems ridiculously Byzantine to me. I am not a coder, I am a hardware guy. I don't have the time to decode that site's format.

 

Please feel free to start a new thread on this to kick start discussion, since it's a much more specific topic than core development funding and/or how to contribute to the development side of the project.

Share this post


Link to post
Share on other sites
Leith Marar    11

This may be a stupid question, but has anyone abstracted a set of component functions of bitcoin core so that they can be documented and standardised, or is this an over simplification?

Share this post


Link to post
Share on other sites
Brian Goss    554

Please post it here, Mike.

 

Thanks

 

https://bitcoinfoundation.org/2014/07/03/mining-decentralisation-the-low-hanging-fruit/

 

 

 

 

Blog

 

Mining Decentralisation: The Low Hanging Fruit

Guest Blogger: Mike Hearn, Developer July 3, 2014

 

In recent days the Bitcoin community has engaged in vigorous debate about how to solve the problem of a small number of players, and GHash.IO specifically, dominating the network’s hash rate. In light of GHash.IO’s decision to do nothing, many suggestions have been made for possible solutions. This blog post will review some and outline what can be done to help rectify the situation.

 

Firstly let’s put aside solutions that don’t address the problem, or that would make things worse. Many alt coins identify “ASIC resistance” as a design goal. It’s tempting to think about rewinding the clock by changing the proof of work algorithm. But this is fraught with difficulty. Firstly, there’s really no such thing as an ASIC-resistant algorithm. Almost by definition, the only way to force someone to use a CPU instead of an ASIC is to use “run an arbitrary computer program” as your proof of work, otherwise, you are doing a specific task and an application specific integrated circuit can always have the advantage. But nobody knows how to design a Bitcoin-like system which requires execution of arbitrarily complex and ever changing programs as the proof of work. Secondly, before the ASIC era Bitcoin had to tackle a different kind of problematic miner: botnets. Several botnets stole electricity to mine coins, which had nasty side effects. For instance, at least one of them mined only empty blocks, which is worse than the situation we have now. Losing ASICs would bring them back. Thirdly, invalidating the huge investment in ASIC farms already made would have profound consequences for the future of the project and people’s willingness to mine in future.

 

So if changing the proof of work algorithm is not a solution, what is?

 

We don’t know for sure, but we can identify useful tasks that help us move towards a more decentralised future. None of them are silver bullets, but most of them are quite easy: they’re what developers call low hanging fruit. By working together and picking enough of this fruit, we might be able to tip the balance in favour of decentralised mining. To save keystrokes, from now on I’m going to call this freemining.

 

Freeminers, unite!

 

Let’s define what this word means. It does not mean no pooling. The problem we face with mining is not pooling to reduce payout variance, it’s that in the process of doing so miners give up their ability to select their own block contents. The purpose of mining is to create blocks, thus answering the question of what happened first. When miners give up control of the blocks they mine en-masse, double spending becomes possible, as does going back into the past and maliciously rewriting the block chain. It’s effectively a form of selling your vote.

 

There are two ways to do pooled freemining. The problem is, both routes are riddled with potholes and speed bumps that cause many miners to not bother.

 

1. You can use p2pool, a peer to peer decentralised mining pool

2. You could (in future) use the getblocktemplate protocol with a regular pool that supports it

 

Freeminers mine in such a way that they both reduce their payout variance but also create their own blocks, a process that always requires running a fully validating p2p node like Bitcoin Core. To repeat: freemining inherently requires running a full node, and today that means Core. If you aren’t running one, you aren’t decentralising the mining process. This requirement puts people off and later I’ll discuss ways we can make it less painful.

 

p2pool’in

 

P2Pool is the best solution currently available. But it faces numerous small yet resolvable problems. Smart volunteers could tackle any of them:

 

1. Problem: installation is too difficult

 

Installation on Windows is possible via a one click installer, but on Linux and MacOS X no such easy setup exists. P2Pool is a Python program and some miners get stuck in dependency hell trying to simply install it.

 

Solution: One click installers need to be developed for both Linux and OS X. Distro-specific packages on Linux are not always sufficient, as they can fall behind upstream or be unavailable for the users specific distribution. Ideally p2pool would have a selection of packages well maintained by the project, and also a generic Linux installer or tarball usable by everyone else.

 

2. Problem: the p2pool website is hard to find

 

Confusingly the P2Pool website is at p2pool.in as p2pool.org was already taken. The .org page dominates the real p2pool.in website on Google. Worse, the website at p2pool.org is merely a proxy to P2pool that encourages people to mine on it without actually running a full node, and charges a 2% fee in the process. Remember, if you aren’t running Core, you are not freemining. This website confuses people into mining in such a way that they think they’re helping decentralise the network, but are not.

 

Solution: The bitcoin.org website needs a slick, simple miners introduction that links to the right website, explains how to use p2pool properly, and shows miners how to get started. Major bitcoin related websites could link to p2pool.in to try and boost the PageRank of the real site. The owner of p2pool.org could be contacted or bought out to try and fix the flow of miners who are getting misdirected.

 

3. Problem: P2Pool feels clunky compared to centralised pools

 

One reason for the popularity of GHash.IO is it presents nice, professionally designed stats pages and monitoring. P2Pool has nice frontends available too, but they aren’t bundled so don’t give users the same slick experience.

 

Solution: the best of the existing p2pool frontends needs to be located and bundled out of the box.

 

4. Problem: P2Pool has an increasingly high share difficulty

 

P2Pool is a single pool and because of how it works, the more people who mine on it the harder it becomes to find a share. This can be discouraging.

 

Solution: P2Pool could split itself into multiple independent p2pools, amoeba style, whenever it gets too large for small miners to benefit.

 

P2Pool has other problems too: it can use a lot of memory and bandwidth, and it needs more miners committing to it for the long haul to reduce the variance. But the above issues are the lowest hanging fruit for improving it.

 

getblocktemplate

 

The getblocktemplate protocol allows miners to mine at a client-server pool but without losing control over the blocks that they are mining. It represents a useful middle ground between the pure decentralisation of p2pool and the centralisation of traditional pools. A GBT pool calculates the coinbase transaction that contains payouts in a centralised manner, and then lets the miner select the rest of the block. Quoting Gregory Maxwell,

 

“Effectively p2pool is getblocktemplate plus a distributed system to compute acceptable coinbase transactions”

 

But there’s a problem: although the protocol theoretically allows for miner-side block creation, the actual software implementations of it don’t do so yet. Nobody has implemented support for freemining with GBT, so pools that support GBT today still do the block creation themselves. This in turn means that with only minor real advantages over its main competitor (Stratum, which doesn’t allow freemining), some pools and mining tools don’t support it.

 

We need volunteers to step up and help make getblocktemplate a competitive solution:

 

1. The software that implements the getblocktemplate protocol needs to be upgraded and finished off, so miners can connect their own copy of Bitcoin Core to the tools and select their own blocks. Fortunately libblkmaker (the reference GBT implementation) is designed with support for this in mind, so it should not require any major refactorings or other surgery to enable. If you’re serious about this, please contact Luke Dashjr (“Luke-Jr” on IRC).

 

2. Mining tools and ASIC systems that don’t natively support GBT can be bridged using a local Stratum-to-GBT proxy, but this is one more system that has to be set up. Ideally this wouldn’t be necessary, or if it must be so, then the proxy should be bundled as well so there’s no setup cost.

 

3. Tutorials and websites need to be created to teach miners how to configure this: ideally, with everything bundled “out of the box” so setting it up can be a one click process after selecting a GBT supporting pool.

 

4. More pools need to be lobbied to support GBT once it’s a suitable solution for freemining. This would reduce the pool’s role to that of tracking share submissions and calculating payouts, but they would have no real power over what blocks are mined.

 

Making Bitcoin Core less hardcore

 

What all the above solutions have in common is that you must run a full node implementation, which should be Bitcoin Core today. You can’t claim to be a decentralised miner if you don’t, because otherwise you aren’t validating blocks and thus have to follow the instructions of someone else who has.

 

Thousands of volunteers around the world run Core for free: node operators, we love you! But we also hear your pain. Core combines many tasks into one: it processes and validates transactions, but it also serves the block chain to newcomers who don’t have it yet. This latter process can consume large amounts of upstream bandwidth, which is often a limited resource.

 

In the short term, miners can stop their bandwidth being used for block chain uploads by blocking inbound connections on port 8333 or using the -nolisten flag. The next version will have a checkbox in the GUI for it. But that means the node won’t contribute to the p2p network at all. A long-term wish of the Core development team has been to teach the software how to use resources more effectively, so making Bitcoin less costly to run. As an example nodes could advertise that they can serve only parts of the block chain instead of all of it. This would allow people to give Core as much bandwidth and disk space as they want to.

 

This is an advanced project that requires work on the guts of Bitcoin Core and the peer-to-peer protocol. The existing developers all have their hands full with other tasks, so we’re looking for someone to step up and prototype. It means extending the IP address broadcasting mechanism so nodes can advertise how much of the chain they’re willing to serve, teaching the software how to find them, and then allowing users to tune things so they can reduce their bandwidth usage to levels they can tolerate. This should make it much less painful to run a full node at home, or in other places where upstream bandwidth is constrained.

 

For developers who want to get their feet wet, there’s an easier piece of low hanging fruit. Some jerk thought it would be funny to put virus signatures into the block chain, and now on Windows a few anti-virus scanners are corrupting Bitcoin Core’s database files. A simple fix is to store data obfuscated with a per-machine key so it’s no longer possible to put particular byte sequences into the database on disk. This would help freeminers who use Windows.

 

Building a community

 

There’s one last way pools like GHash.io build loyalty amongst their userbase: humans are tribal and we love to join a team. GHash/CEX run wallpaper contests and other things to help build their brand. We need a community of people who feel passionately about freemining, and help other people to get started. T-Shirt designers, this means you! But also we need forums and support networks to help people navigate the issues: no matter how easy we make the path, having someone walk it with you will always be better.

  • Upvote 1

Share this post


Link to post
Share on other sites

https://bitcoinfound...-hanging-fruit/

 

Blog

 

Mining Decentralisation: The Low Hanging Fruit

Guest Blogger: Mike Hearn, Developer July 3, 2014

 

In light of GHash.IO’s decision to do nothing, many suggestions have been made for possible solutions.

 

 

Is anyone in Bitcoin Foundation talking to GHash? They stated they wanted to engage with other pool operators / Bitcoin Foundation possibly at the upcoming Coinsummit conference in London. This wasn't doing nothing it was asking for dialogue to collectively find a long term solution. Telling their miners to not use their service is not a solution.

The points in Mike's blog identify the core problem, the success of Bitcoin has accelerated it past the ability of volunteers and part timers to provide solutions and support that enable what is now a multi- million $ industry to be directed / managed efficiently.

I think GHash are perhaps also unsure of what to do for the best.

Is anyone in the Foundation talking to GHash? If no why not.

  • Upvote 2

Share this post


Link to post
Share on other sites
David R Allen    318

Is anyone in Bitcoin Foundation talking to GHash? They stated they wanted to engage with other pool operators / Bitcoin Foundation possibly at the upcoming Coinsummit conference in London. This wasn't doing nothing it was asking for dialogue to collectively find a long term solution. Telling their miners to not use their service is not a solution.

....

Is anyone in the Foundation talking to GHash? If no why not.

 

Ok. I am going to go out on a limb again, and I will send private messages to all of the board and staff to ask this specific question, with a link to this thread.

 

Please keep in mind that there are a number of people with [email protected] emails who are neither paid staff, nor directors. [email protected] is a good example.

 

In light of the fact it is a long weekend in the US, I would not expect any response before next week.

  • Upvote 1

Share this post


Link to post
Share on other sites
Dan Plante    147

https://bitcoinfound...-hanging-fruit/

 

Firstly let’s put aside solutions that don’t address the problem, or that would make things worse. Many alt coins identify “ASIC resistance” as a design goal. It’s tempting to think about rewinding the clock by changing the proof of work algorithm. But this is fraught with difficulty. Firstly, there’s really no such thing as an ASIC-resistant algorithm.

 

I believe there is, and does not require rewinding the clock or excluding miner CapEx. But it is not an algorithmic response in the way that it usually seems to be narrowly defined mathematically by coders, it is simply an algorithnmic translation of some basic socioeconomic axioms, as Satoshi originally conceived. It rests fundamentally on what is widely known as an economic "commodity". Do you want to talk about it Mike?

Share this post


Link to post
Share on other sites
Gavin Andresen    481

Hey David:

 

Jeff Garzik from the core dev team will be at the GHash.io round table in London. Jeff and Greg keep much closer track on the state of miners/mining/pools/etc than I do.

Share this post


Link to post
Share on other sites
David R Allen    318

Hey David:

 

Jeff Garzik from the core dev team will be at the GHash.io round table in London. Jeff and Greg keep much closer track on the state of miners/mining/pools/etc than I do.

 

Excellent Gavin, thank you so much.

 

And Dan, those weren't crickets. It was my winding stop watch. Everybody gets a weekend.

  • Upvote 1

Share this post


Link to post
Share on other sites

×