Annual Fundraiser: Checking Banner Results

Hey All–
We’ve been tracking a huge amount of data during this year’s fundraiser so we can better understand which messages work well and which don’t.  We have two sets of banners that we set each day to run on all Wikipedia languages.  Set one is the English version; set two is all non-English versions.
We have two sets because we want our banners to run globally only if they are translated…which can take some time and volunteer effort.  This is why our non-English banners rotate slowly.  However, with English banners, we can build a banner quickly and put it up to see how it does.
Let’s go into some detail on selecting a rotation.  On December 3rd, our rotation and results (pp = Paypal, cc = credit card):
 
 

% of Total Site Notice Payment Number of Total Average Highest
WP views Type Donations Amount Gift Donation
12/3/09 20% 2009_Notice17 pp 210 $4,933.75 $23.49 $100.00
12/3/09 cc 156 $5,894.76 $37.79 $500.00
12/3/09 20% 2009_Notice18 pp 725 $11,807.41 $16.29 $1,000.00
12/3/09 cc 454 $10,145.52 $22.35 $250.00
12/3/09 40% 2009_Notice30_bold pp 504 $11,023.15 $21.87 $250.00
12/3/09 cc 389 $14,468.07 $37.19 $1,000.00
12/3/09 20% 2009_Notice36 pp 207 $4,650.90 $22.47 $120.00
12/3/09 cc 147 $5,890.83 $40.07 $250.00

As you can see, we had three different notices running at 20% and one banner, taken from one of the better notices from 2008’s fundraiser, 2009_Notice30_bold running at 40%.  It did well throughout last week.
As you can see, 2009_Notice18 pulled in a huge number of gifts despite only showing 20% of the time.   Also, it had a significantly lower average gift…probably as a result of the message itself.   Despite the low average gift, people seemed to really respond to the message…and donated lots.
We are wary of banner-fatigue and saturation, where users might be tired of seeing the same message, so we changed banners around for the next day.
Looking at December 4th, 2009:
 

% of Total Site Notice Payment Number of Total Average Highest
WP views Type Donations Amount Gift Donation
12/4/09 20% 2009_Notice17 pp 192 $4,280.08 $22.29 $250.00
12/4/09 cc 144 $4,778.83 $33.19 $250.00
12/4/09 20% 2009_Notice18 pp 611 $9,511.88 $15.57 $250.00
12/4/09 cc 390 $9,390.74 $24.08 $500.00
12/4/09 20% 2009_Notice30_bold pp 266 $6,573.39 $24.71 $1,024.00
12/4/09 cc 228 $6,696.20 $29.37 $238.75
12/4/09 20% 2009_Notice36 pp 205 $4,399.75 $21.46 $166.53
12/4/09 cc 162 $5,018.47 $30.98 $250.00
12/4/09 20% 2009_Notice40 pp 320 $7,795.45 $24.36 $1,000.00
12/4/09 cc 187 $6,113.04 $32.69 $500.00

We introduced 2009_Notice40 (“Thanks, Wikipedia.”) to the mix and cut back on another.   All five banners in rotation are at 20%.  Two of the banners are greatly outperforming the others.  We like what #40 is doing…but #18 is still rocking…1001 donations while the next closest is 507 donations (#40).
Again, we switched things up, removing #17, and adding 2009_Notice22, a similar, but opposite message to #18, which has been successful.
December 5th, 2009:
 

% of Total Site Notice Payment Number of Total Average Highest
WP views Type Donations Amount Gift Donation
12/5/09 20% 2009_Notice18 pp 518 $8,207.09 $15.84 $250.00
12/5/09 cc 314 $6,866.38 $21.87 $250.00
12/5/09 20% 2009_Notice22 pp 166 $4,634.09 $27.92 $250.00
12/5/09 cc 135 $4,938.74 $36.58 $250.00
12/5/09 20% 2009_Notice30_bold pp 272 $6,989.63 $25.70 $250.00
12/5/09 cc 197 $8,005.98 $40.64 $1,000.00
12/5/09 20% 2009_Notice36 pp 195 $4,440.20 $22.77 $191.00
12/5/09 cc 157 $7,092.57 $45.18 $1,000.00
12/5/09 20% 2009_Notice40 pp 279 $5,881.04 $21.08 $250.00
12/5/09 cc 168 $6,259.51 $37.26 $1,000.00

This day is a fascinating look at our banners and our user population.  Notice the results of #18 and #22…they are similar yet contrasting messages.   #18 is an quote from a small dollar donation (USD 1.95), acknowledging the sincerity of the gift.  #22 is a quote from a high dollar donation (USD 200),  emphasizing that a big gift is a small matter.
Compare the numbers of gifts for the two banners:
#18: 832 gifts, $15073.47 total
#22: 301 gifts, $9572.83 total
Not close right?  But look at the average gift sizes:
#18:  $15.84 for Paypal, $21.87 for credit card
#22:  $27.92 for Paypal, $36.58 for credit card
That’s quite a difference.  What was it about the message that would account for that?  Is it possible that our more affluent donors were more interested in #22, while other donors were affected by #18?
Post your thoughts below.
Rand Montoya, Head of Community Giving

Archive notice: This is an archived post from blog.wikimedia.org, which operated under different editorial and content guidelines than Diff.

30 Comments
Inline Feedbacks
View all comments

You are only measuring one side of the picture, of course. This is saying which banners get how much money – the positive side – but you also have to measure the negative side – which banners do people find the most offputting, and would discourage people from reading or contributing to Wikipedia. Not sure how you can do that, but I think you need to, in order to get teh full picture.

I would expect there to be a strong geographic dependency – both country-based and also sub-country-based – on which donation banner is most appealing. The ratio of paypal to credit card donations also has useful information, particularly on the 5th November. Note how those that use credit cards give a substantially higher average donation. However, interestingly, the highest donations of $1000+ don’t appear to depend on the payment mechanism. There is a rich set of statistical information here that can be used to determine which banners appeal to which types of people. … that’s the statistician side of me talking.… Read more »

I am not sure that averages and max donation is a sufficient metric. What would be interesting would be the median and maybe the 80th percentile or something like this, in order to cut outliers out. Single donations of 500+ $ are probably more often than not thought in advance and are not influenced by the actual banner that happens to be there at that very moment. So removing those outliers should give us much cleaner data on what works and what not… any chance to get access to the raw data? 🙂

Am I the only one, or is there something somewhat gross about engineering messages to extract the greatest possible donations? Maybe this is standard practice in fundraising, maybe it’s the right thing to do…but something about it doesn’t sit quite right. There’s something of a “we’ll say whatever, just hand over the cash” in it. I mean, how far do you take it? Experiment with different formulations of the same message? Randomly insert different adjectives, and measure which is best? It’s a little close to what the most evil marketers do, paying consultants big bucks to tap directly into the… Read more »

Looking at December 5th: #22 is the worst message, and doesn’t raise bigger donations (cf. #30 medium donation): put it off. #36 doesn’t raise much, but it’s important to convey the “non-profit” message and to show the thermometer, so keep it.
Small donations are the way: add other messages like #18, or show it more often.

I really like the idea of quoting donor comments. Here’s a coupla good ones:
(I couldn’t put the complete addresses in because it looked like spam)
http://wikimediafoundation.org/wiki/Special:ContributionHistory?
then add
offset=1260762541#361327
offset=1260762541#361337
offset=1260762541#361301
offset=1260760755#361263
offset=1260760755#361251

I give a donation but felt that is unfair that I donate more and another not donate at all. I think it is better to show banners like #18, beacuse is do my donate more easily.
Thanks for your work
Yoni

Steve (at #4): I think it depends. Even as people, when we ask others for something, we use a lot of well-honed machinery in doing so. As kids, we learn how (and how not) to ask for things. As adults, we’ve mainly forgotten the learning process while retaining the skill, so it seems more natural. But watching my nephew grow up reminds me how much practice has gone into it. The difference for me is to what extent the message stays authentic. Do our numerical methods merely help us to be better at getting across what we want to say?… Read more »

funny, have to say funny. A site that lets anyone add to anything, asking for money. Got to hand it to you wish I had the balls to have money and ask people to give me theirs. Funny world you people live in.

I like the message that encourages any donation, however small. It gets the point across that even if we don’t have much, that “this is our baby”. I remember the crummy old days of Encarta and having to buy new encyclopedia CDs every time something happened in the world. A resource that is ours, and when I say ours I mean everyone on the internet, and we can add entries as needed and change others if they’re incomplete. I donated ten dollars, and I suppose if things don’t go well enough I’ll donate ten more. My comment in the donation… Read more »

The banners discussed in #4 (Steve) and #8 (William) are mostly (all?) donor quotes. There must have been thousands and thousands of these donor comments, and the fundraising team presumably picked about half a dozen they thought were “marketable” or whatever you want to call it. It’s pretty similar to a book publisher printing (excerpts of) reviews on the back cover. Of course the publisher will only print positive reviews, and then only those excerpts of them that they think are most positive/marketable/likely to convince people to buy the book/whatever. People who read the back cover presumably know or at… Read more »

R.Stonecipher (#9), I guess you must be new to civilised society. Wikimedia is what we call a charity, and it relies on goodwill donations of both time and money in order to provide society with a valuable resource free of charge.
If you don’t think Wikimedia is delivering enough value to justify its income, or you don’t have the money to contribute, the fundraiser is not for you and nobody is condemning you for not donating.
But if you really have an issue with the fact that non-profit organisations have operating costs then you’re on your own.

@Will (#12) i would rather they have ad revenue and somehow profit share with the people who have helped build it (the volunteers not jimbo wales the guy who COFOUNDED wikipedia). of course, it probably wouldnt allow him to jet set around the world if he “gave back” to the community that has built wikipedia. why does wikipedia need a spokesperson? to garner more donations? i trust jimbo wales about as much as a politician. @Steve (#4) i have to agree. asking for donations is one thing. i remember quite a while back jimbo wales saying wikipedia would never have… Read more »

There is a cumulative effect to the banners and requests as well. Changing the banner on one single day is not a stand alone event. People (at least those that use wikipedia daily) are also responding to the ongoing message and may give today based on yesterday’s message. Maybe the desire to give is based on one message one day but the means (such as pay day or some other event) may force waiting until another day to actually donate. There is not a direct correlation between each message and each donation.

Wiki has become a proven winner and used by millions with reliable information and up to date
facts. I donated because I pay for things every day that don’t provide me with half of what
Wiki gives me.

The personal appeal from the founder is what got me!! It just showed me the person behind wiki… and the philosophy of it I guess

@yoni bar ilan. Why is it unfair^ If this resource is valuable to you, I dont think itd be unfair to donate. + the whole point is that not everybody is in the same situation. Not everyone CAN donate… so everyone gives what they can to keep it free.
Sure, there are people who have and dont give. But generally, the thought follows the above.

I totally support what Wikipedia is doing, and the insight into the banner compaign is interesting.
Chad (#13) Not sure I understand, in paragraph 1 you say prefer Wikipedia should have ad revenue (if contributors can get some), and in paragraph 2 you criticize Wikipedia for running personal appeals because they are almost like ads?
As for charities not compelling money, of course. The IRS *compels* people to give them money. Wikipedia, like other charities, is asking its users for money (and people who don’t use Wikipedia don’t see these requests). There is a significant difference.

I donated because of #18, and in an amount around equal to the average. I think that what makes Wikipedia truly great is many people making small contributions (even if WP also operates a bit by the Pareto Principle, I doubt that it would have had the impact it had or the popularity it does without its reputation as a source of knowledge from many). Charities must compel people to give. Corporations don’t have to compel, because they are charging for their services. Wikipedia does not charge for its services, and thus has no way to collect revenue other than… Read more »

The reason for relying on (many small) Donations instead of Corporate support / Advertisement lies of course in the question of independence. If you don’t want to go the way of the mass media, which always have to consider the interests and whishes of their corporate customers (and owners) when publishing you need to be free of “market money”. Customer is King, if you don’t want to have a king, you need to have no customers. Donation therefore is the “royal way” for Wikipedia to stay up, running and independent. The guys and girls at Wikimedia have the job to… Read more »

Thank you for your transparency on approach and results of fundraising. Although surprising to see such candor, it is completely in keeping with the spirit of your endeavor. Congratulations! I recently made donation and have two items of feedback: 1) for me, the messages of the banners had a compounding effect and the trigger message to giving was one where I could identify with donor — i.e. I think you need mix of messages: all messages combined give context and help make connection between received value, independence of source, and budgetary need; individual testimonies help to lower threshold to giving.… Read more »

[…] Annual Fundraiser: Checking Banner Results by Rand Montoya (December 11, 2009) Comment! […]

I have written a blog post on this with detailed explanation.
“Wikipedia donators and the Anchoring heuristic”
http://saperduper.org/post/293243288/wikipedia-donators-anchoring-heuristic
Please feel free to post your comments!

We need to review the negative case data – when presented with a banner the user did NOT donate. It is possible that you have 100%conversion on #18 but only 10%conversion on #40. This data leads to the conslusion that ALL VISITS presented with one of the above banners donate. This is not the case. I have NOT clicked to follow any of the banners in this fundraiser.

#19, #20: It is indeed possible to turn off the fundraising banner. There’s a [hide] link at the top right of the banner. #24: AFAICT we’re already implicitly measuring that, right? Say N people visited Wikipedia on a given day and saw a banner. Assuming all banners run at 20% (which is the case most of the time), the number of people seeing a certain banner would be N/5. If that banner got 500 donations, the number of people that saw that banner and didn’t donate is N/5 – 500. So for each banner, the number of people that saw… Read more »

[…]   果然是维基精神,上星期他们还在维基官方部落格刊出了一篇文章,将这些各式各样的捐款广告的“效果”大公开,让人大开眼界!他们基本上只透露了12月3日、4日、5日三天的这些捐款广告效果与转换率,结果如下: […]

[…] Optimizing Wikipedia’s Fundraising Banner Ads […]

[…] that Wikipdeia post I did, Here is some more interesting info via […]

“Am I the only one, or is there something somewhat gross about engineering messages to extract the greatest possible donations? Maybe this is standard practice in fundraising, maybe it’s the right thing to do…but something about it doesn’t sit quite right.” says Steve Bennett. If everyone thought that way charities that have saved millions of lives would all have closed down. And so would Wikipedia. And you wouldn’t be reading this. In fact, on the basis that we should never test what works and what doesn’t, all human progress startinbg with the discovery of fire or how to escape an… Read more »

Absolutely fascinating results! I was drawn by number 18, but finally donated using a different banner. Wikipedia is becoming more and more recognized as a legitimate source, especially for hard science, like finding and using equations, computer programming code, and hard biological facts like cellular respiration. It’s also an interesting source for cultures and country data. I pay hundreds of dollars for textbooks with the same information, or course I would donate.