The unreasonable effectiveness of shared null results:

or, if open science were Wordle we might usually get the answer on the first line.

Wordle is easy. Science is hard.

This is a blog that compares science (the open kind) to cheating at Wordle. But not in a bad way. This is a blog, so I’ll run the narrative first. I have included links to further readings from The Open Scientist Handbook. [OSH] You can find all the other literature references added at the end.

ASAPbio recently (October, 2022) announced a competition to share negative results as preprints. <https://asapbio.org/competition2022>. Sharing negative results is integral for open science to achieve its unreasonably effective potential. The sharing of all research products is one of open science’s main goals.

We can imagine, in an alternative present, an academic publishing endeavor that has long made space for null-results. Ideally, these results would be available in the same journals that publish positive results, and in the same proportion as these are each generated through rigorous scientific methods. Publication in this manner might fairly accurately reflect the sum of knowledge generated by research (if it included data, software, etc.).

Now, let’s look at the academic publishing regime we have today, where null-results are conspicuously absent, and the published corpus reveals a tiny fraction of the work of scientists across the globe. Sources on the topic of “publication bias” outline how this damages the entire academy. A further assortment of bad practices — of bad science — can also be uncovered through methodological and content reviews of published research. This is where Retraction Watch comes into play.

We can find at least two streams of perverse incentives in the current publication situation. The first is an outcome of the arbitrary scarcity of publication opportunities. This warps the whole research landscape and rewards narrowly selected research results, instead of valorizing methodological rigor. Even the available published work rarely includes enough information to allow replicating the research.

The second stream of bad science is the central role the current publication regime plays in career advancement and future funding. By using metrics that are hooked into “journal impact factors,” and other forms of pseudo prestige (e.g., the h-index), universities and funders get to pretend they can evaluate the merits of a researcher’s work without needing to spend the time and effort to make an actual qualitative review.

Apart from the weaknesses inherent in this metric, simply as a metric, when this metric becomes a goal for researchers, Goodhart’s Law predicts that this will be gamed until its original value is erased or even reversed. The unnecessary scarcity of publication opportunities creates an ersatz elite of “published” academics, and a much larger cohort of undervalued, marginalized researchers. [Fierce equality] (Perhaps another blog is needed to show how academy publication is like TikTok.)

OMG! I just got published in Science!

At the same time, the race to get published crowds out the open sharing of research results. The need to be first also prevents collaborative networked interactions with other teams that could greatly accelerate new knowledge discovery. When the great majority of the actual work of science has no place to be shared, the global work of science is fundamentally diminished.

The current availability of preprint servers for a number of disciplines (and also new AI search engines to facilitate discovery) means we are at the front edge of the capability to demonstrate how widespread, open sharing can decenter the current logic of scarcity, in favor of a new, extraordinary abundance.

Back to Wordle

I am going to use Wordle as an analogy that can show the unreasonable effectiveness of open sharing.

Find the right word…

For those of you who don’t Wordle, when you start a Wordle puzzle, you have six chances to uncover the correct five-letter word. Each layer provides information to help you out with the next layer.

Whew! just made it.

You build a solution space for the correct answer by knowing which letters are not used and which are used in the wrong place, or in the correct place.

That’s right. You get to use your own mistakes to learn and improve. In every new layer, the puzzle gets easier. There is also guesswork, and so a bit of luck involved. It’s an elegant design for a short puzzle. You can do one a day.

If Wordle had eight layers, most players would never lose. If Wordle only had two layers, most players would rarely win. Wordle works because its difficulty level is a sufficient challenge to most who play it.

One line is all you get…

But what if your job future depended on you filling in the correct word on the first level? What if you are given this puzzle and told you must guess correctly or find a new position?

You might be tempted to look online and find the answer. Everyone solves the same puzzle each day. Cheat to win? Why not? You can do “bad Wordle.” But then, someone would add a metric based on time, and only the first person who solves the puzzle gets job security. And on and on. The zero-sum-game solution.

However, unlike Wordle, the science puzzle in front of you has never been solved. That’s the whole point. You choose a significant unknown, because that is why you do science. You need to solve something new.

Currently, within the arbitrary scarcity of the publication regime, you need to solve this unknown now, because others are out there looking to solve the same, or related problems. Only the first solution will get published. It’s your lab team against all the others out there. (Of course this is an unnecessary competition, and a hallmark of failed science, but that’s another blog, comparing science with Survivor.) [Playing the game]

Science… only one research project will publish…

When you propose a research experiment, you only get one line: you have one chance to discover a result that explains something new. Nature (actual nature, not the journal) doesn’t give you a lot of hints, and the NSF has (finally) funded this one project for your team.

You are a scientist. You have subscribed to the hardest puzzle anywhere. Your job is to provide the answer to this puzzle. You have lined up all the resources you believe are sufficient. You have a proven methodology and a plan. Your team does the work. You have your results. Now you must publish your work.

science is a lot harder than Wordle… and you only get one line to solve.

Let’s say that getting an article into the academic journal of your choice (the one with a “high impact factor,” or whatever) today requires this:

Today, journals accept only a few research results. They brag about their rejection rate…

After finishing the project your actual research result, the information you found, may look like this:

This finding is as important as any…

Each bit of that finding is as valuable to science as any other finding. It just has no public home to go to. Today you have one sort-of-good — but actually unfortunate — choice, and other “choices” that are not good at all, and totally unfortunate for science.

The “goodish” choice is to keep the data safe, maybe host it up on a repository, and do the write-up for the granting agency. Use the lesson learned for the next research project. Your lab will add this research to its shared knowledge and move ahead. This is the “file-drawer” outcome.

You will wonder how this outcome will affect your ability to get future funding, and realize the lack of a publication might impact your next performance review. You might feel like all the work you accomplished was wasted. Yes, you found out something new, something important on its own terms, only this result was “negative.” It does not spell out a “significant finding” you can use to leverage your career.

Your research revealed a new piece of the larger truth. In terms of the knowledge space of your field, this new information occupies corner of the space of “already-accomplished-research.” It is not any less significant as a finding than any other research. It is another step in the long journey that is science. [Science and the infinite game] However, today, this work matters far less than it should to the academy. And less than it might for your job.

At this point, all of the perverse incentives of current science are now clearly in play if you let them infect what you do next. [Toxic incentives] Perhaps, in some desperation, you go back to the data and revise your hypothesis to match the “findings.”

Maybe you set out to prove that “people who eat their largest meal in the morning gain less weight,” but now your research proves — with great statistical precision — that “people do not eat while they are asleep.”

You shop this “finding” around to journals and one of them publishes it. It is no longer research you are proud of, but your list of publications is larger, and your funder might not notice.

Digging around the data you uncovered something you can show off…

Maybe you cannot find a different hypothesis, so you go back to the data and “regularize” this until some significant pattern pops up. You announce this as a “finding” and shop it to journals. Your lab team will need to be in on this move. They are implicated in your deceit. You figure that nobody will get funded to replicate your work, so this finding will be accepted as legitimate.

You have proven nothing here, but your desperation to save your own career.

Congratulations, your published work will mislead everyone who cites it. You have wounded the body of knowledge you and your colleagues share. Your career now means more to you than the integrity of your work.

Open science can fix this. It must, and it will.

Here is where open science can help. Let us imagine that the academy has promoted publishing null-results on preprint and eprint servers for a decade. With no need for the “file-drawer” option, the number of null-result findings available online is now much larger than the number of recent significant findings. (Note: I use the term “eprint” the signify new publishing efforts that publish all submissions and then do open peer review to add value to these.)

Because nobody needs to contort their research to get published, the actual research statistics being used are much more rigorous, and the data more reusable and available. Let’s add here that a null-result pre/eprint that gets cited is treated the same as any other publication, in terms of career-building metrics. That’s another goal of open science: new institutional cultural practices and norms.

Back to square one… you still need to do the research

Open science still means you are facing the same complex problems

You have just received notice that your funding has been approved. You are still faced with a complex phenomenon to explain, just as before.

Of course, you have already done a complete literature search through all the appropriate journals to see if there are positive findings that would improve the questions you have, and the final hypothesis you will be using. Your research did not end with the already-published positive findings. Before you even wrote you proposal, you expanded your research (using powerful AI-enhanced search routines, and advanced keyword techniques) to include negative-results in related experiments. These are mostly posted on preprint servers.

This is what you discovered: A colleague of yours in Germany did a research project closely related to your work, and this was their team’s result:

You found this on a preprint server…

Another colleague in China also put their negative result up on a preprint server:

This looks very interesting to your team…

A post-graduate researcher in California submitted their research findings to a preprint server:

Unexpected but really valuable

Your team sifts through these findings and their open data. You use protocols developed to match other findings with the phenomenon you are investigating.

You are encouraged when you realize that instead of just this:

The shared resource of null-results has filled in many of the unknowns internal to the unknown you are tackling. You now know so much more about the object/process under study:

You have a better handle on the problem you face. Your team can focus its methods on only those parts that are missing. The rest of the puzzle offers new clues to its solution. You really only need one line to figure this out.

The shared findings occupy much of the original unknown space

You now have a much better starting point, a major advantage, from which to discover something that completes a bit of the landscape of current science knowledge.

and you do not need to do this alone… and there is no race to win (see: R.E. Martin [1998])

Your agency’s program manager is jazzed. Your university puts out a press release. You are networking with new collaborators across the globe, planning the next project together. Of course, you cite the research of all of your sources in your publication. Their shared research products made your findings possible. [Demand Sharing] You add your data to theirs on an open repository. And you pop the resulting publication onto an open, online server.

Science is hard enough. Let’s work in our universities, agencies, and societies to promote the added, unreasonably effective, benefits of open sharing and collaboration.

Certainly the open data you discover from the null-research results cannot be expected to be quite so providential for your work. But these shared resources will offer an abundance of new information and helpful guidance for your own efforts. You are not alone. You don’t have to race. There is no race to win. Your lab has posted seventeen prior research results with data — all of them negative results — up on the web. Your grad students field requests for these data and collect citations for this work. They are making connections across the planet that should enhance their future careers. Curiously, without the race, science moves a lot faster.

A hundred-thousand science research teams working apart, each one of them looking to “win science” by keeping their work secret, would fail constantly against a hundred teams working in concert. [Open Collaboration Networks] The latter gain insights and save time by sharing all of their work toward a common goal of collective understanding.

The unreasonable effectiveness of shared null results is just one example of how embracing abundance instead of scarcity accelerates science knowledge discovery.

CODA: Free riders on the sharing-null-results bus

There is a “what’s wrong with this picture” perspective we can clear up, even if we don’t have an optimal solution space (that space will need to be emergent). Any move from a zero-sum game (e.g., science today) to a non-zero-sum game, allows a few zero-sum game players — those who don’t mind violating cultural norms for their own advantage — to add the shared non-zero-sum assets to their own work, without attribution, and potentially compete more efficiently than before. This is your basic “free-rider” problem. Every commons faces this problem.

Looking at this another way, the free-rider problem becomes a free-rider opportunity within the academy, as long as the cultural norms for sharing are present. [Share like a scientist] Every scientist is a “free-rider” on the discoveries they use in their own research. The real free-rider problem happens when open resources are acquired freely and aggregated by corporations, which want to sell these back to the academy as proprietary property, with some marginal value-added service.

Free-riding is a problem that culture change can help resolve. Yes, there will be those who grab these assets and use them without credit, or massage these and market them. The general strategy for jerks, those who take advantage of a positive cultural change that valorizes sharing, is to marginalize them wherever possible. Academic institutions can cultivate social outrage against those who plagiarize others’ work, including null-results. Agencies can fund open repositories, and require their use. Open means really open. Closed, as John Wilbanks reminded us, means broken.

Additional readings and quotes from them

Bibliographic citations here

On publishing not capturing what science knows, and what reuse requires:

“In present research practice, openness occurs almost entirely through a single mechanism — the journal article. Buckheit and Donoho (1995) suggested that ‘a scientific publication is not the scholarship itself, it is merely advertising of the scholarship’ to emphasize how much of the actual research is opaque to readers. For the objective of knowledge accumulation, the benefits of openness are substantial…

Three areas of scientific practice — data, methods and tools, and workflow — are largely closed in present scientific practices. Increasing openness in each of them would substantially improve scientific progress.”

Nosek, Spies, and Motyl (2012); Buckheit and Donoho (1995)

On publication bias:

“Publication bias is a common theme in the history of science, and it still remains an issue. This is encapsulated in a piece of commentary published in Nature: ‘…negative findings are still a low priority for publication, so we need to find ways to make publishing them more attractive’ (O’Hara, 2011). Negative findings can have positive outcomes, and positive results do not equate to productive science. A reader commented online in response to the points raised by O’Hara: ‘Imagine a meticulously edited, online-only journal publishing negative results of the highest quality with controversial or paradigm-shifting impact. Nature Negatives’ (O’Hara, 2011). Negative results are considered to be taboo, but they can still have extensive implications that are worthy of publication and, as such, real clinical relevance that can be translated to other related research fields.”

Matosin, et al (2014); O’Hara (2011)

On impact factors and the h-index:

“Funders must also play a leading role in changing academic culture with respect to how the game is played. First and foremost, funders have a clear role in setting professional and ethical standards. For example, they can outline the appropriate standards in the treatment of colleagues and students with respect to such difficult questions as what warrants authorship and how to determine its ordering. Granting agencies should clearly emphasize the importance of quality and send a clear message that indices should not be used, as expressed by DORA, which many agencies have endorsed. Of particular importance is for funders not to monetize research outputs based on metrics, such as the h-index or journal impact factor.”

Chapman, et al (2019)

On Goodhart’s Law:

“The goal of measuring scientific productivity has given rise to quantitative performance metrics, including publication count, citations, combined citation-publication counts (e.g., h-index), journal impact factors (JIF), total research dollars, and total patents. These quantitative metrics now dominate decision-making in faculty hiring, promotion and tenure, awards, and funding.… Because these measures are subject to manipulation, they are doomed to become misleading and even counterproductive, according to Goodhart’s Law, which states that ‘’when a measure becomes a target, it ceases to be a good measure’”.

Edwards and Roy (2016)

On the file-drawer problem:

“For any given research area, one cannot tell how many studies have been conducted but never reported. The extreme view of the ‘file drawer problem’ is that journals are filled with the 5% of the studies that show Type I errors, while the file drawers are filled with the 95% of the studies that show non-significant results. Quantitative procedures for computing the tolerance for filed and future null results are reported and illustrated, and the implications are discussed.”

Rosenthal (1979)

On Science and the Infinite Game:

“The paradox of infinite play is that the players desire to continue the play in others. The paradox is precisely that they play only when others go on with the game. Infinite players play best when they become least necessary to the continuation of play. It is for this reason they play as mortals. The joyfulness of infinite play, its laughter, lies in learning to start something we cannot finish”

Carse (1987).

On the free-rider problem:

“But here’s the thing. In addition to the free rider problem, which we should solve as best we can, there’s a free rider opportunity. And while we whine about the problem, the opportunity has always been far larger and its value grows with every passing day.

The American economist Robert Solow demonstrated in the 1950s that nearly all of the productivity growth in history — particularly our rise from subsistence to affluence since the industrial revolution — was a result not of increasing capital investment, but of people finding better ways of working and playing, and then being copied.”

Gruen: We’re All Free Riders. Get over It!: Public goods of the twenty-first century

The Work of Culture in Your Open Science Organization

“Religion is a culture of faith; science is a culture of doubt” Richard Feynman (unsourced).

“Don’t think of culture as other than accumulated learning that sits inside you as one of your layers of consciousness” (Edwin Schein, 2016 <https://www.youtube.com/watch?v=6wJaNKIALLw> accessed April 4, 2019).

“‘Culture’ is everything we don’t have to do” (Brian Eno, 1996; W Magazine)

“‘Culture’ is anything you can get better at” Bruce Caron, 2019.

All the culture that fits: exploring the work of culture to prepare to change it

PLEASE NOTE: This is a draft of a bit of the Open Scientist Handbook. There are references/links to other parts of this work-in-progress that do not link here in this blog. Sorry. But you can also see what the Handbook will be offering soon.

We want culture in the academy to work for us, instead of against us. The many meanings of the word “culture” — each with certain claims to capture essential aspects of this spectrum of human proclivity and activity — make the task of outlining a notion of the “work of culture” also a chore of definitions. What is it about culture that can be said to do work? And what work is important for open science?

One goal of this book is to help scholars who have little or no background in the academic study of culture to gain a sufficient purchase on this notion to become confident, productive agents of culture change for their home institutions, their professional associations and research organizations, and for the academy as a global science endeavor. Like quantum mechanics and machine intelligence, the serious study of culture is not one of these “dip your toes in the shallow end” kind of endeavor. However, with a roadmap through just enough of this contested space, even tenured chemistry professors (or pick your discipline) can become bonafide organizational culture-change agents.

Getting back to basics

Beginning anthropology classes might spend a month covering the “history of the anthropological ideas of culture.” These notions developed first through colonial excursions, and then with missionaries and colonial settlers, and finally ethnographers. Courses on “organizational culture” are now required in MBA curricula and iSchools.

A recent (2017) online book for teaching anthropology in community colleges has distilled culture down to a few pages, entitled “The Culture Concept.” <http://perspectives.americananthro.org/Chapters/Culture_Concept.pdf>. Accessed April 4, 2019.

Arjo Klamer (2017), a Dutch economist, introduces culture to his economics class by adding two meaning domains for this word: culture as the accomplishments of a society (e.g., baroque style as a form of European culture), and culture as creative activity within sectors of the economy (the arts, architecture, music, etc.). His first meaning gives us the adjective “cultured,” applied to individuals who exemplify a certain noticeable style; while his second is where you go to when you click on the “culture” link in an online magazine or newspaper.

“Culture” is a section in your newspaper/magazine/webzine

Culture as a process

Folks who want to use culture and culture change as a resource or a tool to change social groups describe culture as a process. They then offer a method to intercept and guide this process (Marcus and Conner, 2014). Organizational management researchers are full of advice on the culture of organizations, but usually fail to look at how this type of culture fits into the larger sense of culture’s role in society or in individual identity. Anthropologists describe cultures and how these change without intervention, but little advice on how to intentionally change this. Here, you will find both anthropological and organizational perspectives, just so you are fully comfortable that you’ve travelled the entire landscape of the term “culture.”

Do you own your culture, or does your culture own you?

“Culture is public because meaning is” (Geertz, 1973).

Much of the disputed territory for culture, whether as an object of study, or as a field for intentional change, is centered on how culture is carried more or less unconsciously by the individual. Sometimes it feels as though we’ve been “marinated” in cultural practices our entire lives: language, cuisine, music, art, and now online content. There is a part of culture that is tacit, embodied, unspoken, and non-conscious. Culture theories tell us this, and they are not wrong. This aspect of culture is often used to demonstrate how difficult it is to manage culture.

A vague, squishy word, indeed

Jean-Louis Gassée (not an anthropologist; but rather of Apple, BeOS, and Palm fame), in a blog about Intel’s “toxic culture” writes:

“Our powerful human emotions are bundled into something we call Culture, itself a vague, squishy word……Culture develops within us in a manner similar to our taste buds: Our gustatory education starts with Mother’s milk and accumulates over time. The trouble with our acquired tastes, particularly in the realm of ideas, is that they drop below our consciousness: Raw data are filtered, judged, and labeled before being passed to our conscious, ‘rational’ processes.”

Gassée is pointing out that parts of the repertoire of shared meanings, behaviors, and sentiments that people would label “cultural” are known without any explicit knowledge of how and when we came to know these; and even less ability to describe them.

Schein (2010) calls this a cultural “layer.” This layer is learned from birth at home, and then in school, and then in the workplace, where the same tacit layer proves the hardest part to change. When your company/university/agency is running on a tacit culture layer, instead of on a reflexive intentional culture layer, it is most vulnerable to becoming toxic (Deep Dive: Toxic Culture).

Science is a reflexive, interrogative activity

Fortunately, the main aspects of academy culture we are hoping to change can all be made explicit and available to reflexive rebooting. In fact, open science is not reinventing science as much as clearing away the extraneous cultural underbrush (such as journal impact factors) that has collected in the past half-century or so. Scientists can openly interrogate these practices, and collectively move away from perverse incentives, conflicts of interest, and culturally-supported bad behavior in the academy. The leading advice to Silican Valley CEOs today is to avoid “f*cking up your culture” (See also: Don’t F*ck Up Your Culture; Retrieved May 17, 2019). The academy might want to listen here.

You cannot really avoid culture if you want change

A good point is worth saying twice: you may be an open-science pioneer who is eager and intent to bring productive changes to the academy, and yet still be uncomfortable with the notion of culture. You might prefer to offer solutions (e.g., coercive rules enforced by governments and funding organizations, novel technology platforms, and manifestos — so many manifestos) that, you hope, would shape “social behavior” without needing to confront or even consider culture. You look at the term “culture” and see a morass of competing meanings, with tangled and complex implications for the use of the term. How do you defend a program to change culture when you can’t get any three people in a room to agree on what culture means?

Scientists are many things. Each of these things have something in common: a desire for precision. The “vague, squishy” term “culture” offers very little precision and a whole load of ambiguity and complexity. As a scientist, you already have your hands full of ambiguity and complexity; you are striving to understand the inherent, emergent complexity of the universe. You rely on instruments that achieve ever-better accuracy and precision to help you extract some level of near-certainty to observe your object of study.

Many scientists are dismayed by the sheer amount of fuzziness surrounding the notion of culture. So the project at hand is to un-fuzzy that corner of culture where the academy can work on intentional changes to promote open science. The rest can remain terra incognito. The fact is, you don’t need to be an anthropologist to put culture to work in your organization.

In short: the good news is that the cultural work of open science is centered on those aspects of culture that can be intentionally described, discussed, and refactored — even if some of these might later become routine and get framed as default expectations. It’s not a bad thing to have your active culture also inform the tacit level of culture, it’s actually a goal: norms are cultural behaviors and attitudes that have become tacit culture. A norm is when “we open scientists do things like this,” and think: why would we do anything else?

Culture: trimmed down to size for the open scientist

Here we will trim the semantic tangle of the term “culture” to a more specific notion of culture: to the point where it can serve our understanding of how this works and how this fits into the future of the academy. The word “culture” will still hold all of its diverse and multiplex meanings everywhere else, however, here we’ll just agree to use it in one specific way to cut through a lot of the semantic shrubbery it has acquired over the centuries and around the globe.

Learning from anthropology

We can start by looking at some general attributes of “culture.” In his 1993 book, Culture, Chris Jenks notes (following Ralph Parsons):

“…for present purposes three prominent keynotes of the discussion [around culture] may be picked out: first, that culture is transmitted, it constitutes a heritage or a social tradition; secondly, that it is learned, it is not a manifestation, in particular content, of man’s genetic constitution; and third, that it is shared. Culture, that is, is on the one hand the product of, on the other hand a determinant of, systems of human social interaction” (Jenks 1993: 59).

Lets put these verbs into the following order: learn (first exposure) → share (locally) → transmit (across space/time). Repeat as needed. This sounds a lot like education, something the academy already does. For the individual, this process is, or can be, a lifelong activity. What Clifford Geertz reminds us is that these cultural activities are public. Nothing is cultural until it is shared. That means these activities are available to study, and to change, and to be changed through intentional intervention (although somewhat less available when they are only tacit).

One easy way to see what Jenks is proposing here is to substitute “language” for “culture;” after all, language is a good part of any society’s cultural repertoire. Saying that language is transmitted is to acknowledge that we don’t need to invent our own language anew every generation. Saying language is learned explains that we acquire this through learning as children and then hone this learning throughout our lives. To say that language is shared points to a key concept: we need others to make this work; it’s called “conversation”. In many ways, language is primarily a type of sharing. Other skills and cultural content exhibit these same features.

The reverse is also true. If a language is not transmitted over time it “dies”. If a person doesn’t learn a language, they are left outside the conversations that happen in that language. And when a language ceases to be shared in everyday life (e.g., it becomes a “sacred” language that can only be spoken in certain places/times), other language forms will take over in daily life. Languages change all the time. Remember that. They manifest lifelong, tacit cultural practices, and they still change.

Culture comes in community boxes

“Community, therefore, is where one learns and continues to practice how to ‘be social’. At the risk of substituting one indefinable category for another, we could say it is where one acquires ‘culture’” (Cohen, 1985).

The usual container for a culture is called “community.” As an organization grows and governs its own cultural work, you can say that the group becomes a community. You can dive into “community” elsewhere in the Handbook (Deep Dive: Communities, Collectives, and Commons). Notions of community will also be threaded into many of the Handbook chapters.

Meaning, Symbols, and Memes; oh my!

Exactly what is learned, transmitted, and shared as culture is complicated. “Meaning” usually pops up here, together with “symbols” (meaning carriers). In many ways anything that can be learned (anything you can get better at by learning this), and that must be shared in order to make sense as something to do (write a song, choose a new fashion statement, enter a conversation, sports, theatre, etc.) becomes culture when the various meanings of that learned behavior are also shared. You cannot have your own private culture. That said, you can have a very small community with its own distinguished cultural behaviors.

Memes are symbols that have been reimagined as cultural-genetic replicators. The analogy to biology is intentional, and meme theorists also talk of culture change as evolution. Since the 1970s, meme theories have been proposed to explain how certain cultural content packages spread and persist.

“[Richard] Dawkin’s way of speaking was not meant to suggest that memes are conscious actors, only that they are entities with interests that can be furthered by natural selection. Their interests are not our interests. ‘A meme,’ [Daniel] Dennett says, ‘is an information packet with attitude’” (Gleick, 2011).

The notion of a meme is centered on the idea that humans as social beings are shaped by culture the same way their bodies are shaped by their DNA. If you want to explore memes a bit more, here’s a good introduction (by Dennett) and some good counter arguments (by Lanier). Here we will talk about meaning and symbols and culture change, but you are certainly free to talk about memes and evolution. You can also look into “cultural science,” where evolutionary cultural studies are being done.

Culture is a plural noun

Not grammatically, of course, but we have seen and continue to see around us how cultural notions, skills, and activities are typically multiple, contested, fragile, and liable to change. Individuals tend to privilege those notions, skills, and activities they have invested time to learn (so nobody wants to be forced to use a different language). However, since culture must be shared to be viable, individuals continually find themselves in conversation with others who have differing cultural inventories. Culture is like a life-long song we only sing once, and none of us has been handed the score for the next chorus. We just keep on singing, in multipart harmony.

Knowing is the intrinsic work of culture in your organization

Of course, culture is not only a noun. Humans are cultural beings. Humans have culture. Humans do culture. There is a lot of culture going on all the time. More recent takes on organizational culture reject this as being just some packet of ideas that gets passed around.Today, more than ever before, culture is viral, active, flowing (Appadurai, 1996). Today, culture is on the internet too.

The recent work of John Seely Brown, coming out of organizational knowledge theories in the mid 1990s (See: Boland Jr. and Tankasi, 1995), has added (or recovered) a cultural angle on knowledge management (Cook and Brown, 1999). Instead of organizations stewarding an inventory of knowledge objects, what they need to do is open up contexts and spaces for knowing: contexts for the transmitting, learning, and sharing between and among their participants (Thomas and Brown, 2011).

This concept was then picked up by David Snowden and others (Kurtz and Snowden, 2003), who mapped the contexts of knowing and “sense-making” into what they called the Cynefin Framework (https://en.wikipedia.org/wiki/Cynefin_framework, Retrieved May 20, 2019). This framework is largely about identifying types of knowing — and ways of deciding — in corporations, as a corrective to the prior knowledge management systems which only covered tacit and discursive knowledge objects (Wenger et al, 2002).

The Cynefin Framework in 2014 By Snowded — Own work, CC BY-SA 3.0, https://commons.wikimedia.org/w/index.php?curid=33783436

“The framework sorts the issues facing leaders into five contexts defined by the nature of the relationship between cause and effect. Four of these — simple, complicated, complex, and chaotic — require leaders to diagnose situations and to act in contextually appropriate ways. The fifth — disorder — applies when it is unclear which of the other four contexts is predominant” (Snowden and Boone, 2007).

The Cynefin Framework describes several domains of knowing; the core qualities of knowing are different in each of these. Knowing is an activity, an action, not a commodity, not a thing to be managed.

Knowing, or sense-making, is an intrinsic work for organizational culture. This is particularly true in the academy, where new knowledge and learning outcomes are a chief value proposition. Scientific “knowledge” is an output of shared knowing.

The challenge is that these domains are not always fully manageable, and neither are the humans that engage in knowing with each other, most particularly in the complex domain of the infinite game. Knowing is why we might learn more in a 10 minute conversation than we can from a 1000 page book. Knowing is how scientists play the infinite game with one another. You can briefly explore the infinite game by going back to the Things about science section.

Cynefin for the Academy

For now, the main take-aways from using the Cynefin Framework for the academy are the following:

First: it helps to explain the difference between doing science, talking/writing science, and telling others about science. These occur in different domains; and,

Second: it begins to describe the complex, emergent space of the infinite game. Learning this is central to building academy governance for game play. For centuries, most scientists, or earlier, natural philosophers, and before them, philosophers, played the infinite game individually. Today, science and learning is a team sport, and the academy needs to find ways to govern team play (Deep Dive: Knowing to Play the Infinite Game).

The domains of decision-making for open-science organizations. Which domain does your organization currently use to make its decisions?

The Cynefin Framework is explored at length in Deep Dive sections on Leadership and Learning, so we will not pursue it further here, except for this: The Handbook also presents a version of the Cynefin Framework that uses three modal types of cultural activity to represent the framework’s logics (complex, complicated, simple). These modes are: festival, game, and spectacle. You will need to ask this question a lot: upon which logic does your organization base its decisions? Starting with the wrong logic will lead to bad, sometimes very bad, decisions. A lot of toxic culture in the academy is based on decisions arrived at in the wrong domain.

Festival: For those who grew up in the parts of the planet (such as most of North America) without festivals that involve actual danger, nudity, running with fire, social exposure, complex body skills, radical comedy — the various ingredients of festivity that make these events complex, emergent activities — we are not talking about the annual petunia festival here. Also note that the best intellectual conversations are like running with fire.

The Cultural Work of Social Organizations

Cultural practices and social organizations are intertwined in time and space. Social organizations are the social “appliances,” the furniture, that anchor human groups into more durable cultural contexts, which they support and are, in turn, supported by. These contexts expand our capacity for collective action, including economic and political action. Just as we do not need to—or get to—invent our own language, we don’t get to invent most of the social groups we intersect in our lives. But we can change them.

In order to pursue the intrinsic cultural work of the academy, we build communities inside organizations that use governance processes to support sharing knowing. We use can our organizations to manage other, social and economic tasks. If knowing is a dance, then community is the dance floor, and the organization is the dance hall.

In the twenty-five years since Jenks’ book, culture has seen a lot of new attention. From the portmanteau academic discipline of “cultural studies” to the cubicles of Silicon Valley start-up companies, the importance of culture for the everyday life and future prospects of societies and corporations has become a central theme. It’s high time for the academy to take a culture turn. You can help.

Now you know enough about the various aspects of culture to start rolling up your pants and wading in. You know that culture is (and must be) learned, shared, and transmitted. Most of culture is really vulnerable to intervention or substitution. Culture describes a broad range of human activities and a layer of meaning that is spread over (or under) social activities and organizations.

Knowing is an intrinsic work of culture, a primary activity for all cultural activities, but particularly for those, like science, that are involved in the infinite game. Knowing happens in more than one domain. The meanings of culture are all public. You can find them, interrogate them, and, yes, change them. That’s the next topic in the Handbook: The task: culture change.

References

Appadurai, Arjun. Modernity Al Large: Cultural Dimensions of Globalization. Vol. 1. U of Minnesota Press, 1996.

Boland Jr, Richard J, and Ramkrishnan V Tenkasi. “Perspective Making and Perspective Taking in Communities of Knowing.” Organization Science 6, no. 4 (1995): 350–372.

Cohen, A.P. The Symbolic Construction of Community. Chichester, Sussex: Ellis Horwood Ltd, 1985.

Cook, D.N., and John Seely Brown. “Bridging Epistemologies: The Generative Dance between Organizational Knowledge and Organizational Knowing.” Organization Science 10, no. 4 (1999): 381–400. http://www.jstor.org/stable/2640362.

Geertz, C. The Interpretation of Cultures. New York: Basic Books, 1973.

Gleick, James. The Information: A History, a Theory, a Flood. 1st ed. New York: Pantheon Books, 2011.

Jenks, Chris. Culture. London And. New York: Routledge, 1993.

Klamer, A. Doing the Right Thing: A Value Based Economy. 2nd ed. London: Ubiquity Press, 2017. https://doi.org/10.5334/bbb.

Kurtz, Cynthia F, and David J Snowden. “The New Dynamics of Strategy: Sense-Making in a Complex and Complicated World.” IBM Systems Journal 42, no. 3 (2003): 462–483.

Markus, Hazel Rose, and Alana Conner. Clash!: How to Thrive in a Multicultural World. Penguin, 2014.

Schein, Edgar H. Humble Inquiry: The Gentle Art of Asking Instead of Telling. Berrett-Koehler Publishers, 2013.

Thomas, Douglas, and John Seely Brown. A New Culture of Learning: Cultivating the Imagination for a World of Constant Change. Vol. 219. CreateSpace Lexington, KY, 2011.

Wenger, E., R.A. McDermott, and W. Snyder. Cultivating Communities of Practice: A Guide to Managing Knowledge. Harvard Business Press, 2002.

Open Science means leaving the idea desert for the idea farm

Open innovation starts with open idea sharing

“Less than 10 percent of innovation during the Renaissance is networked; two centuries later, a majority of breakthrough ideas emerge in collaborative environments. Multiple developments precipitate this shift, starting with Gutenberg’s press, which begins to have a material impact on secular research a century and a half after the first Bible hits the stands, as scientific ideas are stored and shared in the form of books and pamphlets. Postal systems, so central to Enlightenment science, flower across Europe; population densities increase in the urban centers; coffeehouses and formal institutions like the Royal Society create new hubs for intellectual collaboration.
Many of those innovation hubs exist outside the marketplace. The great minds of the period — Newton, Franklin, Priestley, Hooke, Jefferson, Locke, Lavoisier, Linnaeas — had little hope of financial reward for their ideas, and did everything in their power to encourage their circulation” (Johnson, 2011, emphasis added).

Ideas in the academy are another victim of the logic of arbitrary scarcity. They are also collateral damage in the proximity to the start-up economy. The academy should be an idea hot-house, instead we have an idea desert. The cultural shift to open science will be final when ideas flow across the planet like pinot at a faculty party.

Common-sense on idea sharing

Like talk, ideas are cheap. How many ideas (let’s limit these to synthetic insights about your area of research) do you have in a week? A day? An hour? So many that you really don’t take time to even jot them down? There are the ideas that connect your team (and the data you’ve collected, or plan to collect with funding) to a new hypothesis; that’s your next move in the infinite game. But so many other notions crowd into your thoughts:

  • Ideas you have while listening to a seminar talk outside your field, where you are curious how something you know might be of use or interest;
  • Ideas on where your discipline is headed, and those large-scale issues that might drive funder and association agendas;
  • Ideas that pop up when you read that new journal article (any article that does not give you new ideas is a waste of your time);
  • Ideas about what your graduate students might want to pursue to start their own infinite game play.
  • Ideas you put into that NSF proposal you submitted last month.

Face it: your professional life is brimming with ideas; that’s pretty much the point. Ideas are the starting line for the infinite game (See also: Things about science). There are about ten million science researchers on the planet. Each of them wakes up to a new day filled with new ideas. Almost all of them keep most of their ideas in their head until they are forgotten, replaced with other ideas, similarly forgotten, and a couple recent, unspoken insights.

Right now, today, almost all the ideas you have that are relevant to your work you might share if this were easy enough to do. You have any number of potential solutions for a wide range of issues in your area of research; solutions you have no intention of pursuing, but would really like to have solved — today if possible. For you, these are non-rivalrous ideas. You don’t mind if someone else, or anyone else, takes them and runs. At the same time, your idea might be a catalyst for someone else’s research; just the idea that leads them to a breakthrough.

You already have no reason to not share most of your ideas

Among your ideas are those you might want to propose to operationalize, given funding. So you tuck these away. And while your proposal is being evaluated, you worry that someone in that process will grab them for their own proposal (after down-grading yours: such is the sad state of the academy today).

The RFI idea paradox

Every so often, a funding agency or a foundation asks for feedback: they want your ideas about priorities for future research. Where will the science be in, say, five years? Ideally they would get a vast range of information from the thousands of scientists on their mailing lists. But realistically, they only get granules of ideas that are linked tightly to the goals of the teams/labs that will be angling for funding. “What should we focus on?” they ask. “Me,” you answer, although your text is designed to not say that directly.

Idea-gathering by funders is perhaps the least effective way to assemble knowledge about science. When it opened up an idea-farming platform to gather ideas, one major private foundation recently discovered that almost every idea came with an implied request for funding. So they shut down the service. This is not the fault of the researchers. They have five-hundred words to say what is most important for their discipline. What is most important for their discipline, in their perspective, is more support for an arena of research in which the researcher has already invested. Research funding is firmly embedded into the logic of scarcity today. Open science will explore other funding models.

So many ideas will not be collected by a funder RFI no matter how many responses they get.

The three elephants in the room for science idea sharing

Let’s recap here:

  1. Almost all your important ideas are non-rivalrous for you, so you have no reason to not share them, given the opportunity; and,
  2. Funding agencies are the least efficient organizations when it comes to gathering important ideas: the academy needs more and different idea gathering capabilities.
You already can share almost all of your ideas. 
 Funder RFIs are limited in utility; better to build an independent idea farm. 
 Your own ideas are good; but they are just the short tail of what is happening elsewhere. Add other ideas to them to make your ideas great, and then share these.

These are the two current “elephants in the room” for academy idea sharing. Should an open platform for science idea sharing (along the lines of current idea farming platforms) become available and popular, a third elephant is born: the possibility for open innovation. One of the major changes for corporate R&D in the past twenty years is “open innovation” (Johnson, 2011). This has also become a clarion call for the academy (Europäische Kommission, 2016). On the web, Quora and Stack Overflow offer networked question and answer platform solutions supporting open innovation. For-profit idea farming platforms like IdeaScale offer idea networking for corporate open innovation, through a subscription.

Open innovation starts with the premise “innovation happens elsewhere”:

“Innovation happens everywhere, but there is simply more elsewhere than here. Silly as it sounds, this is the brutal truth: Regardless of how smart, creative, and innovative you believe your organization is, there are more smart, creative, and innovative people outside your organization than inside” (Goldman and Gabriel, 2005).

Idea sharing for open science at an early EarthCube charrette

Ideas happen elsewhere too. The academy has a lot of elsewheres not often heard from. These “long-tail” communities and institutions are the academic homes to the great majority of scientists on the planet; they just happen to not be on the campus of one of the “better known” universities. The even longer tail includes scientists working outside of the academy, and citizen scientists anywhere. They all have ideas.

Put a lot of ideas into a shared, networked (databased, searchable, with discovery tools) environment, and innovation will blossom. This environment will become a place where, as Matt Ridley says, “ideas go to have sex.”

Connect even a million scientists (a small percentage of the total number) across the planet through their most recent ideas, and you should find a select few of them who happen to be considering precisely the same problematic currently puzzling you. Then you can reach out and build collaboratives to explore these together. Thinking of writing a proposal? Mine the combined idea farm of the planet to make your proposal ideas better; and then share these new ideas online (you can embargo them if you are worried). Your graduate students will be looking to see where their ideas are shared elsewhere, and how they can push their own infinite game play into new ground. You can mine the platform to sharpen your paper or your poster. Big data miners can also analyze and model these ideas to create a new form of synthetic learning about how science is done. The new elephant in the room is the potential capacity for communication of ideas, learning opportunities, and collaboration on the internet.

An open platform for your ideas to procreate: one good idea deserves a billion others

What if every day, say at the end of the work day, or after a beer, each scientist on the planet hopped online and added one idea to the global idea-farm platform (with some tags to help discovery)? What if ten-percent of them decided to add lots of ideas every month (the power-law curve suggests this is inevitable)? After a single year, there would be more than three billion ideas on the platform. Lots of overlap and similarities, but a whole lot of variety and difference too; coming from the minds of people who woke up in a hundred different nations. Each idea is time-stamped, with a permanent ID, and linked to its author. Every entry takes a minute or two to accomplish. A phone app lets you speak your idea directly into the mix.

Want to add a crazy good idea, or worried an idea might seem naive? Use your personal alias. Want to add a comment or a question to someone else’s idea? Go ahead. Feeling paranoid? Lock your proposal insight into an embargoed, timestamped vault on the platform. Open this later to show off. Then please try to be less paranoid, and more generous in the future.

Demand sharing means giving what is most valuable to you to the academy. This is a value and a norm for open science. Open science initiatives are building open platforms for a variety of internet services. The platform for open idea farming may not be here now, but can be built with a bit of funding and the right home.

Become an ImagiNative

There is a whole lot of “elsewhere” out there in the global Republic of Science (Polanyi, 1962). You need to be in touch will all these elsewhere ideas and with the people thinking them who also share your disciplinary/theoretical neighborhood. As Clay Shirky noted, “We also have to account for opportunity, ways of actually taking advantage of our ability to participate in concert where we previously consumed alone” (2010). You need to become an ImagiNative; open to new modes of collective knowing. And your lab, your school, your university needs to support open innovation and give up on patents (but that’s another blog). It’s all a part of playing the infinite game.

What if one of your ideas (you had this in the shower yesterday, and spoke it into your phone app over coffee) were picked up by a lab in another county, on another continent, and used to create a new theory that rocked your discipline; and in the paper that announced this theory, your idea was cited as a key element? How rewarding would that be? How many times might this happen across the planet in an open-innovation environment? And what if you searched the platform and found an idea from an early-career scientist in Sri Lanka that gave you a new insight into your current work, so you cited them in your next paper. How great for them. Soon, you might find the courage to give away insights you’ve been holding on to for years, and offering new ideas in response to the ideas of others. Congratulations, you a now an ImagiNative; a passionate knowledge explorer in the infinite game of science.

References:

Europäische Kommission, ed. Open Innovation, Open Science, Open to the World: A Vision for Europe. Luxembourg: Publications Office of the European Union, 2016.

Goldman, Ron, and Richard P. Gabriel. Innovation Happens Elsewhere: Open Source as Business Strategy. Morgan Kaufmann, 2005.

Johnson, S. Where Good Ideas Come from: The Seven Patterns of Innovation. Penguin UK, 2011.

Polanyi, M. “The Republic of Science: Its Political and Economic Theory.” Minerva 1 (1962): 54–73.

Shirky, C. Cognitive Surplus: Creativity and Generosity in a Connected Age. Penguin UK, 2010.

Thoughts on Governance for your New Big Data VO

A well cared for volunteer community is like a great South Berkeley garden!
A well cared for volunteer community is like a great South Berkeley garden!

NOTE: too long for a blog (sorry), but I did want this to be available.
The West Big Data Innovation Hub held its first all-hands-meeting in Berkeley last Thursday. What follows is a short talk I gave to the newly-formed Governance Working Group.
The Hub seeks to become a community-led, volunteer-run organization that can bring together the academy and industry… and that other academy (the one with the statues), and regional and metro government organizations into a forum where new knowledge will be born to build the practices and the technologies for big data use in the western US.
To become this organization it will need to spin up governance. An initial task for the governance working group is to draft a preliminary governance document that outlines the shape of the Hub’s decision space, and the desired processes to enable those HUB activities needed to realize the mission of the organization.
Virtual organization governance is hard. And the knowledge of how to succeed is not well understood.  We do know that the opportunities for failure are numerous. Funders will need to exercise patience and forbearance during the spin-up process. 
I don’t know of any NSF-funded community-led, volunteer-run organization that can be a model for this governance. I would be very happy to hear about one.  It would be great if this Hub becomes that successful organization.    
I have three suggestions (with the usual caveats) to help frame the work of this working group.

NUMBER ONE: Your community does not yet exist.  

There is a quote attributed to either Abraham Lincoln or Darryl Royal (depending if you’re from Texas or not)… “If you have five minutes to cut down a tree, spend the first three sharpening your axe.” 
Community building activities is the hub sharpening its axe.
Right now, when someone talks about the “big data community” that’s just another word for a bunch of people whose jobs or research involve big data. That’s a cohort, not a community. If you want community—and you do want community—you have to build it first.  That’s why you need to spend resources getting more people into the process and give them every reason to stay involved.
The first real job of the hub is to build your member community. 
Part of building your community is to give your members a stage for their vision of the future.  Challenge your members to envision the destination that marks the optimal big-data future for a wide range of stakeholders, then build a model for this destination inside the Hub.  
To meld vision with action and purpose and forge something that is new and useful, that’s a great goal: think of the Hub as the Trader Joes of big data. The place people know to go to… in order  to get what they need.
NOTE: Why do you actually need community? There’s a whole other talk there….  Community is the platform for supporting trustful teamwork… without it, you will not get things done. Without it emails will not get answered, telecons will not be attended, ideas and problems will not surface in conversations… and meetings will be tedious.

NUMBER TWO: Engagement is central. 

ANOTHER QUOTE: Terry Pratchett, the philosopher poet, once wrote: “Give a man a fire and he’s warm for a day. Ah, but set a man on fire and he’s warm for the rest of his life…” 
You governance effort should be centered on maximizing member engagement by giving the greatest number of members opportunities to do what they believe is most important for them to do RIGHT NOW. Invite new members to join and then ask them what the hub can do for them. This is not a Kennedy moment.
Your members want pizza… it’s your job to build them a kitchen and let them cook.
Your steering committee (or whatever this is called) needs to be 90% listening post and 10% command center. It needs to listen and respond to members who want to use the Hub to do what they think the hub should do. It needs to coordinate activities and look for gaps. It needs to remind members of the vision, the values, and the mission goals of the organization, and then remind them that this vision, these values, and the mission belong to them and are open to all members to reconfigure and improve.
The Hub needs to be a learning organization with multiple coordinated communication channels… Members need to know their ideas have currency in the organization.  
Do not be afraid of your members, but do be wary of members that seem to want to lead without first attracting any followers. Spread leadership around. Look for leadership on the edges and grow it.
Engagement will lead to expertise.   Over time, the members will learn to become better members.  The organization should improve over time. It will not start out amazing.  It can become amazing if you let it.
Each member needs to get more than they give to the organization. If they don’t, then you’re probably doing it wrong. This will be difficult at first, so the shared vision will need to carry people through that initial phase.
Creating a bunch of committees and a list of tasks that need to be finished on a deadline is NOT the way to engage members. If you think that’s engagement, you are probably doing it wrong.  YES, some things need to be done soon to get the ball rolling. But remember that volunteers have other, full time jobs.

NUMBER THREE:  There can be a great ROI for the NSF

The Hub’s success will provide the NSF with a return on its investment that is likely to be largely different than what it expects today, but also hugely significant and valuable.
Final quote here: Brandon Sanderson, the novelist wrote: “Expectations are like fine pottery. The harder you hold them, the more likely they are to break.”
The hub is NOT an NSF-funded facility, or a facsimile of a facility…
Unlike a facility, the NSF will not need to fund a large building somewhere and maintain state-of-the-art equipment. The NSF already funds these facilities for its big data effort.  The Hub is not funded to be a facility and will not act like a facility. 
The hub is also not just another funded project… 
Unlike a fully funded project, the NSF will not be paying every member to accomplish work in a managed effort with timelines and deliverables. 
Volunteers are not employees. They cannot and should not be tasked to do employee-style work. They have other jobs.  The backbone coordination projects for the hubs and spokes are paid to enable their volunteer members to do the work of volunteers. The Hub is not a giant funded project. It will not work like a giant funded project. It cannot be managed. It must be governed.  This means it needs to govern itself. 
Self governance is the biggest risk of failure for the hub. That’s why the work you do in this working group is crucial.
Self governance is also the only pathway to success. So, there is a possible downside and potentially a really big upside…
Remember that process is always more important than product.  You may need to remind your NSF program managers of this from time to time.
The Hub needs to take full advantage of the opportunities and structural capacities it inherits as a community-led, volunteer-run organization. It’s goal is to be the best darn community-led, volunteer-run organization it can be.  Not a facility and not a big, clumsy funded project.
Here are Seven Things the NSF can get only by NOT funding them directly, but through supporting the HUB as a community-led virtual organization of big-data scientists/technologists:
1. The NSF gets to query and mine a durable, expandable level of collective intelligence and a requisite variety of knowledge within the HUB;
2. The NSF can depend on an increased level of adoption to standards and shared practices that emerge from the HUB;
3. The NSF will gain an ability to use the HUB’s community network to create new teams capable of tackling important big-data issues (also it can expect better proposals led by hub member teams);
4. The NSF can use the HUB’s community to evaluate high-level decisions before these are implemented (=higher quality feedback than simple RFIs);
5. Social media becomes even more social inside the HUB big-data community, with lateral linkages across the entire internet. This can amplify the NSF’s social media impact;
6. The Hub’s diverse stakeholders will be able to self-manage a broad array of goals and strategies tuned to a central vision and mission and with minimal NSF funding; and,
7. The NSF and the Hub will be able to identify emergent leadership for additional efforts.
Bottom Line: Sponsoring a community-led, volunteer-run big data Hub offers a great ROI for the NSF. There are whole arenas of valuable work to be done, but only if nobody funds this work directly, but instead funds the backbone organization that supports a community of volunteers. This is the promise of a community-led organization.
And it all starts with self-governance…
To operationalize your community-building effort you will be spinning up the first iteration of governance.  If you can keep this first effort nimble, direct, as open to membership participation as you can, and easy to modify, all will be good.  Do not sweat the details at this point.  Right now you are building just the backbone for the organization. Just enough to enable and legitimate the first round of decisions.
Make sure that this document is not set in concrete… it will need to change several times in the next 3-5 years. In the beginning, create a simple process and a low threshold for changes (not a super majority). TIP: Keep all the governance documents on GitHub or something like that. Stay away from Google Docs! Shun Word and PDFs!   

Postscript:

Hallmark moments in the future of this Hub if it is successful:
At some point 90% of the work being done through the Hub will be by people not in this room today. The point is to grow and get more diverse. With proper engagement new people will be finding productive activities in the hub. [with growth and new leadership from the community] 
At some point none of the people on the steering committee will be funded by the NSF for this project…  [this is a community-led org… yes?]… 
At a future AHM meeting more than 50% of the attendees will be attending for the first time.

Yes, your agency/foundation can sponsor world-class virtual organizations to transform the sciences

For VRVOs conviviality is essential
For VRVOs, conviviality is essential

I’ve just returned from the Summer meeting of the Federation of Earth Science Information Partners (ESIP). After nearly two decades of “making data matter”, ESIP continues to show real value to its sponsors. Indeed, the next few years might be a period where ESIP grows well beyond its original scope (remotely sensed Earth data) to tackle data and software issues throughout the geosciences. A good deal of the buzz at this year’s Summer meeting was a new appreciation for the “ESIP way” of getting things done.
ESIP champions open science at all levels, and this openness extends to everything ESIP does internally. ESIP is building a strong culture for the pursuit of open science in the geosciences, and remains a model for other volunteer-run virtual organizations (VRVO) across science domains. There are lessons learned here that can be applied to any arena of science.
I hope other agency sponsors will take note of ESIP when they propose to fund a “community-led, volunteer-run virtual organization.” In this letter I’m going to point out some central dynamics that can maximize the ROI for sponsors and enable these organizations to do their work of transforming science. One note: I am using the term “sponsor” here to designate agencies or foundations that fund the backbone organization, the staff of the VRVO. The work of volunteers is of course, not directly funded (apart from some logistic support).

The biggest picture
The real potential for any science VRVO to return value to its sponsors is realized as this organization develops into an active, vibrant community-led, volunteer-run virtual science/technology organization. To capture this value, the VRVO needs to focus on those activities that leverage the advantages peculiar to this type of organization, with special attention to activities that could not be realized through direct funding as, say, a funded research center. This is a crucial point. The real advantages that the VRVO offers to science and to its sponsors are based on the fact that it is not a funded project or center, and that the difference between it and funded centers (or facilities, or projects) is intentional and generative to its ROI.
The simple truth is that any volunteer-run organization will never be able to perform exactly like a funded center, just as centers cannot perform like VRVOs. Community-led organizations make, at best, mediocre research centers. Volunteers cannot be pushed to return the same type of deliverables as those expected by a center.
The biggest return that any VRVO will provide to its sponsors will come from circumstances where incentives other than funding are in play. In fact, adding money is generally a counter-incentive in these circumstances. Among these returns are the following:

  • A durable, expandable level of collective intelligence that can be queried and mined;
  • An amplified positive level of adoption to standards and shared practices;
  • An ability to use the network to create new teams capable of tackling important issues (=better proposals); and,
  • The ability to manage a diverse set of goals and strategies within the group, each of them important to a single stakeholder community, but all of them tuned to a central vision and mission.

Elsewhere I have outlined a larger number of such returns on investment. I continue to receive comments listing additional ones. I’ll do an updated list before the end of the year.

None of these returns can be funded directly by the sponsors, apart from supporting the backbone organization that in turn supports the VRVO. And none of these could effectively be funded through a center or other entity. They are predictable outcomes only of precisely the type of organization that the VRVO will, hopefully, achieve.

The real test for a science VRVO is to develop fully within the scope and logic of its organizational type. The concomitant test for the sponsors is to understand that sponsoring a new and different type of organization will require some new expectations and some period (a few years) of growth and experimentation to allow the virtual organization to find its own strength and limits.

Experiments, such as micro-funding are easier in a VRVO
Experiments, such as micro-funding, are easier in a VRVO

Governance NOT Management
One important lesson learned at ESIP is this: governance must never be reduced to management. Funded projects and centers are managed. VRVOs are  self-governed. Volunteer-run organizations are intrinsically unmanageable as a whole, and at their best. A VRVO can certainly house dozens or hundreds of small, self-directed teams where real work can be managed. ESIP “clusters” are good example. These teams can produce valuable and timely deliverables for science and for the sponsors.
The style of governance is also very important here. Attempts to shift governance away from the membership and into top-down executive- or oversight committees are always counterproductive. They give the membership a clear alibi to not care about the organization. Academics have enough alibis to not volunteer without adding this one. The members need to own the mission, vision, and strategies for the VO. Successful activities will emerge from initiatives that have been started independently and with some immediate urgency by small groups and which grow into major efforts with broadly valued deliverables. Bottom-up governance will outperform top-down management over the long term.

Science culture shifting
Probably the largest recognized impact that science VRVOs can make here—and perhaps only these can accomplish this—is to model a new, intentional cultural mode of producing science. This new cultural model will likely be centered on sharing (sharing is also one of the oldest cultural traits of science, only recently neglected). Sharing ideas. Sharing software, tools, techniques, data, metadata, workflows, algorithms, methodologies, null data, and then sharing results. Reuse needs to become a key metric of science knowledge (Cameron Neylon noted this at the original Beyond the PDF conference).
Transforming science means changing the culture of science. Science VRVOs must perform real culture work here. This is often a challenge for their sponsors, as these organizations are usually well situated at the center of the existing science culture. The key learning moments and opportunities, and perhaps the highest ROI for sponsoring a science VRVO is when this organization teaches its sponsor to change.

Three critical governance conditions any agency/foundation sponsor needs to heed.

There are three necessary conditions for an agency-sponsored, community-led organization to be accepted as legitimate by a science community.

  1. The sponsoring agency needs to allow the community to build its own governance. Governance documents and practices are not subject to approval or even review by the sponsoring agency, apart from needing to follow standard fiduciary rules. The sponsoring agency can offer input the same way other individuals and groups do, but the community decides its own practices. The metrics for the governance are the growth of volunteer participation, and spread of community involvement, the perceived transparency and fairness of decisions, and the community’s value placed on the work being done.
  2. The sponsoring agency has no right to review or in any way interfere with elections. All organization members have the right to run for office and to be elected.
  3. The agency’s sponsorship is designed to help the organization grow into its potential as a volunteer-run, community-led scientific organization. The returns on investment for the agency are multiple, but do not include tasking the organization to perform specific duties, other than to improve over time.

Postscript: of course, the golden rule of any volunteer organization, new or old, is this: DFUTC.

EarthCube is poised to start its mission to transform the geosciences

The red areas are sandstone.
The red areas are sandstone.

Here is the current vision statement of EarthCube

EarthCube enables transformative geoscience by fostering a community committed to providing unprecedented discovery, access, and analysis of geoscience data.

The primary goal of membership in EarthCube, and indeed of the entire culture of the EarthCube organization is to support this vision. The EarthCube vision describes a future where geoscience data is openly shared, and where a new science, one based on an abundance of sharable data, assembles new knowledge about our planet. Certainly shared open source software and open access publishing are anticipated in this vision. The vision accepts that it will take a committed community of domain and data scientists to realize this goal.

What can we predict about the culture of a community committed to transformational geosciences? How is this different from the culture of a community pursuing geoscience currently? We need to start building out our imagination of what transformative geoscience will look like and do.  One thing we might agree on is that this will be a much more open and collaborative effort.

Unprecedented data discovery, access, and analysis in the geosciences coupled with open science best practices will drive knowledge production to a new plateau. Many of today’s grand challenge questions about climate change, water cycles, human population interaction with ecosystems, and other arenas will no long be refractory to solution. For now, we can call the engine for this process “Open Geosciences” or OG for short.  What will OG pioneers be doing, and how can EarthCube foster these activities?

  • Pioneering OG scientists will collect new data using shared methodologies, workflows, and data formats.
  • These OG scientists will describe their data effectively (through shared metadata) and contribute this to a shared repository.
  • OG scientists will analyze their data with software tools that collect and maintain a record of the data provenance as well as metrics on the software platform.
  • OG scientists will report out their findings in open access publications, with links to the data and software.
  • OG scientists will peer review and add value to the work of others in open review systems.
  • OG domain and data scientists will reuse open data to synthesize new knowledge, and to build and calibrate models.
  • OG software engineers will collaborate on open software to improve capabilities and sustainability.
  • OG scientists will share more than data. They will share ideas, and null results, questions and problems, building on the network effect of organizations such as EarthCube to grow collective intelligence.
  • OG science funding agencies will work with OG communities to streamline research priority decisions and access to funding.

 At this stage, EarthCube is in its most institutionally reflexive moment and is most responsive to new ideas. Like a Silicon Valley start-up flush with cash and enthusiasm, EarthCube is poised to build its future up from the ground. EarthCube can succeed in its vision without attempted to directly influence the embedded cultures of government organizations, tier one universities, professional societies, and commercial publishers. EarthCube will succeed by building its own intentional culture, starting with its membership model and focused on its vision. EarthCube will only transform geoscience by proving that its members can do better science faster and cheaper through their commitment to the modes of scientific collaboration now made possible through EarthCube. EarthCube will transform science by transforming the practices and the attitudes of its own members.

NASA image by Robert Simmon with ASTER data. Caption by Holli Riebeek with information and review provided by David Mayer, Robert Simmon, and Michael Abrams.