We finally published the research roadmap on policy-making 2.0  from the Crossover project – see below.
I’d like to point out one of the conclusion, in the section on evaluation of policy 2.0:

What emerged from the analysis of the prize and the cases is that evidence for uptake is clearly available and now can be considered mature.

However, the evidence presented by the cases and the prize candidates with regard to their impact remains thin and anecdotal in nature. There is no thorough assessment of the impact on the quality of policies. Typically, the impact is demonstrated in terms of:

–       Visits to the website and participation rates

–        feedback and visibility towards media and politicians,

–       actual influence over the decisions taken

while the actual impact on the quality of policies is yet to be demonstrated. Some initial work (in the case of Gleam and Pathways 2050) is focussing on comparing the predictions with the reality as it is unfolding. Only the case of Ideascale presents some tangible ex ante estimates of the advantages of the decisions taken through policy-making 2.0, but no thourough ex post evaluation.

We are therefore elaborating a new evaluation framework, which we will deploy in the context of our new project – EU COMMUNITY. It will be structured alongside the following criteria:

  • number of participants (audience reached, active users, number of comments/input/votes)
  • type of participants (usual suspects vs new players)
  • involvement of decision makers (direct/indirect/non existent)
  • quality of ideas received (qualitative judgement & feedback from policy-makers)
  • actual usage of the output in policies (direct/indirect/non existent influence on policy decisions)
  • actual improvement of policy quality (counterfactual / difference in difference…)

For each of these criteria, we’ll develop a full methodology (questions+indicators+data collection & analysis) based mainly on the literature on democracy and participation. We’ll keep you posted!