Discussion Paper

No. 2017-32 | June 13, 2017
Supporting the UN SDGs transition: methodology for sustainability assessment and current worldwide ranking


The FEEM project APPS – Assessment, Projections and Policy of Sustainable Development Goals lies in the stream of research related to the quantitative assessment of the 17 Sustainable Development Goals (SDGs), adopted by the United Nations at the end of September 2015. The project consists of two phases. The first, retrospective, derives a composite multi-dimensional index and a worldwide ranking of current sustainability. This allows informing on strengths and weaknesses of today socio-economic development, as well as environmental criticalities, all around the world. The second phase, prospective, aims evaluating the future trends of sustainability in the world by 2030. The assessment is carried out by means of an extended version of the recursive-dynamic computable general equilibrium ICES macro-economic model that includes social and environmental indicators. The final goal is to highlight future challenges left unsolved in next 15 years of socio-economic development and analyze costs and benefits of specific policies to support the achievement of proposed targets. The methodology goes through the following steps: screening of indicators eligible to address the UN SDGs; data collection from relevant sources; organization in the three pillars of sustainability (economy, society, and environment); normalization to a common metrics; aggregation of the 25 indicators in composite indices by pillars as well as in the multi-dimensional index. The final ranking includes 139 countries. North European countries are at top of the ranking (Sweden, Norway and Switzerland).

JEL Classification:

O44, O57, Q01


  • Downloads: 268


Cite As

Lorenza Campagnolo, Fabio Eboli, Luca Farnia, and Carlo Carraro (2017). Supporting the UN SDGs transition: methodology for sustainability assessment and current worldwide ranking. Economics Discussion Papers, No 2017-32, Kiel Institute for the World Economy. http://www.economics-ejournal.org/economics/discussionpapers/2017-32

Comments and Questions

Anonymous - Referee report 1
June 30, 2017 - 11:46

The authors have calculated a sustainability index based on identified SDG indicators. While I overall see the importance to do something like this and agree with their general methodological approach, I see a number of weaknesses especially in the structure and writing of the paper.
- ...[more]

... I would not include the modelling approach in this paper. According to your project description, you have these two objectives to 1) do a retrospective evaluation and 2) model the future sustainability. Accordingly, I would also publish two papers for each objective but not mix them up. Without results but just a methodological approach, the second part seems lost otherwise.
- I am missing a sensitivity analysis for your index. You could for example vary the weighting schemes or use a different aggregation method.
- I think it is also necessary to normalize your pillar indicators before calculating your overall index since you will otherwise have an inherent weighting scheme due to different value ranges
- The paper is very light on introduction and discussion as well as literature supporting the statements
- To give the publication a bit more scientific value it would be also good to evaluate the correspondence of the three pillars against other indicators that are not considered in the SDG indicator set but still important e.g. in terms of outsourcing of environmental costs
- Abstract: I find it a bit weird to mention the project in the abstract. I think a project can go into the acknowledgements but should not be part of an abstract of a scientific publication. Furthermore, I think the abstract should focus on what you did in this paper and the results and not on what you want to do in the future. In the end the abstract should put the results in a broader context or discuss their implications, which is completely missing here.
- Introduction
o Page 1: why future tense?
o Page 1: I think a few sentences introducing the reader to the whole discussion about SDGs and MDGs would be good, especially since you are talking about “transition” in your title
o Page 1: why obviously at national level? Some might argue that sustainability can only be achieved at local level
o Page 1: “Each country will choose the national SDG indicators that are best suited to track its own progress towards sustainable development.” Really? If that is/were true wouldn´t it invite countries to choose those indicators which makes them look better?
o Page 2: partly methods are included in the introduction
o Page 2: as I said, I would concentrate on the composite index in this paper but not the modelling framework
o Page 2: you do not have to describe the section of your paper in such detail. Rather state your hypotheses and how you are going to adress them
o There is not one single reference in the introduction. I think it is indispensable to have a thorough literature review and put the study in the context of other similar studies
- Data screening, collection and organization
o Page 3: The first three paragraphs should rather go into the introduction but not in the methods
o Page 3, paragraph 5: You need to be more detailed, e.g. what is the geographical area? The same continent? What does “wen available and reliable” mean?
o Page 3: I would put the sentence about infeasibility of trend analyses in the discussion section
o Page 4 and others: Do not start a sentence with Table 1 or Figure 1, but rather make a statement and put the reference in brackets behind it.
o Page 5: last sentence is unnecessary
- Benchmarking, normalization and aggregation
o Page 6: it is true, that composite indices have a lot of advantages but also weaknesses. You should mention both, but maybe not in the methods part but rather in the intro or the discussion. I think the whole first paragraph should go into the intro or discussion but not the methods and be linked with other literature
o Page 6: first sentence of second paragraph sounds wrong
o Page 6: “First, by considering the different dimensions of sustainability from the indicators listed in Table 1. Second, by building an overall composite index t summarizes the three dimensions.” These are not sentences
o Page 6: better “based on the selected indicators (table 1).”
o Page 7: Indicators are typically ratios? Where do you get this from? They can be but there are lots of indicators that are not ratios. Normalization is a standard step when creating an index since they all have different ranges. This has nothing to do with an indicator being a ratio or not.
o Page 7: The whole second paragraph revers to inverse normalization. I think you can expect the reader to understand what and why you did it from one sentence e.g. “Some indicators were inverted during the normalization process, since higher values express less favorable conditions in terms of sustainability.” This is just one example but the point is that this is also a standard step for composite index creation.
o Figure 1 is unnecessary
o Page 6: Equation: iff??? Do you mean if? Furthermore, if you put equations there, you have to explain the variables.
o Page 7: “Defining and for all indicators is a hard task and possibly the most critical of the present analysis.” Why that? I assume that the variables refer to the minimum and maximum value of your dataset. The difficulty lies, as I understand, in your aim to monitor the progress over time. If you use the minimum and maximum value simply from your dataset for each year, it would not be possible to monitor any progress, since the normalized values always change according to the maximum and minimum value of the respective year. Is that correct? Your explanation is a bit weak on this. Since this is a real problem with composite indices, when monitoring progress over time, which is also not very often discussed, I would suggest to put a bit more effort into explaining this.
o Page 7: “For this reason, benchmarks for sustainable/unsustainable levels have been defined for each indicator relying on specific targets set by SDGs, EU best practices (especially in environmental dimension), scientific literature, as well as the observed data.” This info is a bit thin. Add an extra column in the table and explain your source.
o Page 7, Figure 2: You don´t need 2 tables showing your indicators and another figure. Consider having just one table or one figure and a table or maximum two tables.
o Page 9: It sounds as if you compared two different aggregation methods, but you basically just used two different methods to create your sub-indices and then to create your overall index. Maybe you can write it that way.
o Page 9: “For lack of space, we do not discuss here in detail the methodology behind fuzzy measures, the Choquet integral and fuzzy measure elicitation.” You have to explain the principles behind it. There is so much redundant and unnecessary information you can delete from the paper, but here you have to explain at least a little bit what you did.
o Page 9: So you did not use any weighting scheme for your indicators in the pillars but for the main pillars? I find it a bit difficult to assume that the environmental dimension is less or more important than the social dimension.
o Page 9: “A country is defined as sustainable whenever, to a certain extent, both its environmental and social dimensions are jointly satisfied and, to a lesser extent, when both its social and economic dimensions are jointly fulfilled.”?? I don´t get this sentence.
o Page 10: The Möbius set in Table 3 models the above definition for all the subsets – limited to cardinality two at maximum – that can be formed from the set % = & ', ()*, &*)! containing the three pillars.” Sentence wrong
- Assessing SDGs:
o Maps and figures: try to avoid green and red in the same plots. Many people have a red-green color blindness
o Page 10: You can delete the whole first paragraph
o Page 10: I find it always a bit difficult to say in a results section “unsurprisingly” or “as expected”. If you already knew the outcome, what is the point in doing it? Of course you have an expectation but this depends very much on the information you have in advance. I would stick to reporting simply your results. In the discussion you can say something whether it was surprising or not.
o Page 11, Fig. 3 and also other figures: Caption needs more detail
o Page 11, Fig. 4: use letters for sub-figures. Furthermore, in the caption you are saying you are showing normalized indicators, which you also refer to in the text but in the graph you have the SDGs. This makes it a bit confusing for the reader to follow
o Page 12: “The second sustainability dimension” Delete second or sustainability, otherwise it sounds as if there are two sustainability dimensions
o Page 15: Header sound wrong. Maybe The multi-dimensional composite index of Sustainability
o Page 15: Footnote is unnecessary
o Page 16: Footnote 13: Each pillar covers most countries
o Figure 11: Put the text below the graph in the caption. Delete blue outer line of points
o Why do you show only environmental vs economic pillar? What is the correlation coefficient
o Why don´t you show stacked barplots for all countries with the differents sustainability dimensions? This way you could show for all of them a. the total sustainability value and the shares of the different pillars
o Section 5: I would delete most of this section and concentrate on your results from the index. Part of it could go into the discussion as a future outlook but not before you actually discuss your approach and results from the paper

Lorenza Campagnolo, Fabio Eboli, Luca Farnia, and Carlo Carraro - Reply to referee report 1
September 20, 2017 - 08:35

see attached file

Lorenza Campagnolo, Fabio Eboli, Luca Farnia, and Carlo Carraro - Annex table related to the reply
September 20, 2017 - 08:39

see attached file

Anonymous - Referee report 2
July 24, 2017 - 08:46

The paper makes an interesting and valuable contribution. It is methodologically sound and as a contribution significant. The quantitative analysis is correct and makes sense.

However, I believe that the paper is not yet fit for publication. The reason for this is that is does not adequately ...[more]

... address the political economy behind the SDG - it takes the SDG at face value, so to speak, and constructs a framework of analysis to investigate (according to a selection of SDG indicators) whether specific countries are on track regarding the transformation to sustainable development or not. This is methodologically speaking fine - but it does not do justice to the construct of the SDG themselves. The SDG are not a neutral, objective and transparent set of indicators quantifying sustainable development - they are contradictory, conflictive, potentially damaging (growth benchmarks etc.) and maybe also too broad to serve as a real benchmark. They are the result of an international negotiation process of the UN system and reflect different aspirations and definitions of sustainable development of different countries and regions of the world. Even the fact that each and every country is allowed to formulate their own national indicators regarding the 17 headings of the SDG is problematic.

Within the quantitative Social Sciences, wizzardry is often performed with data and numbers - but without highlighting the political context of the data. Yet, the value added of quantitative analysis is enlarged if the political dimension of the data used is not overlooked but made part of the analysis. If the authors integrate this aspect into the paper - even into the analysis of the data, the paper would make a very useful contribution indeed. It might, however, weaken the idea of creating SDG-Rankings - and this, too, is an aspect the authors should discuss.

Lorenza Campagnolo, Fabio Eboli, Luca Farnia, and Carlo Carraro - Reply to referee report 2
September 20, 2017 - 08:40

see attached file