52019SC0374

Activity	Nr of activities / stakeholders reached
Policy seminars and analysis events	In 2014, eleven events; 2015 ten events were held; in total, close to 3,000 participants
Company visits to address the practical adaptation of Japanese management and production systems to Europe	Around five company visits per year, lasting one day each
Webinars to inform EU SMEs of possible partnering opportunities and support for improving EU manufacturers’ competitiveness	Average nr of participants per webinar: 107
Cluster matchmaking events to inform EU SMEs of Japanese clusters and the opportunities for cluster-to-cluster partnerships	Since 2014, three actions per year, attracting between fifteen to twenty SME

Source: EU-Japan Centre for Industrial Cooperation. Activity Report for Fiscal Year 2015

Under the framework conditions and competitiveness objective, most of the actions were still under implementation at the time of the evaluation. Nevertheless several actions such as the SME envoy Network, the Clusters Go international or the Tourism actions, have reached the milestones in terms of outputs set in the COSME yearly work programmes.

Overall, stakeholders are satisfied with the results of the SME policy actions. The Envoys perceive the network as a highly valuable EU-level forum where ideas and practices are exchanged. The SME Performance Review as well as the factsheets are used by the national representatives and are considered to be of high quality. Many events are organised throughout Europe during the SME Week, and the SME Assembly gathers stakeholders from all MS. The main outreach tool, Business Planet, is broadcasted on Euronews. There is a general perception that the activities organised at the EU-level contribute to a convergence of SME policies across COSME countries.

For the Tourism actions, the consulted stakeholders and beneficiaries indicated as direct results the creation of new strategic partnerships across borders and more visibility in international markets of touristic destinations in the EU. The Tourism actions are geared towards the involvement of actors along the value chain. Based on the data available, they directly involved about 100 companies, active in the tourism sector, 50 private enterprises active in other sectors and 200 organisations ( mainly business associations active in tourism), public agencies, user organisations, and professional service providers (e.g. marketing specialists).

The "Clusters Go International" actions show good potential to lead to an improved strategic position in global value chains and an enhanced access to potential inward investors as survey respondents and interviewees alike highlighted that the activities are based on a strategic analysis. The limited budget is flagged as a hurdle for their internationalisation activities and was reported by 90% of the clusters as an important bottleneck.

In October 2017, the Clusters Go International action involved 25 European Strategic Cluster Partnerships (ESCPs), launched under the "Clusters Go International action representing 145 cluster organisations across 23 European countries and more than 17,000 European SMEs. Only 15 out of the 25 selected ESCPs were funded due to budget constraints. They account for approximately 95 clusters reaching out to over 10,300 European SMEs. The Clusters Go International action succeeded in involving a high number of clusters, some even at a voluntary level, testifying the high interest in this action in the EU cluster community. Moreover, projects show overall good progress in reaching the outputs expected, such as partnership agreements.

The overall achievements of the 1st generation of European Strategic Cluster Partnerships (ESCPs) for going international 2016-2017 prove their success in supporting SME internationalisation. Over a 24 months period, they involved 2000 SMEs in activities targeting international third-markets generating 85 concrete business cooperation cases with international partners. 370 Cluster-to-Cluster events and 3010 Business-to-Business events have been conducted, whilst 39 Memoranda of Understanding and 45 collaboration projects implemented between EU clusters and international peer organisations. The interest in the action remained high for 2018-2019 with 23 new partnerships established since 46 .

The e-skills action reached the outputs expected. Stakeholders interviewed were positive. In 2016, it covered 21 different countries where events were organised and many stakeholders and high-level policy-makers were involved, including members of the European Parliament. Stakeholders and survey respondents unanimously indicated the effective engagement of various stakeholders such as education and training organisations, employees and entrepreneurs as the most important factor in driving progress towards an enhanced awareness and especially an enhanced availability of digital skills.

The two WATIFY campaigns managed to achieve 4 of the 5 specific objectives: to promote the transformation of industry through advanced technologies by sharing success stories ; to stimulate digital entrepreneurial culture by identifying and promoting digital transformation of SMEs and digital start-ups and their success (or failure) stories at pan-European, national, regional and local levels; to spark and support new business ideas in the digital economy and boost their successful commercialisation; and to strengthen business confidence in the application and added value of digital and key enabling technologies, business processes and new business models. However, WATIFY 2 did not succeed in supporting Europe’s regions to translate their smart specialisation strategies in the area of digital and key enabling technologies. Neither desk research, nor interviews were fruitful in collecting information on the support provided by WATIFY to regions for the translation of their smart specialisation strategies into actual projects.

Under the Entrepreneurship objective, the Erasmus for Young Entrepreneurs (EYE) was successful in reaching its expected outputs and outcomes. It involved in total around 2,000 New Entrepreneurs and we estimate that after approximately a year and a half, it led to the creation of 250 new companies, EUR 5 million of additional turnover, and about 1,000 new jobs. The most prominent weakness of this action line lies in the small scale of its activities.

·What are the factors driving or hindering progress and how do they link with different actions of the programme?

The key factor of success for COSME is to rely on an extended network of intermediaries that ensures proximity to SMEs and to stakeholders. This is particularly the case for the EEN, the clusters or EYE but also for the financial instruments.

The LGF has reached a high level of take-up thanks to the flexibility of its design. However, financial intermediaries report high administrative costs for entering into an agreement with the EIF and complying with its requirements are problematic. The threshold of EUR 150 000 above which the financial intermediaries must check if the SME does not meet any of the innovation criteria for funding under the H2020 InnovFin instrument creates the negative effect of inducing the intermediaries to limit their financial offer to this threshold. Most important, the threshold is deemed inadequate to the needs of many SMEs. The EFG is falling behind on its targeted commitments. The growth projects of these SMEs mainly focus on expansion in the existing (local) markets and the development of new products and services. The key barrier identified in the supporting study is the maturity of Venture Capital markets in the EU. Geographically, the beneficiary SMEs are so far concentrated in the more mature financial markets.

The effectiveness of the EEN services relate to the profile of the EEN member organisations and their embeddedness in their national and regional environments. There is a close relationship between the services provided in the context of COSME and these organisations’ ‘usual’ services. The credibility of the organisation in delivering quality services is a primary condition for the use of the EEN services by SMEs. Another enabling factor is the proximity of the members to the local SMEs, as well as the potential for knowledge sharing and information. However, SMEs are not always aware that the EU funds EEN services and SMEs awareness of the existence of the Network could be improved.

Despite the success of the Erasmus for Young Entrepreneurs, the entrepreneurship objective does not seem to have a strategic focus in COSME work programmes. Actions originate from different places and are then managed across multiple Commission services, with only informal and unstructured interaction across them.

Except for the financial instruments and the EEN, the fragmentation of the programme into many smaller activities for 20% of the programme reduces the potential impact of actions. This is particularly the case for EYE and for the Clusters.

·To what extent is the Programme contributing to the five union priorities for a sustainable and inclusive growth?

The COSME intervention logic indicates that contributions by the COSME programme to the higher-level EU policy priorities are expected in four areas: an increase in ‘Jobs and growth’, an enhanced ‘Business creation and growth’, a strengthened ‘Global competitiveness of the SMEs’, and an increase in ‘Inclusive and Sustainable Growth’.

In the graphs below we first set these data in the context of statistical data related to the SMEs’ employment shares at the sectoral level ( Figure 2 ) and the developments from this perspective in recent years ( Figure 3 ). Sectors with the highest shares of COSME beneficiaries are highlighted in dark red; those with high shares are highlighted in pink.

This analysis suggests that COSME has a high potential for contributing to the attainment of the Jobs and Growth objective in sectors that are the most important for the EU economy.

However, as explained in the relevance section, its potential to reach impact in relation to global and societal challenges is limited. The main COSME Instruments, such as the Enterprise Europe Network or the Loan Guarantee Facility target all SMEs without distinction of sector or activity. This universal offer means that the support to policy objectives such as Climate change, Gender Mainstreaming or Youth Unemployment can only be indirect and due to undesired effects.

Figure 2: SME employment shares at sector level

Source: Annual report on European SMEs 2015/2016; Eurostat, Statistical Offices & DIW Econ

Figure 3: Change in employment in SMEs

Source: Annual report on European SMEs 2015/2016; Eurostat, Statistical Offices & DIW Econ

5.3.Efficiency

·Are the costs generated by the programme proportionate to the benefits generated?

COSME is operating at a reasonable level of efficiency overall, with its principal components all being implemented in line with the specific objectives and timetables set out in the annual work programmes. The feedback from beneficiaries is generally positive with some exceptions. A substantial majority is reporting a good cost-benefit ratio for their individual participations. For example, more than two thirds of EEN respondents (68%, 158 respondents) reported that the benefits outweigh the time investment.

Surveyed beneficiaries report some issues of administrative burden. For example, half of the Cluster Go International (CGI) Participants responding to the survey thought the duration of the CGI project funding was not appropriate; about 40% of them thought the funding needed for the internationalisation activities was not correctly estimated. The financial intermediaries indicated that the reporting requirements are burdensome and risk reducing the attractiveness of the LGF. They also argued that the EFG’s administrative burden is substantially higher than is normal in the Venture Capital (VC) sector.

After three years of experience under the COSME programme, the IPR SME helpdesks has been showing effectiveness and should now focus the resources to gain in efficiency. To that effect, improved service quality is needed, as well as tackling the problems of insufficient promotion of activities and improving partnering services. There is no need for additional means in order to achieve more impact, however the use of the current ones could be optimised. The fact that intermediaries act as multipliers, can be seen as an efficiency lever, provided the visibility increases concurrently thanks to more promotion efforts. The strategy to reach the critical mass of SMEs needs to be improved. The evaluation concludes on the need for continued funding for the three regions and considers extending this type of service to India.

The two WATIFY campaigns cost EUR 1,6 million and allowed to reach some 200,000 participants through an initial and closing event, matchmaking events, awareness raising campaign, video clips and case studies. Participants to the matchmaking events insisted on their usefulness and even asked for more of these events. On the awareness raising events there could have been less for the same effect and usefulness. For the launching and closing events, the co-organisers mentioned that it generated some administrative burden. Surveyed SMEs agreed that their participation has resulted or will result in a number of benefits for their company: their increased competitiveness in the market, higher profit margin, higher quality of their products, more clients, new business partnerships, support in introducing technological changes and funding opportunities for technological changes. Overall beneficiaries found the campaigns useful and cost-effective. Several potential improvements were identified: The delayed launch of the WATIFY website prior to the launch of the second campaign created obstacles for the promotional activities; The timing in organizing the events caused some inefficiencies and unexploited potential; The significant Facebook presence in countries outside the EU Member States or COSME programme countries for the second phase of WATIFY shows that this communication channel could have been used in a more efficient way.

·What are the regulatory/administrative costs and the benefits for the different stakeholders?

COSME’s contribution to EASME for the period 2014-2016 was EUR 21,7m, which covered the operational (administrative) aspects of the programme. This is approximately EUR 7,2 million per year and represents 6.6% of the budget committed to COSME in the same period, excluding the budget for ‘Access to Finance’ managed by EIF. 47

For the EEN, approximately 600 Network member organisations are grouped in consortia organised at the national or regional level that have been selected through open calls. On average, approximately 5.5% of the COSME budget made available to the Network member organisations is dedicated to management/coordination activities.

The EYE European Partnerships (EPs) are consortia of Intermediary Organisations officially awarded by EASME, following an open call for proposals. Their staff costs are included in the grant provided by COSME

The European Investment Fund (EIF): The EIF acts as entrusted entity for the implementation of the COSME financial instruments through a mandate given by the Commission. For the period 2014-2016, the costs for the COSME implementation by the EIF amounted in a total of EUR 4.5 million for EFG and EUR 17 million for the LGF. These figures corresponded for EFG to 2.6% of the total EU budget committed and for LGF to 4.5%.

The supporting study 48 presented several Cost Effectiveness Analysis in terms of number of SMEs reached, the increase in turnover or the employment generated.

For example:

Each EUR 1 million invested in the EEN has led to an increase in employment of circa 377 people among client SMEs. This is based on an increase of two employees per company in the period 2014-2016 (calculated from our SME client survey), the total number of companies supported (210,000), and the additionality factor.

Each EUR 1 million invested in LGF has led to an increase in employment of circa 235 people among client SMEs. The evaluators further estimated – after accounting for the additionality – that for each EUR 1 million invested into the LGF (effects fully attributable to the Loan Guarantee Facility):

·additional employment of 491 will be created;

·additional EUR 22 million in turnover will occur in treated SMEs;

·470 SMEs will be supported.

Each EUR 1 million invested in EYE has led to an increase in employment of 33 people among SMEs (Hosts) and New Entrepreneurs. Again, this a very positive result; however, it implies that EEN and LGF are more cost-effective than EYE in terms of supporting job creation.

The Cluster Go International action reached 3,800 SMEs per EUR 1 million invested.

·How efficient is the implementation structure and governance of the programme?

The main strength of COSME is the clarity of its work programme descriptions, leading to a relatively small share of ineligible proposals responding to the calls for grants. No major administrative bottlenecks were identified and the simplification measures that were introduced were overall considered as positive, even though the new IT system clearly had some teething problems.

A major weakness of the programme design is the unbalanced distribution and strong fragmentation of the budget available. As stipulated by the COSME regulation, the large majority of the COSME budget (80%) is used for the two key actions. 49 The remaining 20% of the budget is spread over a large number of small actions. This fragmentation negatively influences the potential for cost-efficiency in the programme implementation and accentuates the limits in strategic steering and coordination of the programme. Competition in some COSME calls can be quite high, especially for the smaller actions, which can be considered a direct consequence of the funding fragmentation. This places an additional burden on applicants, which can be particularly problematic for SMEs and may lead to a narrowing of the applicant base, both geographically and sectorally.

Regarding the contribution of EASME to an efficient management of the COSME programme, two issues can be noted. First, the time-to-grant for the open calls was shorter than the target set by EASME. Second, the number of small actions delegated to the agencies requires the involvement of a proportionally higher number of staff than for bigger actions such as the EEN. In addition, surveyed beneficiaries indicated also the difficulties in using the EASME IT systems in the first years of the programme and the user-friendliness of the IT tools.

Another major weakness of the programme is the quality of data management. The lack in quality and especially completeness of the data on the beneficiaries of the programme constitutes a major hurdle for an efficient management of the programme implementation. The spread of the implementation responsibilities over the many actors involved (the EIF, EASME and in 2014-16, about 15 DG GROW units) creates a considerable challenge to efficient programme management and coordination, with a risk of weakened programme oversight. It also makes it very much harder to evaluate the programme. The various reporting systems’ focus on activities and outputs is equally problematic. A stronger attention for the collection of data on the profile of the beneficiaries is needed to allow for a proper monitoring of the programme outcomes and progress towards reaching its objectives.

5.4.Coherence

·Are the different actions of COSME implemented coherently between them?

Overall, the COSME programme shows a good level of internal coherence: efforts are increasingly being made to create synergies and no substantial overlaps could be identified. The EEN services are the most-often mentioned targets for synergies. The risk is to overcharge the EEN with responsibilities to which the Network cannot respond in an efficient way, and with the risk of losing the focus on its core activities. More synergies could be created by taking a more integrated, cross-thematic approach.

The communication activities related to the financial instruments created a certain abundance of information; those related to the entrepreneurship actions appeared to be run in isolation from each other. The assessment is positive in relation to the synergies and complementarities created between the Your Europe Business portal and the EEN services, and between other COSME activities geared towards internationalisation such as the EEN and Clusters Go International programme.

There is a positive coherence in relation to the complementarities and synergies created between Clusters Go International with other COSME activities geared towards internationalisation and the emerging markets of the main EU trade partners are covered through the action 50 .

A better cooperation between the IPR SME helpdesks and the Cluster Internationalisation actions should be a priority for the future. The interaction between the three helpdesks should be maintained, as well as with other COSME initiatives (mainly the EEN) and other EU and national initiatives.

·How are the different COSME actions overlapping/synergizing with the activities of ESI funds, H2020, EFSI, EASI and other EU programmes?

The division of tasks between COSME and H2020 is clear and we noted no significant overlaps between COSME and H2020 or other EU activities. Nevertheless, there is room for the creation of more synergy and complementarities, especially for the SME internationalisation activities. Overlaps between the COSME LGF and financial instruments established under ESIF may occur when addressing similar SMEs in the regions. The latter was also highlighted in the independent Evaluation 51 of the EFSI Regulation, which was concluded on 4 June 2018, namely that there is a risk of competition between ESIF financial instruments and LGF, and even indirectly EFSI.

The more than 625 members of the EEN help about 250.000 SMEs per year to increase their competitiveness and innovativeness in the Single Market and beyond by offering integrated services. There is a clear market failure for SMEs regarding information, funding, accessing new markets and networking to internationalize their business outside the home country. Network organisations provide information on funding opportunities made available by the various EU programmes such as COSME financial instruments, Horizon2020 SME Instrument and Structural Funds.

The Cluster Go international can also play a major role to incentivise European SMEs across regional industrial ecosystems to enter into joint business projects with partners in third countries 52 for other related actions, such as the EU’s Low-Carbon Business Actions in Brazil and Mexico, supported through the EU Foreign Partnership Instrument 53 .

·To what extent has the coherence been maximised in view of national programmes?

The interplay between the EU, national and regional levels of support for SMEs is not always synergistic and complementary, even though no substantial overlaps were identified. Especially for actions where proximity to the SME communities is a strong enabling factor, better coordination with national and regional actions would further improve coherence. This is especially the case for the EEN services and the COSME actions supporting cluster organisations.

Both EFG and LGF have adopted a bottom up approach, have the flexibility to combine with national programmes and appear to be rather complementary with other schemes. However, Surveyed LGF intermediaries see both complementarities (9 out of 16 intermediaries) and overlaps (10 out of 16) of the LGF instrument with national level actions.

Coherence with initiatives at the national/regional levels is a topic of relevance especially for the EEN services. The Network builds on regional players that are embedded in the local environment. The survey with SME clients of the Network shows that most clients were unsure whether the Network services are complementary to, or overlap with, existing national or regional support in their own country; the findings from the survey with Network members showed similar results. During the interviews, the Network intermediaries explained that they actively seek this complementarity. For example, when prioritising their activities, they aim to focus only on those services that are not yet available in their region. Moreover, many intermediaries explained that they have a good collaboration with other stakeholders that provide services to SMEs in the region; they organise joint events or activities, and actively seek to coordinate their actions. These national and regional stakeholders include ministries, municipalities, chambers of commerce, EU info centres, specialists in IPR, lawyers, etc. By means of signposting, the Network intermediaries refer SMEs to other stakeholders in the field and vice versa when relevant.

In conclusion, while just over half of the Network clients and members feel that the Network services are complementary to services provided at national or regional level, there is scope for improvement.

In the context of the Cluster internationalisation programme, interviewees indicated that there is no clear coordination between EU-level and national level cluster policies, which is perceived as problematic especially in the context of the limited funding available for the cluster internationalisation actions in COSME. The interviewees considered that the limited COSME funding for CGI might be considered as an incentive for clusters to co-invest also their own resources to achieve the desired results. They emphasised that this approach does not sufficiently take account of the diverging levels of funding for clusters in the EU Member States. According to these interviewees, this approach hinders a level playing field for cluster organisations in countries without direct cluster policies. They therefore emphasised the need for an improved coherence and creation of complementarity in the EU funding for cluster internationalisation with the availability of national level funding. In this context, it is to be noted that only 3% of the CGI participating clusters are located in EU13 countries.

The complementarity of WATIFY with similar national initiatives emerges from its European character. Besides these initiatives of a more cross-cutting nature, there are ones focused on specific topics / sectors. Some countries that used WATIFY materials agreed that while there are similar initiatives to WATIFY, it is important to have a European initiative encouraging enterprises to digitalise. Internally, the WATIFY campaign was overall coherent (in terms of there not being conflicting events or activities organised). There is a complementary aspect between awareness raising and matchmaking events. The latter were organised under WATIFY umbrella, and often included the cooperation with local or regional stakeholders. The former, in contrast, were organised by various actors in different European countries, and were used by WATIFY as platforms to introduce and promote the campaign, its objectives and messages.

5.5.EU added value

·What is the added-value of the programme and actions being conducted at the EU level?

COSME has a good level of European added-value. The European dimension constitutes the very essence of the design of its actions and is crucial for their implementation and effectiveness. In most cases, the ‘subsidiarity’ of these actions, i.e. the benefit of their implementation at the European rather than national or regional levels, is therefore very high. While many national and regional initiatives also seek to strengthen the competitiveness of SMEs, the scale of support through COSME and its availability to financial and business support intermediaries in every EU Member State are quite distinct and highly additional. In several cases, the EU level actions have not only been additional to, but have also helped in enhancing national, regional and local level measures.

Indeed, COSME provides important financial leverage that allows national and regional intermediaries to provide lending products, which have a higher risk profile compared to their normal product offering, and it allows offering support to a substantially bigger number of businesses than they might otherwise be able to service. Furthermore, national support programmes, especially in smaller markets, have higher leverage when combined with an EU-level programme. However, the added-value of the LGF suffers from the non-targeted approach, which so far has implied that the support did not sufficiently reach the less mature financial markets. In terms of EFG financing, intermediaries very clearly stated that EU action is additional, despite that as recognised by the independent Evaluation 54 of the EFSI Regulation, which was concluded on 4 June 2018, there were initial overlaps with the Expansion and Growth Window under the EFSI Equity instrument. Despite that the two instruments largely have the same investment focus, operations have been allocated to EFSI since it has greater resources and can offer more favourable conditions.

Moreover, the analysis from the supporting study reveals that the financing – supported by the LGF for example – also has triggered an additional investment from 21 % to 29 % as shown in the figure below.

Source: COSME Interim Evaluation (Technopolis, based on survey data)

Further, by taking into account the actual value of the principal financing received by those SMEs, the evaluators have estimated that LGF has de facto triggered approximately EUR 1,25 billion as additional investments as shown in the figure below.

Source: COSME Interim Evaluation (Technopolis, based on survey data)

The distribution in the two figures above is based on the SME definition. From the graphical presentation in both figures, it is evident that whereas the medium or large enterprises trigger higher additional investment percentage wise, i.e. 29 %, the smaller, but more numerous, the enterprises are, the greater value of additional investments will flow in the EU economy, i.e. EUR 959 million.

The supporting study also has identified via a dedicated survey that 65 % of the respondent SMEs are of the opinion that they would have borne negative consequences for their business plans in absence of LGF support.

Source: COSME Interim Evaluation (Technopolis, based on survey data)

As shown in the figure above, these negative consequences for 65 % of the SMEs mean de facto that 44 % would have continued their business project on a smaller scale and 21 % would not have been able to implement their business plans at all.

The European dimension is at the core of the Enterprise Europe Network. It allows for the attainment of a scale and quality of the services, and in some cases, even the implementation of the service as such, that would otherwise not be possible. The EEN’s role as an intermediary, involving Chambers of Commerce and national/regional innovation agencies, provides the advantage of a gateway for reaching SMEs; its EU added-value could be strengthened by focusing on those services that build upon the transnational characteristics of the Network itself, i.e. the internationalisation and single market penetration. Options to ensure an appropriate level of co-financing and/or funding from EU funds could be examined.

The IPR SME helpdesks add value at EU level by developing and broadening their expertise to all EU SMEs. In order that any SME in any EU MS have access to an equivalent quality of IPR service, it is of specific value to be able to offer them an IPR one-stop-shop to accompany them in their globalisation projects.

The Cluster Go International program (with its fostering of European partnerships among clusters in order jointly to develop and implement internationalisation strategies) shows very high EU added value, where the scale and depth of support goes well beyond the support that can be offered to European SMEs through national or regional initiatives 55 .

A separate evaluation of cluster initiatives from 2014 56 concluded for instance that the European cluster initiatives "generated added value through structuring cluster cooperation contributing to the emergence of European meta-clusters" and have reached out to a wide range of clusters, while flagging out that they lacked the financial means to involve SME sufficiently.

·To what extent and why do the issues addressed by the programme continue to require action at EU level?

The enhanced added-value for the Union of the proposed financial instruments lies in addressing market failures that cannot be addressed by Member States. 57 Funding at the supranational level in Europe allows the continent to match levels of funding seen in the United States. A discontinuation of the instruments would set at risk the current positive developments.

We have seen in the analysis of relevance that the market gap on access to finance for SMEs is still an issue despite the progress made. According to LGF intermediaries, if LGF support were no longer available, SMEs would face more difficulties in getting access to finance. It could be expected that fewer SMEs would get a loan or that interest rates and collateral requirements would rise. The effect of a hypothetical termination of the LGF support on financial intermediaries would be a reduction of activities, both in terms of a lower volume of (counter) guarantees provided and in terms of a reduced number of products offered to higher risk SMEs.

The EFG intermediaries considered action at the European level to be crucial for the European equity market to gain maturity. Equity markets in Europe are still relatively weak and only a few larger Member States can sustain their own equity markets.

The EEN member organisations stated in the survey that the quality, the coverage and the variety of services in their own country or region would decrease or stop if the Commission would lower funding. Full regional coverage and the principle of reciprocity are essential for the effectiveness of the Network. Therefore, if even only a small number of regions would not able to continue their support offer, it would have negative impacts on the service delivery in other parts of the Network as well.

On Entrepreneurship, there is still substantial room for further improvement, with data persistently showing relatively low levels of entrepreneurial interest and activity, as well as high rates of failure and slow rates of growth amongst newly established firms.

Also in the case of the other COSME action lines, the ‘benefit of their implementation at the European rather than national or regional levels, is very high. This is especially the case for those actions where the European scale is of critical importance for the attainment of positive effects also at the national and regional levels. Examples of such actions are those related to eSkills, the Tourism actions bringing together actors in European value chains, the Clusters Go International programme fostering European partnerships among clusters in order jointly to develop and implement internationalisation strategies. Multiple barriers still exist preventing SMEs from undergoing digital transformation. Considerable differences between the MS are still present resulting regions lagging behind in terms of digital transformation. The interviewed stakeholders agreed that a campaign such as WATIFY is needed and met market needs. Last, but not least, the SME policy actions providing strategic information as well as network opportunities for national policy-makers, fostering improved SME policy-making throughout Europe as well as an enhanced convergence. The scale and depth of support that COSME offers goes well beyond the support that can be offered to European SMEs through national or regional initiatives.

6.Conclusions and Lessons Learned

·What is working/what is not working, how it links to the intervention?

COSME actions are highly relevant in addressing the challenges related to fostering economic growth and creating employment opportunities and is strongly aligned to the evolving needs of SMEs. COSME supports competitiveness and growth of all SMEs. The strength of the programme lies in its focus on enhancing the SMEs’ capacities for growth and in its attention to the SMEs’ needs for concrete information and knowledge that are of immediate and practical use in their business practices. The programme also addresses in a direct and concrete manner the need of young entrepreneurs to strengthen their entrepreneurship skills.

However, this approach limits the possibility of specifically addressing other issues. Its two major actions, LGF and EEN, accounting for about 80% of the total budget committed, target all SMEs without making distinctions to reflect the programme’s ‘inclusive and sustainable growth’ and ‘global competitiveness’ policy objectives. The two major actions can therefore create only ‘unintended’ effects in those areas. As a result, the relevance of COSME for policy objectives such as gender mainstreaming or climate change is limited.

The strength of the programme lies in particular in the use of intermediaries who have a direct and longstanding contact with SMEs for the implementation of the programme. COSME exploits the proximity of these intermediaries to SMEs and facilitates the integration of services provided by COSME with services provided by these intermediaries in their national and regional contexts. This allows for the provision of customised SME support tailored to the needs and challenges of specific sectors, such as tourism, textiles, creative industries etc. and to reach a high multiplier effect of actions. COSME has thus a good level of European added-value. The European dimension constitutes the very essence of the design of COSME actions. Moreover, COSME actions have helped to enhance national, regional and local level measures for SME support and the teaming up of actors across EU value chains to boost internationalisation and industrial modernisation. The feedback from beneficiaries is generally positive with a substantial majority reporting a good cost-benefit ratio for their participation.

COSME is complementary to other EU policy instruments and programmes supporting business, without significant overlaps. It is the only EU programme designed to support SMEs' competitiveness and growth. Its ‘non-targeted’ offering addresses the basic needs of enterprises, facilitating access to finance, fostering cooperation and providing access to advice. It responds to existing needs, because a financing gap continues to exist in Europe, depending on the national context, and information asymmetries, specifically those related to market opportunities in the European Single Market and the global market, continue to require policy intervention for facilitating knowledge transfer and exchange as well as cross-border co-operation;

The supporting study highlights several aspects to be improved:

First, as stipulated by the COSME regulation, some 80% of the budget is used for promoting SMEs' access to finance and access to markets. The remaining 20% is allocated between a number of smaller actions with a limited budget and with a low cost-benefit ratio regarding the implementation. EASME (Executive Agency for SMEs) implements 35% of the overall programme budget, with two flagship COSME actions (Enterprise Europe Network and Erasmus for Young Entrepreneurs) as well as a host of smaller projects. The fragmentation of actions and limitations in terms of budget (as flagged out for the EYE and cluster measures) hampers the effective implementation 58 .

Second, while COSME is successful in fostering economic growth and the creation of employment opportunities, there is scope for strengthening its responsiveness to EU objectives related to EU global competitiveness and sustainable and inclusive growth. COSME's main instruments are designed to address all SMEs, independently of their sector and industrial specialisations. They only indirectly address challenges such as climate change, gender mainstreaming or youth unemployment.

Third, while there are no major overlaps with other policy initiatives, there is scope for creating further synergies with other EU, national and regional actions.

Fourth, in relation to monitoring the evaluation points out to a lack of centralised data about implementation and indicators are mostly based on outputs rather than on long-term effects. Some efforts are still needed to centralise the data that at present is dispersed between the coordinating of the programme, the units managing individual actions and the delegated entities.

·Lessons learned:

While acknowledging – based on the available evidence – the strength of the COSME financial instruments to deliver a financial support to the SMEs that most needed it, several areas for improvement are identified:

·Better links between financial instruments and other parts of the future SME programme;

·Better co-ordination between financial instruments and national EU schemes;

·Discontinuation of the EUR 150 000 threshold which has a negative impact on the efficient delivery of the guarantee facility and created significant administrative burden.

A future EU loan guarantee facility should help to ensure a more level playing field for SMEs in those countries where, according to current studies, the needs among SMEs are highest. The existing Equity Facility for Growth is considered to be effective. Nevertheless, it appears necessary to reduce the number of financial instruments and to align the Equity Facility for Growth with the equity facilities established under EFSI.

The limitation of budget beyond the financial instruments and the EEN creates a high competition between the remaining actions. It represents 20% of the budget for a high number of actions. A fragmented budget reduces the potential impact of these actions and it creates a high competition in the published calls.

There is still scope for simplification. Stakeholders report a good cost-benefit ratio for their participation in the programme but report that IT systems could be more user-friendly and there is administrative burden for participating in the programme actions.

At the level of coherence, while actions are rather synergetic with other EU programmes or complement well national and regional instruments for SME support, there are still some overlaps.

Finally, the lack of a centralised database for the implementation of the programme reduces the effectivity of monitoring and evaluation as well as the visibility of the programme.

·Does actual performance meet expectations?

COSME in the period 2014-2016 produced positive results. The programme implementation is on track to reach the objectives set out in the legal basis by the end of 2020.

The COSME LGF has been successfully set-up and is fully on track to delivering on the targets set in terms of SMEs to be financed and volume of financing to be made available. It confirms that the "COSME actions fully respond to the SMEs’ current needs to access finance".

The EYE programme has been established as pilot project in 2008 under CIP and since 2014 incorporated under COSME. It has been particularly effective in reaching its knowledge- and skills-related objectives and has laid the ground for important outcomes in terms of business relationships and internationalization for the participating entrepreneurs. However, the scale of the activities (in total about 2,000 new entrepreneurs) poses limits to the importance of this action line within the overall programme portfolio.

The limited evidence available suggests that COSME will have a positive impact on its beneficiaries’ growth and employment. The EIF has an essential role in the implementation of the financial instruments and EASME plays an important and much appreciated role in creating learning opportunities for the EEN members to enhance their capacities and better serve the SMEs in their local environments.

The EEN is very relevant to the overall need of SMEs in building trust in finding suitable partners. However, it does not address the barrier of high costs that come with building trust in finding suitable partners. As a result, a potential upgrade of the programme could therefore include other types of support, such as demonstration projects, feasibility studies and knowledge-acquisition activities.

COSME meets the expectations in terms of relevance, efficiency, effectiveness and coherence. Its actions show a high level of EU added-value. However, some efforts are still needed to improve efficiency, effectiveness and coherence. Some issues already pointed out by the CIP and EIP evaluations have not been solved during the 2014-2016 period: the fragmentation of the programme into smaller actions and the data management.

·What issues need to be addressed or will it resolve over time?

COSME covers a broad range of actions to implement the objectives set out in the COSME regulation with a limited budget. A clearer definition of the COSME priority areas should enable the programme to reach more efficiency and effectiveness. Several possibilities could be envisaged:

1) Deploying its existing resources in a more focused / strategic manner;

2) Entering new strategic partnerships with other EU programmes by strategically sharing the responsibility for interventions in specific areas; and

3) Increasing its overall budget in order to allow it to run both the non-targeted approach and strategic programmes in parallel.

Before 2020, an increase in the overall budget is not a valid option. The MFF review did not retain this possibility, except for the financial instruments.

A more strategic prioritisation of objectives in the design of the work-programmes could be implemented for the remaining years until the end of the programme in 2020. This could help reduce the fragmentation of the budget. A more centralised approach to data management would ensure higher efficiency in the programme management and a strengthened capacity for a quality implementation of the monitoring and evaluation function. It would also pave the way for the creation of the monitoring system for the post-2020 period.

For the design of the post-2020 period successor to COSME, an ambitious scenario could be proposed in the impact assessment of the successor programme to strengthen the overall effectiveness of the programme. However, considering the potential reduction of the overall EU budget due to the BREXIT and the new challenges the EU has to tackle, (such as Defence and Security), the most realistic option seems to be to better exploit the available budget, and focus on the most successful actions and reserve a budget for new ideas and pilots. Synergies with other programmes could be also exploited in order to really focus on the actions where COSME is the most effective, and to support enterprises in all stages of their life-cycle, from their creation to their growth.

At this stage, substantial changes in the financial instruments or the Enterprise Europe network will be counter-productive to implement. Considering the sophisticated implementation chain and the respective concomitant amendments procedures needed, it is reasonable to expect that any tangible impact would only produce effect after 2020. The potential burden imposed on intermediaries and SMEs following any changes should not be underestimated either. It is more realistic to accept that the conclusions of the evaluation will rather feed the design of the post-2020 instruments.

However, in the impact assessment for the next multi-annual financial framework, the Commission proposes:

·To improve the internal coherence of different parts of the programme, better linkages will be established by further strengthening the existing cooperation with the Enterprise Europe Network 59 (use the Enterprise Europe Network as a stronger signposting element);

·A better upstream co-ordination between financial instruments for SMEs established by Member

· States and those established under the SME window of the InvestEU Fund will be sought by using the existing SME Envoys Network 60 as an information exchange forum;

·To reduce administrative burden for SMEs and financial intermediaries and to improve the impact of the SME guarantee facility, the EUR 150 000 threshold will be eliminated.

This interim evaluation will pave the way for the final evaluation of COSME that will also feed the detailed design of the first work programmes of the period post 2020. The final evaluation should analyse more in detail the wide variety of COSME actions to provide the Commission with data that will support the strategy to select the actions that will compose the future COSME in the next multi-annual financial framework. Preliminary findings of the interim evaluation have informed the impact assessments on Single Market and Invest EU programmes for the next Multi-annual Financial Framework 2021-2027.

Annex 1: Procedural information

1.Lead DG, Decide Planning/CWP references

lead DG: GROW

Decide Planning: PLAN_2016_223

CWP:

2.Organisation and timing

An inter-service steering group was created in Q4 2016. The steering group was constituted of the following DGs:

·BUDG

·SG (A.2 (ex C.1))

·ENV

·EMPL

·REGIO

·RTD

·TRADE

·ECFIN

·ENER

·EAC

In total 7 meetings of the Interservice steering group (ISSG) have been organised between Q4 2016 and December 2017.

After having been consulted with the ISSG and approved by the secretariat general the roadmap was published the 16/01/2017. No feedback was received from stakeholders.

The consultation strategy and the Terms of reference for the supporting study were submitted to the ISSG during Q4 2016. The supporting study was concluded by using a framework contract with Technopolis Group. The Specific contract was signed on 19/12/2016 and the contract started in January 2017.

The inception meeting took place on 27/02/17 and subsequently the inception report was submitted to the ISSG. The inception report was approved on 01/06/2017.

The interim report was submitted on 24/05/2017 and served to prepare the draft final report that was received on 04/10/2017. It was presented to the ISSG (including presentation by the contractor) at the 7th meeting of 13/10/17. The comments from DG GROW and other members of the group have been transmitted to the contractor to prepare the final report. From the ISSG, GROW, ECFIN, REGIO and SG requested changes.

The Final invoice was received the 23/10/17. Contract report approval was suspended 20/10/17 in order to consider ISSG comments. The Final Report was received 20/11/2017. It was sent to the members of the ISSG with deadline of 27/11/17 for comments or approval.

Although comments and corrections had been taken into account by the contractor, there were still corrections and modifications made and requested by the ISSG prior to finalising the document. The revised final report was received the 08/12/17. On 12-13/12/17 the approval of SG, ECFIN, TRADE and REGIO was received, no other member of the ISSG has made any objection so the final report was considered as approved by the ISSG and the final report was sent to all members of the group.

The final versions of all deliverables received from the contractor including IPR annexes from FWC are registered in ARES (ARES(2017)6154624 of 14/12/17).

3.Exceptions to the better regulation guidelines

none

4.Consultation of the RSB (if applicable)

not applicable

5.Evidence, sources and quality

The major difficulty for this evaluation has been the lack in data to measure results and impact. By the time of evaluation, only major and long-standing actions produced sufficient data to allow for measuring results. The evaluation measured on basis of outcomes and outputs produced the probability for the programme's actions to reach the desired effects. The main evidence for this evaluation is the supporting study performed by Technopolis Group. The quality of the supporting study has been positively assessed by the ISSG and other limitations are indicated in the synopsis and methodological report.

The supporting study took into account the below evidence, together with evidence from the evaluation of the IPR SME helpdesks. The complete list of sources is indicated in the methodological report.

·Regulation (EU) 1287/2013 of the European Parliament and of the Council of 11 December 2013 establishing a Programme for the Competitiveness of Enterprises and small and medium-sized enterprises (COSME) (2014 - 2020) and repealing Decision No 1639/2006/EC. 61

·Decision 1639/2006/EC of the European Parliament and of the Council of 24 October 2006 establishing a Competitiveness and Innovation Framework Programme (2007-2013) 62

·Impact Assessment for the proposal for a COSME regulation COM (2011)834 63

·MFF review 64

·Ex-ante Preparatory documents for COSME, including public consultation 65

·CIP Impacts and Results, including evaluations 66

·EACI evaluations 67

·Final evaluation of the Enterprise Europe Network 2008 – 2014 68

·SME Performance Review and Small Business Act Factsheets 69

·The Small Business Act for Europe, including the Report on the 2014 public consultation 70

·Study commissioned by the Committee of the Regions: Implementation of the Small Business Act for Europe (SBA) and Entrepreneurship Policies at Local and Regional Level 71

·The Single Market Strategy 72

·Public consultation under the Start-up Initiative 73

·The Single Market Forum 74

·SAFE (Survey on Access to Finance) 75

·Flash Eurobarometer 354, ENTREPRENEURSHIP IN THE EU AND

BEYOND, 08/2012 76

·Flash Eurobarometer on the internationalization of SMEs 77 :

COSME web page 78

·EASME web page 79

·EEN web page 80

·EYE (Erasmus for Young Entrepreneurs) web page 81

·EIF, Loan Guarantee Facility (including implementation status) 82

·EIF, Equity Facility for Growth 83 :

·Statistics, data and studies on Tourism 84

·Data on regional funds:

-The ex post evaluations of the 2007-2013 ERDF and Cohesion Fund programmes: http://ec.europa.eu/regional_policy/en/policy/evaluations/ec/2007-2013/#1

-ESF and its support to entrepreneurship and micro-finance ( http://ec.europa.eu/esf/main.jsp?catId=3 )

-For the EAFRD: http://ec.europa.eu/agriculture/evaluation/rural-development-reports/index_en.htm

-For the EMFF: http://ec.europa.eu/fisheries/cfp/eff/index_en.htm and http://ec.europa.eu/fisheries/cfp/emff/index_en.htm

-as regards the 20014-2020 programmes:

oThe expected outcomes, planned funding, etc. for all 5 ESI Funds: https://cohesiondata.ec.europa.eu/ (see for instance the search by SME competitiveness: https://cohesiondata.ec.europa.eu/themes/3 ).
http://ec.europa.eu/regional_policy/en/policy/evaluations/data-for-research/

oERDF financial instruments: http://ec.europa.eu/regional_policy/en/funding/financial-instruments/

Annex 2: Stakeholder consultation

Three types of consultations were conducted: the open public consultation, the ‘quantitative’ targeted consultations, and the ‘qualitative’ targeted consultations. In addition, 121 interviews were conducted and carried out seven quantitative consultations, seven qualitative consultations and one public consultation.

The European Commission implemented the questionnaire for the public consultation on the “Your Voice in Europe” website ( https://ec.europa.eu/info/consultations/interim-evaluation-programme-competitiveness-enterprises-and-small-and-medium-sized-enterprises-cosme-2014-2020_en ) on 10 May 2017 until the 31 of August 2017.

Both intermediaries and beneficiary SMEs were approached by targeted consultations. In total, 4,100 responses were received. The response rates among intermediaries was about 45%; it was between 10% and 20% among SMEs. The public consultation received 195 responses and 14 position papers.

For WATIFY, The stakeholder consultation strategy included three main activities:

1.An on-line target survey to the main beneficiaries of the awareness raising campaign

2.A general on-line survey targeting SMEs covering 16 Member States conducted through online panels

3.45 interviews covering all types of direct stakeholders

Synopsis Report

1 Overview of the consultation strategy

2 Stakeholder groups involved in the consultations

2.1 Stakeholders interviewed

2.2 Stakeholders involved in the qualitative targeted consultations

2.2.1 Results for SO1 – Access to finance

2.2.2 Results for SO3 – Favourable environment

2.3 Stakeholders involved in the quantitative targeted consultations

2.4 The public consultation

2.4.1 Overview

2.4.2 Number and profile of the participants

3 Robustness of the consultation results

3.1 The quantitative targeted consultations

3.2 The qualitative targeted consultations

3.3 The public consultation

4 Summary of the results

4.1 Relevance

4.2 Effectiveness

4.3 Efficiency

4.4 Coherence

4.5 European added value

Overview of the consultation strategy

Table 2: Tools and methods used in the evaluation

1.Consultation approach	2.Tools	3.Number
4.Qualitative consultations	5.Interviews	6.121
	7.Qualitative targeted consultation	8.8
9.Quantitative consultations	10.Targeted consultation	11.7
	12.Public consultation	13.1

1.Source: Technopolis

A major purpose of the high-level interviews was to collect background and context information on facilitators and barriers for the attainment of the expected effects as well as the eventual unintended effects. Wherever possible, the interviews were conducted face-to-face. Interviews with EU officials took place in the beginning of the study to reach an improved understanding of the overall context to the programme and the processes for the programme design.

The ‘classical’ targeted consultations were implemented in cases where the population was higher than 100 and would therefore allow for meaningful and robust statistical data analyses. The questionnaires mostly included close-ended questions and only a few open-ended ones. The responses were analysed by means of descriptive statistics, in several cases also including cross-analyses. We foresaw seven quantitative surveys in the Inception Report; they were all implemented.

The public consultation was a 12-week internet-based questionnaire. The objective was to cover all potential relevant stakeholders, in particular those that were not targeted through the targeted consultations and interviews, and to collect feedback on the COSME programme and its objectives. The questionnaire for the public consultation aimed at reaching all types of stakeholders. Given that the targeted consultations focused exclusively on the specific objectives, it was agreed to use the public consultation to collect data that would provide information on the programme as a whole.

Table 3 shows the evaluation questions covered by the different consultation methods. The consultation questionnaires allowed for a proper triangulation of the evidence collected and helped mitigating the risk of lack in data availability. For each evaluation question, the analysis was based on multiple sources and methods

Table 3: Use of the consultations per evaluation question

14.Evaluation question	15.Interviews & qualitative targeted consultations	16.Quantitative targeted consultation	17.Public consultation
18.Q1.1) Programme implementation and evolution since 2014	19.X
20.Q2.1) Ongoing relevance of the objectives	21.X	22.X	23.X
24.Q2.2) Relevance to the stakeholders and the EU citizen	25.X	26.X	27.X
28.Q2.3) Adaptation of the objectives to the political priorities	29.X
30.Q3.1) Progress towards achieving the specific objectives	31.X	32.X
33.Q3.2) Factors driving or hindering progress	34.X	35.X
36.Q3.3) Progress towards contributing to the 5 Union priorities	37.X		38.X
39.Q4.1) Regulatory/administrative costs and benefits for the different stakeholders	40.X	41.X
42.Q4.2) Costs (direct and indirect) proportionate to the benefits	43.X	44.X
45.Q4.3) Efficiency of the programme implementation structure and governance	46.X	47.X
48.Q5.1) Internal coherence	49.X	50.X	51.
52.Q5.2) Coherence with ESI funds, H2020, EFSI, EASI and other EU programmes	53.X	54.X	55.X
56.Q5.3) Coherence with national programmes	57.X	58.X	59.X
60.Q6.1) Added value of implementation at the EU level	61.X	62.X	63.X
64.Q6.2) Ongoing need for action at EU level and why	65.X	66.X	67.X

2.Source: Technopolis

The time plan of the consultation strategy was that the targeted consultations would take place in the month of April, ending in the beginning of May. The statistical analyses of the survey data were scheduled for the month of May, in time for the evaluation team to start drafting the internal reports at the thematic area level in the month of June. This strict time schedule was dictated by the need of the evaluation team to run the triangulation process, eventually conduct additional interviews, and subsequently, various runs of the aggregation of the evaluation findings (at thematic area and programme level), in time for the delivery of the draft final report in the beginning of September.

Table 4 presents an overview of the public consultation process, from start to finish.

Table 4 Overview the open public consultation process

68.Activity	69.Timing
70.Phone call with the EC	71.10 February 2017
72.Delivery of first draft questionnaire	73.24 March 2017
74.Review from the EC	75.29 March 2017
76.Validation of final questionnaire	77.26 April 2017
78.Editing in EN by EC	79.26 April – 5 May 2017
80.Translation in other EU languages	81.6 May – 12 June
82.Deadline of the consultation	83.Launch date: 10 May 84.Closing date: 31 August
85.EC to send inputs to Technopolis Group	86.Latest results received mid-September (13/09/2017)

3.Source: Technopolis

Stakeholder groups involved in the consultations

Stakeholders interviewed

Overall, the contractors' team conducted 121 interviews. These were distributed over the SOs as shown in Table 5 , below.

Table 5: Stakeholders interviewed

87. SO	88. Interviewed stakeholders	89. Nr
90.SO1 – Access to finance	91.Representatives from EU institutions (DG GROW, DG REGIO, DG RTD, DG ECFIN, EIF and EIB)	92.15
	93.intermediary and industry organisations	94.6
95.SO2 – Access to market	96.EU officials who were responsible for the Network and for other programmes aimed at fostering coherence within the Network.	97.9
	98.Network members	99.19
	100.Umbrella organisations were also interviewed so as gain the views of SMEs and intermediaries at a higher aggregated level.	101.5
102.SO3 – Favourable environment	103.EU officials in DG GROW, EASME and other DGs	104.16
	105.Beneficiaries (cluster managers)	106.6
	107.National government representatives	108.4
	109.Umbrella organisations and national experts	110.10
111.SO4 – Entrepreneurship	112.Individuals responsible for the design, management and oversight of COSME entrepreneurship actions – including European Commission officials, EASME staff and individuals within the EYE support office	113.20

	114.Host Entrepreneurs – participants in and beneficiaries of the EYE programme	115.5
	116.New Entrepreneurs - participants in and beneficiaries of the EYE programme	117.6

4.Source: Technopolis

Stakeholders involved in the qualitative targeted consultations

Table 6 Overview of the participation in the qualitative surveys

118.SO	119.Actors & stakeholders targeted	120.Target sample	121.Nr of responses	122.Response rate
123.SO1 – Access to finance	124.LGF intermediaries	125.21	126.16	127.76%
	128.EFG intermediaries	129.5	130.3	131.60%
	132.EFG SMEs	133.12	134.4*	135.33%
	136.Members of European Intermediary Organisations & LGF sub-intermediaries	137.6	138.5	139.83%
140.SO3 – Favourable environment	141.SME Policy: SME Envoys	142.28	143.15	144.54%
	145.SME Policy: SPR Working Group members	146.127 (according to the email list received from EC)	147.23	148.18%
	149.Sectoral competitiveness: European stakeholder organisations, national stakeholder organisations (associations), national and regional public authorities	150.72	151.7	152.10%
	153.E-skills: European stakeholder organisations, national stakeholder organisations dealing with e-skills	154.31**	155.8	156.26%

5.Source: Technopolis - * conducted through a cascade approach **out of the 53 e-mail addresses received, only 37 were valid. Out of them 6 contacts were contacted for interviews and 31 for the qualitative survey.

Results for SO1 – Access to finance

Among the 61 financial intermediaries receiving LGF funds, 20 were approached. Four qualitative surveys were conducted:

1A written questionnaire was sent via e-mail to 21 LGF intermediaries, asking the recipients to complete the questionnaire within two weeks. Five of them were the financial intermediaries (cat 1 promotional banks and cat 2 guarantee societies) and others were selected according to their type (Promotional Institution, Guarantee Institution, Commercial Bank, Leasing Company), country of establishment, direct/counter Guarantees and risk category in order to have a wider and representative possible coverage. Some of them where involved also in the CIP SMEG and overall 50% were currently using also InnoFin SMEG; they gave us more insights on the process and leverage effect.

Two reminders were sent to LGF intermediaries that did not respond by the fixed deadline. Finally, the team received 16 completed questionnaires, corresponding to a response rate of 76%. One LGF intermediary explicitly refused to participate, while four intermediaries did not answer. Eight responding intermediaries provided direct guarantees for loans and another eight intermediaries provided counter guarantees.

2A written survey of EFG intermediaries was conducted. The team originally planned to collect data from five EFG intermediaries. The responding EFG intermediaries make equity and equity-related investments in growth-oriented start-ups and SMEs all over Europe. Only three out of five contacted intermediaries completed the written questionnaire.

3The EFG intermediaries were asked to cascade the survey to selected SMEs 85 that received equity funding from the EFG instrument. It was planned that 12 SMEs were to be contacted by their respective intermediaries. Due to the limited response from the intermediaries, only a limited number of SMEs that were beneficiaries of the EFG (EFG SMEs) could be reached. As a result, the team received only four completed questionnaires (out of 12) from EFG SMEs.

4Five of the six contacted European Intermediary Organisations were asked by the team to cascade the survey to their members. Except for one intermediary, the European intermediary organisations contacted their members prior to the interviews and collected information on their members’ opinions and views. In two cases, the interviews were even conducted in the form of phone conferences in which representatives of both the umbrella organisation and individual members of the intermediary participated. In this way, the opinions and views expressed during the interviews directly reflect those of individual members.

Results for SO3 – Favourable environment

We launched a total of four ‘qualitative’ surveys related to the actions under SO3. A first step in the process was the drafting of the survey questionnaires. Table 7 below, gives an overview of this process, which was rather time consuming due to the number of surveys to be covered.

Table 7: Overview of status of qualitative surveys

157.Survey	158.Date of draft	159.Approval date	160.Launch of survey
161.SME Policy – SME Envoys	162.21.03.2017 - draft 1; 27.03.2017 – draft 2	163.27.03.2017	164.28.03.2017
165.SME Policy – SPR Working Group	166.27.03.2017	167.11.04.2017	168.11.04.2017
169.E-skills	170.10.04.2017	171.10.04.2017	172.28.04.2017
173.Sectoral competitiveness	174.10.04.2017	175.18.04.2017	176.02.05.2017

6.Source: Technopolis

The results of these surveys vary substantially. There was a good level of response to the survey in the field of SME Policy (perhaps due to the fact that it was sent out by the European Commission), while response rates were low for e-skills and sectoral competitiveness. Even though the contacted actors had participated in events or activities of the COSME actions in the field of KETs, e-skills, or sectoral competitiveness, some of them mentioned that they were not aware of the COSME programme nor had they had any interaction with it. In addition, despite being contacted several times, the members of the Digital Coalition for Skills and Jobs (to whom the e-skills survey was targeted) were not responsive – only the Coalition members contacted for interviews were willing to contribute.

Stakeholders involved in the quantitative targeted consultations

Table 8 presents an overview of the quantitative surveys, including the target group, population, target sample, responses, expected response rates and response rates. Out of the seven consultations, five reached or exceeded the expectations in terms of response rate. One did not reach its targets: the survey to the EEN member organisations.

Table 8: Overview – Quantitative surveys

177.#	178.Action	179.Type of stakeholder	180.Pop.	181.Targeted sample	182.Number of responses	183.Expected response rate	184.Response rate reached	185.Closing date in 2017
1	Loan Guarantee Facility	Beneficiary SMEs	140,000	3,875	356*	6.3%	9%	18 July
2	Enterprise Europe Network (EEN)	EEN member organisations	523**	530	270	~60%	48%	17 May
3	Enterprise Europe Network (EEN)	Beneficiary SMEs (EEN clients)	100,000	50,000	2,334	~5%	5%	17 May
4	Clusters Go International	Cluster managers	136	136	48	25-30%	35%	5 May
5	Tourism (Calls)	Beneficiary SMEs and Intermediary agencies / organisations	377	389	162	20%	41%	10 May
6	Erasmus for Young Entrepreneurs	New Entrepreneurs	2,343	2,117	595	25%	28%	5 May
7	Erasmus for Young Entrepreneurs	Host Entrepreneurs	1,800	1,620	245	10%	15%	5 May

7.Source: Technopolis - *Reached via 20 financial intermediaries. **Reached via 92 coordinators.

The public consultation

Übersicht

The approach of the public consultation followed the consultation process defined by the European Commission (Better Regulation Toolbox/Tool #50). The questionnaire was available in all EU languages. The European Commission implemented the questionnaire for the public consultation on the “Your Voice in Europe” website and the questionnaire was available in English on 10 May 2017. It remained accessible for 12 weeks, starting from the date on which the last translation was made available online. The dissemination of the public consultation fell under the responsibility of the European Commission. EASME and the EC’s communication units advertised the public consultation on COSME, EASME and EIF websites; through EEN networks and relevant national contact points; through other agencies and networks under COSME; and via other EC social media.

Number and profile of the participants

The public consultation was closed on the 31 August 2017. A total of 195 responses were received, as well as 14 position papers. The survey responses were analysed using descriptive statistics.

The questionnaire also included questions that allowed to profile the respondents to the public consultations. Half of the respondents were active in industry/business; the second largest group comprised of financial intermediaries (financial institutions or finance providers). The respondents under ‘others’ included law firms, research centres, innovation centres, NGOs and associations; business organisations; public authorities; and citizens (see Figure 4 ). Five countries account for two thirds of the answers (France, the UK, Italy, Spain and Germany).

Figure 4 Public consultation - profile of the participants

Source: COSME public consultation, 2017, n=195

Two-third of the respondents, i.e. 88 respondents, indicated that their organisation had benefitted from the COSME programme. Five respondents were involved in more than one action (e.g. EEN as a member organisation and beneficiary, etc.). The Loan Guarantee Facility (LGF) is, by far, the most recurrent action.

Table 9 Number of respondents per type of financial support

186.Type	187.Action	188.Number of beneficiaries
189.Access to finance	190.Loan Guarantee Facility	191.55 in total: 192.29 companies 193.16 financial institutions 194.10 others
	195.Equity Facility for Growth	196.5
197.Access to markets	198.Enterprise Europe Network (EEN) – EEN member organisations	199.15
	200.Enterprise Europe Network (EEN) – beneficiaries	201.13
202.Entrepreneurship	203.Erasmus for Young Entrepreneurs – New Entrepreneurs	204.4
	205.Erasmus for Young Entrepreneurs – Host Entrepreneurs	206.4
207.Favourable environment	208.Cluster internationalisation	209.2
	210.Tourism (calls)	211.2
212.	213.Other: 214.Selected as an expert 215.NCP for COSME including financial instruments and publishing calls for proposals in all areas / Coordinator EEN including extra support for EYE-participation 216.COS-DESIGN	217.3 218.

8.Source: COSME public consultation (n=88)

Robustness of the consultation results

A key method to ensure robustness of the analysis in an evaluation is the triangulation of the findings, i.e. the validation of information collected through one method by comparing it with the information collected through another method. In this section, instead, we expand on the robustness of the findings collected through the single consultation methods.

The quantitative targeted consultations

Apart of the more than satisfactory response rates reached for the quantitative targeted consultations, the profile of the responding organisations is also in close to all surveys very similar to the target sample, thus guaranteeing the statistical representativeness. Exceptions to the rule are the survey of the Network client SMEs, managed through a cascade approach, for which less responses than expected were reached; in this case, also the representativeness of the responses cannot be checked. Another exception is the survey related to the Clusters Go International programme where more funded ESCPs than non-funded ones responded compared to the target sample. In this case, however, this does not imply forcefully a lack of representativeness for the funded ESCPs.

Table 10: Criteria and results regarding the statistical representativeness analysis

219.Survey	220.Response rate	221.Criteria for representativeness	222.Result
223.SO1 - LGF beneficiaries	224.9%	Country of residence Sector of activity	A higher response rate than recorded in previous exercises The distribution of the survey responses is well balanced in terms of country and sector of activity in comparison with the target sample. One notable exception is France (no responses received)
225.SO2 - Network member organisations	226.48%	Country grouping Network member role	Response rates from both coordinator (47%) and partner groups (48%) are roughly equal Distribution of the survey population is very similar to the target sample
227.SO2 - Network client SMEs	228.5% of the estimated target population (cascade approach)	n.a.	Over two thirds of respondents located in EU15 Members States, but sufficient respondents located in EU13 MS to analyse questions based on country groupings Number of respondents from COSME-participating third countries too low to make meaningful analysis.
229.SO3 – Clusters Go International	230.35%	Funded versus voluntary ESCPs	The response rate for funded organisations was higher than that for voluntary organisations
231.SO3 –Tourism	232.41%	Country grouping	High level of alignment between the country groups split between the target sample and respondent groups, for those for which this information was available.
233.SO4 - New entrepreneurs 234.	235.28%	Country grouping	Distribution of responses (among those respondents for which the geographical location is known) is very similar to that of the target sample
236.SO4 - Host entrepreneurs 237.	238.15%	Country grouping	Distribution of responses (among those respondents for which the geographical location is known) is very similar to that of the target sample

The qualitative targeted consultations

As shown in Section 0, the qualitative consultations gave different results. While three out of the four surveys related to SO1 reached a reasonable level of response rate, the same can be said only for one out of four surveys related to SO3.

The qualitative surveys for which the number of returned questionnaires was below expectation are the ones targeting the EFG beneficiary SMEs, and the surveys related to SME policy actions (one out of two), the sectoral competitiveness actions, and the e-skills actions. A factor playing a role for the EFG beneficiaries was the cascade approach used; for the sectoral competitiveness one, the wide variety of sectors to be covered was a hindering factor, as well as the related difficulty to identify the relevant stakeholders to include, even with the support from the EC.

Overall, the quality of the responses was also often below the expected level; the full exploitation of this evaluation tool was hindered by the impossibility for the evaluation team to set up a direct follow-up interaction with the respondents. These consultations did not have the intention to reach ‘statistical representativeness’.

The public consultation

Public consultations present limits for use in the context of an evaluation due to the impossibility of controlling and checking the ‘statistical representativeness’ of the information provided. For example, the profile of the respondents shown in Section 0 clearly indicates a strong representation of stakeholders involved in the financial instruments and respondents involved in a mix of other COSME actions. A use of these data in a statistical manner would therefore have been inappropriate. Instead, we considered the responses as well as the opinion papers submitted as illustrations of a viewpoint, in contrast to our original intentions, used the data exclusively to corroborate findings deriving from various other data collection tools used.

Summary of the results

Relevance

Quantitative consultations

Looking at the financial instruments, it appears that they matched the beneficiaries’ needs. More than half of the respondents indicated that they preferred the option that included the EU-COSME guarantee, though other sources of finance were available to them that would cover all or part of their required amount. The survey shows that the EEN member’s view on the extent to which Network services help SMEs in addressing their needs is in general quite positive. There is a good match between the type of services financed by the Network and the demand of companies. When it comes to tourism, respondents were positive, with each of 12 needs rated as being addressed ‘to a great extent’ or ‘to some extent’ by at least two thirds of them. All cluster organisations responded positively to the benefits resulting from collaboration, calling for the creation of stronger linkages between locations brought about by COSME actions. When it comes to the EYE, the respondents’ answers provide widespread confirmation of the existence of market and systemic failures in the area of barriers to entrepreneurship and entrepreneurial culture were present in their country today.

Interviews

Financial intermediaries interviewed mentioned that some financial intermediary needs – and therefore, in cascade, SME needs - are not sufficiently supported by the LGF. Interviewed stakeholders regarded the focus of the EFG instrument as appropriate. Interviewed intermediaries indicated that their EEN services matched the needs of companies and SMEs in the field very well. Interviewees indicated the creation of knowledge-exchange opportunities is perceived as critical in the field of KETs. They considered there was a gap in the support from COSME from that perspective; more support for awareness raising activities was seen as beneficial. Stakeholders interviewed agreed that support by the SME policy actions is highly relevant for the dissemination of the ‘Think Small First’ principle.

Public consultation

Networking actions rank as the most relevant actions of the programme amongst the respondents. Two thirds of them considered network-building and cooperation as relevant. Financial instruments come second, with on average, a little less than two thirds of the respondents thinking grants and loan portfolio guarantees are relevant.

Effectiveness

Quantitative consultations

The EU-COSME guarantee appears to enable focused investments in purchases or improvements/developments among our survey respondents in areas of capital expenditure, regardless of business size. Respondents were rather positive about the potential for the financing to further enable expansions in existing markets and the development of products or services that are new to the market. Results from the survey show that the services related to knowledge enhancement of the EEN are rated highly by the SMEs. Moreover, Network members stated that their participation in the Network allowed them to build stronger networks for their organisation. When it comes to tourism, respondents’ answers suggest that COSME-funded tourism projects are achieving their expected objectives in relation to visibility and capacity. Looking at cluster organisations, over half of the responding organisations reported that they had, at least to some extent, achieved (or were expected to in the near future) the objectives of the actions. A majority of EYE beneficiaries, i.e. the New Entrepreneurs, reported improvements in all of the suggested skill areas, and in particular various personal and language skills. Host Entrepreneurs reported widespread benefits for their business activities as a result of hosting a New Entrepreneur.

Interviews

According to interviewees, the EEN services had a positive indirect effect on business cooperation at the regional level thanks to participation in the information events. The interviewees emphasised on the importance of the knowledge- and network-sharing activities among the Network members for the overall functioning and quality of the services provided to SMEs. The interviewees considered the e-skills actions effective especially in raising awareness on the needed skills and in engaging the relevant stakeholders and policy-maker communities. Most national representatives interviewed stated that the SME performance review provides reliable statistical data and analysis on EU SMEs and SME policies. Some interviewees mentioned that the SME Envoys network could be more exploited. The interviewees find that the clusters and cluster managers are well placed to facilitate the internationalisation of their member SMEs as cluster managers are acquainted with the technologies of the companies in their network and can promote the “ecosystem” to outside markets.

Public consultation

COSME fully responded to its key objective of supporting jobs, growth and investment, with half of the respondents who believe that the programme largely contributed to it. Additionally, almost half of the respondents believe that the COSME programme supports the creation of start-ups, the growth of SMEs and business transfer. As much respondents believe that the programme eases access to finance for SMEs through loans. Improved international visibility and more cooperation with partners from other countries are the two main expectations respondents have regarding COSME

Efficiency

Quantitative consultations

Survey responses paint a positive picture of the EU-COSME guarantee. The majority of respondents reported that they had not experienced additional costs, administrative burden or complexity as a result of their EU-COSME guarantee. When it comes to the EEN members, the survey results show that only 9% of the Network members state that benefits of the Network do not outweigh the time investment. Regarding the tourism actions, respondents were generally positive about both (financial and non-financial) aspects, with 85% (122 respondents) either strongly agreeing or agreeing that the Programme provides funding in a time-efficient manner. Looking at cluster organisations, over a third of them thought the reporting and monitoring requirements set by the programme were reasonable. However, many of the organisations felt the funding requirements were not correctly estimated by the ‘Clusters Go International’ programme. Though New Entrepreneurs do receive programme funding to cover certain travel, accommodation and other costs during their placement, our survey suggests that this is often not sufficient to cover costs during their stay. Most respondents were broadly satisfied across all of the aspects covered by the survey, including the timing and administration, as well as the support and assistance that was provided by intermediary organisations at different stages.

Interviews

The interviews indicated that EASME is quite efficient in the tasks they run (EEN) thanks to the experience built up during the CIP programming. The transition to the COSME programme did not have a significant impact on their tasks, in which efficiency was preserved. The availability of EASME to respond to questions from the Network members was highly praised by the interviewees. CGI participants interviewed felt unrealistic expectations were set by the EC especially for Strand 2 actions.

Public consultation

According to the respondents, the COSME procedures and administrative documents are fair, transparent and clear. In addition, the overall programme is considered as well structured. On the other hand, the programme’s biggest weakness is in its capacity to communicate and reach the potential beneficiaries (less than one in five respondents thinks it has the capacity). A couple of questions were targeted at respondents from non-EU participating countries. Overall, it appears that the COSME programme has not been very convincing to those respondents.

Coherence

Quantitative consultations

When it comes to the LGF, it is difficult to make robust conclusions about the complementarity of the EU-COSME guarantee and other initiatives, as there is little uptake (and thus opinion on) other sources of funding (both European and national or regional) for the same or similar activity among beneficiaries. Looking at the EEN and other EU programmes, survey results show a high level of complementarity between the Network and H2020. At national level, the maturity of the role of the Network members needs to grow, while the recognition of the Network by stakeholders in the region needs to be further developed. Though at least 50% of COSME tourism actions beneficiaries said they were aware of the campaigns, the respondents indicated a generally low uptake of the tools. Survey respondents were rather more positive when asked to what extent they use the tourism-related networking opportunities offered by the European Commission through the COSME programme. Regarding the cluster organisations, the proportion of respondents reporting complementarity between Clusters Go International and other activities greatly outweighed those reporting any overlap with any of these other schemes. When it comes to the Entrepreneurship TA, survey respondents did not point at specific overlaps between individual COSME entrepreneurship actions.

Interviews

Desk research and interviews showed a high level of synergy with the EEN by allowing SMEs to send out questions from the portal to the Network members. Interviews show that there is a high level of complementarity between the Network and H2020. Interviewees considered that there is room for creating more synergies between the cluster internationalisation actions and the actions supporting cluster value chains for S3. A more ‘holistic’ integrated approach to supporting Cluster internationalisation as part of the COSME Programme would be beneficial. Interviewees mentioned the need to coordinate the approach to cluster internationalisation with the new COSME cluster support actions targeting the strengthening of value chains for smart specialisation. In the context of the Cluster internationalisation programme, interviewees indicated that there is no clear coordination between EU-level and national level cluster policies.

Public consultation

On average, two thirds of the respondents did not know what to answer regarding the coherence of COSME with other EU programmes. Nonetheless, Horizon 2020, the European Fund for Strategic Investment (EFSI) and the European Structural and Investment Funds (ESIF) are the programmes with which COSME works the most in synergy/is the most complementary.

European added value

Quantitative consultations

Survey responses show that the EU-COSME guarantee creates significant value for beneficiaries. The EU-COSME guarantee both provides funding for those that had struggled to receive funding for their projects from other sources, as well as supporting projects to be conducted at a larger scale than would be possible without the funding. The survey results show that responding SME clients held quite positive views about the importance of the EU level Network services as opposed to only national support. Survey respondents were very positive about the value of EU level support in the field of tourism. Survey respondents were also very positive when asked to what extent they agree that the tourism-related actions funded by COSME have resulted in specific developments that would not have happened otherwise. Four out of five cluster organisations, part of the Clusters Go International programme, strongly agree that cluster cooperation is essential to create a more competitive European Union, while none of the organisations disagreed with this statement. Host and New Entrepreneurs who have participated in the EYE programme were nearly unanimous in their belief that the EYE scheme should continue in the future. Indeed, the vast majority of both groups stated that this was ‘very important’.

Interviews

Interviewees generally agree that there are no private institutions nor national governments that can achieve the results that COSME (financial instruments) has managed to achieve in terms of leverage effects and scope and risk profile of the SMEs involved. In the interviews, the intermediaries clearly indicated that for all the ‘structural’ elements of the Network, the European dimension constitutes a pre-condition. Many interviewees indicated a shortage of funding in their countries or regions for this type of SME support; in this case, the additionality is therefore of a financial kind.

Public consultation

Overall, a third of the respondents considers that the overall value of COSME compared with national and/or regional SMEs and entrepreneurship support programmes is higher. More than half of the respondents believe the programme helped financing projects which otherwise could not be supported at national and/or regional level.

Annex 3: Methods and analytical models

Table of Contents

1 Introduction

2 The methodological framework

2.1 Methodological approach

2.2 Overview of the evaluation tools and methods

2.2.1 Brief description of tools and methods

2.2.2 Matrix for the triangulation

2.3 Robustness of the results

2.3.1 The quantitative targeted consultations

2.3.2 The qualitative targeted consultations

2.3.3 The public consultation

2.4 Flow of the analysis

3 Desk research

3.1 Overview

3.2 Details at the Thematic Areas level

3.2.1 SO1 – Access to finance

3.2.2 SO2 – Access to market

3.2.3 SO3 – Favourable environment

3.2.4 SO4 – Entrepreneurship

4 Statistical data analysis

4.1 Sources

4.2 Portfolio analysis

4.3 Composition analysis

4.4 Analysis of COSME programme management and proposal data

4.5 Cost-Effectiveness analysis

5 The consultations

5.1 The targeted consultations – quantitative

5.1.1 Overview

5.1.2 Sampling strategy and results

5.1.3 Robustness of the results

5.2 The targeted consultations - qualitative

5.2.1 Overview

5.2.2 Sampling strategy

5.2.3 Results for SO1 – Access to finance

5.2.4 Results for SO3 – Favourable environment

5.3 The public consultation

5.3.1 The structure of the public consultation questionnaire

5.3.2 The implementation of the public consultation

5.3.3 Outcomes of the public consultation

5.3.4 Statistical analysis

6 Interviews

6.1 Overview

6.2 Results

6.2.1 SO1 – Access to finance

6.2.2 SO2 – Access to market

6.2.3 SO3 – Favourable environment

6.2.4 SO4 – Entrepreneurship

Appendix A - COSME programme KPIs, and the extent to which these KPIs have been realized

Appendix B - Statistical data analyses results

Appendix C - List of documents analysed

Appendix D – Lists of interviewees

List of Tables

Table 1 Evaluation questions

Table 2 Tools and methods used in the evaluation

Table 3 Use of the methodological mix per evaluation question

Table 4 Literature list at the programme level

Table 5 Evidence table for the desk research and literature review

Table 6 Preliminary list of the available relevant databases

Table 7 Portfolio analysis levels

Table 8 Overview of available data on beneficiaries

Table 9 Overview of available data on intermediaries

Table 10 Calls on which time to grant analysis is based

Table 11 Time to grant by thematic area and action line

Table 12 Cost-effectiveness analysis

Table 13 Overview – Quantitative surveys

Table 14 Overview of the participation in the qualitative surveys

Table 15 Overview of status of qualitative surveys

Table 16 Overview of responses to SO3 short survey consultations

Table 17 Overview of the open public consultation process

Table 18 Number of respondents per type of financial support

Table 19 Overview of the interviews conducted, by SO

Table 20 SO3 – Performed interviews

Table 21 Legend to Table 22 below

Table 22 Main activities, KPIs, and the extent to which these KPIs have been realised

Table 23 Country penetration rates of LGF guarantee 2014-2016

Table 24 Geographical distribution - shares by country aggregation of number of beneficiaries for 2014-2016

Table 25 Geographical distribution – estimated number of final recipients by country in EU for 2014-2016

Table 26 Geographical distribution – weighted shares of participants by country in EU for 2014-2016

Table 27 Geographical distribution - shares by country aggregation of cumulative budget for 2014-2016

Table 28 Geographical distribution (money allocated to final beneficiaries in EUR k for 2014-2016)

Table 29 Sectoral involvement (estimated shares of cumulative budget 2014-2016)

Table 30 Sectoral involvement (shares of final beneficiaries 2014-2016)

Table 31 Sectoral involvement (budgets in EUR m)

Table 32 Sectoral involvement (number of final beneficiaries)

Table 33 List of documents analysed SO1

Table 34 List of background documents SO2

Table 35 List of background documents SO3

Table 36 List of documents analysed SO4

Table 37 Interviews performed SO1

Table 38 Interviews performed SO2

Table 39 Interviews performed SO3

Table 40 Interviews not performed SO3

Table 41 Interviews performed SO4

Table of Figures

Figure 1 Evaluation criteria and intervention logic in an interim/mid-term evaluation

Figure 2 Intervention logic of the COSME programme

Figure 3 Flow of the analysis

Figure 4 LGF beneficiaries: comparison of the country distribution of the whole population, target sample and survey population

Figure 5 LGF beneficiaries: comparison of the sector distribution of the whole population, target sample and survey population

Figure 6 Network member survey target sample and respondents, by county

Figure 7 Network member survey target sample and respondents, by role

Figure 8 Network client SME company survey respondents, by country group

Figure 9 Distribution of population in sample and responses between type of cluster organisation

Figure 10 Profile of target sample and SO3 tourism survey respondents, by country group

Figure 11 Distribution of population in sample and responses by types of countries

Figure 12 Distribution of population in sample and responses by types of countries

Figure 13 Public consultation - profile of the participants

List of abbreviations

Abbreviation	Full name
CEA	Cost-Effectiveness Analysis
CIP	Competitiveness and Innovation Framework Programme
COSME	Programme for the Competitiveness of Enterprises and Small- and Medium-sized Enterprises
DG CONNECT	Directorate-General for Communications Networks, Content and Technology
DG GROW	Directorate-General for Internal Market, Industry, Entrepreneurship and SMEs
DG REGIO	Directorate-General for Regional and Urban Policy
DG RTD	Directorate-General for Research and Innovation
DG TRADE	Directorate-General for Trade
DGT	Directorate-General for Translation
DG EAC	Directorate-General for Education, Youth, Sport and Culture
EASME	Executive Agency for SMEs
EC	European Commission
EEN	European Enterprise Network
EFG	Equity Facility for Growth
EIB	European Investment Bank
EIF	European Investment Fund
EIP	European Innovation Partnerships
ESCPs	European Strategic Cluster Partnerships
EU	European Union
EYE	Erasmus for Young Entrepreneurs
FWC	Framework Contract
HE	Host Entrepreneur
ICT	Information and Communication Technology
IPR	Intellectual Property Rights
KETS	Key-Enabling Technologies
LGF	Loan Guarantee Facility
NACE	Statistical Classification of Economic Activities in the European Community
NCP	National Contact Point
PES	Performance Enhancement System
RPAS	Remotely Piloted Aircraft Systems
SBA	Small Business Act
SMEs	Small- and Medium-sized Enterprises
SO	Specific Objective
SPR	SME Performance Review
UEAPME	European Association of Craft, Small- and Medium-sized Enterprises

Einführung

This annex is structured as follows:

·The methodological framework used for the evaluation is presented in Section 1

·We then describe the approach taken and provide information on the results at SO level for the methods used for the evaluation (including the case studies), i.e. the desk research (Section 2), the statistical data analysis (Section 3), the targeted and public consultations (Section 4) and the interviews (Section 5).

There are four appendices to the report, providing additional data (Appendix A and Appendix B), lists of documents analysed (Appendix C) and interviewees (Appendix D), and the targeted (quantitative) and public consultation questionnaires).

1.The methodological framework

In this chapter, we give an overview of our approach taken to the evaluation. Section 1.1 presents the key concepts that guided our evaluation, including the programme’s intervention logic and the evaluation questions.

Section 1.2 summarises the evaluation tools and methods we used, the triangulation matrix we designed and the flow of analysis we followed.

The objective of the study was to evaluate the relevance, effectiveness, coherence, efficiency and EU added value of the programme activities funded in the 2014-2016 period. The study findings should contribute to the decision-making on the renewal, modification or suspension of the programme actions and measures, to set the basis for the next evaluation of the COSME programme until 2020 and contribute to the preparation of the future programme in the next Multiannual Financial Framework (MFF), and to feed into the mid-term evaluation of EASME.

The scope of this evaluation was the COSME activities conducted in the period 2014-2016.

1.1.Methodological approach

The Terms of Reference (ToR) indicated the evaluation criteria of relevance, efficiency, effectiveness, coherence and EU added value as overall framework for the study. These evaluation criteria assess the relationships between the various components of the intervention logic ( Figure 5 ).

Figure 5 Evaluation criteria and intervention logic in an interim/mid-term evaluation

Source: Technopolis

During the inception phase, the scope for the analyses against these evaluation criteria was defined as follows:

·Assessment of the relevance, coherence, and efficiency in implementation criteria covering all actions

·Assessment against the effectiveness criterion focused on the Key Actions, complemented by four case studies

·The European added value and the efficiency / cost-effectiveness criteria remain covered through the analysis of the Key Actions only

The Key Actions included: the financial instruments (LGF and EFG), the Enterprise Europe Network services, the SME Policy actions, the Cluster Internationalisation programme, and the Erasmus for Young Entrepreneurs programme.

Actions covered through case studies are: the Vulcanus scheme in the EU-Japan Centre action; the e-skills/digitisation actions, the Tourism actions (grants), and the Women Entrepreneurship action.

The supporting study was performed on the basis of six leading evaluation questions and fifteen sub-questions for this evaluation.

Table 11 Evaluation questions

Key question	Evaluation question
1) Implementation - State of play	Q1.1) How has the programme been implemented and how has it evolved since the start in 2014?
2) RELEVANCE: Do the objectives correspond to the current needs?	Q2.1) To what extent are COSME’s initial objectives still pertinent to the needs, problems and issues it was designed to address?
	Q2.2) How relevant is the programme to the stakeholders (public and private) and the EU citizen?
	Q2.3) How has the programme adapted to the political priorities, for instance to the new Single Market Strategy and its particular objective to support Start-ups and Scaling-ups?
3) EFFECTIVENESS: How effective is the programme in achieving its objectives?	Q3.1) To what extent is the COSME programme progressing towards achieving its specific objectives?
	Q3.2) What are the factors driving or hindering progress and how do they link with different actions of the programme?
	Q3.3) To what extent is the Programme progressing towards contributing to the 5 Union priorities for smart, sustainable and inclusive growth (incl. climate objectives and gender mainstreaming)?
4) EFFICIENCY: To what extent were the effects (benefits) achieved at a reasonable cost?	Q4.1) What are the regulatory/administrative costs and the benefits for the different stakeholders?
	Q4.2) Are the costs (direct and indirect) generated by the programme proportionate to the benefits generated?
	Q4.3) How efficient is the implementation structure and governance of the programme?
5) COHERENCE: To what extent are the individual actions financed by COSME internally coherent and to what extent is COSME externally coherent with other EU and MS programmes/policies with similar objectives?	Q5.1) Are the different actions of COSME implemented coherently between them?
	Q5.2) How are the different COSME actions overlapping/synergizing with the activities of ESI funds, H2020, EFSI, EASI and other EU programmes?
	Q5.3) To what extent has the coherence been maximised in view of National programmes?
6) EU ADDED VALUE: What is the EU value added?	Q6.1) What is the added value of the programme and actions being conducted at the EU level?
	Q6.2) To what extent and why do the issues addressed by the programme continue to require action at EU level?

Source: Technopolis

Time is needed for effects such as mid-term results and long-term impacts to occur. In the case of interim evaluations such as the current one focusing on the COSME programme, most of the expected mid-term results are not yet be measurable. The focus is therefore on the assessment whether the programme succeeded in setting the conditions that can be expected to lead to these expected results. This implies an emphasis in this evaluation on the adequacy and quality of the programme design, i.e. the formulated objectives, the invested resources, and the activities implemented – in other words, the relevance and coherence criteria.

In terms of effectiveness, the assessment focused on the attainment of the expected outputs and short-term outcomes, and the enabling factors or barriers for the attainment of the outcomes and expected results, such as the capacity of the programme to reach and involve the stakeholders needed.

The intervention logic was a critical component of our methodological framework. The consistent use throughout our analyses of the categories and types of effects that were defined in this intervention logic allowed us to structure our findings and draw conclusions at the programme level, going beyond a mere aggregation of the findings at the SO level. Figure 6 , below, sets out the intervention logic of the COSME programme and its expected effects.

Figure 6 Intervention logic of the COSME programme

Source: Technopolis

1.2.Overview of the evaluation tools and methods

The consultant used a mix of qualitative and quantitative methods for the evidence collection and analysis. We briefly describe our approach for these evaluation tools and methods in Section 1.2.1, below. A full description of each tool and method is provided in the next chapters. In Section 1.2.2 we present the matrix that constituted the framework for the triangulation of the evaluation findings. Section 1.3 sets out the flow of analysis in the evaluation and its workplan.

1.2.1.Brief description of tools and methods

Table 12 , below, presents the tools and methods used in this evaluation.

Table 12 Tools and methods used in the evaluation

Methodological approach	Tools & methods	Number
Qualitative methods	Desk research	260+
	Interviews	121
	Case studies	4
	Workshop	1
	Qualitative targeted consultation	7
Quantitative methods	Targeted consultation	7
	Public consultation	1
	Statistical analyses of secondary data	5

Qualitative methods

The desk research encompassed a broad range of sources, ranging from programme and policy documents to market studies, impact assessments and preceding and/or other evaluations. The list has been in constant expansion throughout the evaluation. The review of the literature and desk research was an activity that took place throughout the entire lifespan of the evaluation. Nevertheless, it was concentrated especially in the first stages of the evaluation, enabling the study team to conduct the interviews with the stakeholders in an informed manner.

A major purpose of the high-level interviews was to collect background and context information on facilitators and barriers for the attainment of the expected effects as well as the eventual unintended effects. We also conducted some interviews at the overall programme level. These interviews with EU officials took place in the beginning of the study to reach an improved understanding of the overall context to the programme and the processes for the programme design.

In this evaluation, we conducted four case studies covering three SOs: SO2 – Access to market (the trainee programme “Vulcanus in Japan”, run by the EU-Japan Centre), SO3 – Favourable environment (the Tourism actions under the programme and the E-skills for competitiveness and innovation actions), and SO4 – Entrepreneurship (the Women Entrepreneurship action). Each case study covered the overview of the programme’s progress in attaining the objectives set out in the intervention logic, and to the extent possible, an appraisal of their progress towards attaining their expected results. Another important focus of the case study was the analysis of the relevance and coherence of the COSME actions. The results fed into the wider analysis of SO2, SO3, and SO4, as well as of the COSME programme overall.

A day-long workshop was organised on May 31, 2017, involving representatives of the Member States and the COSME Third Countries as well as Key Stakeholders. The objective of the workshop was two-fold: presenting the then-current results to the stakeholders as well as receiving feedback from them, especially regarding the relevance, coherence and European added value of the COSME programme.

The first half of the workshop was dedicated to the presentation of the COSME interim evaluation (the objectives and the methodology) and the first findings of the team for each SO, followed by a Q&A session. During the second half, participants split into four groups. Positive and negative aspects of the COSME programme were discussed as well as the ideal design of an SME support programme. A plenary session concluded the workshop where the outcomes of the breakout sessions were discussed.

On request of the EC, this evaluation included the conduct of ‘qualitative’ targeted consultations, replacing interviews. This type of surveys will be implemented where the population is lower than 100; the small numbers therefore did not allow for meaningful statistical analysis. The questionnaires mostly include open-ended questions, geared at collecting qualitative information.

Quantitative methods

The ‘classical’ targeted consultations were implemented in those cases where the population was higher than 100 and would therefore allow for meaningful and robust statistical data analyses. The questionnaires mostly include close-ended questions and only a few open-ended questions. The responses were analysed through descriptive statistics, in several cases also included cross-analyses.

The public consultation was a 12-week internet-based questionnaire. The objective of the public consultation was to cover all potential relevant stakeholders, in particular those that were not targeted through the targeted consultations and interviews, and to collect feedbacks from them on the COSME programme and its objectives. More specifically, we received input from:

·Business and trade associations

·Citizens of the EU

·Non-profit organisations

·Financial intermediaries (private equity, business angels, venture capital funds, banks, promotional banks, guarantee institutions)

·Intermediary structures such as incubators/clusters/innovation & growth agencies

·Public authorities (member states, European agencies)

The questionnaire for the public consultation was quite straightforward as it aimed at reaching all types of stakeholders. Given that the targeted consultations focused exclusively on the specific objectives, it was agreed to use the public consultation to collect data that would provide information on the programme as a whole.

In total four types of analyses were conducted on secondary data:

·The portfolio analysis entailed the analysis of the COSME budget distribution over the different structural components of the programme, ie the Strategic Objectives and their operational (sub)objectives, the policy mix used in the programme (ie the policy instruments) and the modes of implementation. We covered both the planned budget distribution (based on the Work Programmes) and the actual spending.

·The composition analysis focused on the stakeholders that benefit of the COSME programme funding. The outcomes of this analysis informed especially the analysis of effectiveness, in terms of the capacity of the programme to attract the type of stakeholders needed for the attainment of the desired effects and the analysis of cost-effectiveness. Dimensions for analysis were the geographical location of the beneficiaries, the type of stakeholders involved in the actions funded, such as SMEs, professional associations, intermediaries, national/regional administration officials, and consultancies providing services to the EC, and the sector of activity of the SMEs and other private sector stakeholders involved.

·The time-to-grant analysis, based on EASME management data, and the success rate analysis of the open calls for grants, based on the EASME proposal database, were conducted to feed into the assessment of the COSME efficiency in the implementation of the programme.

·The cost-effectiveness analysis (CEA) entailed quantifying the not-monetised benefits that would be generated by one Euro of total costs. Typically, this entails defining the benefit-cost ratio, i.e. dividing the benefits by costs. CEA is normally used to identify the “value for money” of programmes. The CEA focused on SO actions that have similar objectives, rather than on the SOs in general. In addition, for the CEA to provide a meaningful result, a measure to identify ‘good performance’ was needed; typically, this is obtained through comparison. Our approach was to assess the cost-effectiveness of the SO Actions in such a way that the outcomes for the different SOs can be compared to each other. This implied a focus of the ‘effectiveness’ factor on the only comparable outcome indicator possible, the number of SMEs and individuals supported.

1.2.2.Matrix for the triangulation

For each evaluation question, the analysis was based on multiple sources and methods ( Table 13 ). This allowed for a proper triangulation of the evidence collected, ensuring the robustness of the findings. It also helped mitigating the risk of lack in data availability.

Table 13 Use of the methodological mix per evaluation question

	Qualitative				Quantitative
	Desk research	Interviews (including the “qualitative” surveys)	Case studies	Workshops	Secondary data analysis / Composition & portfolio analysis	Other secondary data analysis	Cost-effectiveness analysis	Statistical analysis targeted consultation data (“quantitative” survey)	Public consultation
1) Implementation - State of play
Q1.1) How has the programme been implemented and how has it evolved since the start in 2014?	X	X	X		X
2) RELEVANCE: Do the objectives correspond to the current needs?
Q2.1) To what extent are COSME’s initial objectives still pertinent …	X	X		X	X			X	X
Q2.2) How relevant is the programme to the stakeholders (public and private) and the EU citizen?	X	X		X	X			X
Q2.3) How has the programme adapted to the political priorities …	X	X		X
3) EFFECTIVENESS: How effective is the programme in achieving its objectives?
Q3.1) To what extent is the COSME programme progressing towards achieving its specific objectives?	X	X	X	X				X
Q3.2) What are the factors driving or hindering progress and how do they link with different actions of the programme?	X	X	X	X	X			X
Q3.3) To what extent is the Programme progressing towards contributing to the 5 Union priorities ….	X	X	X	X	X
4) EFFICIENCY: To what extent were the effects (benefits) achieved at a reasonable cost?
Q4.1) What are the regulatory/administrative costs and the benefits for the different stakeholders?	X	X					X	X
Q4.2) Are the costs (direct and indirect) generated by the programme proportionate to the benefits generated?	X	X				X	X	X
Q4.3) How efficient is the implementation structure and governance of the programme?	X	X			X	X	X	X
5) COHERENCE: To what extent are the individual actions financed by COSME internally coherent and to what extent is COSME externally coherent with other EU and MS programmes/policies with similar objectives?
Q5.1) Are the different actions of COSME implemented coherently between them?	X	X		X	X			X
Q5.2) How are the different COSME actions overlapping/synergizing with the activities of ESI funds, H2020, EFSI, EASI and other EU programmes?	X	X		X		X		X	X
Q5.3) To what extent has the coherence been maximised in view of National programmes?	X	X		X		X		X	X
6) EU ADDED VALUE: What is the EU value added?
Q6.1) What is the added value of the programme and actions being conducted at the EU level?	X	X		X				X	X
Q6.2) To what extent and why do the issues addressed by the programme continue to require action at EU level?	X	X		X				X	X

Source: Technopolis

1.3.Robustness of the results

In this section, we discuss the measures of triangulation and the findings of the robustness checks related to the consultations that were implemented in this study.

·Triangulation is especially important for the validation of findings that derive from the statistical analysis of survey responses, due to the potential for statistical bias (see Section 2.3.1, below).

·However, robustness checks and quality assurance measures should be applied also for the other consultation methods (see Section 2.3.2 and Section 2.3.3, below).

More detailed information is provided also in the sections for the individual actions in the sections below.

1.3.1.The quantitative targeted consultations

It is well known that surveys may suffer from a series of statistical biases that make the treatment of responses quite delicate for statistical purposes. These include:

·Sampling bias – the distortion arising from the non-randomness of the sample extraction, or the patent under-representation of relevant geographic areas, sectors, etc.

·Non-response bias – the distortion arising from the possibility that the non-respondents share a characteristic that is important but will be missed in the results

·Response bias – the answer distortion arising from the way the questionnaire is structured, or the perceived expectations of the sponsor/surveyor, or laziness (primacy, extreme responding, fatigue, etc.)

Some of these standard statistical problems (sampling bias, data errors) were present also in the surveys conducted in this study.

However, a part of the more than satisfactory response rates reached, the profile of the responding organisations was in close to all surveys very similar to the target sample, in general guaranteeing statistical representativeness.

A first exception to the rule was the survey of the Network client SMEs. This survey was managed through a cascade approach and reached less responses than expected. In addition, because of the cascade approach and therefore the unknown size of the population, the study team could not check the representativeness of the responses. In the main report for this study (section 2.3.4), we emphasise the negative consequences of this approach for the statistical value of survey data in the context of evaluations and endorse the recommendation made by the CIP and EIP final evaluation teams to ensure the possibility for evaluation teams directly to contact beneficiary SMEs.

A second exception is the survey related to the Clusters Go International programme where more funded than non-funded ESCPs responded to the survey, compared to the target sample. In this case, however, this does not imply forcefully a lack of representativeness for the funded ESCPs.

Surveys to stakeholders involved in the programme are often considered to be unreliable sources for information due to the expected bias in the responses by beneficiaries in the programme. We took this into account by analysing the data in terms of distribution of the responses in the different groupings along a 5-point Likert scale and by considering especially the ‘extremes’ in these distributions.

Most important, input provided through the surveys was always corroborated through the other methods used

1.3.2.The qualitative targeted consultations

The qualitative consultations gave different results. While three out of the four surveys related to SO1 reached a reasonable level of response rate, the same can be said only for one out of four surveys related to SO3.

These consultations did not have the intention to reach ‘statistical representativeness’, so there is no reason to doubt the value of the input provided. However, the efficiency of this type of exercise is highly questionable.

1.3.3.The public consultation

Public consultations present limits for use in the context of an evaluation due to the impossibility of controlling and checking the ‘statistical representativeness’ of the information provided. In the case of this evaluation, for example, the profile of the respondents clearly indicates a strong representation of stakeholders involved in the financial instruments and respondents involved in a mix of other COSME actions. A use of these data in a statistical manner would therefore have been inappropriate. Instead, we considered the responses as well as the opinion papers submitted as illustrations of a viewpoint, and used the data exclusively to corroborate findings deriving from various other data collection tools used.

1.4.Flow of the analysis

The evaluation was implemented in three phases: the inception, data collection and analysis, and reporting phases, whereby the latter two phases took place in two steps each ( Figure 7 ).

In the data collection and analysis phase we focused our activities on setting out the ‘state-of-play’ of the COSME implementation and the conduct of the analyses (desk research and interviews) focusing on the relevance and coherence criteria. This allowed us to report on our preliminary findings for these criteria in the Interim report (D3). In the interim report we also reported on the progress in the targeted and public consultations.

These preliminary findings were further validated in the next step, in particular during the workshop with the COSME committee members. The collection of evidence through the other methods and its triangulation led to the formulation of findings at the thematic area level. These findings were reported internally, allowing for an aggregated analysis and the drawing of conclusions and drafting of recommendations at the COSME programme level, which were then reported in the draft and final version of the Final report.

Figure 7 Flow of the analysis

2.Desk research

In this chapter we present the desk research strategy followed for the evaluation. We present an overview of this strategy in Section 2.1 while we detail it at thematic level in Section 2.2.

2.1.Overview

In Appendix B we provide the full list of all documents consulted per SO.

Table 14 , below, lists the documents and studies that were of relevance at the overall programme level.

Table 14 Literature list at the programme level

Type of document

Examples of relevant documents

COSME programme documentation

EU (2013), Regulation (EU) No 1287/2013 of the European Parliament and of the Council establishing a Programme for the Competitiveness of Enterprises and small and medium-sized enterprises (COSME) (2014 - 2020) and repealing Decision No 1639/2006/EC.

Ex-ante Preparatory documents for COSME, including public consultation: Final report, annex 1, 1a and 2;

Impact Assessment for the proposal for a COSME regulation:

Impact Assessment for the proposal for a COSME regulation COM (2011)834;

Impact Assessment for the proposal for a COSME regulation SEC(2011) 1452;

Impact Assessment for the proposal for a COSME regulation SEC(2011) 1453;

COSME work programmes: 2014, 2015 and 2016;

M&E available at the programme level

COSME monitoring reports 2014 & 2015 (2016 under development during the study)

Documents and sources for programmes prior to COSME – CIP:

Impact assessment (2005); Interim evaluation (2010); Final evaluation (2011);

CIP performance review 2007-2012;

Implementation, evaluation, performance and beneficiaries reports for each year of 2007 – 2012: http://ec.europa.eu/cip/documents/implementation-reports/index_en.htm

Decision 1639/2006/EC of the European Parliament and of the Council of 24 October 2006 establishing a Competitiveness and Innovation Framework Programme (2007-2013);

EU Policy documents

EC (2014), For a European Industrial Renaissance, COM/2014/014 final

EC (2012), A Stronger European Industry for Growth and Economic Recovery Industrial Policy Communication Update, COM/2012/0582 final

EC (2010), An Integrated Industrial Policy for the Globalisation Era Putting Competitiveness and Sustainability at Centre Stage, COM/2010/0614 final

EC (2008, 2011), Small Business Act

EC (2015), Single Market Strategy

EC (2016), Europe's next leaders: the Start-up and Scale-up Initiative, COM/2016/0733 final

EC (2012), Entrepreneurship 2020 Action Plan

EC (2011) Industrial Policy: Reinforcing competitiveness

EU Policy Studies

Pellegrin, J. et al (2015), EU Industrial Policy: Assessment of Recent Developments and Recommendations for Future Policies, CSIL, CASE, University of Milan, Economisti Associati, a study for: European Parliament, Directorate General for Internal Policies, Policy Department A: Economic and Scientific Policy

Dhéret, C., Morosi, M. (2014) Towards a New Industrial Policy for Europe, EPC Issue Paper No.78

Sylvest, J. et al. (2014) How can European industry contribute to growth and foster European competitiveness?, Danish Technological Institute, A study for: European Parliament, Directorate General for Internal Policies, Policy Department A: Economic and Scientific Policy

Source: Technopolis

A concrete output of our desk research was the overview of the achievements of the KPIs that are listed in Appendix A. This analysis was performed in the context of our assessment of the efficiency of the COSME programme, specifically related to the monitoring activities.

The table below gives an overview of the types of documents used for the assessment of COSME against the evaluation questions and topics.

Table 15 Evidence table for the desk research and literature review

SO1

SO2

SO3

SO4

Nr. question

Evaluation topics

Programme documents

Studies / secondary data

Other studies

Programme documents

Studies / secondary data

Other studies

Programme documents

Studies / secondary data

Other studies

Programme documents

Studies / secondary data

Other studies

IMPLEMENTATION STATE OF PLAY

Rationale of the programme and its actions

Objective hierarchy

Link with actions funded under CIP

Budget allocation over objectives

Budget allocation over types of actions

Budget allocation over types of calls and instruments

Budget allocation over types of stakeholders

RELEVANCE

Q2.1)

Ongoing validity of the rationale

Link developments/changes in budget allocations

Q2.2)

Alignment needs/actions/stakeholders

Deficiencies in alignment needs/actions

The decision-making process for the actions undertaken

The societal relevance of the programme

Q2.3)

The policy relevance of the programme

EFFECTIVENESS

Q3.1)

Outputs so far

Results so far

Alignment with expectations

Q3.2)

Appropriate stakeholders targeted

Appropriate stakeholders reached

Appropriate policy mix

Appropriate budget allocation

Q3.3)

Progress towards policy objectives

EFFICIENCY

Q4.1)

Administrative costs / stakeholders

Affordability / burden

Simplification measures

Q4.2)

Administrative costs / implementation structure

Cost-effectiveness

Comparison across actions

Q4.3)

Time-to-grant / / use of funding instruments / distribution of funding

Implementation / delegation

Monitoring system

COHERENCE

Q5.1)

Internal coherence - SO

Internal coherence - communication instruments

Internal coherence - programme

Internal coherence - coordination process

Q5.2)

External coherence - EU

External coherence - EU process

Q5.3)

External coherence - MS

External coherence - MS process

EU ADDED VALUE

Q6.1)

Importance of action at EU level

Additionality

Q6.2)

Ongoing need for action at EU level

Source: Technopolis

2.2.Details at the Thematic Areas level

2.2.1.SO1 – Access to finance

Desk research and secondary data analysis were conducted based on 60 documents identified during the inception phase and data collection phase.

Overall the exercise of gathering the documentation was challenging as EU officials did not have at end a knowledge database concerning the “Access to Finance” actions. This resulted in collecting documents throughout the overall study and up to September 2017. All available documentation was carefully processed and related to other findings.

The collection of secondary data is primarily important for the portfolio analysis as well for assessment of the effectiveness and the efficiency of the actions. Secondary data on the LGF and EFG were obtained from EIF notably Quarterly reports from 2014 to end of 2016, Annual reports 2014 to 2016, and Growth & Employment report 2015.

DG GROW shared a selection of other monitoring data of both instruments:

·Extraction of DG GROW commitments to both EFG and LGF

·Extraction of remuneration and fees paid to EIF for both instruments

The list of documents analysed is presented in Appendix A.1

2.2.2.SO2 – Access to market

One of first steps in the evaluation was the collection of the relevant documentation for the desk research on the actions under SO2. We contacted the relevant EC officers with a documentation request related to the following actions:

·Enterprise Europe Network

·Your Europe Business portal

·Improved Points of Single Contact

·Facilitating Access to Light Remotely Piloted Aircraft Systems (RPAS) Regulation

·Exchanges of Good Practices in the Area of Compliance Assistance and Schemes

·Improving SMEs’ Access to Public Procurement

·SME Internationalisation Portal

·EU-Japan Centre

·China, Latin America and South-East Asia IPR SME Helpdesks

·Support to SME Internationalisation

·Industrial Policy Cooperation

Most secondary data were identified in a meeting with Unit H2 on 9 February 2017 and have reached us via DG GROW. Additional secondary data were sent by EASME.

Overall the exercise of gathering the documentation was challenging, as many EU officials were very busy, especially within the holiday period that overlapped with the short time period between the inception report and the interim report. Furthermore, although many actions come from the Work Packages of 2014 and 2015, not all actions have been implemented to the extent that was initially expected. By the end of the evaluation, while information was made available on all the actions, for some actions the information remained limited. All available documentation was carefully processed and related to other findings.

The collection of secondary data is primarily important for the portfolio analysis as well for assessment of the effectiveness and the efficiency of the actions. Secondary data on the Enterprise Europe Network were obtained from EASME. for;

The largest source of collected information is the Performance Enhancement System (PES) database, which relies on regular reporting done by the Network members. The aim of the PES is to measure the achievements of the Network members, and to monitor the overall Network performance. It includes an Inception Report (6 months after the start of the Consortium), quantitative reporting on the 16 key variables of the PES (12 months, as well as 24 months after the start of the Consortium), self-assessments by the Consortium, as well as some other reflective elements. 86

It often takes quite some time to finalise all the data for the PES, therefore the time available to prepare the data within the evaluation was short. A selection of the preliminary PES data that were available in time to be included in the final report was provided to the evaluation team. EASME also shared a selection of other monitoring data of the Network, namely:

·Extraction of Achievements, providing insight into the number of Partnership Agreements and Advisory Service Outcomes realised per consortium

·Extraction of the Partnership Opportunity Database, providing insight into the number of partnering opportunities that were published per consortium, as well as the number of expressions of interest that those profiles generated

·Extraction of the new Impact Survey results. This is a new survey and therefore the results are still incomplete. It does provide insight into the way impact data are currently collected and what kinds of data are produced.

The list of documents analysed is presented in Appendix C.2

2.2.3. SO3 – Favourable environment

Desk research has been completed for all actions, and has aimed to provide an overview of the economic developments and policy issues within the different thematic areas under SO3: SME Policy, regulatory and administrative burden, innovation performance in SMEs, digitisation and KETs, as well as sectoral competitiveness.

The secondary data we used came from the COSME Work Programmes 2014-2015 and the COSME Activity Reports 2014 and 2015 for all actions in focus. Where available, Calls for Proposals / Projects for the various streams of actions (especially for the cluster internationalisation actions, tourism, and e-skills actions), and technical progress reports or final implementation reports were consulted. Moreover, for SME Policy, the SME Performance Review publications, documents published by SME Envoys and the description of the mission of the SME Envoys were used.

Beyond the COSME work programme and activity reports, the level of access, utility and comprehensiveness of further secondary data varies per field. In some cases, it has been difficult to obtain a comprehensive overview of the status of the implementation and interim results achieved, especially for the measures where the effectiveness dimension was in focus. In particular, synthesis reports of the projects funded under the Cluster Internationalisation call were only partially available, as the data collection for the first-year performance review had not been completed by the time of the evaluation, as several cluster partnerships had yet to submit their second progress reports. In the case of the tourism actions, no synthesis reports of the individual project monitoring and activity reports were available and, due to their non-standardised format, individual progress reports could not be used to provide a more comprehensive overview of the progress of actions and outputs in the Tourism sector. To counterbalance the lack of data in these cases, the results of the survey with beneficiaries of these two actions were used to illustrate the level of progress achieved in a more aggregated manner.

The list of documents analysed is presented in Appendix C.3

2.2.4.SO4 – Entrepreneurship

Desk research activities have been ongoing throughout the evaluation, but were a particular focus of the first stages of the study, to inform consultation activities and early analysis. The evaluation team collected and analysed secondary sources in relation to all the entrepreneurship actions in scope. Information was sought from the European Commission and EASME to ensure that both technical and policy-relevant information was considered. In addition, online searches and interviews were used to complete the review.

Additional information was obtained in relation to the key entrepreneurship action (EYE). This included information from the EYE programme database (containing information on all intermediaries and participants involved in EYE placements), past evaluations (the interim evaluation of the EYE pilot and the EYE statistical analysis), a selection of recent six-monthly monitoring reports produced by the EYE support office (Eurochambres) and the EYE programme guide and quality manual.

Information on the individual actions was captured in a structured format to facilitate analysis along the main evaluation questions. These secondary sources are drawn upon and referenced throughout the report, but particularly when analysing the context, relevance and coherence of the entrepreneurship actions.

The list of documents analysed is presented in Appendix C.4

3.Statistical data analysis

In this chapter we begin with an overview of our sources of information (Section 3.1). Then we present the portfolio analysis (Section 3.2), the composition analysis (Section 3.3), the programme management data (Section 3.4) and the cost-effectiveness analysis (Section 3.5).

3.1.Sources

Ten structured databases have been identified for the programme and the different SOs, based on input provided by the officials in charge of the databases at DG GROW, EASME and EIF ( Table 16 , below). Contacts with these officers and the EC officers acting as the SO contact persons continued throughout the study for the retrieval of some more detailed information and the completion of this list.

These datasets provided input for the composition and portfolio analysis depicting the programme’s ‘state-of-play’ as well as for the other analyses, especially those related to the relevance, effectiveness and efficiency questions. Some may serve also for the targeted consultations.

Table 16 Preliminary list of the available relevant databases

Database	Name	SO	Ebene	Kontakt
1	Data hub for COSME	SO2, 3, 4	Projects	EASME
2	Data from Annual operational reports	SO1	Aggregated + financial intermediaries	DG GROW
3	Data from the Growth and employment report 2015 (LGF)	SO1	Aggregated	DG GROW
4	Data from Quarterly reports	SO1	Aggregated + financial intermediaries	DG GROW
5	Data from EIF	SO1	Intermediaries and recipients	EIF
6	Achievement database	SO2	Partnership agreements and Advisory service outcomes	EASME
7	Performance Enhancement System	SO2	Consortia and network partners	EASME
8	Various data for EEN	SO2	/	EASME
9	EYE database	SO4	HE, NE, EP, IO	DG GROW
10	Statistics	SO1, 2, 3, 4	Countries	Publicly available

Source: Technopolis

3.2.Portfolio analysis

For the portfolio analysis, both the data on planned and committed budgets were collected. The budgetary distribution was calculated across policy instruments, implementation modes, and thematic areas (and corresponding action lines) and combinations thereof.

The outcomes of this analysis especially informed:

·the analysis of the relevance of the programme, in terms of the budget allocated for the attainment of specific objectives and sub-objectives as an illustration of importance attributed

·the analysis of effectiveness, in terms of the level of the specific effects that can be expected

·the analysis of implementation efficiency, in terms of the extent to which the COSME programme allocated sufficient levels of budget to specific actions and the projects within these actions, and the eventual fragmentation of the budget over multiple actions with similar objectives

Table 17 Portfolio analysis levels

Policy instruments

Implementation modes

Thematic areas

Financial instrument

Information instruments

Knowledge & networking instruments

Ad hoc grant Art. 190 RAP

Administrative arrangement

Calls for tender (service contracts)

Direct contracts

FWC (specific contracts)

Grants

Indirect Management

Membership fees

Not defined

Reimbursement experts / members

Remuneration experts / evaluators

Specific grant agreement under Framework

Partnership Agreement

Business mgt capacity

Digitisation & KETs

Unternehmertum

Information campaign

Internationalisation

Regulations & administrative burden

Responsible innovation

Sectoral competitiveness

SME policy

Support to access finance

Source: for financial instruments: EIF quarterly report Q4/2016; for EEN and EYE: databases provided by COSME

3.3.Composition analysis

The composition analysis, i.e. the profiling of the stakeholders involved in COSME, has been conducted; however, the predominant use of agreements for the allocation of the COSME budget, in particular their delegation agreements (indirect management) or Framework Partnership Agreements, constituted a challenge for the identification and analysis of the final beneficiaries’ profile.

This analysis was therefore conditioned by the integration of the COSME data hub with the databases related to the specific actions (financial instruments, EEN, EYE, etc.) as well as, most importantly, by the availability of detailed information on the stakeholders involved, covering the intended dimensions for analysis. Particularly in the case of the type of stakeholders involved the original databases were insufficient for composition analysis due to the lack of a predetermined typology of stakeholders. To amend this, database entries were manually checked in order to attribute stakeholders to the typology designed for this evaluation (i.e. industry, industry association, association, public administration/agency, research/RTO).

Similarly, the sectoral representation in the original databases was insufficient for the composition analysis due to the absence of a systematic sectoral identifier, such as the NACE rev.2 classification, which was only available for SO1. To amend this, a concordance with NACE rev.2 codes was developed. It should be noted, however, that the concordance provides only for a rough approximation, given that some sectors do not correspond to a NACE rev.2 classification (e.g. Organic/Bio/Eco products and services, etc.) or some of the entries were too broad and required a matching with multiple NACE rev.2 levels (e.g. Mining, machinery for mining, basic metals and related products: B + C28.92 + C24).

This analysis encountered difficulties in terms of data availability related to the types of stakeholders and sectors of activity. No data was available in relation to gender, except for the EYE participants.

The information available by Action Line is described in Table 18 and Table 19 , below.

Table 18 Overview of available data on beneficiaries

SO	Standort	Type of stakeholder	Sector	Budget	Gender
Financial instruments	Yes	Yes	Yes	Yes	No
EEN services	Yes	No	Yes	Not applicable: EEN network member organisations receive funding to provide services to final beneficiaries	No
Cluster Internationalisation/ Tourism (grants)	Yes for type of call=calls for proposals	Yes	Yes for type of call=calls for proposals	Yes for type of call=calls for proposals	No
EYE	Yes	Yes	Yes for Host entrepreneurs. It is not applicable for New entrepreneurs.	Yes for New Entrepreneurs. This is a fixed monthly amount adjusted by country. Host Entrepreneurs do not receive funding.	Yes

Source: for financial instruments: EIF quarterly report Q4/2016; for Cluster Internationalisation: European Cluster Collaboration Platform; for EEN and EYE: databases provided by COSME; for Tourism: COSME data-hub.

Table 19 Overview of available data on intermediaries

SO	Location (country)	Type of stakeholder	Budget	Gender
Financial instruments	Yes	Yes	Yes	No
EEN services	Yes	Yes	Yes	No
EYE	Yes	Yes	No	No

Source: for financial instruments: EIF quarterly report Q4/2016; for EEN and EYE: databases provided by COSME.

Additional analyses on geographical distribution and sectoral involvement are available in Appendix B.

3.4.Analysis of COSME programme management and proposal data

The time to grant is measured as the number of months between the deadline for the submission of proposals and the signature of grants for successful proposals. The empirical scope of the time-to-grant analyses covers the calls listed in Table 20 .

Table 20 Calls on which time to grant analysis is based

Business mgt capacity

EEN Administration & services

COS-Adhoc-2014-2020

COS-EEN-2014-2-01 FPA

COS-EEN-2014-2-04

COS-EEN-SGA-16-B-07-2016-1 SCALE UP CALL

COS-EEN-SGA2-2016-2-01

EEN-SGA-COSME-2015-01-1

Unternehmertum

Entrepreneurship education

COS-ENTEDU-2014-4-06

EYE

COS-EYE-2014-4-05

COS-EYE-2015-4-01

COS-EYE-FPA-2016-4-01

COS-EYE-SGA-2016-4-01

Migrants

COS-MigrantsENT-2016-4-02

Second starters

COS-EarlyWarningEU-2016-4-01

Internationalisation

Cluster internationalisation

COS-CLUSTER-2014-3-03

IPR Helpdesks

COS-IPR-2014-2-05

SME Internationalisation

COS-ETPO-2015-2-01

Regulations & administrative burden

Access to public procurement

COS-APP-2016-2-05

Access to regulations

COS-RPAS-2014-2-03

Privacy

COS-DRONES-2016-03-02

Single points of contact

COS-SPOC-2016-02-03

Sectoral competitiveness

Clusters

COS-CLUSTER-2014-3-04-02-1

COS-CLUSTER-2015-3-02

Design-based consumer goods

COS-DESIGN-2015-3-03-1

COS-DESIGN-2015-3-06

Einzelhandel

COS-IPPDS-2015-2-02-1

Tourism

COS-TEDEN-2014-3-15

COS-TEDEN-2015-3-05

COS-TFLOWS-2014-3-15

COS-TOUR-2015-3-04-1

COS-TSUST-2014-3-15

COS-TSYNER-2014-3-15

SME policy

SBA implementation

COS-DCFB-2015-3-01-1

Source: Technopolis elaboration based on raw data provided by COSME

The table below presents a complete and detailed time-to-grant analysis for each action line (compared to the tables in the core body of the report, where only the main action lines are listed). For transparency purposes, the table includes also the action lines for which data are not available, in order to provide a complete overview of the data provided.

Table 21 Time to grant by thematic area and action line

Themes	Action lines	Average Time to grant in days	Average Time to grant in months (assumption month=30 days)	Min Time to grant in months (assumption month=30 days)	Max Time to grant in months (assumption month=30 days)
Business mgt capacity	EEN Administration & services	234.8	7.8	3.9	26.8
Business mgt capacity overall average		234.8	7.8	3.9	26.8
Unternehmertum	Entrepreneurship education	209	7.0	7.0	7.0
	EYE	218.6	7.3	3.8	10.0
	Migrants
	Second starters	178	5.9	5.9	5.9
Entrepreneurship overall average		217.6	7.3	3.8	10.0
Internationalisation	Cluster internationalisation	222.6	7.4	6.7	8.0
	IPR Helpdesks	190.7	6.4	2.7	10.0
	SME Internationalisation	114.5	3.8	3.7	4.0
Internationalisation overall average		205.3	6.8	2.7	10.0
Regulations & administrative burden	Access to public procurement	277.5	9.3	8.3	10.5
	Access to regulations	226	7.5	7.5	7.5
	Privacy
	Single points of contact	147	4.9	4.9	4.9
Regulations & administrative burden overall average		247.2	8.2	4.9	10.5
Sectoral competitiveness	Clusters	181.5	6.0	5.4	7.0
	Design-based consumer goods	235.4	7.8	7.6	8.3
	Einzelhandel	174	5.8	5.8	5.8
	Tourism	250	8.3	5.6	14.1
Sectoral competitiveness overall average		239.9	8.0	5.4	14.1
SME policy	SBA implementation	215.4	7.2	6.1	9.1
SME policy overall average		215.4	7.2	6.1	9.1
Grand Total (overall average for all thematic areas)		232.8	7.8	2.7	26.8

Source: Technopolis elaboration based on raw data provided by COSME

3.5.Cost-Effectiveness analysis

Our indicators reflect the number of outputs and outcomes produced by the (million) Euros invested (e.g. “how many SMEs have been effectively supported by Euro spent?”). This approach provides metrics on efficiency that can be compared across actions.

To conduct the analysis, we have brought together the costs and benefits reported for each action. The analysis focus on four main actions: LGF (Access to finance), EEN (Business Management Capacity), Cluster Go International (Internationalisation), and EYE (Entrepreneurship). Together these four actions account for 62% of the budget committed to COSME in 2014-2016 (EUR 547 million out of EUR 879 million).

In terms of costs, we have considered the (i) direct cost to the programme (i.e. budget committed to COSME in 2014-2016 and administrative costs paid to EASME and EIF), and (ii) the indirect cost to beneficiaries to account for the cost to the ‘system’. In the case of EEN and Cluster Go International this includes the co-funding from partner / member organisations; in the case of EYE this includes additional costs incurred by the host and new entrepreneurs; and in the case of LGF this includes the additional guarantee provided under EFSI. Two measures of CEA indicators have been calculated based on those costs (with and without indirect costs). Only the indicators that include indirect costs are presented in the Main Report.

In terms of benefits, we have arrived at grossed-up estimates of benefits considering information from existing monitoring data (such as the population of beneficiaries) and the results obtained from our surveys.

As such, the analysis does not pretend to be exhaustive. In terms of cost, the analysis focuses mainly on the costs and investments sustained by the EC and participant companies / organisations. The analysis does not consider the potential costs to wider society. Similarly, the analysis of benefits focuses on the impact on participant organisations and does not include the analysis of externalities (i.e. the potential benefits on organisations and on communities not directly involved in any of the specific programmes).

We have mainly used four sources of information in our analysis:

·Information on budget committed (and additional EFSI guarantee used in the case of LGF)

·Information on administrative costs paid to EASME (proportionally distributed across actions to equate to 6.6% of budget committed) and EIF

·Monitoring data from the different actions, which have provided figures on total participations and have been used to arrive at grossed-up estimates.

·Our beneficiary surveys (including, the LGF SME survey, EEN SME client survey, Cluster organisation survey, HE surveys and NE surveys), which have provided key parameters for our estimations, including indirect costs

We have reached estimates for a total of 22 indicators. The section below provides full overview of the costs and benefits mapped for each of those actions. In the case of LGF we have estimated 8 additional indicators to account for the fact that the current 2014-2016 budget envelope (from COSME and EFSI guarantee) will continue to deliver results as the financial intermediaries have 2 to 3 years to build up loan portfolios under the guarantee agreements signed under 2014-2016 budget envelope. So far, a total of 143,000 SMEs has been supported, and it is expected that a total of 300,000 SMEs will be supported under guarantee agreements signed with financial intermediaries within the same budget envelope.

Table 22 Cost-effectiveness analysis

Note that any discrepancies between the parameters and the final figures are due to rounding.

CEA Indicator	Costs		Vorteile	CEA Indicator I		CEA Indicator II
Description	Direct (Budget committed) Value (mEUR ) [+ administrative fees]	Indirect Value (mEUR ) (Cost to the system)	Value / unit	Value /unit per EUR million invested		Value /unit per EUR million invested
	[A]	[B]	[C]	[C]/ [A]		[C] / ([A] + [B])
Support to access finance – LGF (achieved so far)
[1] Number of SMEs supported through COSME financing, per EUR million invested	COSME Budget committed: 349.2 mEUR [+26.3 mEUR ] COSME Budget committed: 349.2 mEUR [+26.3 mEUR ]	EFSI guarantee: 262.4 mEUR No data on the value of costs for SMEs has been collected however evidence suggests that additional costs to SMEs are small EFSI guarantee: 262.4 mEUR No data on the value of costs for SMEs has been collected however evidence suggests that additional costs to SMEs are small	143,344 SMEs SMEs supported so far as reported in monitoring data	381.7		224.7
[17] Number of SMEs supported through COSME financing (that would not have had access to other sources of funding), per EUR million invested			90,307 SMEs 143,344 SMEs multiplied by the percentage of SMEs that stated that the financing supported by a EU-COSME guarantee was the only option available or that other options were available but did not cover the full amount (63% according to the LGF SME survey)	240.5		141.6
[18] Value of credits leverage, per EUR million invested			5,547.2 mEUR Value of Principal Amount as reported in monitoring data	14.8		8.7
[19] Value of private funds leverage, per EUR million invested			1,247.6 mEUR Grossed-up figures based value of private funds leveraged, as a percentage of Principal Amount (as reported by survey respondents) multiplied by the value of the Principal Amount, considering size of the company.	3.3		2.0
[5] Increased turnover by companies that had access to LGF (in EUR ), per million EUR invested			10,653 mEUR Average value of turnover at the time of first inclusion (362 kEUR ), multiplied by the average growth on turnover (20. 5%) as reported in the “COSME Loan Guarantee Facility – Employment and Growth report, 2015”. This figure is then grossed-up by multiplying the average value of the increased employment with the total number of SMEs (143,344)	28.4		16.7
[11] Increased employment by companies that had access to LGF, per million EUR invested			237,431 people Average number of employees at the time of first inclusion (3.9), multiplied by the average growth on employment (42.7%) as reported in the “COSME Loan Guarantee Facility – Employment and Growth report, 2015”. This figure is then grossed-up by multiplying the average value of the increased employment with the total number of SMEs (143,344)	632.3		372.2
[8] (Additional) Increased turnover by companies that had access to LGF (in EUR ), per million EUR invested			6,711.3 mEUR Grossed-up value of increased turnover (as shown in indicator [5]) multiplied by % of SMEs that stated that ‘the financing supported by a EU-COSME guarantee was the only option available or that other options were available but did not cover the full amount’ (63% according to the LGF SME survey)	17.9		10.5
[14] (Additional) Increased employment by companies that had access to LGF, per million EUR invested			149,581 people Grossed-up value of increased employment (as shown in indicator [11]) multiplied by % of SMEs that stated that the financing supported by a EU-COSME guarantee was the only option available or that other options were available but did not cover the full amount (63% according to the LGF SME survey)	398.3		234.5
Support to access finance – LGF (expected)
[1b] Number of SMEs supported through COSME financing, per EUR million invested	COSME Budget committed: 349.2 mEUR [+26.3 mEUR ]	EFSI guarantee: 262.4 mEUR No data on the value of costs for SMEs has been collected however evidence suggests that additional costs to SMEs are small	300,000 SMEs expected to be supported As estimated by the European Commission. All the estimates below use a rule of three to recalculate the estimates based on 143,000 SMEs. In practice, all the above indicators are multiplied by a factor of 2.09.	798.9		470.3
[17b] Number of SMEs supported through COSME financing (that would not have had access to other sources of funding), per EUR million invested			189,000 SMEs 300,000 SMEs multiplied by % of SMEs that stated that the financing supported by a EU-COSME guarantee was the only option available or that other options were available but did not cover the full amount (63% according to the LGF SME survey)	503.3		296.3
[18b] Value of credits leverage, per EUR million invested			18,904.7 mEUR Based on EIF Q4 2016 operational report. A simple ‘rule of three’ gives a value of 11bnEUR , but we use the 18bn figure as it considers the real implementation conditions under each Guarantee Agreement with the respective Financial Intermediaries and (ii) it is based also on a previous experience gained in the implementation of similar debt financial instruments.	50.3		29.6
[19b] Value of private funds leverage, per EUR million invested			2,611.0 mEUR Grossed-up figures based on the average value of private funds leveraged by survey respondents and number of SMEs expected to be supported (300,000), considering size of the company	7.0		4.1
[5b] Increased turnover by companies that had access to LGF (in EUR ), per million EUR invested			22,295.1 mEUR Average value of turnover at the time of first inclusion (362 kEUR ), multiplied by the average growth on turnover (20. 5%) as reported in the “COSME Loan Guarantee Facility – Employment and Growth report, 2015”. This figure is then grossed-up by multiplying the average value of the increased employment with the total number of SMEs expected to be supported (300,000)	59.4		34.9
[11b] Increased employment by companies that had access to LGF, per million EUR invested			496,911 people Average number of employees at the time of first inclusion (3.9), multiplied by the average growth on employment (42.7%) as reported in the “COSME Loan Guarantee Facility – Employment and Growth report, 2015”. This figure is then grossed-up by multiplying the average value of the increased employment with the total number of SMEs expected to be supported (300,000)	1,323.2		778.9
[8b] (Additional) Increased turnover by companies that had access to LGF (in EUR ), per million EUR invested			14,045.9 mEUR Grossed-up value of increased turnover (as shown in indicator [5]) multiplied by % of SMEs that stated that ‘the financing supported by a EU-COSME guarantee was the only option available or that other options were available but did not cover the full amount’ (63% according to the LGF SME survey)	37.4		22.0
[14b] (Additional) Increased employment by companies that had access to LGF, per million EUR invested			313,054 people Grossed-up value of increased employment (as shown in indicator [11]) multiplied by % of SMEs that stated that the financing supported by a EU-COSME guarantee was the only option available or that other options were available but did not cover the full amount (63% according to the LGF SME survey)	833.6		490.7
Business management capacity - EEN
[2] Number of SMEs reached and supported, per EUR million invested	COSME Budget committed: 144.5 mEUR [+ 9.6 mEUR ]	57.8 mEUR [Based on co-funding from EEN members, equivalent to 40% of the EC contribution]	210,000 SMEs Estimations prepared in the context of this evaluation	1363		991
[6] Increase in turnover linked to services provided by EEN (in EUR ), per million EUR invested			34,785 mEUR Grossed-up figure of turnover based on average turnover growth per SMEs in the period 2014-2016, as reported in the EEN SME client survey (696 kEUR ), multiplied by the total number of SMEs supported (50,000 SMEs).	949		690
[12] (Additional) Increase in turnover linked to services provided by EEN (in EUR ), per million EUR invested			1,739.2 mEUR Grossed-up figure of turnover (as shown in indicator [6]) multiplied by an additionality factor of 6.5%. The additionality factor is based on our calculations on growth presented in EEN Thematic Report, where we show that turnover among EEN SME clients has grown 6.5 percentage points more than a benchmark group. This estimate excludes outliers and considers death rates.	2727		1983
[9] Increase in employment linked to services provided by EEN, per million EUR invested			200,000 employees Grossed-up figure of employment based on average growth on number of employees per SMEs in the period 2014-2016, as reported in the EEN SME client survey (4 multiplied by the total number of SMEs supported (50,000 SMEs).	62		45
[15] (Additional) Increase in employment linked to services provided by EEN, per million EUR invested			19,000 employees Grossed-up figure of turnover (as described above in indicator [9]) multiplied by an additionality factor of 9.5%. The additionality factor is based on our calculations on growth presented in EEN Thematic Report, where we show that employment among EEN SME clients has grown 9.5 percentage points more than a benchmark group. This estimate excludes outliers and considers death rates.	518		377
Internationalisation - Cluster Go International
[20] Number of cluster supported, per EUR million invested	2.3 mEUR [+ 0.1 mEUR ]	0.6 mEUR [Based on co-funding from partners, as funded ESCP received 75% co-funding from the EC]	93 Based on number of partners as reported in ESCP-4i website	38.8		31.4
[3] Number of SMEs supported by Cluster, per EUR million invested			11,246 Based on estimated number of SMEs supported in Strand 1 and 2, according to ESCP-4i 2nd synthesis report	4,998.2		3,798.0
Entrepreneurship - EYE
[4] Number of SMEs supported, per million EUR invested	24.3 mEUR [+ 1.6 mEUR ]	12.2 mEUR [Based Calculated based on the information provided by HE and NE via the survey, and the number of HE and NE involved in the EYE in the period of analysis]	3,737 2,117 New entrepreneurs and 1,620 Host Entrepreneurs, as reported in monitoring data	144.4	98.0
[21] Number of new entrepreneurs that have created their own business (thanks in part to their participation in EYE), per million EUR invested			241 New entrepreneurs Based on percentage of NEs creating a business according to our survey (11.4%), multiplied by the number of new entrepreneurs (2,117). Our calculation (11.4%) includes those NE that have reported turnover in 2016 (ie had a business generating turnover in 2016).	9.3	6.3
[22] Number of entrepreneurs engaging in at least one business cooperation agreement, per million EUR invested			993 Entrepreneurs Based on percentage of NE engaging in at least one business cooperation agreement according to our survey (46.9%) multiplied by the number of new entrepreneurs (2,117)	38.4	26.0
[7] Increase in turnover linked to EYE (in EUR ), per million EUR invested			209.0 mEUR Grossed-up turnover figures reported in the HE and NE surveys, and number of HE and NE. The figure includes the increase in total turnover for both HE (203.8 mEUR ) and NE (5.2 mEUR ). The estimated average increase in turnover for HE is 125.7 kEUR and for NE is 2.5 kEUR ]	8.1	5.5
[13] Increase in employment linked to EYE, per million EUR invested			122.2 mEUR Based on grossed-up figure of turnover and an additionality factor of 58% and 73% for HE and NE respectively. These factors are calculated based on survey responses. 58%of HE that agree that NE contributed to growth in companies' turnover, while 73% % of NE that agree that EYE has helped them to establish a new business. We use these parameters and assume that in 58 per cent of the cases attribution for growth is 100% while in 47 per cent attribution is 0%. A similar assumption is applied to NE	85.6	58.1
[10] (Additional) Increase in turnover linked to EYE (in EUR ), per million EUR invested			2,216 employees Grossed-up turnover figures reported in the HE and NE surveys	4.7	3.2
[16] (Additional) Increase in employment linked to EYE, per million EUR invested			1,247 employees Based on grossed-up figure of turnover and an additionality factor of 41% and 73% for HE and NE respectively. These factors are calculated based on survey responses. 41%of HE that agree that NE contributed to growth in companies' employment. In the case of NE we use the same measure for turnover. We use these parameters and assume that in 41 per cent of the cases attribution for growth is 100% while in 59 per cent attribution is 0%. A similar assumption is applied to NE.	48.2	32.7

Source: Technopolis (2017)

4.The consultations

This chapter presents the different types of consultation we used. We begin with the quantitative targeted consultations (4.1): first an overview, then the sampling strategy, then the results for each SO. In section 4.2 we present the qualitative targeted consultations. The last section deals with the public question (0): its structure, its implementation, the outcomes and its limits.

4.1.The targeted consultations – quantitative

4.1.1.Overview

This section presents an overview of the targeted consultation, specifically the seven ‘quantitative’ surveys that have informed this evaluation.

Table 23 presents an overview of the surveys, including the target group, population, target sample, responses and response rates, while a more detailed reporting on the sampling methods, data collection process and response rates is provided in the sub-sections below.

All draft questionnaires were included in the Inception Report and presented in further iterations; final approvals of the questionnaires were obtained from the corresponding contact points at the EC. The final questionnaires are provided in Error! Reference source not found. to this report.

Table 23 Overview – Quantitative surveys

#	Action	Type of stakeholder	Population	Target sample	Number of responses	Response rate	Closing date in 2017
1	Loan Guarantee Facility	Beneficiary SMEs	70,000	3,870	356*	9%	18 July
2	Enterprise Europe Network (EEN)	EEN member organisations	523**	530	270	48%	17 May
3	Enterprise Europe Network (EEN)	Beneficiary SMEs (EEN clients)	210,000	50,000	2,334	5%	17 May
4	Cluster internationalisation	Intermediary agencies and cluster organisations	136	136	48	35%	5 May
5	Tourism (Calls)	Beneficiary SMEs and Intermediary agencies / organisations	377	389	162	41%	10 May
6	Erasmus for Young Entrepreneurs	New Entrepreneurs	2,343	2,117	595	28%	5 May
7	Erasmus for Young Entrepreneurs	Host Entrepreneurs	1,800	1,620	245	15%	5 May

Source: Technopolis - *Reached via 20 financial intermediaries. **Reached via 92 coordinators.

4.1.2.Sampling strategy and results

The following paragraphs explain our sampling approach for the quantitative surveys. In the case of the short surveys no sampling is needed as all stakeholders will be approached.

We have taken careful consideration of potential issues related to ‘survey fatigue’ when designing our sampling strategy.

SO1 - Sampling of the beneficiary SMEs of Loan Guarantee Facility

The ToR provides a series of good recommendations regarding the sampling strategy for the financial instruments. There are also lessons to learn (in terms of expected response rates) from the “Final Evaluation of the Entrepreneurship and Innovation Programme” (2011) 87 . Taking those elements into account, we estimated that we would need to address 70,000 companies to obtain a response of 3,150.

We requested EIF/ DG Grow for access to the full list of LGF recipient companies, including all contact details necessary for implementing the survey (name, email address, telephone number).

We followed the following steps:

·We drew a stratified random sample of circa 70,000 companies from the amended list and invite them to participate in our (targeted) consultation. Stratification ensured the sample has a good spread across countries and sectors (provided this information is recorded alongside contact details).

·To arrive to a sample of 3,870 companies, we needed an (implicit) response rate of 5.6%, which was virtually the same as the response rate from the EIP. This sample size guaranteed a good representation across sectors and countries.

·A response of 3,150 represented a representative sample, with 95% confidence interval and a margin of error of +/- 6%.

We monitored responses in real time to make sure we are achieving a good spread of responses.

The Commission signed off on the survey of stakeholders of the SME Loan Guarantee Facility on 5 May 2017.

The survey was translated into 16 languages and released to an estimated 3,870 beneficiaries via 20 financial intermediaries in 19 countries. The study team constructed a random sample of 250 beneficiary companies for those intermediaries that had served a greater number of clients. Those intermediaries that had served fewer than 250 clients were asked to distribute the survey to all beneficiary clients.

Via this method, the companies that had received financing supported by an EU-COSME guarantee were originally invited to contribute to the study via an online survey between 19 May and 23 June 2017. Subsequent surveys were launched over the following weeks; to boost the number of responses and facilitate participation, the survey deadline was first extended to 30 June and then to 7 July 2017.

The survey received 356 responses, equivalent to a response rate of 9%.

SO2 - Sampling of the EEN member organisations

We approached EEN members through a broad call that was distributed by the EEN coordinators. We received access to the full list of EEN members, including all contact details necessary for implementing the survey (name, email address).

Enterprise Europe Network member organisations were originally invited to contribute to the study via an online survey between 12 April and 3 May 2017. The survey was initially released to 92 coordinators, and, to boost the number of responses and ease of participation, the survey was subsequently released to 431 partner organisations on 19 April. A further reminder e-mail was circulated to both groups on 26 April.

Over the course of implementation of the survey, the study team worked directly with both coordinators and partners to ensure that all relevant parties received the survey invitation. This resulted in additional partners receiving the invitation that were not initially included in the data, and subsequent greater access to client groups for the second survey. It also ensured that the survey has reached individuals that had replaced colleagues that were no longer in relevant positions.

Survey progress was briefly interrupted when the EEN members communicated through the SAG bureau that the timing of the survey was a bit inconvenient given the holiday period, and that they had received word from EEN members that were not in the position to send out the SME survey in time. An additional reminder was set out with May 17 as a new end date of the survey, providing the EEN members with an additional opportunity to participate in the survey and to send out the SME survey to their clients.

Overall, we received 270 responses, accounting for 48% of the target sample (the highest response rate of the targeted quantitative consultations).

SO2 - Sampling of the EEN client SMEs

All 92 coordinators were approached in the first instance, with a request to distribute details of our survey of client SMEs through the network.

Working closely with these coordinators we reached out to most of EEN’s 523 organisations, both within and outside of the EU. Through these organisations, we also issued further requests to around 50,000 client SMEs. Based on our experience of the previous EEN evaluation we expected the response rate from EEN member agencies to be high. At the same time, we expected SME response rates to be much lower, and we aimed for a ~5% response rate from those contacted.

Prior to the dissemination of both surveys, a letter from the Head of Unit H2 was posted on the Enterprise Europe Network intranet, advising Network members of the upcoming consultation, and seeking their support to disseminate the SME survey to their client base. The time available to answer the survey was extended once to accommodate for the busy schedules of the stakeholders.

Enterprise Europe Network client SMEs were also invited to participate in the study via an online survey questionnaire. The survey of client SMEs was disseminated as a cascade survey, i.e. Network members invited to participate in the first survey (above) were asked to support the study in distributing an open web link to their client base.

As with the survey of Network members, the survey of client SMEs was released in two phases, the first on 12 April 2017, when the study team wrote to all individual Network coordinators following their own survey invitation, and subsequently, on 19 April, when the study team wrote to the group of Network partners, following their own invitation to participate in the survey of Network members. We provided a text that Network members could forward to their own clients to participate in the study.

The study team worked closely with Network members to ensure as high as possible participation of SMEs in the survey. To accommodate national holidays in many countries, the study team further wrote to all 530 coordinators and partners on 26 April 2017. The survey closed at the end of 17th May.

Of the 50,000 SMEs of the target sample, we received 2,334 responses or 5%. Though rather low, this result met our expectations.

SO3 - Cluster organisations

We addressed the 136 cluster organisations that are funded through the Cluster Internationalisation calls. We requested access to the full list of cluster organisations, including all contact details necessary for implementing the survey (name, email address).

In our survey of cluster organisations in 2014, in the context of the “Evaluation of cluster initiatives managed by DG Enterprise and Industry”, we had obtained a response rate of circa 8%. This time, we expected a higher response rate as the 136 clusters had only recently received support from COSME. We therefore aimed at reaching a response rate of at least 25-30%, or 34-40 responses.

We received 48 answers, or a 35% rate, thus higher than our expectations.

SO3 – Tourism actions

We addressed the 377 organisations that were successful in their applications to the open calls for proposals in the field of Tourism. The list of these beneficiaries is available in the COSME Data Hub.

Participants and beneficiaries of actions funded through COSME tourism calls were invited to contribute to the study via an online survey that took place between 26 April and 10 May 2017. To boost the response rate, the study team sent a reminder on 3 May 2017. The survey was closed on the 10 May 2017. In total, 389 participants and beneficiaries – both intermediary organisations and SMEs – across 33 countries were invited to contribute to the online survey consultation for this interim evaluation.

We received 162 answers, or a 41% rate.

SO4 - EYE Host entrepreneurs

As the key action funded within the entrepreneurship area, the EYE programme was the focus of targeted consultation surveys. Specifically, we invited all Host and New Entrepreneurs in the programme during the 2014-2016 period to provide information and feedback on their participation in this action. As was reported above, these surveys received a good response rate from both groups of beneficiaries (with 841 Host and New Entrepreneurs responding in total). Analysis of the responses to these questionnaires has fed into various parts of the evaluation, particularly around the effectiveness and efficiency of the programme. Follow-up interviews were also conducted with a number of the beneficiaries (as reported below).

We asked EASME/ DG Grow for access to the full list of Host entrepreneurs, including all contact details necessary for implementing the survey (name, email address).

To avoid survey fatigue, we selected (randomly) a proportion of the total number of Host entrepreneurs involved in EYE in the period 2013-2016. We reached a sample of 1,620 Host entrepreneurs. Based on the response rates obtained in the prior exercise, we expected to attain a response from 250 HE as a result. In the end, we received – in line with our expectations - 245 answers which make up for 15% of the target sample.

Host Entrepreneurs were invited to contribute to the study via an online survey that was open between 12 April 2017 – 5 May 2017. To boost the response rate, the study team sent out a single reminder on 26 April 2017.

SO4 - EYE New entrepreneurs

We asked EASME/ DG Grow for access to the full list of Host entrepreneurs, including all contact details necessary for implementing the survey (name, email address).

To avoid survey fatigue, we selected (randomly) a proportion of the total number of New entrepreneurs involved in EYE in the period 2013-2016. We reached a sample of 2,117 New entrepreneurs. Based on the response rates obtained in the prior exercise (10%), we expected to attain a response from 100 NE as a result. Of the 2,117 entrepreneurs contacted, we received 595 answers (much higher than our expectations) or a response rate of 28%.

New entrepreneurs were invited to contribute to the study via an online-survey between 12 April 2017 – 5 May 2017. To boost the response rate, the study team released a single reminder on 26th April 2017.

4.1.3.Robustness of the results

SO1 – Access to finance

Overall, the survey received 356 responses, equivalent to a response rate of 9%. This represents a higher response rate than recorded in previous exercises such as the 2011 Final Evaluation of the Entrepreneurship and Innovation Programme (where the response rate was approximately 6.3%). 88

Figure 8 below shows a comparison by country of the distribution of the LGF beneficiaries as a proportion of the population, of the target sample and the survey population, while Figure 9 , displays the distribution of LGF beneficiaries within sectors of activity. These figures illustrate the degree of the representativeness of the survey population, and we emphasise that the survey population should not be used to profile all beneficiaries.

The survey did not target all countries in which there are beneficiaries of EU-COSME guarantees. 89 For the most part, the shares of companies in each country are higher within the survey population than in the overall beneficiary population, except in Spain and Italy.

Overall, the distribution of the survey responses is well balanced in terms of country and sector of activity in comparison with the target sample. One notable exception is France, where despite our best efforts we did not receive any response as the financial intermediary did not circulate the survey before the deadline.

The distribution of sectors is largely consistent between the population and the survey respondents. This is particularly evident in the most prominent sectors, such as ‘Wholesale and retail trade; repair of motor vehicles and motorcycles’, and ‘Manufacturing’. 90

Figure 8 LGF beneficiaries: comparison of the country distribution of the whole population, target sample and survey population

Source: Technopolis, based on provided beneficiary details and survey data

Figure 9 LGF beneficiaries: comparison of the sector distribution of the whole population, target sample and survey population

Source: Technopolis, based on provided beneficiary details and survey data. NB[1]: there is a slight difference between the sectors in the provided information (where ‘Wholesale and retail trade’ is combined with ‘Repair of motor vehicles and motorcycles’ as per the NACE classification) and the list of sectors within the survey questionnaire, where these sectors are presented separately. The survey data responses have been combined for the purposes of this comparison. NB[2]: The reclassification of survey respondents resulted in the addition of one sector not originally offered in the survey questionnaire: ‘Water supply; sewerage; waste management and remediation activities’. This is not shown in this chart.

SO2 – Access to market

Survey of Network Members

In total, the survey targeted 531 Network members: 89 coordinators and 442 partners across 37 countries. At the time of reporting (May 17), the survey has received 270 responses from across 35 countries. This represents an overall response rate of 48%. The response rates from both coordinator (47%) and partner groups (48%) are roughly equal.

The distribution of the survey population is very similar to the target sample in its two main characteristics: country grouping and Network member role. Figure 10 and Figure 11 , below, summarise this.

Figure 10 Network member survey target sample and respondents, by country

Source: Technopolis, based on survey data. Base = 270

Figure 11 Network member survey target sample and respondents, by role

Source: Technopolis, based on survey data. Base = 270

Survey of Network client SMEs

The cascade approach meant that the study team did not have direct control over the number of SME clients that are invited to participate in the survey. We estimated the target sample for the client SME survey to be 50,000, and set out a target to achieve a response rate of 5% (2,500 client companies). The survey of SME clients received 2,334 SME responses across 34 countries, 91 representing a response rate of 5% from the estimated target population.

Over two thirds of respondents (1,639) are located within EU15 Members States, though there are sufficient respondents located in EU13 Member States (25%, 568 respondents) to analyse questions based on country groupings. The number of respondents from COSME-participating third countries (4%, 98 respondents) is too low to make meaningful analysis. The distribution of the survey respondent base is shown in Figure 12 , below.

Figure 12 Network client SME company survey respondents, by country group

Source: Technopolis, based on survey data. Base = 2,309

SO3 – Favourable environment / Cluster internationalisation

A total of 48 responses were received, equivalent to a 35% response rate, which is higher than the desired response rate of 25%.

The response rate varied between the two main groupings: ‘funded’ and ‘voluntary’ ( Figure 13 ). The response rate for funded organisations (51%) was higher than that for voluntary organisations (15%), which is to be expected, given the closer involvement of funded organisations with the programme.

Figure 13 Distribution of population in sample and responses between type of cluster organisation

Source: Technopolis

Respondents were asked to characterise the main sector of activity of their company. Respondents were spread fairly evenly across the sectors, with a relatively high representation within the information and communication (ICT) sector (16.7%). Nearly half of the respondents placed the activities of their company in the ‘Other (please specify)’ category, which refers to transport (aerospace and rail), sports, and high-tech industry.

SO3 – Favourable environment / Tourism

The survey received 162 responses from 32 countries 92 , equivalent to a 41% response rate. Respondents have been split into three main country groupings: EU15 countries (those that were Member States prior to 2004), EU13 countries (those that joined as Member States in 2004 and subsequently), and COSME-participating third countries. 93 Figure 14 , below, illustrates the high level of alignment between the country groups split between the target sample and respondent groups, for those for which this information was available.

Figure 14 Profile of target sample and SO3 tourism survey respondents, by country group

Source: Contact data provided by the Commission, and Technopolis, based on survey data

SO4 - Entrepreneurship

New entrepreneurs

In total, 2,117 New Entrepreneurs from 37 countries were invited to contribute to the survey consultation for this evaluation. In the three weeks that the survey was open, 595 responses were received, equivalent to a 28% response rate (and higher than the expected response rate of 25%, based on results obtained in prior studies).

Most countries in the sample population were represented by at least one respondent, apart from two EU countries, Malta and Luxembourg. However, it is worth noting that these countries only had two contacts each, or 0.1% of the sample population. The numbers for some countries are rather small for the purposes of analysis, so we divided them into three groups as explained above.

Figure 15 shows that the distribution of responses (among those respondents for which the geographical location is known) is very similar to that of the target sample when considering the three country groupings: pre-existing EU Member States (as of 2004), expansion countries (those joining the EU post-2004), and COSME-participating third countries.

Figure 15 Distribution of population in sample and responses by types of countries

Source: Contact data provided by the Commission, and Technopolis, based on survey data

Host entrepreneurs

In total, 1,620 Host Entrepreneurs from 36 countries were invited to contribute to the survey consultation for this evaluation. In the three weeks it was open, the survey received 245 responses, equivalent to a 15% response rate, which is the expected response rate (based on the results obtained in prior studies).

Respondents were asked to characterise the main sector of activity of their company. All but three of the categories provided were represented by at least one respondent. These three were: ‘Repair of motor vehicles and motorcycles’, ‘Electricity, gas, steam and air conditioning supply’, and ‘Mining and quarrying’. The highest share of respondents reported their main sector of activity as ‘Other’, followed by ‘Arts, entertainment, and recreation’ and ‘Information and communication (ICT)’.

Figure 16 shows that also in this case, the distribution of responses (among those respondents for which the geographical location is known) is very similar to that of the target sample when considering the three country groupings: EU13 MS, EU15 MS and COSME-participating third countries.

Figure 16 Distribution of population in sample and responses by types of countries

Source: Contact data provided by the Commission, and Technopolis, based on survey data

4.2.The targeted consultations - qualitative

4.2.1.Overview

The qualitative targeted consultations were used only for two SOs: SO1 on Access to finance and SO3 on Favourable environment. As mentioned above, this type of survey was implemented where the population was lower than 100.

Table 24 below presents an overview of the participation in the qualitative surveys.

Table 24 Overview of the participation in the qualitative surveys

		Target sample	Number of responses
SO1 – Access to finance	LGF intermediaries	21	16
	EFG intermediaries	12	4
SO3 – Favourable environment	SME Policy	28	15
	SME Policy Sectoral competitiveness	127 (according to the email list received from EC)	23
		72	7
	E-skills	31*	8

Source: Technopolis

4.2.2.Sampling strategy

SO1 - Sampling financial intermediaries and sub-intermediaries of Loan Guarantee Facility

Among the 61 financial intermediaries receiving LGF funds, we decided to approach 21 of them. Five of them (the financial intermediaries) had been already identified (cat 1 promotional banks and cat 2 guarantee societies) and other were selected according to their type (Promotional Institution, Guarantee Institution, Commercial Bank, Leasing Company), country of establishment, direct/counter Guarantees and risk category in order to have a wider and representative possible coverage; but also some of them who implemented as well CIP SMEG and overall 50% currently implementing also InnoFin SMEG. These gave us insights of the process and leverage effect.

4.2.3.Results for SO1 – Access to finance

Four qualitative surveys were conducted:

·qualitative surveys of the LGF and EFG intermediaries

·qualitative survey of the EFG SMES

·qualitative survey of the members of European Intermediary Organisations

A written questionnaire was sent via e-mail to 21 LGF intermediaries, asking the recipients to complete the questionnaire within two weeks. Two reminders were sent to LGF intermediaries that did not respond by the fixed deadline. Finally, the team received 16 completed questionnaires, corresponding to a response rate of 76%. One LGF intermediary explicitly refused to participate, while four intermediaries did not answer. Eight responding intermediaries provide direct guarantees for loans and another eight intermediaries provide counter guarantees.

A written survey of EFG intermediaries was conducted. The team originally planned to collect data from five EFG intermediaries. The responding EFG intermediaries make equity and equity-related investments in growth-oriented start-ups and SMEs all over Europe. The latter were asked to cascade the survey to selected SMEs 94 that received equity funding from the EFG instrument via the intermediary. It was planned that 12 SMEs were to be contacted by their respective intermediaries. However, only three out of five intermediaries completed the written questionnaire for EFG intermediaries. This also implied that only a limited number of SMEs that were beneficiaries of the EFG (EFG SMEs) could be reached. As a result, the team received only four completed questionnaires (out of 12) from EFG SMEs.

Five of the six contacted intermediary organisations were asked by the team to cascade the survey to their members. Except for one intermediary, the European intermediary organisations contacted their members prior to the interviews and collected information on their members’ opinions and views. In two cases, the interviews were even conducted in the form of phone conferences in which representatives of both the umbrella organisation and individual members of the intermediary organisations participated. In this way, the opinions and views expressed during the interviews directly reflect those of individual members.

4.2.4.Results for SO3 – Favourable environment

Table 25 Overview of status of qualitative surveys

Survey	Date of draft	Approval date	Launch of survey
SME Policy – SME Envoys	21.03.2017 - draft 1; 27.03.2017 – draft 2	27.03.2017	28.03.2017
SME Policy – SPR Working Group	27.03.2017	11.04.2017	11.04.2017
E-skills	10.04.2017	10.04.2017	28.04.2017
Sectoral competitiveness & KETS	10.04.2017	18.04.2017	02.05.2017

Table 26 Overview of responses to SO3 short survey consultations

Survey	Actors targeted	No of targeted actors	No of responses received
SME Policy	SME Envoys	28	15
	SPR Work group members	127 (according to the email list received from EC)	23
Sectoral competitiveness	European stakeholder organisations, national stakeholder organisations (associations), national and regional public authorities	72	7
E-skills	European stakeholder organisations, national stakeholder organisations dealing with e-skills	31*	8

Source: Technopolis - *out of the 53 e-mail addresses received, only 37 were valid. Out of them 6 contacts were contacted for interviews and 31 for the qualitative survey.

4.3.The public consultation

The objective of the public consultation was to cover all potential relevant stakeholders, including those who were not covered by the targeted consultations and interviews, and to collect feedback from them on the COSME programme and its objectives. More specifically, we expected inputs from:

·Citizens

·Companies

·European business organisations

·National business organisations

·Trade unions

·Non-governmental organisations

·National public authorities

·Regional/local public authorities

·Financial institutions/Finance providers

·Academia/Research centres

·Professional consultancies/law firms

·Incubators/clusters/innovation support centres

4.3.1.The structure of the public consultation questionnaire

Because public consultations are available in all EU languages, translation of open responses can become very time-consuming, can critically delay the treatment of responses, and undermines comparability between Member States. Therefore, we collected all questions by means of closed questions (multiple choices). As it is still important to provide stakeholders with the opportunity to express themselves in an open structure, we included an open question at the end of the questionnaire to collect additional information and all types of comments from the respondents.

4.3.2.The implementation of the public consultation

The approach of the public consultation followed the consultation process defined by the European Commission (Better Regulation Toolbox/Tool #50). Following an initial phone conversation with the client specifically regarding the public consultation in early February 2017, Technopolis Group submitted a draft questionnaire on the 24 March 2017 and feedback was shared by the European Commission on the 29 March 2017. A final questionnaire, resulting from an internal EC drafting and validation process, was validated by the end of April 2017.

As required in the Terms of References, the questionnaire was available in all EU languages. The Commission translation service (DGT) took care of the English version of the consultation and further translations in all other EU languages followed. The European Commission implemented the questionnaire for the public consultation on the “Your Voice in Europe” website and the questionnaire was available in English on 10 May 2017. It remained accessible for 12 weeks, starting from the date on which the last translation was made available online.

The dissemination of the public consultation fell under the responsibility of the European Commission. EASME and the EC’s communication units advertised the public consultation on COSME and EASME websites; through EEN networks and relevant national contact points; through other agencies and networks under COSME; and via other EC social media.

Responses have been monitored regularly (brief update reported by the EC every two weeks) to identify the types of stakeholders (for example, stakeholders in specific countries) from which fewer responses were received and to dedicate efforts to advertise the public consultation to them. After the end of the consultation period, the Commission forwarded to our team the raw data collected in the original language, in both Word and Excel formats.

Table 27 Overview of the open public consultation process

Activity	Timing
Phone call with the EC	10 February 2017
Delivery of first draft questionnaire	24 March 2017
Review by the EC	29 March 2017
Validation of final questionnaire	26 April 2017
Editing in EN by EC	26 April – 5 May 2017
Translation in other EU languages	6 May – 12 June
Deadline of the consultation	Launch date: 10 May Closing date: 31 August
EC to send inputs to Technopolis Group	Latest results received mid-September (13/09/2017)

Source: Technopolis

4.3.3.Outcomes of the public consultation

The public consultation was closed on the 31 August 2017. We received the final results by mid-September (13/09) 2017. A total of 195 responses were received, as well as 14 position papers.

The questionnaire also included questions that allowed us to profile the respondents to the public consultations.

Half of the respondents are active in industry/business; the second largest group is comprised of financial intermediaries (financial institutions or finance providers). The respondents under ‘others’ included law firms, research centres, innovation centres, NGOs and associations; business organisations; public authorities; and citizens (see Figure 4 ). Five countries account for two thirds of the answers (France, the UK, Italy, Spain and Germany).

Figure 17 Public consultation - profile of the participants

Source: COSME public consultation, 2017, n=195

Table 28 Number of respondents per type of financial support

Typ	Action	Number of beneficiaries
Access to finance	Loan Guarantee Facility	55 29 companies 16 financial institutions 10 others
	Equity Facility for Growth	5
Access to markets	Enterprise Europe Network (EEN) – EEN member organisations	15
	Enterprise Europe Network (EEN) – beneficiaries	13
Unternehmertum	Erasmus for Young Entrepreneurs – New Entrepreneurs	4
	Erasmus for Young Entrepreneurs – Host Entrepreneurs	4
Favourable environment	Cluster internationalisation	2
	Tourism (calls)	2
	Other Selected as an expert NCP for COSME including financial instruments and publishing calls for proposals in all areas / Coordinator EEN including extra support for EYE-participation COS-DESIGN	3

Source: COSME public consultation (n=88)

4.3.4.Statistical analysis

The survey responses were analysed using descriptive statistics.

The large number of respondents enabled us to undertake cross-analyses. We compared their views on the relevance of COSME actions to their views on the results (and the other way around) and to the benefits they expect from the programme (and the other way around). We also compared the expectations of the respondents with the areas of improvement they identified (and the other way around).

5.Interviews

In this chapter, after a brief overview (5.1), we present the results from our interview strategy (5.2), here again for each SO.

5.1.Overview

Overall, our team conducted 121 interviews. Table 29 provides an overview of the interviews conducted for each SO.

Table 29 Overview of the interviews conducted, by SO

SO	Interviewed stakeholders	Nr
SO1 – Access to finance	representatives from EU institutions (DG GROW, DG REGIO, DG RTD, DG ECFIN, EIF and EIB)	15
	intermediary and industry organisations	6
SO2 – Access to market	EU officials who were responsible for the Network and for other programmes aimed at fostering coherence within the Network.	9
	Network members	19
	Umbrella organisations were also interviewed so as gain the views of SMEs and intermediaries at a higher aggregated level.	5
SO3 – Favourable environment	EU officials in DG GROW, EASME and other DGs	16
	Beneficiaries (cluster managers)	6
	National government representatives	4
	Umbrella organisations and national experts	10
SO4 – Entrepreneurship	Individuals responsible for the design, management and oversight of COSME entrepreneurship actions – including European Commission officials, EASME staff and individuals within the EYE support office	20

	Host Entrepreneurs – participants in and beneficiaries of the EYE programme	5
	New Entrepreneurs - participants in and beneficiaries of the EYE programme	6

Source: Technopolis

5.2.Results

5.2.1.SO1 – Access to finance

The team conducted 21 phone interviews with high level stakeholders. These comprised 15 interviews with representatives from EU institutions (DG GROW, DG REGIO, DG RTD, DG ECFIN, EIF and EIB) and six interviews with intermediary and industry organisations. The rate of response was equivalent to 100%, i.e. all relevant stakeholders could be reached and interviewed as originally planned.

See Table 47 , Appendix D for the list of interviewees.

5.2.2.SO2 – Access to market

For the evaluation of the Access to Markets actions, the interviews conducted mainly targeted the Enterprise Europe Network and other programmes/actions related to the Network. We focused on interviewing EU officials who were responsible for the Network and for other programmes aimed at fostering coherence within the Network. We spoke to a large number of Network members in order to gain deep knowledge of how the Network works in practice. Finally umbrella organisations were also interviewed so as gain the views of SMEs and intermediaries at a higher aggregated level.

We have been successful in conducting interviews during the evaluation despite the coinciding holiday period. Almost all interviews that were foreseen were conducted during the time set out in the project planning, and two additional interviews were conducted to provide further insight. The list of the 34 interviews is shown in Table 48 , Appendix D.

5.2.3.SO3 – Favourable environment

For SO3, we have performed 36 interviews to cover the SME Policy and cluster internationalisation key actions, the case studies on tourism and e-skills, as well as the thematic areas such as KETs and sectoral competitiveness. Some interviews have been performed, due to various difficulties in reaching the interviewees. Bottlenecks encountered included the limited availability of some of the interviewees (e.g. in the field of cluster internationalisation and e-skills), as well as potential lack of interest or suitability to contribute. The latter was especially the case in the field of sectoral competitiveness and KETs, where stakeholders are not aware of COSME actions in this field and are reluctant to answer questions. Where possible, replacements were found, however, in most cases, it was not possible to find a replacement in due time.

An overview of the number of interviews performed is shown in Table 30 .

Table 30 SO3 – Performed interviews

		No. interviews conducted
EU officials	DG GROW	8
	EASME	2
	Other DGs	6
Beneficiaries (intermediaries)		6
National governments		4
Umbrella organisation & national experts		10
Total		36

Source: Technopolis

Table 49 , Appendix D lists the people with whom interviews were conducted for all the actions foreseen for the evaluation under the COSME SO3 work stream.

Table 50 (same appendix) lists the persons with whom interviews were planned initially, but not performed by the end of the evaluation due to lack of availability, refusal, or other reasons.

5.2.4.SO4 – Entrepreneurship

The team undertook interviews with three main groups of stakeholders, focusing mainly on the key entrepreneurship action (EYE), but also covering many of the other actions in the portfolio. A small number of interviews were also undertaken with other relevant stakeholders. The main groups of interviewees were:

·Individuals responsible for the design, management and oversight of COSME entrepreneurship actions – including European Commission officials, EASME staff and individuals within the EYE support office

·Intermediary organisations - which are responsible for matching, managing and monitoring EYE relationships between New and Host Entrepreneurs.

·Host Entrepreneurs – participants in and beneficiaries of the EYE programme

·New Entrepreneurs - participants in and beneficiaries of the EYE programme

Table 51 , Appendix D lists the 30 individuals interviewed in relation to the EYE programme and other entrepreneurship actions.

Appendix A- COSME programme KPIs, and the extent to which these KPIs have been realized

Table 31 Legend to Table 32 below

	COSME's Medium term KPIs' targets for 2017 met.
	KPIs have changed. Even though goal attainment cannot be measured, new indicators show promising developments.
	Targets not met or no info at all available

Table 32 Main activities, KPIs, and the extent to which these KPIs have been realised

Result indicator	Situation in 2011	COSME's Medium term KPIs for 2017	Did COSME meet the KPI targets?	Source
Activities to improve Competitiveness
Number of simplification measures adopted	The Commission's simplification programme was updated in 2010 and is on track to cut red tape by 25% in 2012. There were 5 simplification measures per year done until 2010.	About 7 simplification measures per year.	2014: 27% reduction of the burden	monitoring report
Number of "fitness" checks on quality and value-added of activities	Four "fitness" checks including stakeholders were launched in 2010 for environment, transport, employment and industrial policies. Feedback included comments on legislation and value-added of activities.	The feedback approach with "fitness" checks will be extended to other policies and lead to simplifications impacting positively on industry. Up to twelve "fitness" checks are foreseen, with the objective of better regulation.	2014-2016: 15 regulation impact assessments 2014: 8 2015: 4 2016: 3	Data provided by DG GROW
Level of adoption by companies of European sustainable production and product tools, including EMAS, eco-label, and eco-design	Approximately 35,000 ISO 14001 EMS certifications and 4,500 EMAS registration, 18,000 licences for the EU Ecolabel	Significant number of companies monitor their performance, apply environmental management systems and achieve improvement in resource productivity and environmental performance. Significant part of production are resource efficient and environmentally friendly products	(2005-)2015: 39,340 EMAS/ISO 14001 certifications	http://ec.europa.eu/environment/emas/emas_registrations/statistics_graphs_en.htm
Developing SME policy
Number of Member States using SME test	Number of Member States using SME test: 15 MS	Number of Member States using SME test: 21 MS	2015: "Of the 24 responding countries, 20 countries reported that they already had an SME-test in place"	EU MSs reporting about their SME test http://ec.europa.eu/DocsRoom/documents/19261
Increased EU-wide publicity of the European Enterprise Awards with media publications/clippings in all Member States	Number of media publications/clippings in all Member States: 60 in 2010	Number of media publications/clippings in all Member States: 80	2014: 6 2015: 15	Meltwater (maybe they have the data internally)
Reduction of start-up time and complexity for new enterprises	Reduction of start-up time: 7 working days	Reduction of start-up time: 5 working days	In 2015, the average time to start a private limited company in the EU was 3.4 days and the cost was EUR 315. The reduction in average time is mostly due to simplifications implemented in Austria, Bulgaria and Sweden. The slight augmentation of average cost is mostly due to increasing costs in Hungary	http://ec.europa.eu/growth/smes/promoting-entrepreneurship/advice-opportunities/start-up-procedures_en
New business concepts
Number of new products/services in the market	So far, this activity was restricted to analytical work of limited scale.	Target for the cumulative number of new products/services to be 5 in 2017 (increasing to 15 in 2018 and 25 in 2019).	Design-based consumer goods? 2015: EUR 11.2 Mio budget, 27 proposals, 4 awarded projects 2016: EUR 4 Mio, 26 proposals, 6 awarded projects	Call COSME Design Based Consumer Goods - Main outcomes & 10 success stories (EASME presentation)
Level of additional exports and corresponding monetary amounts		As for exports, no expected impact in 2017 yet. The share of exports of the first generation of participating SMEs will appear in 2018 with a target increase of 20%.	K.A.	K.A.
Feedback from stakeholders on quality and value-added of activities		At least 70% of SMEs participating in 2014 shall express a positive impact on their turn over in a survey done end 2017	Design-based consumer goods? 2015: "The stage of implementation does not allow measuring other relevant indicators"	monitoring report
Tourism
Number of applications to funding	Number of applications to funding (to all calls for proposals) in total: around 75 per year (average for 2011)	Number of applications to funding (to all calls for proposals) in total: more than 100 per year	2014: n/a 2015: 216	monitoring report
Percentage of SMEs (and trend) in applications for tourism-related funding opportunities	Up to date, no calls for proposals were directly addressed to SMEs	30% of calls for proposals directly addressed to/involving SMEs in partnerships	Most of calls for proposals (except ad-hoc grants) required the involvement of SMEs in the projects consortia. SMEs account for around 150 out of 377 organisations which were beneficiaries of the open calls for proposals in the field of tourism, which stand for roughly 40% of all beneficiaries. (Out of the SMEs beneficiaries around 100 companies were active in the tourism sector and additional 50 private enterprises were active in other sectors)	EASME/ COSME data hub
Number of destinations adopting the sustainable tourism development models promoted by the European Destinations of Excellence	Number of European Destinations of Excellence awarded in total 98 (on average 20 per year – in 2007- 10, in 2008-20, in 2009-22, in 2010-25, in 2011-21, 2013 - 19)	158 winners and up to 500 runners-up destinations adopting the sustainable tourism development models promoted by the European Destinations of Excellence initiative.	2014: new 18 destinations selected 2015: 13 projects on promotion of the EDEN destinations awarded 2016: new 17 destinations selected 2017: up to 17 projects on promotion of the EDEN destinations to be awarded	EASME
Support for entrepreneurship
Feedback on the public perception of entrepreneurship (% of EU citizens that would like to be self-employed as measured by Eurobarometer)	Figures from 2007 and 2009 are stable at 45%	Increase of EU citizens that would like to be self-employed to 50%	Most recent Eurobarometer on entrepreneurship is 2012 2012: 37% of EU27 would prefer to be self-employed 32% of EU27 find it desirable (fairly + very) to become self-employed within the next five years 53% of EU27 have a broadly favourable opinion of entrepreneurs	Flash Eurobarometer 354 (2012) (volume AA) http://data.europa.eu/euodp/en/data/dataset/S1024_354
Number of states implementing entrepreneurship solutions developed at EU level: 22 (2010)	Number of nationally run programmes available to SMEs from other MS: 5	Number of nationally run programmes available to SMEs from other MS: 10	Entrepreneurship Education? 2015: "to early to assess how many countries participated in the activities of the network"	monitoring report
Number of simplification measures adopted for SMEs	5 simplification measures per year (2010).	About 7 simplification measures per year	No information available	K.A.
Financial Instruments for growth
Number of firms receiving loan (credit) guarantees and value of lending	Proposed instruments are not yet launched and different from current instruments, so data from current instruments may not be comparable	Number of firms receiving loan (credit) guarantees (+/-95 000) and value of lending (+/- EUR 10.7 billon)	2014: 3 operations signed, EUR 851 Mio of financing to around 13,000 SMEs (leverage: 1 to 10) 2015 (aggregate): EUR 1,292 Mio of financing to more than 51,000 SMEs 2015 (expected): 24 operations, EUR 6,200 Mio of financing to around 95,000 SMEs (leverage: 1 to 30)	monitoring report
Number of VC-backed firms and value of investments (of which cross border deals)		Number of VC-backed firms: (+/- 180) and value of investments (+/- EUR 220m)	2014 (expected for the lifetime of the programme): EUR 2,6 to 4 billion, risk capital to 362 to 544 firms 2015: 5 operations, EUR 395 Mio, risk capital to 53 SMEs	monitoring report
Enterprise Europe Network
Number of partnership agreements signed	Partnership agreements signed: 1.950 (2010)	Partnership agreements signed: 3.000/year	2013-2014: 5,273 SMEs 2015-2016: 6.223 SMEs (2 years periods)	monitoring report
Increased recognized Network brand and brand Culture (e.g. brand awareness among SME population)	Not measured yet	30% of SMES reached	2014: 411,000 SMEs involved in promotion and information local events 2015: 8% recognition among SME population (Eurobarometer survey n°421) 2015: 10,5 mio SMEs reached via (digital) information services	monitoring report
Clients satisfaction rate (% SMEs stating satisfaction, added-value of specific service)	Clients satisfaction rate (% SMEs stating satisfaction, added-value of specific service): 78%	Clients satisfaction rate (% SMEs stating satisfaction, added-value of specific service): >80%	2014: 85% 2016 : 86% of respondents declared to be satisfied or very satisfied with EEN services 93% would recommend it to others	monitoring report
Number of SMEs receiving support services	Number of SMEs receiving support services: 435.000 (2010)	Number of SMEs receiving support services 250.000/year (revised indicator)	2014: 132,668 SMEs 2015-2016: 254.057 SMEs/Y	monitoring report
Number of SMEs participating in brokerage events and company missions	Number of SMEs participating in brokerage events and company missions: 45.000 (2010)	Number of SMEs participating in brokerage events and company missions: 60.000/year	2013-2014: 50,719 SMEs 2015-2016: 45.953 SMEs with 156.408 meetings	monitoring report
SME business support in markets outside the EU
Share (%) of SMEs involved in international activities (exports, imports, FDI and other activities) outside the EU)	13 % (2009)	17 % (2017)	2015: - Exported: 20% - Imported: 19% - Subcontractor based abroad: 7% - Worked as a subcontractor: 5% - Worked for R&D with a partner abroad: 4% - Invested in a company abroad: 2%	Flash Eurobarometer 421 (2015)
International Industrial Cooperation
Number of cases of improved alignment between EU and third countries’ regulations for industrial products	It is estimated that in regulatory co-operation with main trading partners (US, Japan, China, Brazil, Russia, Canada, India) there is an average of 2 relevant areas of significant alignment of technical regulations	3 relevant areas of significant alignment of technical regulations with main trading partners (US, Japan, China, Brazil, Russia, Canada, India) (2017)	No information available	K.A.
Number of areas and good practices of the EU Small Business Act which have been introduced in neighbourhood and candidate countries	It is estimated that on average in the three policy region (candidate countries region, neighbourhood East and neighbourhood MED) of the 10 policy areas of the SBA at least 3 of those have been regulated in these countries.	5 policy areas of the SBA in the three policy region (candidate countries region, neighbourhood East and neighbourhood MED) (2017)	No information available	K.A.

Source: Techonopolis

Appendix B- Statistical data analyses results

2.1 Analysis of the geographical distribution

The action lines considered in the analysis, for which data on final beneficiaries are available, are:

·LGF

·EFG

·EEN

·EYE

·Cluster Internationalisation

·Tourism

Table 33 below presents the penetration rate by country for the action line LGF. The penetration rate measures the percentage of SMEs, employees and SME turnover reached by LGF compared to total number of SMEs, employees and SME turnover in the country.

Table 33 Country penetration rates of LGF guarantee 2014-2016

LGF SMEs (all)					Penetration rates… based on:
Land	Total number of SMEs	Total number of employees (First Inclusion)	Total value of SME Turnover (First inclusion) (in EUR)	Average age	number of SMEs	number of employees	SME Turnover
Österreich	1,176	8,110	907,520,155	7.6	0.40%	0.40%	0.00%
Belgien	471	4,062	955,471,071	5.1	0.10%	0.20%	0.00%
Bulgarien	660	12,797	949,484,069	9.1	0.20%	0.90%	0.01%
Tschechische Republik	1,812	16,371	2,459,727,894	11.3	0.20%	0.70%	0.01%
Dänemark	209	840	6,404,467	6.9	0.10%	0.10%	0.00%
Estland	255	4,370	576,987,811	10.4	0.40%	1.30%	0.01%
Frankreich	51,047	155,539	8,577,346,400	9.5	1.60%	1.60%	0.00%
Deutschland	8,840	25,475	3,568,813,371	2.3	0.40%	0.10%	0.00%
Griechenland	322	3,972	523,598,067	16.1	0.00%	0.20%	0.00%
Ungarn	1,475	10,290	741,963,920	11.5	0.30%	0.60%	0.00%
Italien	28,518	142,168	19,111,421,957	13.3	0.80%	1.30%	0.00%
Lettland	67	1,453	111,387,715	12.8	0.10%	0.30%	0.00%
Litauen	71	3,770	253,300,460	13.2	0.00%	0.50%	0.00%
Montenegro	521	3,563	191,344,923	10	1.60%	3.80%	0.02%
Niederlande	440	916	50,546,318	5.3	0.00%	0.00%	0.00%
Polen	3,351	24,381	2,325,042,247	7.4	0.20%	0.40%	0.00%
Rumänien	642	6,665	408,234,103	10.8	0.10%	0.30%	0.00%
Slowakei	146	521	31,315,673	8.7	0.00%	0.00%	0.00%
Slowenien	392	8,767	723,330,826	15.6	0.30%	2.10%	0.01%
Spanien	40,166	102,973	7,287,511,069	6.8	1.70%	1.30%	0.00%
Vereinigtes Königreich	2,763	19,041	2,204,655,663	8.6	0.20%	0.20%	0.00%
Total	143,344	556,044	51,965,408,179	9	0.60%	0.70%	0.00%

Source: Technopolis, based on EIF monitoring data at end of 2016 and National statistics 2016 (2017)

Table 34 presents, for each action line analysed, the shares of beneficiaries belonging to a country aggregation (EU15, EU13, COSME 3rd countries and Other countries) over the total number of beneficiaries reached by that thematic area.

Table 34 Geographical distribution - shares by country aggregation of number of beneficiaries for 2014-2016

Action line	EU15	EU13	3rd countries	Other
Access to finance / LGF	80.9%	49.3%	18.6%
Access to finance / EFG	0.01%
EEN	11.1%	40.7%	73.5%	100.0%
EYE	2.4%	7.6%	7.7%
Cluster internationalisation	5.4%	1.8%
Tourism grants	0.2%	0.5%	0.2%
Grand total	100%	100%	100%	100%

Source: for LGF and EFG: EIF quarterly report Q4/2016; for EEN and EYE: databases from COSME; for Cluster Internationalisation: European Cluster Collaboration Platform. For all the other action lines: datahub. Note: the datahub only contains information on calls for proposal.

Table 35 reports the estimated number of beneficiaries reached in each EU country by the different action lines.

Table 35 Geographical distribution – estimated number of final recipients by country in EU for 2014-2016

Land	Access to finance / LGF	Access to finance / EFG	EEN	EYE	Cluster Internationalisation	Tourism grants
AT	1,176		351	125	250	12
BE	471		590	149	586	19
BG	660		553	60	114	8
CY	0		84	84	0	2
CZ	1,812		863	82	52	3
DE	8,840	2	2,630	345	1,050	9
DK	209		198	56	914	4
EE	255		177	30	0	3
EL	322	2	815	227	112	13
ES	40,166		3,174	951	1,375	57
FI	0		231	36	140	2
FR	51,047	4	2,449	174	2,171	21
HR	0		293	76	0	9
HU	1,475		692	84	43	9
IE	0	1	219	66	0	7
IT	28,518	1	2,011	1,025	879	80
LT	71		406	135	25	5
LU	0		19	9	0	1
LV	67		351	74	0	8
MT	0		67	10	0	6
NL	440	1	1,275	199	440	10
PL	3,351		2,013	246	88	2
PT	0		450	142	254	11
RO	642		1,025	253	0	5
SE	0		558	75	480	1
SI	392		413	126	0	25
SK	146		388	114	0	2
UK	2,763	1	3,392	393	301	18
Total	142,823	12	25,687	5,346	9,274	352

Source: for LGF and EFG : EIF quarterly report Q4/2016; for EEN and EYE: databases from COSME; for Cluster Internationalisation: European Cluster Collaboration Platform. For all the other action lines: datahub. Note: the datahub only contains information on calls for proposal.

Table 36 provides the percentages of beneficiaries reached by each action line, weighted by the number of SMEs in each country.

Table 36 Geographical distribution – weighted shares of participants by country in EU for 2014-2016

Land	Access to finance / LGF	Access to finance / EFG	EEN	EYE	Cluster Internationalisation	Tourism grants
AT	0.4%	0.0%	0.1%	0.0%	0.1%	0.0%
BE	0.1%	0.0%	0.1%	0.0%	0.1%	0.0%
BG	0.2%	0.0%	0.2%	0.0%	0.0%	0.0%
CY	0.0%	0.0%	0.2%	0.2%	0.0%	0.0%
CZ	0.2%	0.0%	0.1%	0.0%	0.0%	0.0%
DE	0.4%	0.0%	0.1%	0.0%	0.0%	0.0%
DK	0.1%	0.0%	0.1%	0.0%	0.4%	0.0%
EE	0.4%	0.0%	0.3%	0.0%	0.0%	0.0%
EL	0.0%	0.0%	0.1%	0.0%	0.0%	0.0%
ES	1.7%	0.0%	0.1%	0.0%	0.1%	0.0%
FI	0.0%	0.0%	0.1%	0.0%	0.1%	0.0%
FR	1.6%	0.0%	0.1%	0.0%	0.1%	0.0%
HR	0.0%	0.0%	0.2%	0.1%	0.0%	0.0%
HU	0.3%	0.0%	0.1%	0.0%	0.0%	0.0%
IE	0.0%	0.0%	0.1%	0.0%	0.0%	0.0%
IT	0.8%	0.0%	0.1%	0.0%	0.0%	0.0%
LT	0.0%	0.0%	0.2%	0.1%	0.0%	0.0%
LU	0.0%	0.0%	0.1%	0.0%	0.0%	0.0%
LV	0.1%	0.0%	0.4%	0.1%	0.0%	0.0%
MT	0.0%	0.0%	0.3%	0.0%	0.0%	0.0%
NL	0.0%	0.0%	0.1%	0.0%	0.0%	0.0%
PL	0.2%	0.0%	0.1%	0.0%	0.0%	0.0%
PT	0.0%	0.0%	0.1%	0.0%	0.0%	0.0%
RO	0.1%	0.0%	0.2%	0.1%	0.0%	0.0%
SE	0.0%	0.0%	0.1%	0.0%	0.1%	0.0%
SI	0.3%	0.0%	0.3%	0.1%	0.0%	0.0%
SK	0.0%	0.0%	0.1%	0.0%	0.0%	0.0%
UK	0.2%	0.0%	0.2%	0.0%	0.0%	0.0%
Total	0.6%	0.0%	0.1%	0.0%	0.0%	0.0%

Source: for LGF and EFG : EIF quarterly report Q4/2016; for EEN and EYE: databases from COSME; for Cluster Internationalisation: European Cluster Collaboration Platform. For all the other action lines: datahub. Note: the datahub only contains information on calls for proposal.

Table 37 shows the estimated shares of cumulative budget allocated for the action lines analysed, over the cumulative budget allocated in each group of countries. The cumulative budget is the budget allocated for each action line from January 2014 to December 2016.

The action line EYE is not analysed because the amount of money received by the final beneficiaries (the young entrepreneurs) depends on the country in which they work.

Table 37 Geographical distribution - shares by country aggregation of cumulative budget for 2014-2016

Action line	EU15	EU13	3rd countries
Access to finance / LGF	96.5%	96.3%	58.5%
Access to finance / EFG	1.5%
Business mgt capacity (EEN)	1.7%	3.1%	38.3%
Unternehmertum	0.2%	0.4%	1.6%
Tourism	0.1%	0.3%	1.1%
Grand total	100.0%	100.0%	100.0%

Source: for LGF and EFG : EIF quarterly report Q4/2016; for EEN and EYE: databases from COSME; For all the other action lines: datahub. Note: the datahub only contains information on calls for proposal.

Table 38 presents the total overview of budget allocated to final beneficiaries through the four action lines LGF, EFG, Tourism and EEN, between 2014 and 2016.

Table 38 Geographical distribution (money allocated to final beneficiaries in EUR k for 2014-2016)

Land	LGF*	EFG*	Tourism	EEN	Grand total
AL	-	-	-	233	233
AM	-	-	-	144	144
AT	216,065	-	490	3,960	220,515
BE	81,172	-	583	3,719	85,474
BG	91605	-	163	2,112	93,880
CY	-	-	103	649	752
CZ	199,655	-	102	2,736	202,493
DE	552,956	36,011	322	22,676	611,965
DK	17,968	-	113	2,074	20,155
EE	73,580	-	130	510	74,220
EL	34,189	11,237	425	0	45,851
ES	1,011,408	-	1 421	14,125	10,255,33
FI	-	-	55	1,784	1,839
FR	1,274,179	81,52	668	15,821	1,298,820
HR	-	-	303	1,157	1,460
HU	49,878	-	345	2,711	52,934
IE	-	1,000	225	0	1,225
IS	-	-	53	527	580
IT	1,532,541	1,250	2 197	17,373	1,551,164
LT	12,226	-	201	898	1,099
LU	-	-	16	705	721
LV	6,177	-	169	649	6,995
ME	11,012	-	-	179	11,191
MK	-	-	43	273	316
MT	-	-	205	0	205
NL	17,318	3,000	370	0	20,688
PL	138,928	-	99	8,285	147,312
PT	-	-	204	2,844	3,048
RO	42,545	-	86	2,177	44,808
RS	-	-	-	482	482
SE	-	-	38	2,959	2,997
SI	106,561	-	590	1,161	108,312
SK	3,318	-	28	1,170	4,516
TR	-	-	85	5,578	5,663
UK	74,050	3,364	439	0	77,853
Grand Total	5,547,173	64,014	6,653	119,671	5,725,443

Source: for LGF and EFG - EIF quarterly report Q4/2016; for Entrepreneurship education - EYE database; for all the other action lines - datahub. Note: the datahub only contains information on calls for proposal. * The figures presented in the table for LGF and EFG refers to investments into eligible final beneficiaries (financing transactions).

2.2 - Analysis of the sectoral involvement

This section provides the distribution by sector of the number of beneficiaries reached and budget allocated to final beneficiaries, for the action lines LGF, EFG, EEN, EYE, Cluster Internationalisation and Tourism grants.

Table 39 and Table 40 present the breakdown of, respectively, the estimated number and the shares of final beneficiaries reached, by the principal NACE sectors.

Table 39 Sectoral involvement (estimated shares of cumulative budget 2014-2016)

Sector	Description	Access to finance / LGF	Access to finance / EFG	EEN	EYE	Cluster internationalisation	Tourism grants	Grand total
A	Agriculture, forestry and fishing	5,874	1	508	172	1,291	2	7,848
C	Fertigungsindustrie	16,564	2	8,276	625	3,076	-	28,543
D	Electricity, gas, steam and air conditioning supply	128	1	2,132	69	1,444	-	3,774
E	Water supply; sewerage; waste management and remediation activities	487	-	-	26	349	-	862
F	Bauwesen	18,057	-	-	647	190	-	18,894
G	Wholesale and retail trade; repair of motor vehicles and motorcycles	36,830	3	-	133	-	-	36,966
H	Transporting and storage	8,907	-	-	42	373	5	9,327
I	Accommodation and food service activities	14,666	-	-	381	-	11	15,058
J	Information and communication	2,881	1	4,255	647	2,196	2	9,982
M	Professional, scientific and technical activities	11,626	2	2,540	1,303	-	42	15,513
N	Administrative and support service activities	5,751	-	4,350	-	-	114	10,215
N79	Tourism services	-	-	-	-	-	130	130
Q	Human health and social work activities	4,221	-	2,913	150	264	23	7,571
S	Other services activities	9,697	2	5,235		-	-	14,934
Other	Other sectors	7,655	-	499	1,366	91	29	9,640
TOTAL		143,344	12	30,708	5,561	9,274	358	189,257

Source: for LGF and EFG : EIF quarterly report Q4/2016; for EYE and EEN: data from EASME; for all the other action lines: datahub. Note: the datahub only contains information on calls for proposal. Note: the sectors for the final beneficiaries of the calls for proposals have been attributed on the basis of the information available; data for the sectoral involvement in LGF includes also Montenegro.

Table 40 Sectoral involvement (shares of final beneficiaries 2014-2016)

Sector		Description	Access to finance / LGF	Access to finance / EFG			EEN	EYE	Cluster internationalisation	Tourism grants		Grand total
A	Agriculture, forestry and fishing		4%		5%	2%		3%	14%	1%	4%
C	Fertigungsindustrie		12%		16%	27%		11%	33%		18%
D	Electricity, gas, steam and air conditioning supply		0%		11%	7%		1%	16%		3%
E	Water supply; sewerage; waste management and remediation activities		0%					0%	4%		0%
F	Bauwesen		13%					12%	2%		8%
G	Wholesale and retail trade; repair of motor vehicles and motorcycles		26%		16%			2%			16%
H	Transporting and storage		6%					1%	4%	1%	4%
I	Accommodation and food service activities		10%					7%		3%	7%
J	Information and communication		2%		21%	14%		12%	24%	1%	7%
M	Professional, scientific and technical activities		8%		16%	8%		23%		12%	9%
N	Administrative and support service activities		4%			14%				32%	6%
N79	Tourism services									36%	0%
Q	Human health and social work activities		3%			0%		3%	3%	6%	5%
S	Other services activities		7%		16%	17%		0%			8%
Other	Other sectors		6%		31%	2%		25%	1%	8%	4%
TOTAL			100%		100%	100%		100%	100%	100%	100%

Source: for LGF and EFG - EIF quarterly report Q4/2016; for EYE and EEN: data from EASME; for all the other action lines - datahub.

Table 41 presents the amount of budget (in million Euros) allocated by each action line to beneficiaries, distributed by NACE sector, while Table 42 shows the shares of budget allocated in each sector for each of the action lines analysed.

Table 41 Sectoral involvement (budgets in EUR m)

Sector	LGF	EFG	EYE	Tourism	Grand Total
A - Agriculture, forestry and fishing	327.2	4	0	0	331.2
C - Manufacturing	1,053	3	1		1,057
F - Construction	584.3		0		584.3
G - Wholesale and retail trade; repair of motor vehicles and motorcycles	1,401.40	35.9	0		1,437.3
H - Transporting and storage	389.4		0	0.1	389.5
I - Accommodation and food service activities	447.4		0	0.2	447.6
M - Professional, scientific and technical activities	312.7	14.8	3	1.3	331.8
N - Administrative and support service activities	180.8			3	183.8
N79 - Tourism services				3.5	3.5
Q - Human health and social work activities	159.4		0	0.5	159.9
R - Arts, entertainment and recreation	84		0.1	0.8	84.9
S - Other services activities	244.7	6.7			251.4
Other sectors	362.8	12.6			375.4
Grand total	5,547.1	77	4.1	9.4	5,637.6

Source: for LGF and EFG: EIF quarterly report Q4/2016; for EYE: data from EASME; for all the other action lines: datahub. Note: the datahub only contains information on calls for proposal. Note: the sectors for the final beneficiaries of the calls for proposals have been attributed on the basis of the information available.

Table 42 Sectoral involvement (number of final beneficiaries)

Sector	LGF	EFG	EYE	Tourism	Grand total
A - Agriculture, forestry and fishing	5.9%	5.2%	0.0%	0.0%	5.9%
C - Manufacturing	19.0%	3.9%	24.4%	0.0%	18.7%
F - Construction	10.5%	0.0%	0.0%	0.0%	10.4%
G - Wholesale and retail trade; repair of motor vehicles and motorcycles	25.3%	46.6%	0.0%	0.0%	25.5%
H - Transporting and storage	7.0%	0.0%	0.0%	1.1%	6.9%
I - Accommodation and food service activities	8.1%	0.0%	0.0%	2.1%	7.9%
M - Professional, scientific and technical activities	5.6%	19.2%	73.2%	13.8%	5.9%
N - Administrative and support service activities	3.3%	0.0%	0.0%	31.9%	3.3%
N79 - Tourism services	0.0%	0.0%	0.0%	37.2%	0.1%
Q - Human health and social work activities	2.9%	0.0%	0.0%	5.3%	2.8%
R - Arts, entertainment and recreation	1.5%	0.0%	2.4%	8.5%	1.5%
S - Other services activities	4.4%	8.7%	0.0%	0.0%	4.5%
Other sectors	6.5%	16.4%	0.0%	0.0%	6.7%
Grand total	100.0%	100.0%	100.0%	100.0%	100.0%

Source: for LGF and EFG - EIF quarterly report Q4/2016; for EYE: data from EASME; for all the other action lines - datahub. Note: the datahub only contains information on calls for proposals; the sectors for the final beneficiaries of the calls for proposals have been attributed to LGF and EFG based on the NACE rev2 classification and for all other action lines an approximate concordance to NACE rev 2 was constructed based on the available sectoral information.

Appendix C- List of documents analysed

3.1 SO1 – Access to Finance

Table 43 List of documents analysed SO1

List of documents analysed

A Single Market Strategy for Europe - Analysis and Evidence, SWD(2015) 202 final

Action plan to improve access to finance for SMEs – Communication from the Commission to the Council, European Parliament, Committee of the Regions, and to the European and Social Committee COM(2011) 870 final SEC (2011) 1527 final)

CIP Results 2015 and 2016: CIP Results 180416.pdf (there are some updates in the web: http://ec.europa.eu/growth/access-to-finance/cip-financial-instruments_es)

COM (2014) 4993 final, COSME Work Programmes (2014-16)

COM (2015) 3767 final, COM (2016) 4344 final, COM (2014) 247 final, COM (2014) 8044 final, COM (2016) 63 final: Annexes to the Commission Implementing Decisions concerning the adoption of the work programmes for 2014 / 2015 / 2016 and the financing for the implementation of Programme for the Competitiveness of Enterprises and small and medium-sized enterprises.

COM_Presentation_EFSI.pdf (https://era.gv.at/object/document/2505/attach/COM_Presentation_EFSI.pdf)

DG GROW Presentation on draft internal regression analysis <<SMEloans_over1year.pptx>> It gives an example on potential calculation of the gap in question.

DIFFERENCES AND SIMILARITIES BETWEEN CIP AND COSME, Briefing Note IP/A/ITRE/2012-14

Draft alternative conclusion provided by other colleagues. <<gap 375.docx>> It points at the same direction, i.e. almost bi-angulation, however, they might have used other data and/or methodology

DRAFT REPORT on the implementation of the EFSI, European Parliament (http://www.europarl.europa.eu/sides/getDoc.do?type=COMPARL&reference=PE-597.724&format=PDF&language=EN&secondRef=01)

ECB is calculating external current financing gap, for example on page 4, table A and bottom of the page: https://www.ecb.europa.eu/pub/pdf/other/accesstofinancesmallmediumsizedenterprises201611.en.pdf

EFG calls for expression of interest, jointly with InnovFin: InnovFin Equity and COSME EFG Call for Expression of Interest at the bottom of the websitehttp://www.eif.org/what_we_do/equity/single_eu_equity_instrument/index.htm

EFSI: an early proposal to extend and expand (2016); http://www.eca.europa.eu/Lists/News/NEWS1611_11/OP16_02_EN.pdf

EIB Operations Evaluation - independent evaluation report on EFSI (2016); http://www.eib.org/attachments/ev/ev_evaluation_efsi_en.pdf

EIF, 2015: The Economic Impact of EU Guarantees on Credit to SMEs. Evidence from CESEE Countries, http://www.eif.org/news_centre/publications/eif_wp_29_economic-impact-guarantees_july15_fv.pdf

Euro area bank lending survey https://www.ecb.europa.eu/stats/ecb_surveys/bank_lending_survey/html/index.en.html

Europe's next leaders: the start-up and scale-up initiative. EC Communication COM(2016) 733 final

European Small Business Finance Outlook (ESBFO), December 2016, EIF Research & Market Analysis, Working Paper 2016/37 (http://www.eif.org/news_centre/publications/eif_wp_37_esbfo_dec16_final.pdf)

Evaluation of EIF Own Resources Activity (2012); http://ec.europa.eu/dgs/economy_finance/evaluation/pdf/final_reprt_eval_eif.pdf

Ex-ante assessment of the EU SME Initiative, SWD(2013) 517 final

Final evaluation of the Competitiveness and Innovation Framework Programme

Final evaluation of the Entrepreneurship and Innovation Programme (April 2011)

http://www.nefi.eu/index.php?eID=tx_nawsecuredl&u=0&g=0&t=1493922032&hash=aaab5903abcdbc636efefc7217c179b2fc0f8dca&file=fileadmin/Redaktion/Images/publications/Doc_08_-_NEFI_HLM_Paris_-_Joint_Statement_of_the_NEFI_HLM_Final.pdf"

https://ec.europa.eu/commission/publications/investment-plan-europe-evaluations-give-evidence-support-its-reinforcement_en; Page 9, second paragraph

Impact Assessment accompanying the document Proposal for a regulation establishing a Programme for the Competitiveness of Enterprises and small and medium-sized enterprises (2014 - 2020). COM(2011) 834 final, SEC(2011) 1452 final.

Implementation Report 2012 EIPC of 14 March 2013 (Version revised 21 May 2013)

InnovFin SMEG Implementation Status (http://www.eif.org/what_we_do/guarantees/single_eu_debt_instrument/innovfin-guarantee-facility/report-quarterly-innovfin-smeg-implementation-status.pdf)

Interim evaluation of the Competitiveness and Innovation Framework Programme

Interim evaluation of the Entrepreneurship and Innovation Programme (April 2009)

Joint Committee Complementarity COSME LGF-InnovFin SMEG, Presentation 2015

Key role of National Promotional Institutions in the EFSI SME Window, Joint Statement of the NEFI High Level Members (May 2016)

Kraemer-Eis, H., Signore, S. and D. Prencipe (2016), The European venture-capital landscape: an EIF perspective, Volume I: The impact of EIF on the VC ecosystem, EIF Research and Market Analysis, Working Paper 2016/34.

Latest annual report for GIF facility: CIP GIF Annual Report as at 30.09.2016.pdf

Latest annual report for SMEG facility: CIP SMEG 2016 Annual report.pdf

LGF calls for expression of interest: bottom of the website http://www.eif.org/what_we_do/guarantees/single_eu_debt_instrument/cosme-loan-facility-growth/index.htm

Regulation establishing a Programme for the Competitiveness of Enterprises and small and medium-sized enterprises (COSME) (2014 - 2020). Regulation (EU) No 1287/2013

Report on the public consultation under the Start-up Initiative

Research for REGIO COMMITTEE - Maximisation of synergies between European Structural and Investment Funds and other EU instruments to attain Europe 2020 goals", http://www.europarl.europa.eu/RegData/etudes/STUD/2016/585872/IPOL_STU(2016)585872_EN.pdf

Restoring EU Competitiveness, EIB (2016); http://www.eib.org/attachments/efs/restoring_eu_competitiveness_en.pdf

Revision of the Regulation on the EFSI– towards an EFSI 2.0?, Briefing implementation appraisal, European Parliamentary Research Service (EPRS), September 2016; http://www.europarl.europa.eu/RegData/etudes/BRIE/2016/587342/EPRS_BRI%282016%29587342_EN.pdf

SAFE survey 2016

SBA Fact Sheet 2016 (http://ec.europa.eu/growth/smes/business-friendly-environment/performance-review-2016_en#interactive-sme-database)

Scale-up manifesto. http://scaleupeuropemanifesto.eu/

Small Business Act for Europe: access to finance; https://ec.europa.eu/growth/access-to-finance_en

Special report no 19/2016: Implementing the EU budget through financial instruments – lessons to be learnt from the 2007-2013 programme period; http://www.eca.europa.eu/Lists/ECADocuments/SR16_19/SR_FIN_INSTRUMENTS_EN.pdf

Statistics from the access to finance website: web page of access to finance in European countries; http://europa.eu/youreurope/business/funding-grants/access-to-finance/index_en.htm; Latest data <<a2f stats.xlsx>>

Study on ''Got rejected? Real effects of not getting a loan'':https://www.ecb.europa.eu/pub/pdf/scpwps/ecbwp1960.en.pdf

Study on 'Building Momentum in Venture Capital across Europe', https://www.kfw.de/PDF/Download-Center/Konzernthemen/Research/PDF-Dokumente-Studien-und-Materialien/Building-Momentum-in-Venture-Capital-across-Europe.pdf

The audit of the SMEG Facility (2011); http://www.europarl.europa.eu/meetdocs/2009_2014/documents/cont/dv/sr4_/sr4_en.pdf

The EFSI, maximising its potential, European Political Strategy Centre (EPSC) Strategic Notes, Issue 11 (2016); https://ec.europa.eu/epsc/sites/epsc/files/strategic_note_issue_11.pdf

The Shortage of Risk Capital for Europe’s High Growth Businesses (2017), https://www.afme.eu/globalassets/downloads/publications/afme-highgrowth-2017.pdf?dm_i=3TYX,674O,2T0PMT,LD2H,1

Upgrading the Single Market: more opportunities for people and business, EC Communication, COM(2015), 550 final

Web page Statistical Data Warehouse: Customizing the results of the Bank Lending Survey and the Survey on the Access to Finance of Enterprises (http://sdw.ecb.europa.eu/browse.do?node=9691105)

Web Page: SME Performance Reviews 2015/2016 (https://ec.europa.eu/growth/smes/business-friendly-environment/performance-review-2016_en)

Work Programme: They plan to audit 'EU’s venture capital interventions' and 'EFSI: management of investment plan and EU added value' as we can see from their priority tasks available. In addition they plan to produce an special report to assess the impact of the SMEG on access to finance for SMEs and the effectiveness of the SMEG in the 2007-2013 period. http://www.eca.europa.eu/Lists/ECADocuments/WP2017/WP2017_EN.pdf

Source: Technopolis

3.2 SO2 – Access to markets

Table 44 List of background documents SO2

List of documents analysed

BusinessEurope, (2011), “Are the Points of Single Contact truly making things easier for European companies”

BusinessEurope, (2012), “BusinessEurope contribution on the redesign of the enterprise europe network (EEN)”

Centre of Strategy & Evaluation Services, (2011), "Final Evaluation of the Competitiveness and Innovation Programme"

Council of the European Union, (2015), "OUTCOME OF THE COUNCIL MEETING 3371 Council meeting Competitiveness (Internal Market, Industry, Research and Space)"

DroneRules.EU, (2016), "Consortia response to: EASME Letter - Result of the Review of your COSME project 680960 — DRONE-RULES.eu"

EASME, (2009), "Promotion document: “An Introduction to the Enterprise Europe Network”

EASME, (2013), "Enterprise Europe Network Activity report 2008-2012"

EASME, (2013), "Enterprise Europe Network Progress Report 2013"

EASME, (2014), "Brochure Enterprise Europe Network 2015 - SMEs step up a gear with the Enterprise Europe Network"

EASME, (2014), "Template: EASME MULTI-PARTNER MODEL FRAMEWORK PARTNERSHIP AGREEMENT FOR THE ENTERPRISE EUROPE NETWORK (EEN FPA — MULTI)"

EASME, (2015), "Your Europe Business Portal - Exit survey results 2015"

EASME, (2016), "Enterprise Europe Network Progress Report 2015"

EASME, (2016), "Grant Application - EU Japan Centre for Industrial Cooperation"

EASME, (2016), "Letter - Result of the Review of your COSME project 680960 — DRONE-RULES.eu": Ares(2016)695609462913

EASME, (2016), "PO Assessment Report/657846/LA IPR SME HD": Ares(2016)7015710

EASME, (2016), "PO Assessment Report/657914/SEA HD 2015-2017": Ares(2016)6286280

EASME, (2016), "Your Europe Business Portal - Exit survey results 2016"

EASME, (2017), "Coaching under the SME-Instrument": https://ec.europa.eu/easme/en/coaching-under-sme-instrument.

EASME, (2017), "Progress report - DRONE-RULES.eu - RPAS rules regulation portal": Ares(2017)327875

EASME, (2017), "Progress report - Request for revision of deliverable submission - DRONE-RULES.eu - RPAS rules regulation portal": Ares(2017)740655

Ecorys and IDF GHK, (2014), "SMEs' access to public procurement markets and aggregation of demand in the EU"

EU Japan Centre, (2012), "Vulcanus in Japan - Key figures"

EU Japan Centre, (2014), "EU Japan Centre - Activity Report 2014"

EU Japan Centre, (2014), "Presentation: EU Japan Centre - Activity Report 2014"

EU Japan Centre, (2015), "EU Japan Centre - Activity Report 2015"

EU Japan Centre, (2015), "Presentation: EU Japan Centre - Activity Report 2015"

EU Japan Centre, (2016), "EU Japan Centre - Work Programme 2016"

EU Japan Centre, (2016), "Traineeship in Japan for students from the EU & COSME* countries"

EU Japan Centre, (2017), "Application form: Vulcanus Programme"

EU Japan Centre, (2017), "Vulcanus in Japan - Frequent Questions and Answers"

EUROCHAMBRES (2011) “Enterprise Europe Network: Chambers’ views on its services post 2014”

EUROCHAMBRES, (2012), “Response to the Commission consultation on the Enterprise Europe Network”

Euroepan Commission, (2015) "Call for Expressions of Interest ‘Business Cooperation Centres’ in third countries for the Enterprise Europe Network 2015-2020 - Annex 1 EU Business Support in Markets Outside the EU"

Euroepan Commission, (2015) "Call for Expressions of Interest ‘Business Cooperation Centres’ in third countries for the Enterprise Europe Network 2015-2020": COS-Art-7-001

Eurofound (2015), “Job creation in SMEs ERM annual report 2015”

European Commission (2001), “European Charter for Small Enterprises – Annual Implementation Report”

European Commission, (2005), “Proposal for a Decision of the European Parliament and of the Council establishing a Competitiveness and Innovation Framework Programme (2007-2013)": COM(2005) 121 final

European Commission, (2006), "Call for proposals: Services in support of business and innovation": 2006/C 306/07

European Commission, (2006), "DECISION No 1639/2006/EC OF THE EUROPEAN PARLIAMENT AND OF THE COUNCIL of 24 October 2006 establishing a Competitiveness and Innovation Framework Programme (2007 to 2013)": 2006/C 306/07

European Commission, (2007), "Call for proposals: Services in support of business and innovation": 2007/C 227/04

European Commission, (2007), "Submission Set - Call for proposals (ENT/CIP/070001a): Services in support of business and innovation"

European Commission, (2008), "Points of Single Contact: Doing business made easier": http://ec.europa.eu/internal_market/eu-go/docs/psc_en.pdf

European Commission, (2009), "Interim evaluation of the Enterprise Europe Network"

European Commission, (2010)," Evaluation of EIP indicators"

European Commission, (2011), "Proposal for a REGULATION OF THE EUROPEAN PARLIAMENT AND OF THE COUNCIL establishing a Programme for the Competitiveness of Enterprises and small and medium-sized enterprises (2014 - 2020)": COM(2011) 834 final, SEC(2011) 1452, SEC(2011) 1453

European Commission, (2012), "Competitiveness and Innovation Framework Programme Performance from 2007 to 2012"

European Commission, (2012), "Implementation of the Small Business Act for Europe (SBA) and Entrepreneurship Policies at Local and Regional Level"

European Commission, (2013), "CIP Performance Report": JCM-02-2013

European Commission, (2013), "REGULATION (EU) No 1287/2013 OF THE EUROPEAN PARLIAMENT AND OF THE COUNCIL of 11 December 2013 establishing a Programme for the Competitiveness of Enterprises and small and medium-sized enterprises (COSME) (2014 - 2020) and repealing Decision No 1639/2006/EC"

European Commission, (2014), "COSME Work Programme 2014 - Commission Implementing Decision concerning the adoption of the work
programme for 2014 and the financing for the implementation of Programme for the
Competitiveness of Enterprises and small and medium-sized enterprises"

European Commission, (2014), "Statistical Report About Your Europe Business Portal 2014"

European Commission, (2014), “Call for proposals – COSME Enterprise Europe Network - COS-EEN-2014-2-01”

European Commission, (2014), “Call for proposals – COSME Enterprise Europe Network - COS-EEN-2014-2-04”

European Commission, (2014), “Call for Proposals Facilitating access to regulation for light Remotely Piloted Aircraft Systems (RPAS)": COS-RPAS-2014-2-03

European Commission, (2014), “Call for proposals for China, Latin America And South-East Asia IPR SME Helpdesks": COS-WP2014-2-05

European Commission, (2015), "COSME Work Programme 2015 - Commission Implementing Decision concerning the adoption of the work
programme for 2015 and the financing for the implementation of Programme for the
Competitiveness of Enterprises and small and medium-sized enterprises"

European Commission, (2015), "Enterprise Europe Network survey 2016 – Your Europe Business Portal 2015"

European Commission, (2015), "PERIODIC REPORT COSME PROGRAMME - Action Title: China IPR SME Helpdesk"

European Commission, (2015), "PERIODIC REPORT COSME PROGRAMME - Action Title: Latin America IPR SME Helpdesk"

European Commission, (2015), "PERIODIC REPORT COSME PROGRAMME - Action Title: South-East Asia IPR SME Helpdesk"

European Commission, (2015), "Proposal Evaluation Summary Report - DRONE-RULES.eu - RPAS rules regulation portal - COS-RPAS-2014-2-03": Ares(2015)2940919

European Commission, (2015), "Statistical Report About Your Europe Business Portal 2015"

European Commission, (2015), "Study on good practices in the area of compliance assistance and compliance schemes FRAMEWORK CONTRACT-ENTR/300/PP2013/FC" Ares (2015)3332549

European Commission, (2015), "The Performance of the Points of Single Contact An Assessment against the PSC Charter"

European Commission, (2015), “Flash Eurobarometer 421 – Internationalisation of Small and Medium-sized Enterprises”

European Commission, (2015), “GRANT AGREEMENT NUMBER — 680960 — DRONE-RULES.eu": Ares(2015)3949307

European Commission, (2015), “Innobarometer 2015 – The innovation trends at EU enterprises”

European Commission, (2016), "AMENDMENT Reference No AMD-657846-4 Grant Agreement: 657846 — Latin America IPR SME Helpdesks (LA IPR SME HD)": Ares(2016)3256986

European Commission, (2016), "CALL FOR PROPOSALS - Supporting the businesses' expansion in the Internal Market by improving Points of Single Contact – Increasing awareness among entrepreneurs about the Points of Single Contact": Ares(2016)4745300

European Commission, (2016), "COSME Work Programme 2016 - Commission Implementing Decision concerning the adoption of the work
programme for 2016 and the financing for the implementation of Programme for the
Competitiveness of Enterprises and small and medium-sized enterprises"

European Commission, (2016), "Description of the action: EU Japan Centre for Industrial Cooperation - GRO/SME/16/B/041": Ares(2016)2787248

European Commission, (2016), "Enterprise Europe Network survey 2016 – Your Europe Business Portal 2016"

European Commission, (2016), "Europe’s Next Leaders: The Start-Up And Scale-Up Initiative": COM(2016) 733 final

European Commission, (2016), "GRANT AGREEMENT NUMBER — 657914 — SEA HD 2015-2017": Ares(2015)1197896

European Commission, (2016), "Statistical Report About Your Europe Business Portal 2016"

European Commission, (2016), "Technical Final Report - Study on Good Practices in the Area of Compliance Assistance and Compliance Schemes"

European Commission, (2016), “Call for proposals for Improved Points of Single Contact": COS-SPOC-2016-02-03

European Commission, (2016), “Call for proposals for Improving access to SMEs in public procurement": COS-APP-2016-2-05

European Commission, (2017), "Enterprise Europe Network - Key results & figures for COSME committee"

European Commission, (2017), "Explanatory Memorandum of the EC Proposal for a Regulation of the European Parliament and of the Council on establishing a single digital gateway to provide information, procedures, assistance, and problem solving services": COM(2017) 256 final

European Commission, (2017), "Official website on the EU Single Market Scoreboard": http://ec.europa.eu/internal_market/scoreboard/performance_by_governance_tool/points_of_single_contact/index_en.htm.

European Commission, (2017), "Presse release: Aviation: Commission is taking the European drone sector to new heights"

European Commission, (2017), "Single Market Scoreboard - Points of Single Contact": http://ec.europa.eu/internal_market/scoreboard/_archives/2013/07/performance_by_governance_tool/points_of_single_contact/index_en.htm

European Commission, (2017), "SME Performance Review and Small business act Factsheets": https://ec.europa.eu/growth/smes/business-friendly-environment/performance-review-2016_en

European Commission, (2017),"Proposal for a Regulation of the European Parliament and of the Council on establishing a single digital gateway to provide information, procedures, assistance, and problem solving services": COM(2017) 256 final

OECD (2004), “Promoting Entrepreneurship and Innovative SMEs in a Global Economy - Towards a More Responsibleand Inclusive Globalisation – 2nd OECD Ministerial Conference on SMEs – June 2004 – Istanbul Turkey”

SESAR Joint Undertaking, (2016), "European Drones Outlook Study - Unlocking the value for Europe"

t33, SWECO and OIR, (2012), "Implementation for the Small Business Act for Europe (SBA) and Entrepreneurship Policies at Local and Regional Level"

Technopolis Group (2015), "Final evaluation of the impact of the Enterprise Europe Network 2008-2014"

Technopolis Group, (2009), "Interim Evaluation of the Competitiveness and Innovation Framework Programme (2007-2013)"

Technopolis Group, (2015), "Evaluation of the three years of operation (2011 – 2013) of the Executive Agency for Competitiveness and Innovation (EACI)"

The British Standards Institution (BSI) and WIFO, (2015), "Proposal for an analytical study - Study on Good Practices in the Area of Compliance Assistance and Compliance Schemes"

The British Standards Institution (BSI) and WIFO, (2016), "Final Report - Study on Good Practices in the Area of Compliance Assistance and Compliance Schemes"

The British Standards Institution (BSI) and WIFO, (2016), "Mid-Term Report - Study on Good Practices in the Area of Compliance Assistance and Compliance Schemes"

UEAPME, (2012), “Position Paper - proposals for a renewed EEN”

3.3 SO3 – Favourable environment

Table 45 List of background documents SO3

Thematic area	List of main background documents used
Cluster internationalisation	Analysis of the consultation launched by ECCP amongst the ESCP-4i Partnerships in June 2016 (published Oct 2016)
	Background documents:
	Call for Proposals – Clusters Go International (COS-CLUSTER-2014-3-03)
	European Cluster Collaboration Platform (clustercollaboration.eu) resources:
	Evaluation of European Cluster Initiatives managed by DG Enterprise and Industry (2015), performed by Technopolis Group
	Iszak, K. (April 2016): European Strategic Cluster Partnerships Going International: common opportunities for internationalisation, Technopolis Group paper.
	Monitoring / analysis documents published on the European Cluster Collaboration website, containing statistics on the ESCP4i clusters
	Monitoring reports of the implementation of the ESCP-4i projects, provided by EASME
SME Policy	2014 Report on public consultation on SBA
	2016 SME Envoys report on the implementation of SME Test in EU countries and presentations of the participants in the technical seminar on the SME Test (Oct 2016)
	Commission Decision of 4.7.2012, setting up the group of experts on Enterprise and SME Policies, C(2012)4386final
	Documents of the SME Envoys Network, 2014-2016
	European Parliament reports on the European Semester for economic coordination 2016, 2015 and 2014
	Further reports and sources
	Meeting minutes of the SME Envoys Network available on the European Commission Transparency Register, http://ec.europa.eu/transparency/regexpert/index.cfm?do=groupDetail.groupDetail&groupID=2666
	Policy documents
	Results of the Annual Growth Survey 2014-2017
	SME Envoys mission description
	SME Performance Review resources: Small Business Act Factsheets, Annual Report on European SMEs 2014-2016
	SME Policy website of DG GROW
	The SBA for Europe and SBA Review, 2011
	The Single Market Strategy for Europe
Tourism	Tourism actions documents
	Accessible Tourism Services report (2015)
	Ad-hoc grant (2014-2016) awarded to the European Travel Commission for Destination Europe 2020 Campaign
	Call for Proposals to select EDEN destinations COS-WP2014-03-15.01 and call COS-TEDEN-2014-3-15 for communication support
	Calls for proposals for the tourism sector grants actions: COS-WP2014-3-15.02 (synergies with high-end industries and creative industries), COS-WP2014-3-15.03 (diversifying EU tourism offer through sustainable products), COS-TOUR-2015-3-04 (supporting competitive and sustainable growth in the tourism sector – 3 themes);
	CIP Evaluation (2013), including references to actions on Tourism
	COS-TFLOWS-2014-3-15
	COS-TOUR-2015-3-04-1
	COS-TSUST-2014-3-15
	COS-TSYNER-2014-3-15
	DG GROW – Guide to EU funding for the Tourism sector
	DG GROW, Tourism website
	Eurobarometer surveys of the Tourism sector
	European Commission, 2010 Communication on Tourism
	European Travel Commission reports and statistics
	Mapping of skills and training needs for accessibility in tourism services (2014)
	Monitoring reports of the calls (a sample of 4 reports per call, due to the lack of a synthesised report on the progress of the actions).
	OECD reports on Tourism sector (e.g. 2016 OECD report on Tourism Trends and policies)
	Policy studies, reports and websites:
	Travel patterns of accessible tourism in Europe (2013)
	Virtual Tourism Observatory
	World Tourism Organisation UNWTO – World Tourism Barometer
e-skills	Cattaneo, G., Aguzzi, S, et. Al, (2014): Digital Agenda for Europe – Engagement for Growth and Jobs
	Digital Agenda for Europe (2010).
	E-leadership skills: Second draft interim report, 2016, together with the Technical Assessment Sheet e-leadership; Final report (May 2017)
	e-Skills for Jobs Campaign website, http://eskills4jobs.ec.europa.eu/
	E-skills for Jobs Campaign, Final Report (Part 1 and 2), 2016
	e-skills for Jobs Manifestos (2014-2016)
	Empirica (2015): e-Skills in Europe: Trends and Forecasts for the European ICT Professional and Digital Leadership Labour Markets
	Empirica et al, (2015): Final Report - e-Leadership Skills for Small and Medium Sized Enterprises
	European Framework for the IT profession: Interim report, June 2016 together with the Technical Assessment Sheet IT profession; Final report & brochure (forthcoming, May 2017)
	German and Austrian SME Envoys report (2016): Avoiding skills shortages and mismatches – New strategies for meeting companies’ skilled labour needs
	Innovation Value Institute, IDC, Empirica, CEPIS (2014): e-Skills: The International dimension and the impact of globalisation
	Policy Documents
	Project implementation documents received from DG GROW
	Reports
	Reports of the Grand Coalition for Digital Jobs
	S. Robinson, L. Hendrix, B. Hanny, W. B. Korte, T. Hüsing (2015): New Curricula for e-Leadership Skills. Guidelines and Quality Lables for new Curricula for e-Leadership Skills in Europe. Final Report. Bonn. http://eskills-lead.eu/documents.html
	T. Hüsing, W.B. Korte, E. Dashja (2015): E-skills and e-leadership skills 2020. Trends and forecasts for the European ICT professional and digital leadership labour market. Empirica Working Paper. Bonn.
	The Commission Communication of 2007, 'e-Skills for the 21st Century: Fostering Competitiveness, Growth and Jobs' - a long term e-skills agenda for the period 2008-2010.
	The Employment Package (2012)
	The European e-Competence Framework - a reference of 40 skills required in the ICT workplace (http://www.ecompetences.eu/).
	Websites and further resources:
	www.leadership2015.eu and www.eskills-lead.eu - website of European Conferece on digital and key enabling technologies skills 2015 and of the service contract to promote e-leadership in Europe (LEAD)
Sectoral competitiveness (except for tourism)	Call for Tenders specifications on Establishment of the "European Sustainable Chemicals Support Service", EASME/COSME/2015/01,7, Phase 1 and Phase 2
	Presentation of results of the Call for Proposals on design-based consumer goods, received from DG GROW
	The Construction 2020 Action Plan and reports on its implementation from DG GROW website.
KETs	Projects’ implementation documents, websites and policy documents
	EC Communication 2016, on “Digitising European Industry. Reaping the full benefits of a Digital Single Market”
	Coordination framework for Digital Single Market
	DG GROW, KETs Tools Website
	Digital Agenda for Europe (2010).
	KETs High Level Work Group reports – final results and recommendations
	KETs strategy, 2015
	The Commission Communication of 2014, 'An Industrial Renaissance for EU”

3.4 SO4 – Entrepreneurship

Table 46 List of documents analysed SO4

List of documents analysed

Annex 1 to the Commission Implementing Decision concerning the adoption of the work programme for 2014 and the financing for the implementation of Programme for the competitiveness of Enterprises and small and medium-sized enterprises. C(2014) 247 final.

Annex 1 to the Commission Implementing Decision concerning the adoption of the work programme for 2015 and the financing for the implementation of Programme for the competitiveness of Enterprises and small and medium-sized enterprises. C(2014) 8044 final.

Annex 1 to the Commission Implementing Decision concerning the adoption of the work programme for 2016 and the financing for the implementation of Programme for the competitiveness of Enterprises and small and medium-sized enterprises. C(2016) 63 final.

Annex 1 to the Commission Implementing Decision concerning the adoption of the work programme for 2017 and the financing for the implementation of Programme for the competitiveness of Enterprises and small and medium-sized enterprises. C(2016) 7033 final.

Attitudes towards the planned EU pilot project - Erasmus for Young Entrepreneurs; Report 3: Survey among young would-be and existing entrepreneurs in all EU Member States: Analytical Report” produced by the Gallup Organisation for Directorate-General for Enterprise and Industry, November 2007

Commission staff working document accompanying the document Report from the Commission to the European Parliament and the Council. COSME – Programme for the Competitiveness of Enterprises and SMEs 2014-20 – Monitoring Report 2014. SWD (2016) 274 final.

Communication “Think Small First – A ‘Small Business Act for Europe’ COM(2008) 394 final

COSME Monitoring Report 2015.

COSME programme documentation (annual work programmes, texts of the calls for proposals and tenders, annual monitoring reports). Key text referenced included:

COSME. Programme for the Competitiveness of Enterprises and SMEs 2014-2020. Monitoring Report 2014. Annex 1 – Details of COSME Actions 2014. European Commission.

EIP Implementation Report 2011. EIPC of 15 March 2012. (European Commission, 2012)

EU and other international policy studies, evaluations and assessments on the topics. Key text referenced included:

EU policy documents, strategies and action plans. Key text referenced included:

European Commission, Entrepreneurship 2020 Action Plan

European Commission: Call for Proposals, Migrants Entrepreneurship Support Schemes, Europe: 2016

Eurostat, 'Statistics in focus: the entrepreneurial gap between men and women' (30/2007)

Evaluation and Analysis of Good Practices in Promoting and Supporting Migrant Entrepreneurship. EU conference on migrant entrepreneurship. Background Paper. 23 February 2016. European Commission.

Feasibility study for the Pilot Action “Erasmus for young entrepreneurs" Part A: Scenarios and Costs. A Final Report to DG Enterprise of the European Commission (ECOTEC Research and Consulting, 2008)

Final evaluation of the Entrepreneurship and Innovation Programme – Final report. (CSES, EIM, April 2011)

Final Report. The future of Europe’s rural periphery, the role of Entrepreneurship in responding to employment problems and social marginalization. EU research on social science and humanities. European Commission (EUR 21704)

Green Paper – Entrepreneurship in Europe. COM(2003) 27 final

Interim evaluation of the Erasmus for Young Entrepreneurs Pilot project / Preparatory action (CSES, 2011)

Past studies and evaluations (of CIP, EIP and EYE). Key text referenced included:

Project level information – including information accessed from the project websites

Report from the Commission to the European Parliament and the Council. COSME. Programme for the Competitiveness of Enterprises and SMEs 2014-2020. Monitoring Report 2014. COM(2016) 526 final.

Statistical Analysis of the Erasmus for Young Entrepreneurs (EYE) Programme (Technopolis, 2014)

Appendix D– Lists of interviewees

4.1 SO1 – Access to Finance

Table 47 Interviews performed SO1

Interview category	Interview ID	Nachname	Vorname	Institution	Role
A: Representatives from EU institutions	A-SO1-1	Schreiber	Kristin	DG GROW	Director, H, COSME program
A: Representatives from EU institutions	A-SO1-2	Cristea	Ciprian	DG GROW	Head of Unit, H3, Financial Instruments
A: Representatives from EU institutions	A-SO1-3	Diss	Martine	DG GROW	Deputy head of Unit, H3, Financial Instruments
A: Representatives from EU institutions	A-SO1-4	Bartels	Astrid	DG GROW	Team leader
A: Representatives from EU institutions	A-SO1-5	Squilloni	Matteo	EIF	Mandate Guarantees
A: Representatives from EU institutions	A-SO1-6	Boregas Amaro	Luis	EIF	Mandate VC investments
A: Representatives from EU institutions	A-SO1-7	Gresko	Patrick	EIF	Investments, Technology and innovation
A: Representatives from EU institutions	A-SO1-8	Manca	Silvia	EIF	Mandate Management, Institutional Business Development
A: Representatives from EU institutions	A-SO1-9	Mai	Gunnar	EIF	Head of Division, EU Guarantee Facilities
A: Representatives from EU institutions	A-SO1-10	Gonzalez Martin	David	EIF	Institutional Business Development Manager
A: Representatives from EU institutions	A-SO1-11	Appel	Stefan	DG REGIO	Head of Unit B3 - Financial Instruments and relations with International Financial Institutions within Directorate 'Policy'
A: Representatives from EU institutions	A-SO1-13	Clausen	Merete	DG ECFIN	Head of Unit 'Financing of innovation, competitiveness and employment policies' within Directorate 'Treasury and financial operations'
A: Representatives from EU institutions	A-SO1-14	Muent	Gunnar	EIB	Director of innovation and Competitiveness
A: Representatives from EU institutions	A-SO1-15	Cannenterre	Karen	EIB	Intermediated Financing to Micro, Small and Medium Size Entreprises
A: Representatives from EU institutions	A-SO1-16	Stoykov	Hristo	EIB	Equity, New Products and Special Transactions
B: Intermediary organisations	B-SO1-1	Klein	Germaine	European Association of Public Banks (EAPB)	EAPB interlocutor
B: Intermediary organisations	B-SO1-2	Sturm	Katrin	European Association of Guarantee Institutions (AECM)	AECM Secretary-General
B: Intermediary organisations	B-SO1-3	Tschofen	Nina	Network of European Financial Institutions for Small and Medium Sized Enterprises (NEFI)	Coordinator
B: Intermediary organisations	B-SO1-4	Collins	Michael	InvestEurope	Chief Executive
B: Intermediary organisations	B-SO1-5	Huemer	Gerhard	European Association of Craft, Small and Medium-sized Enterprises (UEAPME)	Research and Innovation
B: Intermediary organisations	B-SO1-6	Cloquet	Daniel	BusinessEurope	Director SMEs and entrepreneurship

4.2 SO2 – Access to market

Table 48 Interviews performed SO2

Stakeholder type	Organisation	Name
EU officials	EC – DG GROW	Giacomo MATTINO (Head of Unit – H2)
		Patrick DE SMEDT (Deputy Head of Unit – H2)
	EASME	Jose PUIGPELAT (Deputy Head of Unit – A1)
		Muriel DE GRANDE (Team leader evaluation and monitoring)
	EC – DG RTD	Steve ROGERS (Deputy Head of Unit – B3)
	EC – DG DEVCO	Soyes THIERRY (Cooperation Officer)
	EC – DG GROW	Christophe GUICHARD (Deputy Head of Unit – F2)
	EC – DG REGIO	Katje REPPEL (Deputy Head of Unit – G1)
	EC – Service for Foreign Policy Instruments (FPI)	Daniel VAN ASSCHE (Programme Officer – EU Policies)
Beneficiaries (intermediaries)	Center for Technology Transfer and Innovation (CTT)	Spela STRES
	Hanze Hogeschool	Matthijs PLIJNAAR
	Croatian Chamber of Economy	Vesna TORBARINA
	ARC Consulting	Daniela TCHONKOVA
	Agencia IDEA	José A. Pascual SANCHEZ
	IAPMEI	Vanda NARCISO
	CCI Auvergne Rhone Alpes	Salvatrice BUFALINO
	Eurosportello-Servizi Informativi Per Le Imprese	Maria Christina RAFFONE
	Fundatia Centrul Roman Pentru Interprinderi Mici Si Mijlocii	Cristian ORMINDEAN
	Hessen Trade & Invest Gmbh	Khaled SNOUBER
	Investment and Development Agency of Latvia	Jekaterina VANAGA
	Politechnika Wroclawska	Agniezska TURYNSKA-GMUR
	Scottish Enterprise	Caroline GRAY-STEPHENS
	Technologicke Centrum Akademie Ved Ceske Republiky	Petr HLADIK
	Tillvaxtverket	Tony MEURKE
	Fonden Vaeksthus Hovedstadsregionen	Simon POULSEN
	Wirschaftskammer Oesterreich	Cristoph HUTER
	Ss Cyril and Methodius University in Skopje	Beti KOSADINOVSKA
	Camera de Comert Si Industrie A Republicii Moldova	Ludmila BOTOSAN
Umbrella organisation	UEAPME	Luc HENDRICKX Veronique WILLEMS
	BusinessEurope	Daniel CLOQUET
	EBN (Innovaiton Network)	Giodrano DICHTER
	EURADA	Esteban PELAYO VILLAREJO
	Eurochambers	Silvia Caneva

4.3 SO3 – Favourable environment

Table 49 Interviews performed SO3

Stakeholder type	Organisation	Name	Topics
EU officials	EC – DG GROW	Maarit Nyman (Deputy Head of Unit – H1)	SME Policy
		Christophe Guichard (Unit – F2)	Clusters
		Andre Richier (Unit – F3 – dealing with E-skills)	E-skills
		Heidi Moens (Unit – F3 – dealing with KETs)	KETs
		Domenico Lombardi	SME Policy
		Iuliana Aluas (Head of Unit – F4)	Tourism
		Piedad Rivas, F4 (together with Iuliana Aluas)	Design-based consumer goods
		Timoteo de la Fuente (Unit – D2)	Sectoral competitiveness – chemical regions service
	EASME	Valentino Izzo, Giovanna D’Addamio, Antonis Christopoulos	Tourism
		Ramunas Kuncaitis (Head of Sector A.1.4.- Entrepreneurship / for Clusters)	Clusters
	EC – DG RTD	Ana Grigore, Unit D1 (KETs)	KETs
	EC – DG REGIO	Katja REPPEL (Deputy Head of Unit – G1) – together with SO2	Clusters, KETs
		Joanna Mouliou (D1)	Tourism
	DG CONNECT	Helen Kopman, Deputy Head of Unit, Startup Europe	SME Policy
	DG EAC	Barbara Gessler, Head of Unit, Creative Europe Programme	Tourism
	DG TRADE	Martin Pilsner	SME Policy, Clusters, Sectoral Competitiveness
Beneficiaries (intermediaries)	European Travel Commission	Eduardo Santander, executive director, and Teodora Marinska	Tourism
	European Strategic Cluster Partnerships coordinators	Vladimir Gumilar, ECCA	Clusters
		Emilie Romeo, BioX
		Patrick Vuillermoz, WIINTECH
		Anna Naydenova, ICT Cluster Bulgaria
		Johan Lecocq, Silicon Europe
National and regional governments	National representatives Tourism	Lithuania: Lidjia Bajaruniene	Tourism
		Romania: Iulia Dangulea	Tourism
		Portugal: Joanna Pinto Coelho	Tourism
	National Representative SME Policy	Gabriele Schmid, Federal Ministry of Science, Research and Economy, Austria	SME Policy
Experts	Regional Representative	Jan Larosse, Expert, DG Regio and Flemish Government	Clusters and Sectoral Competitiveness
Umbrella organisation	UEAPME	Luc HENDRICKX, responsible for SBA & SME policy– together with SO2	SME Policy
	Business Europe	Daniel Cloquet, Director, SME competitiveness	SME Policy and Sectoral Competitiveness
	European Cluster Collaboration Platform	Mark Pattinson, INNO AG	Clusters
	TCI Network members	Joep Browers, Brainport Development	Clusters
	SPIRE	Angels Orduna	Sectoral competitiveness
	CIONET Netherlands	Frits Bussemaker, ICT Programme Manager	E-skills
	ECDL Foundation	Austeja Trinkunaite	E-skills
	Academy Cube	Lisa Keiler	E-skills
	CEPIS	Fiona Fanning	E-skills

Table 50 Interviews not performed SO3

Typ	Org.	Name	Topics	Status
EU Officials	EC- DG CONNECT	Alexander Riedl	E-skills	Refused
		Ronan Burgess	KETs	Did not reply despite repeated contacting
	EC – INTERREG	Magdalini Anagnostou	Clusters	Did not reply, replaced with DG REGIO – Joanna Mouliou Niessler (D1)
National governments	National representatives Tourism	Poland : Ewelina Macierzynska	Tourism	No answer despite emails and phone calls
Umbrella Organisations	NECSTOUR	Cristina Nunez	Tourism	Refused, postponed to after 15 June and refused again
	Eurochambers	Slivia Caneva	SME Policy	Postponed a couple of times then no answer
	TCI Network members (x3)	Patricia Valdenebro, CEO, TCI Network	Clusters	No availability before end of June. Accepted for 20 June, when phone call and emails weren’t answered
		Dorota Fraczek, Polish Agency for Enterprise Developent,	Clusters	Refused. Team attempted to replace with Mateja Dermastia, Slovenian cluster expert, who was not available
	EFFRA	Zeljko Pazin	KETs	No answer despite repeated emails and phone calls.
	CECIMO	Filip Geerts	KETs	No answer despite repeated emails and phone calls
	EIT Digital	Anders Flodstrom, director education	E-skills	Replaced with Fiona Fanning, CEPIS

Source: Technopolis

4.4 SO4 – Entrepreneurship

Table 51 Interviews performed SO4

Group	Name	Organisation	Relevance
Programme management	Michael Berz	DG GROW F3	Digital entrepreneurship
	Patrick Klein	DG GROW F2	Social entrepreneurship
	Simone Baldassari	DG GROW F2	Various entrepreneurship actions
	Katerina Nedjlova	DG GROW H2	EYE
	Ivelina Fedulova	Eurochambres	EYE Support office
	Pascale Gaucher	DG GROW	Women’s entrepreneurship
Intermediary organisations	Marie Fallgaard	Aalborg University	EYE IO - Denmark
	Nadine Bettens	Agence Bruxelloise pour l’Entreprise	EYE IO - Belgium
	Yavor Profirov	Bulgarian Management Association	EYE IO - Bulgaria
	Blanka Bendlová	JCI Czech Republic, z.s.	EYE IO - Czech Republic
	Pamela Fombuena	Chambre de commerce et d'industrie de région Paris Ile-de-France	EYE IO - France
	Thomas Berger	Hochschule Fulda	EYE IO - Germany
	Knieža Aivaras	Kaunas Chamber of Commerce, Industry and Crafts	EYE IO - Lithuania
	Susanne Marklund	European Minds	EYE IO - Sweden
	Anna Pajarón	Cambra Oficial de Comerç, Indústria I Serveis de Terrassa	EYE IO - Spain
	Alexandra Yaghil	Southbank Technopark	EYE IO - UK
Beneficiaries – Host companies	Daniele Trevisan	Kairos Europe	EYE HE - Belgium
	Guerric Gautier	Ozfair	EYE HE - Spain
	Francisco Sarrias	MSI	EYE HE - Austria
	Oliver Rauch	Cow Style	EYE HE - UK
	Martin Stevens	IT IS 3D	EYE NE - Italy
Beneficiaries – New Entrepreneurs	Matteo Schipsi	n/a	EYE NE - Romania
	Alexandra Nicorici	n/a	EYE NE - Spain
	Alberto Rodriguez Arias	n/a	EYE NE - Germany
	Heidi Storr	n/a	EYE NE - Bulgaria
	Vassil Tzanov	n/a	EYE HE - Belgium
	Matteo Schipsi	n/a	EYE HE – Spain
Other stakeholders	Efka Heder	South East European Centre for Entrepreneurial Learning	Advisory committee
	Maria Lustri	WES	Advisory committee
	Xavier Cobbaers		Contractor – women’s entrepreneurship
	Olena Beck	European Training Foundation	Advisory committee

[Catalogue number]

Annex 4: Interim Evaluation of the International Intellectual Property Rights (IPR) SME Helpdesk, a specific COSME measure

Table of contents

1. INTRODUCTION

2. BACKGROUND to the IHD measure

3. STATE of PLAY

4. METHODOLOGY

5. Analysis and answers to the evaluation questions

6. CONCLUSION

7. Annexes

Glossary

Term or acronym	Meaning or definition
AHK	German Chambers of Commerce Worldwide Network
ASEAN	Association of Southeast Asian Nations
ASEANHD	ASEAN helpdesk
BCCs	Business Cooperation Centres
CIBEPYME	Plataforma Iberoamericana de Propiedad Industrial Dirigida a Empresas
CIP	Competitiveness and Innovation Programme
CNHD	China IPR SME Helpdesk
COSME	Programme for the Competitiveness of the Enterprises and Small and Medium-sized Enterprises
EABC	EU Business Centres in Thailand
EASME	The Executive Agency for Small and Medium-sized Enterprises
EC	European Commission
ECCP	European Cluster Collaboration Platform
EEN	Enterprise Europe Network
EFTA	European Free Trade Association
EKONID	EU Business centres in India
ESCPs	European Strategic Cluster Partnerships
EUIPO	EU International Property Office
EVBN	EU Business centres in Vietnam
FAQ	Frequently asked questions
HD	Helpdesk
IBEPI	Ibero-American Industrial Property Programme
IHD	international helpdesk
INPI	National Institute of Industrial Property (IP office France)
IP/IPR	Intellectual Property/Intellectual Property Rights
ITA IPR Desks	Italian Trade Agency Intellectual Property Rights Desk
KPI	key performance indicator
LA	Latin America
LAHD MERCOSUR	Latin America IPR SME Helpdesk Mercado Común del Sur (Southern Common Market 95 )
MS	Member States
SEA	South-East Asia
SEAHD SME(s)	South-East Asia IPR SME Helpdesk small and medium-sized enterprise(s)

1.INTRODUCTION

The measure "COS-WP2014-2-05 China, Latin America and South-East Asia IPR SME Helpdesks" within the COSME programme 96 set up the International Intellectual Property Rights SME Helpdesks (IHD) with the aim to offer free information and advice services on relevant Intellectual Property (IP) issues to European Union (EU) SMEs doing business in China, Latin America (LA) and South-East Asia (SEA).

The aim of this evaluation is to check to what extent the IHD services have reached their objectives in terms of relevance, effectiveness, efficiency, coherence and EU value added. The present evaluation supports the mid-term review of the Competitiveness of SMEs (COSME programme and the discussions on the future multiannual financial framework.

The results of this evaluation will also help decide on how best to provide IP support with different types of partners in a more efficient and effective ways.

The evaluation covers 28 months of the IHD activities in China and SEA and 22 months for LA i.e. the years since the pre-COSME activities of the IHD (China from 2010, SEA and LA from 2013) and the current COSME grants i.e. from 2015 until its situation is captured in April 2017.

2.BACKGROUND to the IHD measure

The IHD initiative found its roots in the European policy relating to research and SMEs support, industrial policy, Intellectual property Right (IPR) 97 policy and innovation policy. Before the initiative started, the landscape was a more limited exploitation of international opportunities and innovation prospects both in- and beyond the Single Market. SMEs usually received help in relation to IPR via their national export agencies, EU MS support administrations; EU and MS trade representations, chambers of commerce and in-site via bilateral chambers of commerce or EU and MS trade representations. Against this framework, the first European IPR Helpdesk was created in 1999. The aim was to assist EU SMEs and beneficiaries of EU-funded programs which frequently do not have the knowledge and resources to protect their intellectual assets through Intellectual Property Rights (IPR) and build their competitiveness effectively around it, and need to reduce the risk of their intellectual assets being used by competitors or, if protected, being infringed; in dealing with IPR matters.

Next, in the framework of increasing globalisation, to support EU SMEs both protect and enforce their IPR while trying to enter and grow in third countries in particular on the Chinese 98 , SEA and LA markets, the IHD, were established. The three regions targeted by the action comprised high-growth markets where EU businesses, and especially SMEs, could find many expansion opportunities. The initial experience with the China IPR SME helpdesk (CNHD) run as a pilot until mi-2008 and continued under the Competitiveness and Innovation Programme (CIP) 99 from December 2010. The ASEAN and MERCOSUR HDs started under CIP 100 in 2013. This help was however limited regionally and in scope. The CNHD covered mainland China while the MERCOSUR HD was limited to the MERCOSUR countries and Chile. The number of materials that was produced and the range of activities were also more limited. Under the CIP programme, the European Commission (EC) was responsible for the implementation of the China, ASEAN and MERCOSUR helpdesks.

Since 2015, the IHD project is funded by the EU via the COSME programme. The COSME programme takes a more targeted approach and is aimed at improving the business environment to enhance competitiveness by focusing on framework conditions, entrepreneurship, and access to finance and/or markets. It is the Executive Agency for Small and Medium Enterprises (EASME) 101 that is entrusted by the EC with the implementation of, inter alia, the supervision of the IHD. The new version of the CNHD covers Hong-Kong, Macau and Taiwan. Similarly, the MERCOSUR grew up into the LAHD, covering more countries. Under the COSME programme the IHD were asked to develop a more customised approach including e.g. sectorial publication. In 2015 102 , a common web portal started to provide a single-entry point for all three regional entities of the IHD.

The mission to the IHD set in COS-WP2014-2-05 was to

1.Set-up and operate the IHD

2.Provide advice and support services to EU SMEs facing IPRs difficulties in or arising from China, SEA and LA

3.Maintain the link with intermediaries and stakeholders including the Enterprise Europe Network (EEN), industrial associations, and pan-European business support organisations in the target regions

4.Adapt the work to the target group on demand incl. by developing a categorisation according to the sector and geographical area to which users pertain

Each IHD has in particular the specific objectives to:

1.reduce the risk of counterfeit products entering the European market from the concerned Markets;

2.protect SME's valuable IPR before entering and while doing business in the concerned markets (e.g. developing the market there for their own products or when following a company to which they have traditionally been a supplier when it enters China, LA and SEA);

3.help SMEs to find and constructively work with local administrations or service providers to enforce their IPR protection

These specific objectives had to be achieved through a number of activities

1.Provide 1st line advisory services on IPR protection and enforcement free of charge

2.Develop and publish background materials (incl. publications, factsheets) on IPR for EU SMEs

3.Provide training to EU SMEs on IPR protection and enforcement

4.Monitor national IP policies and legislation

5.Provide partnering services

6.Communicate and disseminate including by hosting and operating a single multilingual web portal

7.Put in place joint actions and other innovative services to EU SMEs

Objectives of the IHD

3.STATE of PLAY

3.1.The functioning of the IHD and to what extent they meet the requirement set in COSME

·The IHD started function in 2008 in China with the CNHD, followed by the ASEANHD and Mercosur HD in 2013. The latter grew up into the LAHD in 2015. One consortium implements each of the regional entity of the IHD. There are three consortiums in total. The LAHD consortium has a stronger academic component than the two other entities. The IHD currently offer, free of charge, first-line support in the area of IPR to EU SMEs involved in business activities in or relating to China, LA and SEA.

·Each regional entity of the IHD provides a helpline, trainings, publications, a website, and partnering services, monitors national IP policies and legislation, and cooperates with other existing organizations and institutions which deal with IPR protection in China 103 , SEA 104 or LA 105 . The cooperation consists in providing complementary services and signposting users to each other depending on their needs.

It is difficult to quantify the amount of SMEs benefiting from the IHD services. However, the quantity of users registered for the helpline gives some indication on the quantity of SMEs reached by the IHD. In April 2017 the IHD had registered 3154 helpline users (1347 for China; 590 for SEA and 1217 for LA).

Users are mainly SMEs: 75% of the survey's respondents were SME and the IHD members confirm that they are in daily contact essentially with EU SMEs. On a closer look at the survey's respondents, SMEs from the manufacturing branches stand out as sector (31%), while the second largest sector of respondents are service providers of different kinds (i.e. Legal, business/management services or R&D consultancy firms).

·The IHD operate under a grant agreement 106 signed with EASME, which handles the technical and financial management. The EC provides general policy guidance. The IHD measure is steered with the help of around 50 different KPIs. Based on the performance reported to EASME in April 2017, the table below shows selected KPIs and the main achievement of the IHD as of that date. At that time, there were eight months left to reach the indicated goals (i.e. less than 30% of the contract time left).

·Table 1: IHD Main achievements, in April 2017

	CHINA	SEA	LA
Number of registered helpline users	1347	590	1217
Number of enquiries to be answered	1509	652 107	687
Trainings modules to be developed	20	11	25
Training sessions to be delivered	125	60	60
Webinars to be delivered	37	38	6
Number of training participants	7682	3578	1655
Number of partnering events attended	78	38	0
Number of visits to the website	119.692	32.479	20.000 (estimated)
New guides & other publications	20	32	10
Articles published	239	230	K.A.

Source: EASME Policy Reports

·In April 2017, the state of play was that the helplines, overall, experienced high levels of demand, with the number of registered users and enquiries exceeded in two regions, and despite related challenges reported by the LAHD 108 . The websites will in all likelihood not be used as much as expected (in terms of page views and visitors), judging by the data available. No more than 50% trainings had been organised and only one IHD entity had held more than 50% of the webinars. This is also reflected in an overall low number of participants in the training. For publications, the relevant KPI seems slightly more positive than for the training.

·As can be seen from the Figure below, out of the range of services offered by the IHD, the website and the publication are the most used. More than 50% of the users surveyed have occasionally visited the website, 10% visit it frequently. Around 45% have used the publication offerings at least occasionally, and around 10% use them frequently. The helpline services are ranked in third position as regards the frequency of use. Around a quarter of the respondents have used them occasionally.

·Figure 18 Usage frequency of different types of services offered by the HDs, SMEs in absolute numbers, overall

·Source: Stakeholder's consultation (n=71)

·From a regional perspective, the online training offerings appear as more frequently used by SMEs in SEA and China, whereas the figures in LA are significantly lower (around 20% occasional or frequent users versus 32% and 45%, respectively). Some intermediary organisation reported indeed webinars organised by the LAHD with a low number of participants. It appears also that the direct training offerings of the CNHD and the SEAHD are used more frequently than in LA (over 30% of the respondents using in Asia at least once against only 25% in LA). The number of training events organised in China is the highest. The SEAHD has, on the one hand, organised fewer training events than the LAHD. However, these events have reached a higher total number of participants.

·Use of the helpline appears as more frequent in Asia than in LA: 70% of the Chinese and 50% of the SEA HD survey' respondents have used the helpline at least once. In LA this number drops to around 38%.

Figure 19 Usage frequency of selected services offered by the HDs, by region

Source: Stakeholder's consultation n=24-36

3.2.Previous evaluations and reports

This is the first evaluation of the three regional entities of the IHD. However, the CNHD was already evaluated once.

·According to the assessment of the CNHD that was conducted in 2010 109 by an external contractor, among SMEs users, 75% believed that the CNHD is needed and at least 60% that it provided good services. Face-to-face events, training and workshops generate more added-value.

However, 40% of users expressed a certain degree of dissatisfaction, which might have been linked to the difficulties they face to understand the content of the CNHD offer. Future help should focus on enabling SMEs understand and comply with Chinese administrative process including by putting them in contact with the relevant organisations/person. The acknowledged high-quality information services could benefit from a categorisation according to the sector and geographical area to which users pertain; increasing the range of materials provided to SMEs that have never been confronted before with IPR issues, with information produced in Chinese and in relation to the situation in China. A list of lawyers in China could help with infringement issues. The users are very satisfied with the high quality of the experts used in China, but would appreciate the inclusion of a commercial and multicultural profile, and a consistent presence in Europe to develop relationships with intermediaries at EU level.

·More generally, building relationship with intermediaries by notably increasing presence in trade fairs and cooperation with trade missions, as well as sustaining this relationship by notably increasing the training for intermediaries' staff would help raise awareness amongst SMEs and the visibility of the CNHD.

A LAHD 110 conducted from November 2015 to March 2016 with a total of 344 LA SMEs respondents, distributed across most of the LA countries analysed that efforts should be focused on providing tailor made actions in the form of helpline support and training preferably via Webinars, employing a jargon-free language, tailor-made for an SME and ideally to its sector; providing access to material such as fact sheets or tips, videos and case studies, and complementing trainings with networking activities like meetings, B2Bs where SMEs will have the possibility to connect with each other and with other intermediaries dealing with innovation as well as universities, technology and innovation centres, chambers of commerce, trade associations, etc. in strong synergy with key actors in the IP context including the LA countries’ National Patent Offices.

On the basis of the 2010 assessment of the CNHD and of the Pilot Study in LA, under the COSME programme, the IHD were asked to develop a more customised approach including e.g. sectorial trainings or publication.

Previous evaluations or studies have also been carried out by the Commission. In relation to the same activity but in a distinct geographical area, the evaluation of the European IPR Helpdesk 111 run by an external contractor in 2013, concluded that some aspects of the services had to be improved in particular that a greater emphasis should be put on the promotion of the EU Helpdesk services, via in particular the implementation of a clear and effective marketing strategy and Communications Plan, the development of closer relationship with multipliers, and a promotion within the Commission services; that the possibility of providing greater flexibility within the contractual form, while still complying with procurement law needed to be explored further, streamlining the reporting requirements of the Helpdesk contract, including in the report information on performance indicators.

In addition, the IHD themselves carried out an own Internal impact assessment exercise in 2016 112 .

4.METHODOLOGY

The evaluation of the IHD was conducted between 2 December 2016 and January 2018 and was steered by the members of an Inter Service Steering Group involving various services of the Commission. It has been partially supported by an evaluation study i.e. "Support Study for the Evaluation of the International Intellectual Property Rights SME Helpdesks" prepared by an external contractor 113 . The contractor was primarily asked to get stakeholders views (through surveys and interviews) on the functioning of the IHD and on the impacts on the objectives in terms of relevance, effectiveness, efficiency, coherence and value added and provide an analysis of their replies; and to analyse the coherence of the IHD with other COSME, EU and/or MS SME support measures, as well as other relevant programmes contributing to the internationalisation of European businesses and their competitiveness in China, SEA and LA. Both the external study and the current staff working document of the Commission services have been reviewed by three independent experts selected among a list of five proposed by the external contractor. Their views on the relevance and coherence of the IHD initiative can be summarised as follow:

On substance, the three experts limit their analyse of the relevance aspect to analysing whether the IHD initiative can still be considered relevant for achieving the objective of helping SMEs dealing with IP matters in the relevant territory. One expert applying a systemic approach and considering the Chinese angle only considers that at overall level, the EU IP system is obsolete. Because of the sheer market size and relevance for value chains, companies are forced to become active in China, and to cope with detrimental policies and regulations that lead to actually unwanted know-how flows. Measures like the IHD initiative can at best slow this trend down but cannot help in avoiding it. A second expert views the initiative as overall still relevant for achieving the desired objectives, also in the context of China which has shown considerable changes in its IP system. At micro level and under the assumption that the IHD would be largely functioning as intended, the first (China focussed) expert, still views the IHD initiative as not relevant. Reasons put forward are that SMEs are mostly not in a position to enter China. IP issues apart, these companies would find it difficult to enforce their rights because of lack of resources particularly financial ones. Moreover, the IPR system conceived in Europe in the 18th century is outdated. It cannot match current shorter and shorter innovation cycles. As a result it limits innovation. Innovation requires increasingly permeable markets. The second expert, a patent attorney, views SMEs from big EU countries as less in need of the IHD services because they benefit from a very dense and professional network of IP advisors. His personal practical experience (in Germany) shows a lack of visibility of the IHD. His clients hardly mention the IHD services, while his colleagues have usually not heard about the IHD (at least in sufficient details). The third expert considers that the little use and take–up of the IHD services, acknowledged by the KPIs, could indicate a limited relevance. This expert focuses on the methodological aspect of the analysis and considers that most elements presented in the external study and the staff working document of the Commission services are available for an adequate assessment of relevance of the IHD initiative. However, the analysis of the quantity of SMEs benefitting from the service should be expanded to a more detailed analysis of the type of enquires/type of topics covered – which could inform an assessment of the nature of needs which SMEs have in relation to IP issues.

Having acknowledged their lack of expertise in general EU policies, the experts analyse that the coherence with other EU policies could be limited due to a lack of collaboration with the European Patent Office resulting from an originally present personality having gone, while the fact that the China IPR SME helpdesk does not differentiate Taiwan and mainland China as different territories would be a clear sign of in-coherence. From a methodological point of view, overall, the data which is presented in the external study and the staff working document provides a good basis for assessing coherence of the IHD initiative. The finding of the staff working document that IHD should ensure better communication with other COSME initiatives and improve collaboration seems relevant given the findings on coherence. However, given that all the initiatives considered are funded and managed by the EC, there would be benefit in considering how the EC could promote and facilitate collaboration. 114

The following evidence has been used in the evaluation:

-COSME Regulation (EU) No 1287/2013 115

-Competitiveness and Innovation Programme (CIP) 116

-Competitiveness of SMEs (COSME) programme 117

-For China, the CIP work programme 2010 (ENT/CIP/10/B/N02C00) 118 and 2013 (ENT/CIP/13/B/N02C00) 119

-For LA and SEA, the CIP work programme 2012 (ENT/CIP/12/B/N04C00) 120

-2012 Grant Agreement for the SEA IPR SME HD 121

-2013 Grant Agreement for the Mercosur IPR SME HD 122

-2015 Grant agreement for the China IPR SME HD 123

-2015 Grant Agreement for the SEA IPR SME HD 124

-2015 Grant agreement for the LA IPR SME HD i.e.

-Original Grant Agreement 125

-Annexe 1 to original Grant Agreement 126

-Amended Grant Agreement 127

-As well as information given by EASME including quarterly, annual, and biannual interim reports submitted by the three IHD entities, as well as Policy Officer (PO) reports and other deliverables.

A number of different stakeholders were consulted through web surveys and 39 interviews. Separate ad-hoc survey were run for SMEs users of the IHD and potential SMEs user of the IHD located in the EU and already doing business in the target region(s) or considering internationalisation. The first category was reached through each IHD, while the survey addressed to the potential users of the HID was circulated via intermediaries like, IP support services, embassies, chambers of commerce, etc.

Other stakeholders consulted were: business representatives of SMEs like associations; chambers of commerce, industry associations, sectorial associations, etc.; national export promotion agencies of EU MS and bilateral EU MS-China, SEA and LA business associations; experts/consultants/service providers on IP, SME and internationalisation; MS ministries steering the COSME programme and MS trade counsellors from embassies in the target countries. In addition, a public consultation was carried on for the overall COSME programme 128 .

It is important to note that the data from the surveys come from a non-representative sample of SMEs. Moreover, despite the efforts to maximise the replies to the surveys, the overall number remained quite low: 70 replies for the users survey and only 41 for the non-users survey.

The limitations of the dataset collected via the survey was balanced, where feasible, through triangulation with complementary data sources, first of all the Key Performance Indicators reported for the evaluation period by each consortium to the Executive Agency for Small and Medium-sized Enterprises (EASME) and secondly interviews with the helpdesk personnel, relevant staff of the EC namely DG GROW, DG TRADE and EASME, SMEs, intermediaries e.g. business consultancies, stakeholders and similar service providers. Other sources of information came from existing materials i.e. for each regional entity of the IHD, the grant agreement (defining in particular the KPIs), the table describing the work in terms of a list of deliverables and KPIs and their current status (pending, submitted or approved), the Progress Report, the Periodic Report(s); EASME's three "PO Assessment Report" assessing the first Periodic Report of each regional entity; other background documents i.e. agenda, minutes, mission report and action plan of the monitoring visits performed to Alicante (LAHD) and Beijing (CNHD); handover documentation of the CNHD funded by the CIP programme; and the results of an internal impact assessment exercise carried out by the IHD themselves in 2016.

While the interpretation of the data presented needs to be done with care some important conclusion could be drawn.

The survey with users and non-users of the IHD was executed electronically between August 2017 and September 2017. Information about the synopsis report on the consultations 129 , as well as the survey respondent recruitment, response rates and respondent characteristics 130 and the survey questionnaires 131 can be found as annexes to the Study. In the same way, the Study provides information about the interview programme, interview guidelines, and interviews synthesis 132 .

5.Analysis and answers to the evaluation questions

In accordance with the Better Regulation guidelines 133 , the evaluation has to determine on an evidence-basis whether the actual performance of the IHD has been at least relevant given the needs and its objectives, effective and efficient, coherent both internally and with other EU policy interventions, and has achieved EU added-value. To guide this analysis, a set of evaluation questions have been asked in relation to each of the five criteria. They are mentioned below.

5.1.RELEVANCE

A key consideration for the evaluation is whether the results obtained by the IHD are still regarded as useful and pertinent to the current needs of users. Evidence collected from existing studies as well as the result of the evaluation show that the IHD serves the most relevant needs of SMEs in the area.

5.1.1.Is the need to protect IP still relevant?

Overall IP remains a major part of the value of companies, and IPR an important element for a significant amount of businesses: several recent studies 134 , among which the OECD/EUIPO 2016 135 , show that as much as 5.1% of EU imports in 2013 was in counterfeit and pirated goods, amounting to Euro 85 Billion. China is the Number one provenance economy of counterfeit goods entering the EU while SEA (Singapore 9th, Thailand 15th) and LA countries appear in the top-15 list. Similarly, the 2017 edition of the special 301 report by the United States Trade Representative – a report on the state of IP protection and enforcement in US trading partners around the world – places 11 countries on the top priority watch list for issues in relation to IP enforcement. This list includes China; Indonesia, Thailand, Argentina, Chile, and Venezuela, i.e. six countries that are addressed by the IHD. 136 The 2016 "European SME Exporting Insights Study", a survey of 2016 of 12 815 SMEs, conducted by logistics firm UPS, in eight major EU countries shows that the share of exporting SMEs in the EU has steadily increased between 2014 and 2016. There may be as many as 1 million active exporting SMEs in Belgium, France, Germany, Italy, the Netherlands, Poland and the UK 137 . Similary, a survey of KfW Group in 2016 (n=600) revealed that about 50% of German SME turnover in the manufacturing industries is accounted for through export activities, and “…89% of manufacturing SMEs sell into foreign markets, either directly or via their exporting customers.” 138 . Hence there is a strong rationale for still supporting EU firms in enforcing their IP rights in these regions. In addition, all these studies point out a) that there is still a problem with IP infringements, in particularly in third countries targeted by IHD and that b) a sizable number of EU SMEs is likely to face in particular while exporting 139 .

·A large majority of the users surveyed (80%) consider the assistance that the IHD provides is essential both when planning to expand to a third country and once they have already entered the market. This could indicate that support for IP management among SME is still a relevant issue in the context of their international activities while still operating in the EU and planning to expand outside Europe and once they have entered a third country.

Five of the seven specific actions of the IHD are oriented towards satisfying the SMEs’ need for information and advice. The results of the survey rank as first priority the "information on the difference formal ways to protect IP" (59%), followed by "information on registration and other procedures" (41%). This can be seen from the following figure.

Figure 20 Share of users with needs for advice and support on specified topics, by HD, in %

Source: Stakeholder's consultation

Interviews corroborate this finding as well. Overall the need to get reliable information to protect and manage adequately IPR in the different targeted markets is the most important mission of the IHD for SMEs users. This comes over operational assistance like assistance with licensing (16% for the three regions) or confidentiality agreement (15%) or in situations of infringements. The IHD are also not especially asked for support in implementing IP management (28% for all regions) strategies (except for the LAHD) nor for mediation (6%). This shows that the focus on provision of IP expertise offering a relevant information and support adapted to the targeted markets (via the helplines, trainings, publication and website services) remains pertinent.

The survey gives also interesting information about how important the EU SME’s users view the specific features of the IHD services.

Figure 21 Importance of various aspects of the services provided in the view of users

Source: Stakeholder's consultation. n=56-58.

Overall, user and non-user SMEs particularly value the importance of having local support outside the EU (around 60% rate it as "very important"), the quality of advice (more than 70% of the users’ sample), and the depth of information of the services (around 50% users and non-users). EU SMEs using the IHD value also highly the fact that their services are offered free-of-charge (more than 50% rate it as "very important"). Above all, these findings confirm that all the characteristic features of the IHD offer, as initially thought, are regarded as important and, in a high proportion "very important, by around 90% of the surveyed SMEs. This is clearly a clue of relevance.

Another indicator for a high relevance of the IHD is the finding that more than around 80% of the survey' respondents thinks that the services should be extended to other areas of the world, India coming as first mentioned country.

5.1.2.Should the scope of the IHD be broadened to any IPR issue potentially faced by EU SMEs (including litigation)?

·Evidence shows that there is an interest in deepening the services within the scope currently offered. In general, a large majority of SME users (more than 80%) responding to the survey indicate that it will remain of special importance to obtain further information on the different ways to protect IP, as well as further assistance with registration and other procedures. This matches the top two needs already expressed 140 . Between 70% and 80% of the survey' respondents consider also important or very important for the future to obtain more assistance with confidentiality agreement, licensing, alternative dispute resolutions or litigation and enforcement. Disparities appear between the different geographical areas. In particular, EU SME in China strongly claim for more assistance with confidentiality agreements (95%), while litigation and enforcement issues, including alternative dispute resolutions, appear to be a of special interest in SEA (more than 80% of the replies). From the interviews, the IHD members, based on their experience, see an interest, for users, in benefitting from support during mediation efforts in SEA (where it could be offered during trade fairs), and in China (where such service is, as a matter of fact, already currently provided). This evidence shows that when the respondents did not rank these items among their priority needs, they still value them for the future.

·Overall, the original range of services as defined in the framework of the COSME programme seems to fulfil the needs of EU SMEs facing IPR difficulties in or arising from China, SEA and LA. Interviews do not specifically outline new services to be added to the current offer from the IHD. The demand focuses more on a deepening of the IHD expertise than on a broadening of their current range of services.

Another issue is whether the SME focus is still relevant. The target group of the IHD is SMEs according to the EU definition and firms slightly above this size limit. National support services usually do not have such a company size restriction. Question arises as to whether this target group definition limitation still makes sense. Interviews evidence that the IHD members consider the SME focus to be very relevant first because SMEs are the backbone of the EU economy, second because the high number of inquiries and participations in their activities shows that EU SMEs feel concerned. A possible inclusion of larger companies, which might miss expertise about new markets, is also voiced. On a closer look, the CNHD and SEAHD are aware of the SMEs specific needs and use a tailored approach whereas the LAHD considers it to some degree challenging to cater for SMEs as they are very heterogeneous. SMEs tend to perceive the topic of IP as relevant only to larger firms, which, in itself, shows the need to make SMEs sensitive to it. Most stakeholders, especially similar service providers, argue that SMEs have a higher need since they tend to be under-informed about IP matters. Some chambers of commerce especially in SEA argue that it is usually larger companies who have high investments in the area that should be protected. However, these companies do usually have their own lawyers.

To conclude on this issue some elements need to be further analysed, in particular, statistics on the quantity of SMEs benefiting from the IHD services. There is no indicator on the quantity of SMEs facing IPR difficulties in or arising from China, SEA and LA that the IHD have reached. The number of users registered for the helpline only gives some information about the quantity of SMEs reached by the IHD. In April 2017 the IHD had registered 3154 helpline users (1347 for China + 590 for SEA + 1217 for LA), and this often despite the absence of an efficient IP tool for such registration. The CNHD and LAHD over performed their target, while the SEAHD achieved less due to a belated implementation of the helpline. With 1347 registered helpline users against 1000 targeted, China performs well. According to the European SME Exporting Insights Study 2015 141 , only around 42.900 out of the 390.000 SMEs of seven large EU MS 142 from four sectors 143 export to China (which represents 11%). With 1347 SMEs registered as helpline users, the CNHD reaches around 3% of these EU SMEs. On such basis, the IHD could still substantially increase the number of their users in the future.

Conclusion on relevance:

Evidence shows that the IHD have been well designed to make EU SMEs aware about the need to take care of their IP rights before and after entering the targeted third countries markets. The IHD do this by providing EU SMEs appropriate information and encouraging them to take the rights measures to protect their assets. In this, the IHD initiative potentially contributes to the overall objective of the COSME programme to strengthen competitiveness of EU SMEs. It could contribute as well to the specific objective of reducing counterfeit. For both objectives the IHD need to reach the critical mass of EU SMEs.

5.2.EFFECTIVENESS

5.2.1.How effective the IHD action has been in achieving or progressing toward their objectives?

The COSME’s programme set a mission to the IHD which require the implementation of specific actions: first line advisory services, trainings, publications, website, partnering services, communication and dissemination, joint action with other services to EU SMEs.

In terms of general assessment of effectiveness, the evidence suggests that the IHD successfully achieved providing useful advice and support services to EU SMEs facing IPR difficulties in or arising from China, SEA and LA. The main positive effect resulted in actually raising the SME’s awareness about the necessity to take care of their IP rights.

According to the survey, over 80% of the user SMEs responding consider the IHD overall to be "effective" or "very effective" in raising awareness about the value of intellectual assets. 70% find them effective in raising awareness of the necessity to secure and manage IP in the countries covered.

The effectiveness analysis focusses on the results that the IHD have been delivering during the evaluation period in the light of the mission set by the COSME Programme. It should allow determine also to what extent these achievements have positive effects on the three specific objectives of the IHD initiative 144 .

The efficiency analysis considers first whether the IHD optimised the use of their resources given the obtained (quantitative and qualitative) results.

Effectiveness of the IHD measure in achieving its mission

a.Objective 1: Operate the international IHD

As evidenced from their website, KPIs, quarterly reports and other deliverables, the IHD went operational. By the end of 2016, an IHD has been settled and offered to SMEs in Mainland China, Hong Kong, Macao, Taiwan; LA (covering Brazil and Mexico as well as other countries in the area like Argentina, Bolivia, Chile, Paraguay, Peru, Uruguay and Venezuela, and some central American countries depending on EU business interest); and SEA (covering Indonesia, Thailand and Vietnam as well as other countries in the area like Brunei, Cambodia, Laos, Malaysia, Myanmar, the Philippines and Singapore). To offer such services the main project teams and physical contacts points in Beijing; Brazil, Mexico and Chile; Indonesia, Vietnam and Singapore; and Belgian and Spanish European-based offices, started to operate in 2015, after appropriate staff recruitment and equipment bought.

Its implementation experienced however some slow performance in certain areas. It is only in October 2015 that the IHD services could become fully available in SEA. The IHD experienced also some difficulties to develop activities in Brazil and Mexico due to external hindering factors. The most critical problem that hindered the implementation was the absence of a common website during more than one year. It is only in March 2016 that a common website portal was developed.

It is to be noted that despite this major drawback, all key services i.e. the enquiry helpline, publications on IPR-related issues, policy input, the provision of training to EU SMEs have however been delivered.

b.Objective 2: Provide advice and support services to EU SMEs facing IPRs difficulties in or arising from China, SEA and LA

To answer this question, four points must be checked:

1.Did the IHD succeed in reaching the targeted EU SMEs facing IPR issues?

2.To what extent the targeted SMEs are aware of the existing IHD services?

3.Have SMEs users found the services that meet their needs?

4.How do the surveyed users assess the IHD initiative?

1.The IHD reach out the targeted EU SMEs and intermediaries

As already mentioned, the IHD progress reports do not provide data about the quantity and the typology of SMEs that use their services. The only available indicator is the number of registered helpline users. It is reported that the CNHD and LAHD have already over performed their target (1347 registered against 1000 targeted for the CNHD and 1217 registered against 800 targeted for the LAHD). Due to the belated implementation of its helpline, the SEAHD reached only 590 users (1000 targeted) but should increase significantly this number.

These results are encouraging and might increase considering the potential of EU SMEs facing IPR protection problems (as already seen above the CNHD reaches only around 3% of the EU SMEs that export to China).

The recurrent problem is that it remains complex to know precisely what kind of users call upon IHD services.

The survey' respondents gives a first idea. As mentioned in the state of play, the largest group of respondents (around 75%) is made up of companies having less than 50 employees.

·Among these EU SMEs, the different sub-sectors of the manufacturing branches dominate the sample. 31% or the respondents come from this group. Logically, the manufacturing sector gathers most of the SMEs facing IPR issues. The second largest sector in the sample is represented by SMEs operating as service providers of "legal, business/management services" or "Engineering, R&D consultancy services". The two sectors represent cumulatively 35% of the sample. Such a high proportion is questionable. While the COSME Programme does not list law firms among the list of its beneficiaries but does not exclude them either 145 , it is dedicated to support entrepreneurs and their representatives. The primary objective of the IHD is not to provide advice to law firms, or business consultancies intermediaries. When such intermediaries are involved, the link between the service provider and SMEs is not direct. In such cases, the target (i.e. SMEs) could appear as somehow missed. On the other hand, this could show that the IHD offer their IP services to law firms, which in turn act as multipliers by offering to their final clients (i.e. SMEs) a broader service including IP. Interviews among the IHD members and a panel of intermediaries support this view. The role of intermediaries could reveal problematic in case they would divert SMEs from the free of charge IP support services offered by the IHD. To avoid this negative impact, care should be taken to insist on the promotion of the IHD services to increase their visibility. This is all the more true considering that, as already mentioned (see relevance part), there is a gap between the number of EU SMEs registered as users of the helpline of the IHD and the number of SMEs facing IPR difficulties in or arising from China, SEA and LA that the IHD could potentially reach.

Both the sample composition and the number of helpline registered users concur in showing that the IHD might to some extent have been effectively reaching the objective of providing IP support and advice to an adequate number of EU SMEs users.

2.The IHD do not get sufficient visibility

·The effectiveness of the IHD depends on how they reach out to their beneficiaries notably by promoting efficiently their services. The visibility of the IHD is limited. In the views of the SME users responding to the survey, overall, around 50% find the IHD to be promoted very well or well. On the other hand, nearly 30% find the promotion to be insufficient and another 14% do not know how well the IHD were promoted. In comparison to other survey responses, these numbers seem overall low and could indicate room for improvement. This reveals particularly true in LA, where less than 50% of the users surveyed judge the promotion to be very good or good. The lack of attendance to partnership events reported by the LAHD could corroborate further this finding (see below "efficiency" part). Around 30% of all respondents think that the promotion of the IHD is insufficient. Some interviews point out that this could be related to the fact that the LAHD is managed from Europe and not "on the ground" in LA, others link this to the fact that LAHD is the youngest, covers more countries and does not benefit from local political support as much as other regions do e.g. China. The interviewed stakeholders from the EC confirm that the IHD have difficulties in reaching out to SMEs and creating sufficient visibility. Of even more concern is that among the sample of 43 SMEs non-users surveyed, no one answer that they know the services of the IHD.

It is interesting to see below the difference in perception of the IHD promotion between the group of SMEs and the group of ‘intermediaries’ (expert promotion agencies, business association, innovation support organisation, law firms, IP professionals).

Figure 22 Promotion of the helpdesks in the view of user SMEs and user intermediaries

·Source: Stakeholder's consultation=64 (user), n=27 (intermediary).

·Among the intermediaries group, more than 80% state that the IHD are promoted well or very well. This suggests that the IHD promotion has more effect on intermediary organisations than on the SMEs themselves. As such the intermediaries appear as a successful alternative way to compensate a lack of direct contact with SMEs. However, since the main target group of the IHD are EU SMEs, it appears pertinent to develop communication and dissemination to gain more visibility as well as to provide partnering services.

3.The IHD meet the EU SMEs needs especially by raising their awareness for IPR protection

·With the objective to provide effective supports to EU SMEs, the IHD are requested to implement different specific actions such as the provision of first line advisory services, development of trainings, publications and of websites. The figure below provides a view on the effects of these actions.

Figure 6 Effectiveness of helpdesks in the view of users across various aspects

·Source: Stakeholder's consultation. n=45-47.

·Over 80% of the user SMEs responding to the survey consider the IHD overall to be at least effective 146 in raising awareness of the value of intellectual assets; 70% find them effective in raising awareness of the necessity to secure and manage IP in the countries covered. The IHD are viewed as less effective in solving concrete IP problems in the countries covered (just under 60% find them at least effective) and in developing capacities in SMEs to deal with IPR (just over 50%).

·These results make sense while considering the most important needs for IPR advice and support ranked by the respondents in the survey as shown below.

Figure 7 Share of users with needs for advice and support on specified topics, overall, in %

·Source: Stakeholder's consultation. n= 60

·The parallel display of the two above Figures tends to shows that the IHD are effective in catering the major need of the EU SMEs. EU SMEs first require being in a position of managing their IP issues in a specific geographical area by accessing the relevant information, as well as adequate support services and advices. The IHD appear as less effective on other aspects that the survey' respondents consider less important such as assistance in dealing with concrete IP problems. EU SMEs value more receiving a reliable upstream of information and advice for implementation than to obtain a downstream assistance for concrete IP management.

·A comparative analysis of the effectiveness of the three entities of the IHD shows significant regional differences. The users surveyed homogeneously value the "raising awareness of the value of IP" and "raising awareness of the necessity to secure…" aspects. On all the other aspects listed under Figure 6 above, the respondents systematically rate the LAHD lower than the two other regional entities. About the objective to "Provide advice and support services to EU SMEs facing IPRs difficulties” the LAHD appears significantly less effective on the "solving concrete IP problems" and "developing internal capacities” aspects. Around 45% of the survey' respondents finds the LAHD effective whereas the SEAHD reaches more than 70%. For the CNHD, the survey' respondents assess the "solving concrete IP problems" aspect in a contrasting way: 58% find the CNHD at least effective, whereas 25% find it "ineffective". This is the highest rate of recorded ineffectiveness.

·Interviews provide some elements of clarification on the level of effectiveness reached by the LAHD. According to some interviews, the LAHD would be over academic and less operational. One interviewee mentions "…that the material provided for by the LAHD is too basic. I would like to have more concrete information on IP protection. However, I am mostly explained that there are things like patents." This view needs however to be tampered. When a low number of respondents (as compared to the CNHD an SEAHD) find the LAHD to be effective on the "solving concrete IP problems" and "developing internal capacities” aspects, the number of respondent finding the LAHD to be "not effective" is the same as for the two other regions. The LAHD simply totalise more "do not know" responses. The gap found on the effectiveness of the operational support provided by the LAHD is more likely to be linked with a later start (6 months) on a wider and heterogeneous territory (19 countries covered against 10 for the SEAHD). The LAHD needs more time to face a wider range of specific situations and to improve its visibility across its specific geographical area.

To conclude, the objective ‘provide supports to EU SMEs’ leads the three regional entities of the IHD to achieve important efforts to implement the first line advisory services, to develop publications and IPR materials, to adapt and deliver training sessions and to put in place a relevant website. These specific actions have a direct effect on raising awareness for IP management issues that could explain the overall positive assessment in term of effectiveness from the survey' respondents confirmed by the interviews. Survey respondents and interviewees agree that helpline, publication and training services are important and useful regardless of the significant gaps that can be observed between targets and achievements (see efficiency analysis below). Despite these gaps and delays in implementing the services, the IHD seem to have successfully provided supports services to SMEs facing IP issues in or arising from the target region and have put into place a mechanism which enables cater SMEs information and advice needs related to IP protection.

4.The survey respondents' views on the IHD effectiveness, by service

The survey and interviews contribute to give an overview of how users might perceive the benefits of the main IHD services i.e. helpline, publication, website and training, in terms of usefulness and quality.

oEffectiveness of the helpline services

The level of effectiveness of the helpline services varies according to their level of use and the degree of usefulness acknowledged by the users.

The “deliverables and KPIs” documents provide information on the level of use. Data reported 147 show that the number of enquiries to be answered already exceeds the target set for the IHD (three entities included) and this long before the end of the contracting period of the IHD initiative. This is a great achievement. The level of users survey satisfaction with response to enquiries is only mentioned about LA (95,7% achieved against 99% targeted). The corresponding input of the other regions is unfortunately missing.

From a qualitative angle, the survey reveals in practice that around 80% of the respondents assess the listed features of the helpline services as "highly important" and "important". In particular, providing "business oriented information in an understandable way for non-legal experts", and offering a "first-line support" in a "user-friendly" way, are the highest ranked aspects (around 90%). Asked about the usefulness of the same features, nearly all survey' respondents find the helplines provided by the IHD to be "useful" or "very useful" (more than 90% at least useful). In China more than 60% of the respondents rate these services as "very useful", whereas only 40% of LA ones and 0% of SEA ones do. These findings are corroborated by interviews of the IHD according to which the need among SMEs for the helpline is high.

·Similarly, in the views of most respondents to the survey, overall, the IHD helpline service sufficiently covers the four main IP topics (i.e. IP Protection, IP Checking of agreements, IP Management and IP Exploitation). In particular, the coverage of IP protection topics is considered to be sufficient by more than two thirds of the respondents. This can be seen as a positive result, considering that IP protection is one of the top needs for advice and support mentioned by the respondents. This indicator drops to 50-60% for the other IP topics but significant numbers of respondents were not able to answer this question and thus opted for the "do not know" option. This high percentage of "do not know" answers is not surprising when considering that a significant part of survey' respondents considers that the IHD services are not sufficiently promoted. Improved promotion of the IHD services might help more users and EU SMEs in general to know about the IHD and/or to know well about the scope of services they offer.

·Most survey' respondents indicate that an avenue for improvement could be to expand the subject scope of the helplines (nearly half of them). This evidence an interest in the development of the helpline services, as well. In addition, neither the survey nor the interviews reveal any dysfunction or notable inadequacy that would need to be fixed. This tends to confirm also a significant level of satisfaction with the helpline service.

·To sum up, for the respondents to the survey, IHD helpline services are useful. They cover aspects and topics that they consider important in relation to their IP management needs.

oEffectiveness of the publication services

The level of effectiveness of the IHD can also be assessed from the number of publications (guides, training materials, newsletter, articles) written by IP experts for SMEs that they offer, the level of importance given to them by SMEs and their usefulness.

·As for the previous services, a high amount of survey' respondents (around 80%) qualifies the different kind of publication services offered (online library, factsheets, FAQs, events lists) as "important" or "very important" (almost one third). In addition, overall, around 90% of the respondents find the publication services of the IHD to be "very useful" or "useful". However, on a closer look, a gap appears between 100% respondents for China on the one hand and 75% for SEA on the other hand. Interviews with the IHD confirm that publication services are ranked as an important part of the offer, just after the helpline.

·Two factors prevent us from appraising in a positive way the level of effectiveness of this service. Firstly the "deliverables and KPIs" documents show a limited level of achievement for the publications services as compared to the targets set 148 . It seems that it will be difficult to fil the gap for the SEA and LA regions. Secondly, a user satisfaction indicator is missing for this service.

oEffectiveness of the online service, website

·Granting access to a useful and performing website is also a key feature of an effective service offered to SMEs. The IHD have appropriate financial means to design and maintain such a website. These means would be considered as well employed when users value high the quality and usefulness of the provided tool. The survey includes one question listing possible main quality features of a website like "The website is easily accessible" or "The content is relevant for us" or "the website has a clear layout", etc. Height items are listed for valuation. A high portion of survey' respondents (between 60 and 80%) agree that the IHD website services display the listed quality features. The item "the content is relevant for us" is unanimously agreed to. This is a very good result considering that it is the most important criteria. Quality features like "the information is easy to find" or "the website has a clear layout" are also well rated. This outcome is however more observable for China and LA respondents surveyed than for the SEA ones. On the other hand, ergonomic criteria like search facilities, design, and consistency of navigation throughout website are more positively rated in relation to the SEA website than for the China and LA one. Indeed, the last two websites gather more negative assessments.

·This overall good qualitative assessment of the IHD websites also appears from the "deliverables and KPIs" documents. They show a high satisfaction rate for China (4,4/5 against 3,5/5 targeted and for LA (80% satisfied in agreement to the target). There is no evaluation of the satisfaction of users about the SEA Website.

·Overall, the websites, as designed, satisfy the respondent to the survey. They acknowledge in unison that the content is clear, relevant, and easily accessible. By contrast some ergonomic features, which collect more negative assessments, should be analysed to find accurate solutions.

Beyond the qualitative aspects, the level of use must be analysed in order to achieve the performance assessment of the websites. In the "deliverables and KPIs" documents, the level of use is assessed by reference to the number of visits. From reported data, it can be derived that the results are well below the targets 149 . This is particularly true for the LAHD and SEAHD. The two IHD entities that tend to lag behind the targeted level of use are the ones which gather the highest number of negative responses on ergonomic features like "website easy to find" and "website easily accessible". From the interviews with the EC, it appears that despite the fact that having a single-entry point for companies via a unique IHD website is a plus, it is a challenge to have a one-size-fits-all website for the three regional entities 150 .

·Although, the websites apparently match the users’ needs as appears from the survey and the interviews, it would be recommended to take appropriate measures to improve their visibility and accessibility, and by extension their effectiveness.

oEffectiveness of the training services

The IHD provide different kinds of training services on IPR issues to their customers, tailored to the needs of SMEs. The effectiveness of the training services depends on some criteria like the different training formats offered, the number of completed actions and the number of trainees, as well as the level of participants' satisfaction.

The survey provides interesting answers about the importance and usefulness of the different training formats offered by the IHD. The formats cover, for instance, live workshops as well as more innovative ones such as online webinars. From the survey it appears that overall more than 60% IHD respondents find five of the six listed formats as "very important" or "important". The response rate for the "not important" option varies from 5 to 15%. The survey' respondents finds workshops and webinars to be equally important to serve their needs. These two formats are the ones which are rated as "most important".

The survey respondents also find the webinars to be the most useful training service (more than 70% of them) but more than 60% of respondents consider also the others mentioned formats to be useful.

Although the IHD entities have properly diversified the training means to adapt to the IPR skills development needs of SMEs, the level of achievement appears more problematic. Here again, the "deliverables and KPIs" documents reveal an important possible future gap between targets and achievements 151 . The IHD have reached less than 50% of the trainings targets. This is true for the number of developed modules, as well as for the number of delivered sessions. The number of participants is proportionally low as compared to the targets. Nevertheless, the reported number of participants, in comparison to the number of delivered training actions reveals a relatively important level of attendance. This finding could be opportunely validated by providing additional information on the average attendance rate e.g. from the attendance sheets.

The third element to be considered to analyse the effectiveness of the IHD training services is the valuation made by trainees based on the evaluation sheets. This data is only provided by the LA entity. It reports that 60,6% of participants are highly satisfied (against a 95% targeted). The other IHD entities do not provide any equivalent information.

To conclude, the IHD receive grants to implement, among other services, training facilities aimed at enhancing SME IPR skills. Various formats (workshop, webinar etc.) were designed and are viewed as important and useful by the survey' respondents. However, the number of developed training modules, delivered actions, and as a consequence of EU SMEs trained remains limited. This finding needs to be taken into account to increase the performance of the IHD in the future.

c.Objective 3: Maintain the link with intermediaries and stakeholders

As already highlighted, the COSME programme mentions that the IHD services will be offered in co-ordination with and in direct response to requests from European SMEs and their representatives (e.g. sectoral, regional, local associations of SMEs, chambers of commerce) 152 .

Checking how the IHD have implemented the specific actions ("provision of partnering services" and "joint actions and innovative services to SMEs") gives a first indication on how the link with intermediaries and stakeholders in the EU or in the target country has been maintained. According to the PO Assessment Reports 153 , the link with intermediaries and stakeholders including the Enterprise Europe Network (EEN), industrial associations, and pan-European business support organisations in the target regions has been maintained during the period 154 . CNHD and SEAHD have attended a certain amount of partnering services and matchmaking events and conferences (e.g. until April 2017, the IHD attended 116 events in Asia and in Europe). They did provide participants with supporting IPR packs of documents and guides. These actions helped facilitate the strengthening of links between EU SMEs and companies in the targeted area. In LA, this activity has essentially taken the form of an initial survey on the training needs of SMEs followed by a few trainings without however reporting any participation in matchmaking events. The CNHD, LAHD and SEAHD progress reports 155 indicate that the three regional entities of the IHD have also successfully developed common actions together or with other relevant initiatives (EU SME Centre, EU Gateway projects, EEN, ELAN network…) both in Europe and in the targeted areas. This collaboration has been fruitful with development of joint trainings and dissemination activities, mutual web linking, joint content materials and joint participations in events and trainings. However, the development of a joint website was not successful. The efficiency analysis of the objective to maintain the link with intermediaries and stakeholders will however reveal discrepancies between the regional performances.

On the qualitative aspect, the above Figure 7 highlights that just over 50% of the survey' respondents assesses as effective the capacity of the IHD in "building long term partnership with relevant EU networks". Additionally, only a bit more than 40% of the respondents assesses the IHD to be effective in "building networks and relations to experts and institutions in the relevant countries". This last finding is somewhat worrying since the specific objective involves the IHD to help EU SMEs work well together with local authorities and service providers in IP. On a closer look, in comparison to the other users in the sample, in LA only a very low number of respondents (30%) found the IHD to be effective in building networks and relation to experts, and long-term partnership with relevant EU networks. This finding could be linked to the fact that LAHD does not report any participation to partnering services and matchmaking events.

According to the interviews and in line with the survey, the IHD could become more effective by increasing cooperation with other EU activities, as well as with national activities. Around 85% of the IHD survey' respondents rate highly the importance of the extent of the network managed by the IHD and that of having local support in the region.

To conclude, while this objective of the IHD service is rated highly by the respondents to the survey, the IHD are seen only effective to a certain extent in building long term partnership with relevant EU/other stakeholders.

d.Objective 4: Adapt the work to the target group on demand

Overall, during the period, the IHD have tried to answer the requirement for improving the quality of their services to the group of users on demand 156 . Publication services provide significant examples of such implementation with efforts in using a jargon-free language understandable by SMEs having no or only basic knowledge of IPRs, or translating where demands exist. Overall, the quality of the language has improved as compared to the previous edition of the IHD. The project focussed also in developing new publications centred on particular economic sectors relevant to the geographical area at issue (e.g. guide on IP considerations for Automotive Industry in SEA or guide to IPR protection in China for the Fashion and Design Industry) or about country(ies) not previously geographically covered (e.g. factsheets and FAQs related to Mexico, Colombia, Peru and Costa Rica, or IP factsheet on Macao). The sectorial approach and extension to new countries has also been followed in the training activities offered (e.g. nano and biotechnologies sector-oriented modules developed in LA or development of webinars capturing a wide diversity of sector in China). To be mentioned as well: in LA the partnering activity started with a survey to understand the training needs of SMEs, after which a few trainings were successfully developed. These examples show that the IHD pay specific attention to adapt their activity to the group of SMEs on demand. The outcome is that the current action of the IHD is not a mere continuation of the services previously provided: it has extended in scope by covering both specific economic sectors and new geographical areas.

Evidence tend to show that the respondents to the survey and interviewee value the customization aspect of the current IHD services which varies over time. Each time the survey' respondents are asked about the relevance and usefulness of the services offered, the responses are positive in a high proportion. The publication services in general are perceived as useful by the respondents. 90% find them useful or very useful, while 45% of the respondents state that they use these services at least occasionally (against only 25% for the helpline). Eventually, the interviews of the IHD reveal that one factor that makes the IHD particularly relevant for businesses is that the IHD services are customised (by key sector, MS, target country) and that the IHD provide their services accordingly. Interviews from the EC services confirm this trend as IHD are seen there as having a general ability to communicate in an SME like language and to adapt easily to new partners/environments.

In addition to publication service, the IHD are asked to offer services like trainings or advice (through the helpline), customized to the need of their users. Between 52 and 55 % of the survey' respondents rate as effective the IHD capacity to solve concrete IP problems in the third countries targeted and in developing internal capacities in SMEs to deal with IP matters.

The IHD took care to tailor their work to the target group on demand. The external feedback shows that it remains adapted to the work of this target group to some extent (publication services and provision of first line information). However, the overall perception of the entire IHD offer seems more nuanced. As already highlighted, the capacity of the IHD to deal with specific IP problems faced by their users and to help them develop internal capacities to deal with IP protection could apparently be improved. On these aspects (support to IP problem solving and development of specific IP skills) coherence needed with the services that law firms and consultancies offer, implies some limitations (see coherence part).

Further analysis on regional variations will be provided in the efficiency part of the report.

Effectiveness of the IHD measure in achieving the three specific objectives

e.Specific objective 1: reduce the risk of counterfeit products entering the EU market from the concerned markets

The specific actions implemented by the IHD (helpline, publication, trainings, website, communication and dissemination) are aimed at enabling the EU SMEs to control their IPR management and defend their intellectual assets before entering and while doing business in the target regions.

The following chart 157 on IPR Infringements shows in particular a slight increase in the high number of articles suspected of violating IPRs (while the number of cases opened by the EU customs remains stable).

6.Number of registered cases and articles

6.Source: European Commission: IPR Infringements: Facts and figures 2016 - https://ec.europa.eu/taxation_customs/business/customs-controls/counterfeit-piracy-other-ipr-violations/ipr-infringements-facts-figures_en

The 2016 Report on EU customs enforcement of IPR at the EU border 158 , for its part, stresses in particular that

- China, Hong-Kong, Singapore, Thailand, Vietnam and Malaysia are among the countries of origin,

- China continues to be the main source country from where IPR infringing goods are shipped to the EU (except for certain goods like alcoholic beverages).

Furthermore, according to the 2017 Situation Report on Counterfeiting and Piracy in the European Union 159 prepared by the EUIPO and Europol, "seizures at the EU external border are estimated to be only a small part of the total amount of imported counterfeit goods". Such goods are otherwise sold online, subsequently distributed via parcel and postal services and difficult to seize as "targeting and controlling suspicious shipments in postal freight are a great challenge".

In front of such evidences no conclusion can be derived in relation to the link with IHD intervention because of its inner limited scope.

However, based on several observable factors, it is possible to conclude that the IHD initiative might contribute to this result. By helping EU SMEs to be increasingly aware of the value of IPR, and to adopt adequate protection behaviours and methodology, the IHD, to some extent, contribute to reducing risks of counterfeit products entering the market. Evidence shows for instance that around half of the sample of SMEs surveyed having experienced the IHD services express that they start to "take steps to manage IP” or that they "consistently now manage IP in or outside the EU”.

During the stakeholder's consultation, 28 companies reported that – before the contact with the (IHD) – they were "only just aware that IP is an issue". Out of these 28 companies, 12 reported that after the contact with the (IHD) they were beginning to take steps to manage IP inside and outside the EU. Therefore, the survey evidence seems to indicate that almost half of the respondents (43%) reported an improvement in their ‘IP management status’ after the contact with the IHD. For respondents citing a higher level of IP management skills for the situation before the contact with the HD, these improvement rates are similar (around 40%).”

·Despite the low number of responses, it seems that the action of the IHD has a positive effect in developing the IP management skills among users of the IHD. This result should obviously help SME’s users to come to have a better control on the risk of infringement.

However, as previously analysed, in a territory like China where infringement is a huge issue, the IHD reaches only 3% of EU SMEs exporting. The more this percentage increases, the more impact on reduction of risks of counterfeiting products entering the EU market will be observable.

a.Specific objective 2: Protect SME's valuable IPR before entering and while doing business in the concerned markets

As highlighted above, the specific actions of the IHD should generate a positive effect in developing SMEs IP management skills to reduce the counterfeit risk and to put their IPR protection under control. Evidence shows an interest in users getting support in IP protection and registration both before and after doing business in the targeted areas. For instance, a large majority (80%) of survey' respondents agree that the assistance provided by the IHD is essential when planning to expand to a third country as well as that it is important when having entered a third country. As already seen, the respondents to the survey need advice and support mostly in relation to IP protection (59%) and registration (41%) 160 . From a purely qualitative angle, several case studies presented by IHD in their Impact Assessment run in 2016 show examples of how the IHD helped EU SMEs to protect their IPR before entering the target area.

As a matter of facts, the IHD have put into place means to protect IP and to register IPR both before entering the targeted area and after EU SMEs did. The presence of local support, as well as presence in the EU is key to maintain services continuity both in time and space. At first sight, from the evidence collected, there is no feedback showing that EU SMEs would benefit from better IP support services before than after entering the targeted area. The opposite is also true. Some interviews imply however that the LAHD meets difficulties in providing local support to SMEs with the same degree of effectiveness as in Europe. Some justification like a "too academic approach", "insufficient country specific publications" as well as the location of the experts’ offices in Europe and not in LA, are put forward to explain these difficulties.

b.Specific objective 3: Help SMEs to find and constructively work with local administrations or service providers to enforce their IPR protection

While more than 80% survey' respondents find that the assistance that the IHD provide is essential once they have entered the third country, and the IHD members themselves consider them as directly impacting on increasing the ability of EU SMEs to collaborate with local authorities and service providers for enforcement purposes 161 , the survey shows on the other hand that only a bit more than 40% of the respondents consider that the IHD are effective as a mechanism to building networks and relations to experts and institutions in the targeted countries covered. The interview with the IHD and the EC services corroborate this finding by showing that awareness of the IHD services in Europe is limited.

This could indicate only a moderate level of effectiveness of the IHD in reaching the effect of helping EU SMEs to find and constructively work with local administrations or service providers to enforce their IPR protection in the views of the respondents to the survey.

6.2.1.Where expectations have not been met, what factors have hindered their achievement?

Focusing on the specific actions (helpline, trainings, publications, website and partnering services), the effectiveness analysis shows that only one service offered by the IHD i.e. the helpline fulfils the targets set. The IHD face problems in providing the remaining services in a full satisfactory way. External or internal hindering factors explain them.

A first hindering factor occurred in SEA. The IHD was not operative in SEA until October 2015 (instead of January 2015). This delay was caused primarily by changes in the constitution of the consortium which occurred during grant preparation. In addition, it is only as from December 2015 (when the new IP advisor became part of the EuroCham Vietnam payroll) that publications, trainings, policy feedback and IP support during matchmaking events services started to gain momentum. This explains that for these services, the consortium is still lagging behind the targets 162 .

Another external hindering factor lies with difficulties in LA in reaching out to Brazilian and Mexican stakeholders essentially due to a lack of cooperation of stakeholders who perceive the IHD as a competitor. This factor impacted negatively the level of achievement of trainings 163 .

Among the internal hindering factors, the IHD initiative has not been implemented satisfactorily in relation to the development of a joint website 164 . This task has been severely delayed (completed in March 2016 i.e. almost 12 months after the expected date). In addition, the quality level of the joint portal did not meet the expectations of the IHD. Several functionalities were missing resulting in various dysfunctions and extra work (manual registrations of training participants after an event, or need to use a previous blog in LA). Eventually, the procedure for addressing the issues/bugs was not quick and effective enough. The delay in making the website available and following implementation problems might have partially hindered the achievement of certain services e.g. website and trainings.

Another concern in LA lies with a lack of attendance of the IHD to matchmaking events. This hindering factor might have restricted the possibility for EU SMEs trading (or planning to trade) in this area to match (e.g. when EU SMEs would refrain from participating because they do not benefit from support in relation to IP protection) or match in a safe way (e.g. when they would not have taken care to IP protect their assets before negotiating with trade partners because they would not have been made aware of the need to protect them due to the lack of attendance of the IHD) trading partners.

Conclusion on effectiveness

The IHD are expected to serve the COSME programme general objectives of strengthening competitiveness and promoting growth and specifically improving access to markets at global level. They should thus help SMEs to enter and grow on the Chinese, SEA, and LA markets, while protecting their IP assets, preventing them from risk of counterfeit, and building constructive relations with local administrations and services providers.

From the evidence reviewed, and with the usual caveat as regards in particular the representativeness of the sample analysed, the following trends appear:

-EU SMEs would particularly value getting advice and support on the different format of IP protection as well as information on registration and other procedures.

-The IHD would be perceived as specifically effective in increasing awareness for IP. The tools implemented to that end, like the helpline, trainings and publications effectively would contribute to cater the EU SMEs need for advice and reliable information both before and after entering the targeted third countries markets.

-A significant part among users of the IHD report having improved their IP management after having been in touch with the IHD.

-The level of effectiveness appears to be lower as regards solving concrete IP problems in the countries covered by the IHD and developing internal capacities to deal with IP issues. This last finding appears however acceptable, considering first that the IHD rightly prioritised the provision of information on IP (helpline, publications, trainings). Second, the range of services offered by the IHD is not supposed to overlap with the services provided by the private sector. It will always be necessary to have in mind the border between providing first-line support and raising IP awareness on one hand, and delivering legal support services on the other hand.

-An element that the IHD should take care of is to develop network with local experts both in Europe and in the targeted countries. Results between the three targeted regions are very heterogeneous. The LAHD underperforms as compared to the two other regional entities of the IHD. Improving the performance of the IHD in networking development could in particular involve increasing significantly their promotion at local level.

Although the delay in implementing the IHD services in SEA, as well as the lack of a joined website for one year impacted the figures (e.g. First six months in SEA and absence of on line registration of users in the entire IHD), positive factors could lead to assume that, on the medium/long term, the IHD could fulfil the objectives of the initiative. Basic instruments have been put into place to reach SMEs and enable them efficiently face IPR issues before and after entering the targeted third countries markets. These instruments are an effective helpline service, trainings, useful publications and a website. Key success factors of the IHD on the long term lies on their capacity to first improve the quality of these instruments already largely valued by the users, second, tackle the problems of insufficient promotion of their activities, and third improve partnering services.

6.1.EFFICIENCY

6.1.1.What are the costs incurred to the IHD, the public authorities, and the other stakeholders to implement the initiative (as defined by the Regulation, and follow up grant agreements)?

The IHD project is funded by the European Union via the COSME programme established by the COSME regulation (EU) No 1287/2013. The budget involved is Euro 7 200 000 (Euro 2 400 000 for the CNHD, Euro 2 600 000 for the SEAHD, Euro 2 000 000 for the LAHD), plus an additional contribution of Euro 200 000 to develop and maintain the common portal. The EU grant is limited to a maximum co-funding rate of 90% of eligible costs. As already noted, the contracts are for 36 months (CHINA and SEA) or 30 months (LA).

The COSME programme requires from the IHD periodical and detailed reports (quarterly, annual, and biannual interim reports) of their activity reviewed through 7 KPIs that match with the specific actions.

The achievement of these objectives implies the use by the IHD of their European funding, to provide SMEs with the following four main services:

§Helpline services to answer enquiries and give confidential advice

§Publication services (training materials, guides, articles, newsletter content or other publications)

§Online (website) services

§Training services

The efficiency analysis aims at checking how optimal the use of the allocated financial resources by the IHD is, in view of the achievements.

The table below shows how the IHD split their costs among various expenditure items.

Table 2: Costs share among various services

	Grants (euros)	Direct personnel costs (euros)	Direct costs of subcontracting (euros)	Helpline	Trainings	Publications & materials	Overhead costs (dissemination, website, partnering…)
CHINA	2.400.000	2.154.217	308.426	48%	19%	14%	19%
SEA	2.600.000	1.984.590	421.250	31%	33%	17%	20%
LA	2000.000	1.346.243	235.345	31%	23%	10%	36%

Source: IPR Helpdesks

It shows that there is not a uniform allocation of expenses among the three IHD regional entities. In China almost a half of the cost (48%) relates to the helpline service. This benefits the CNHD which reaches higher amounts of registred helpline users and handles more enquiries than the SEAHD or LAHD do (see Table 3). The SEAHD evidences a more balanced cost allocation between helpline (31%) and training (33%) services. The SEAHD reaches however lower results than the CNHD for these two services. More than two-thirds of the cost of the LAHD are dedicated to helpline service (31%) and the "overheads costs" (36%). The LAHD reaches a satisfactory amount of registred helpline users but less important ones in terms of website use and partnering service, despite significant expenditures incurred. For the three regions, publications services are the poor relative expenditure item. This could explain why results in this area are well below the targets (see Table 3). Two cost allocation strategies can therefore be distinguished. The CNHD one which favours the helpline service, that becomes key actor to the overall performance of the HD. Partnering and dissemination efforts help the CNHD attract a significant amount of users and handle a large number of enquiries. Next, the useful helpline generates a positive effect on trainings. Publications and website maintain the relationship with the IHD users. In SEA and LA, cost allocation strategy is distinct. The helpline service does not play a pivotal role as important as for the CNHD. KPIs from the other services benefit less from the good helpline results. In terms of efficiency, results show that the cost allocation strategy in CHINA seems more rewarding. However, analysis of KPIs and, above all, of average costs achieved masks more contrasted patterns.

6.1.2.To what extent are the costs justified and proportionate, given the benefits achieved by the IHD?

To ensure a proper monitoring and evaluation of the specific measure, each regional entity of the IHD was asked to measure and record the following essential performance indicators.

• Number of inquiries from EU SMEs answered and SME satisfaction and feedback on quality;

• Number and quality of training and awareness events, including virtual events organised as well as number of SMEs and business support organisation participants

• Number of visits and hits on the website and SME user satisfaction of the website;

• Number and quality of guides, factsheets, e-learning modules and other publications targeted at SMEs and business support organisations;

• Improved performance of end-users in relation to targeted issue;

• Clients performance assessment (growth of sales, jobs created/maintained, new products or services developed).

Other performance indicators were detailed in the framework of the requested quarterly, interim and final reports. The level of achievement of the above mentioned KPIs is a significant way to analyse whether costs are justified and proportionate.

A comparative analysis of the most significant targeted/achieved results of the specific actions of the IHD was performed on the basis of the three "deliverables and KPIs" documents (one for each IHD region) covering the reporting period. These results appear in the following table:

Table 3: Most significant targeted/achieved results of the specific actions of the IHD

	CHINA		SEA		LA
Reporting period covered (months)	28 (out of 36)		28 (out of 36)		22 (out of 30)
GRANT (euros)	2.400.000		2.600.000		2.000.000
	Targeted	Achieved	Targeted	Achieved	Targeted	Achieved
	WP1 FIRSTLINE ADVISORY SERVICES
Numbers of enquiries	1500	1509	550	652 165	800-1200	687
Number of helpline users	1000	1347	1000	590	600-800	1217
Users satisfaction with response	3.75/4	K.A.	3.75/4	K.A.	99%	95,70%
	WP3 TRAININGS
Training sessions to be delivered	260	125	260	60	122+107	60
Webinars to be delivered	60	37	72	38	36	6
Participants	15000	7682	15000	3578	3660	1655
Quality assessment	K.A.	K.A.	K.A.	K.A.	95% highly satisfied	60,6% good
	WP5 PARTNERING SERVICES
Number of events attended	30	78 (4 in EU)	60	38	30	0
	WP6 WEBSITE
Number of visits	240.000°	119.692	125.000°	32.479	180.000 (estimated)	20.000 (estimated)
User satisfaction	3,5/5	4,4/5	3,75	K.A.	80-100%	80%
	WP2 PUBLICATIONS
New guides & other publications	35	20	60	32	25	10
Case studies	12	8	15	9	30	9
Articles published	450	239	450	230	236	211
Trainings modules developed	35	20	39	11	K.A.	25
	WP4 MONITORING OF IP POLICIES & LEGISLATION
Nb of subscribers to Newsletter	5.000	K.A.	5.000	975	K.A.	K.A.
	OVERALL CLIENT SATISFACTION ASSESSMENT
Number of SME reporting taking follow-up actions	60%	K.A.	60%	K.A.	1000	K.A.
Number of customers reporting positive development of their business situation	80%	K.A.	70%	K.A.	400	K.A.

Source: IPR Helpdesks

This table presents a selection of most relevant KPIs. It gives an overall vision of the achievements of the IHD in relation to the resources granted.

Regarding the assessment of progress toward the objectives, the figures show that globally the IHD strive to achieve the specific actions with varying degree of success to fulfil their KPIs.

The CNHD appears as the most efficient IHD. It over performs in relation to the number of handled inquiries and registered users. Similarly, it made significant promotion efforts favouring dissemination of IP content in particular thanks to an active participation to events (78 events attended against 30 targeted). As regards other KPIs, the level of achievement is below the targeted objectives. However, these results are not worrying. They could imply that the CNHD is on an upward trajectory, which would probably lead to achieving closer to the targets by the end of the period considered.

The SEAHD ranks as a second with a medium level of efficiency with significant gaps between targets and achievements. These backlogs are partly due to a belated start 166 , which hampered its pace of development. Nevertheless, the level of achievement on the different KPIs are less worrying considering that the SEAHD reached a normal cruising speed as from the second year only. Now it is clear that the helpline works well, the website is in place, trainings are delivered and the publications are well on track. The SEAHD is also concerned with promotion as evidenced by their events attendance efforts. The SEAHD experiences more significant gaps with the targets than the CNHD, but seems to follow a good development rhythm.

The LAHD performs below the targets on almost all KPIs. Fortunately, the LAHD scores very well on the registered number of helpline users (1217 out of 800 targeted) with a very high satisfaction rate. On the first line advisory service, the LAHD performs better than the SEAHD, considering that the LAHD has achieved a similar number of enquiries in six months less of operation. As regards the other specific actions, underperformances are clearly perceptible. On trainings for instance, the LAHD mentions 25 trainings modules developed, leading to 60 training sessions delivered, gathering 1655 participants (27 per session), while the CNHD with 20 modules developed, trained 7682 participants in 125 sessions (61 per session). While the SEAHD and the LAHD have delivered the same amount of training session, the SEAHD has a number of participants twice as important (3578 against 1655). A critical issue for the LAHD is the lack of participation in business events organised by third parties (0 events attended) from which the other IHD regional entities get most of their enquiries. It is easier for the CNHD than for the SEAHD and LAHD to achieve the specific action "provision of partnering services". This has to do, to some extent, with the higher interest that EU companies have in the China market. The LAHD and SEAHD encounter difficulties when trying to convince stakeholders to organise meetings targeting these two regions. China is an easier market. On this basis, a lower performance from the LAHD might be understood (like that of the SEAHD which shows 38 events attended against 60 targeted) but not a complete lack of participation. The LAHD should be more pro-active in engaging in third-party events. The very low amount of visits on the website is another issue for the LAHD. This can be related to the insufficient and delayed development of the website by the external responsible entity. As a result, the LAHD blog was available only one year after the start of the project. In light of these results, it is difficult to conclude for the LAHD on an onward trajectory which could lead to catching up on the backlog by the end of the period considered.

To assess cost/benefit ratio, comparing data on Tables 2 and 3 allow determine the average cost per unit of various services. This gives a deeper insight into the level of efficiency required from each regional entity of the IHD and reached by them.

Table 4: Average costs (euros) of the services by work unit

	CHINA		SEA		LA
	Targeted	Achieved	Targeted	Achieved	Targeted	Achieved
Average cost of one registered helpline user	1034	597	615	693	596	251
Average cost of one handled enquiry	689	533	1119	867	417	445
Average cost of one trainee	31	47	53	173	99	161
Average cost of one training session	1462	2246	2391	6301	2302	4042
Average cost of one piece of publication	612	838	685	985	595	491
Average cost of one website visit	1,26	1,96	2,22	6,65	1,05	6,91

This table evidences first that the overall efficiency of the IHD is not very high. The targets appear as not properly defined considering the distinct cost allocation schemes. For instance, as regards the helpline, by dividing the targeted KPIs by the costs to reach them, the average targeted cost for handling one enquiry can reach until euro 1119! Similarly, by devoting 48% of its costs to the helpline service, the CNHD reaches an average target cost of euro 1034 for each registered user. Fortunately, by largely exceeding their target, the CNHD limits this cost to euro 597 (over the 28 months reporting period). However, the LAHD performs apparently better in this area. By assigning 31% of its budget to the helpline service, the LAHD reaches almost the same amount of users in 22 months than the CNHD does in 28 months. This way, the LAHD limits the average cost of one user to euro 251. The LAHD appears as the most efficient, in relation to the average cost for one enquiry, by reaching euro 445 (against euro 533 and 867 for respectively China and SEA). In contrast, the CNHD reaches an average cost for training and website use which outperforms the two other regions. This result from the CNHD allocating a higher portion of their budget to the helpline service 167 and to the positive impact generated on the other services. Question arises as to why the LAHD, which achieves a number of helpline users that is almost as important as the one that the CNHD reaches, does not benefit from an identical knock-on effect on other services. One explanation could be that, while having a more efficient helpline, the LAHD deals with a relatively low number of enquiries (less than half of the CNHD one). The knock-on effect on trainings or website use can therefore be limited. Similarly the SEAHD which has the most expansive and less developed helpline, benefits only to a very limited extent from this knock-on effect. The CNHD, for its part, largely compensates a relative loss of efficiency, due to a high level of expenses being focussed on the helpline services (48% of cost) with significant efficiency gains on the training and website services. As a matter of fact, a training participant costs euro 47 in China, against euro 173 and 161 in SEA and LA, respectively. Similarly, the CNHD performs one training session for a cost of euro 2246, while the equivalent cost is euro 6301 for the SEAHD and euro 4042 for the LAHD. Therefore the cost allocation in China appears as globally more efficient. The publications item shares the lowest cost expenditure in the three IHD regions (ranking from 10 to 17%). None of the three IHD entities reaches the targeted KPIs. This could mean that the IHD entities do not devote sufficient time and means to this service. Actually, they have to follow numerous KPIs in relation to the publication services. Preparing and publishing all the articles, guides, video/audio podcasts, newletters, FAQs, infographics, case studies... that they are asked for, would force them to dedicate an important portion of their resources to the detriment of other services. The IHD have not abandoned this specific action. They have clearly not prioritised this action to the extent required by the KPIs. The LAHD displays the lowest unitary cost (euro 491) but limits the amount of time-and resource-consuming publications (news guides, video podcasts, specific materials…). In China and SEA, the time devoted to publications compared to the number of items produced does not reveal a very efficient result. However, without reaching the numerous amounts of KPIs for the various publications targeted, the IHD offers to their users an acceptable amount of information that the user survey finds important and useful. It is likely that the IHD cannot dedicate more than a minimum threshold of cost to publications. Their level of efficiency might deteriorate even more considering that time for preparing certain publication items might appear disproportionate in light of the expected level of consultation on the website.

The efficiency analysis shows that the IHD are struggling to optimise the cost/benefit ratio within the framework of the KPIs, a system which forces them to arbitrate between various targets that are not always appropriate considering potential use by SMEs. Some targets are not sufficiently ambitious. In particular, it is more than likely that the three regional entities of the IHD will reach a number of registered users and handled enquiries significantly higher than the targets set. The average cost will decrease accordingly. Other targets appear, to the opposite, largely oversized (KPIs for publications, amount of website visits) and require huge resources to implement them even only partially.

While the average unit/cost analysis evidences that the cost allocation is more efficient in China, where the helpline plays a pivotal service paper, it would be excessive to devalue the efficiency level of the LAHD or the SEAHD as compared to that of the CNHD. The SEAHD and LAHD cover various, heterogeneous countries with different jurisdictions requiring also distinct expertise. In this context, the two regional entities need to gain more experience to set appropriate organisation, processes, tools and networks. To improve the costs/benefits ratios, experience shows now, that it will be necessary to focus on the most appropriate tasks that provide the best level of effectiveness in relation to the keys objectives. The authors of the external study underline that "inefficiencies arise in relation to some sub-services, where costs might be higher than possible benefits. This relates, for the China helpdesks, for example to the development of e-learning modules. Developing e-learning modules is costly and the respective additional benefits are hard to measure. For the SEA helpdesk, partner bulletins – where the helpdesk tries to insert information into the publications and info channels of partner organisation – were also a difficult terrain. Many partner organisations were not particularly open to that kind of activity, and it proved difficult to sync the timing and scope of the respectively to-be-inserted information between the HDs and the partner organisations involved. The issue of the website of the LAHD clearly impacted also efficiency of (web-site) service provision negatively. And it is also clear that serious games, and to a lesser extent the podcasts, have proven not only largely ineffective but also inefficient activities. We deduct this from the very low level of demand, as evidenced by the respective KPIs".

To conclude, the assessment of the results achieved in light of the KPIs show that during the period considered, the IHD experimented difficult implementation phases which led to more or less delays. The CNHD has apparently found an organisation and a strategy that should help it to reach the expected targets with better efficiency than both others. The SEAHD and LAHD are seriously involved in the implementation of some specific actions i.e. helpline, publications, and trainings. The one-year under performing website hindered their communication and dissemination actions. The SEAHD and LAHD should seek ways to participate more to business events. The "provision of partnering service" is one of the specific actions that the IHD have to implement. Although, on the basis of the evidence gathered, this action seems more difficult to implement in SEA and LA than on the Chinese market, a "zero participation" policy like that implemented by the LAHD is certainly not the right approach.

Eventually, an efficiency analysis only based on some cost/benefit ratios obtained by dividing spent resources by number of SMEs advised or SME users or trainees give a relative unfavourable efficiency assessment. This is evident by considering the overall high level of ten average costs per unit. But this has to be relativized. Firstly, as commented by the authors of the Study "Because IP has an insurance function, the possible benefits even for a small number of SMEs which have managed to avoid being infringed and/or otherwise involved in litigation/enforcement of IP rights can outweigh the expenses." Secondly, assessing the efficiency of the IHD in light of productivity ratios or gap analysis based on too ambitious targets (and probably too numerous as well) seems irrelevant. It is more important to ensure that the resources that the IHD use to settle an organisation, develop tools, get expertise, and provide services, result in improving quantitatively and qualitatively their performance. The benefits evidenced in the effectiveness analysis show an increased awareness of EU SMEs of the need to protect their intellectual assets before and while entering the targeted markets. The survey and interviews show that EU SMEs which have used the services of the IHD almost unanimously value the usefulness of their services. After three years of experience under the COSME programme, the IHD have been showing effectiveness and should now focus their resources to gain in efficiency.

6.1.3.Can the IHD achieve more impact (within current or increased resources)?

The grants contributing to financing each IHD entity seem sufficient to achieve the objectives of the initiative within the agreed KPIs. Overall, the current IHD entities do not find the budget too low to accomplish their tasks 168 . However, Table 3 above shows that the IHD properly fulfil their targets in relation to one and only service i.e. the helpline. For the remaining services, the IHD do not reach their targets. This is not due to lack of financial resources. Within the current budgets, progresses could be made to move closer to the targets.

This would imply concentrating efforts on:

·Trainings: by making a better use of resources to develop more modules and deliver more actions. Considering the good attendance rate, three times more participants might possibly be reached.

·Promotion: as indicated above the survey shows that an important portion of respondents consider that the IHD are insufficiently promoted (more than 50% in LA, between 30 and 40% in China and SEA). In addition, promotion would probably be better when the entity in SEA and particularly in LA could improve their partnering service, as the CNHD does. Another aspect that could help improvement of promotion would be the publication services. The gap between the publications achieved and targeted remains too important, while the survey' respondents value these services as particularly useful and important.

·Customer satisfaction: while certain KPIs have been set to measure the users' satisfaction levels, they are not equally reported by the three IHD entities. Putting into place a shared and homogeneous SMEs satisfaction barometer would be an interesting efficiency tool.

Consequently, there is no need for additional means in order to achieve more impacts, however the use of the current ones could be optimised as was done for the current helpline services.

6.1.4.Governance: What implementation mode is optimal: grant, contract or other options?

A basic consideration for the efficiency of the operation of the IHD is the contractual framework within which they operate. Currently the IHD operate based on three grant agreements, one for each IHD regional entity, plus an additional grant for the development and maintenance of the common web portal. As can be seen from the interview synthesis, the EC informed about a possible other option i.e. a contract agreement. Question arises as to whether, to implement the IHD initiative, a contractual framework would be better than the on-going grant agreement.

In case of a grant, the Commission makes a contribution to project carried out by an organisation (i.e. the beneficiary) because its activities contribute to the EU policy aims. Such funding is considered in particular as typically fit to beneficiaries like Research Centres or University covering a broad range of theoretical projects. The application for financing originates with the beneficiary, who submits a proposal for support for activities it plans to carry out in response to a Commission call for proposals. The proposal sets out the specifications for the action to be performed within the framework laid down in advance by the Commission. The grant must not have the purpose or effect of producing a profit for the beneficiary. A pre-financing (by the Commission) usually takes place prior to (or at the start of) the provision of the services 169 . Save in exceptional cases, the grant may not finance the total costs of the action. Ownership generated by the action (like logo, trademarks, website) as a rule remains with the beneficiary. Due to this financing scheme, funding theoretically appears as encouraging less a maximised delivery of services, than a payment which takes into account delivery. The grant option should normally imply less control for the Commission and more flexibility for beneficiaries in defining the action. However, other factors in particular the way the implementation of the action is actually monitored by the Commission or its agency, have to be taken into account. In the framework of the current evaluation period, Article 6 of the COSME Regulation allows entities established in third countries to participate in actions under the COSME Programme, even when the third country in question does not. In accordance with Article 7 profit-making entities established outside countries participating in the COSME Programme cannot receive any EU funding. This limitation does not concern subcontracting. Subcontracting may cover a limited part of the action (not core business and 20% of the eligible costs).

In case of a contract, it is the provision of the service which generates the payment. Such payment per delivery is usually seen as providing incentives to maximise delivery. There is normally no pre-financing, while the remuneration of the contractor includes an element of profit. In addition, ownership generated by the action belongs to the contracting authority. The core services to be provided are pre-defined in detailed specifications prepared by the Commission (or its agency). There is not much room for applicants to propose the type of services or the way of implementing them. Such implementation form where the contracting authority has the initiative and control over the action, while the selected applicant has to concentrate on its delivery, looks more adapted to services offered by profit-making entities. In case of a contract with the EASME Agency, participation is open to "all natural and legal persons coming within the scope of the Treaties, as well as to international organisations", and to "all natural and legal persons established in a third country which has a special agreement with the Union in the field of public procurement on the conditions laid down in that agreement 170 ". Other entities can be subcontractors. Subcontracting is unlimited.

As the IHD services contribute to the EC policy aim to support EU SMEs in relation to IP, a grant agreement is an option for implementing the corresponding services. To implement the IHD, partners located in China, SEA and LA (i.e. in non-COSME countries) are absolutely needed. Public entities established in (non-COSME) countries can participate. Subcontracting could be opened to some extent (for non-core tasks) to profit-making entities that are otherwise excluded from EU funding in these countries. During the evaluation period, this implementation form appeared as specifically appropriate for the consortium which has been operating the LAHD. As it includes two universities and an institute for legal and technological studies, it has a strong academic component. Profit-making entities are, however, usually less interested in participating in actions funded by grants under the COSME programme, which require a co-financing by the applicants (in the current period 10%) and does not allow them to generate any profit. In addition, unless an exception to the model grant agreement is created 171 , ownership of the website or IPR generated by the IHD action would normally stays with the beneficiary. This could be seen as problematic to ensure in particular good continuity in the service offered to EU SMEs. During the evaluation period for instance, different approaches to the website design generated difficulties in the transition from the predecessor IPR Mercosur helpdesk to the LAHD website.

The very practical aspect of the services to be offered by the IHD (e.g. providing advice in less than 3 days, in a jargon-free language, or publications adapted to sectorial concerns) requires a concrete approach that might appear less adapted to the grant agreement framework usually reserved for more academic projects. Similary, the need for the Commission (or its agency) to detail the specification of the IHD services in a contract could be seen as more burdensome than drafting the main lines of a grant proposal. Resorting to a service contract could possibly increase the range of applicants, by not refraining profit-making entities from applying as there is no need for them to co-finance the action and they can generate profit. Non-EU based entities, or entities from countries like China, Vietnam and a number of countries of the IHD, that did not sign the Government Procurement Agreement (GPA) cannot be partners of a consortium. However, subcontracting is fully open to them. Eventually, to avoid discontinuity in the IPR website service offered to EU SMEs, it is critical that any ownership generated by the action remains with the EU, as provided by a service contract.

The question of the choice of implementation is closely related to the extent to which payment is linked to delivery of certain services or not. While payment per delivery as usual in service contract could be seen as the best option to maximise the quality and timely delivery of the IHD services, the current grant agreement offers a quite strict link between delivery and payment as well. Payment is subject to the approval of the periodic reports. For instance, part of the grant allocated to cover the development of a joint website has been disallowed from the staff costs claimed by the beneficiaries as that key deliverable was submitted almost one year late from the expected delivery date. This factor does not lead to choose one option or the other.

In any event, the way the implementation actually takes place will depend also on additional factors like deliverables and KPIs and, more generally, the way the Commission (or its agency) monitors the implementation. Practical experience shows that the efficiency of the operation of the IHD does not lead only in the contractual framework within which they operate. The choice between the two options was discussed within the framework of the evaluation of the European helpdesk, as can be seen from the report published in 2014. The interviews run within this previous evaluation revealed a clear administrative burden for both the Contractor and EASME. Some interviewees passed the blame for this burden on the then service contract because a service contract requires a much more 'hands-on’ management than is the case with a grant agreement. Some interviewees suggested that the contractual form made it difficult to shift resources major priorities over the period of the contract. However, there was no consensus on these two points: other interviewees suggested that the form of the agreement did not pose a major problem and that there was sufficient flexibility within the current contractual arrangements. For the IHD, the COSME programme requires periodical and detailed reports (quarterly, annual, and biannual interim reports) of the activity of the IHD reviewed through 7 KPIs that match with the specific actions. However, this compulsory reporting was not qualified as a burden during the interviews.

The only source of concern was voiced by a member of one of the consortia running the IHD i.e. in September 2017, the European Chamber of Commerce in Vietnam feared that, under a contract option, as a third country organisation, they might be restricted to a subcontractor position 172 having a minor role in the project (e.g. their logo might not be visible any more). In any event, there was no claim as to a possible better/worst performance of the IHD under one implementation framework or the other.

It is unclear that the efficiency of the IHD would improve with the choice of another implementation framework. In particular, there is no evidence that the administrative burden linked to the monitoring of the IHD would be lower/higher under one or other implementation option. In any event, as already highlighted in the evaluation report of the EU helpdesk, the quality, effectiveness and efficiency of the reporting system could be improved by developing a simplified structure [based on an annual planning cycle, in which reports provide information on progress in achieving longer term objectives, as well as key outputs]. Developing such a simplified reporting system can be implemented under one or the other implementation option.

The choice of an implementation method or another is essentially linked to the internal monitoring and management control of the EC. High administrative constraints could indeed divert the IHD from focusing on their main objective. However, such diversion from core task can be limited by reorganising the reporting system, whatever the implementation form.

In addition, having a third country partner as a subcontractor does not mean automatically that the attractive activities will be kept by the main contractor for themselves or that they might even outsource activities to others than the third country subcontractor. It is the content of the contract which establishes such rules. As it is binding, the selected contractor will have to justify each time they diverge from the contract e.g. when they want to assign some work to another subcontractor. On the other hand, a subcontractor will have less financial reporting.

To conclude, it is unclear based on the evidence collected whether, to implement the IHD initiative, a contractual framework would be better than the on-going grant agreement. This was also the conclusion of the authors of the evaluation previously run on the European helpdesk.

6.1.5.What factors influenced the efficiency of the results achieved by the IHD?

The objective of the IHD is to offer services in co-ordination with and in direct response to requests from EU SMEs and their representatives (e.g. sectoral, regional, local associations of SMEs, chambers of commerce). As already highlighted, companies from the private sectors like lawyers or consultant firms can also represent EU SMEs. They act as multipliers by offering to their final clients (i.e. SMEs) a broader service including IPR aspects that they normally offer against payment. This could reveal problematic considering that this companies earn money notably by re-selling to EU SMEs information or advice that they obtain free of charge from the IHD. This could negatively impact the efficiency of the IHD initiative in two ways. First the grant is not supposed to feed the on-going business of the intermediaries (which makes, for instance, around 24% of the survey sample). Second, such a portion of intermediaries could be seen as depriving EU SMEs from a direct and free of charge IPR management services from the IHD. Seen from another angle, intermediaries can be perceived as an efficient way to reach more SMEs. The problem is not so much that SMEs rely on intermediaries that benefit from IP support from the IHD. It is more that due to the lack of visibility, EU SMEs are not necessarily aware that they have the option to turn directly to the IHD. The final objective is that a maximum number of SMEs are directly or indirectly made capable of facing IP protection issues in or related to the targeted area. Therefore, the fact that intermediaries act as multipliers, can be seen as an efficiency lever, provided the visibility of the IHD increases concurrently thanks to more promotion efforts.

Another issue that might bear consequences on the efficiency lies in the monitoring the IHD activity. The huge amount of KPIs (44 for LA, 47 for China, 48 for SEA) to be reported on, in addition to the number of periodical reports requested (monthly reports, quarterly, interim and final reports) represents a heavy workload that generates costs. Dedicating a part of the time and resources used for these reporting tasks to the core activity could probably benefit more final users i.e. EU SMEs. Eventually, each IHD entity reports in its own way. This heterogeneous way of reporting, complicates any comparison between the three regions results and does not lead to a clear overall picture of the IHD performances. Additionally, instead of focusing them on the most effective services, all these KPIs split the activity and resources of the IHD among a wide range of services at times costly, without significant results (e.g. e-learning, partner bulletins modules, serious game, podcasts). Such a dispersion of the means and attention of the IHD is likely to affect its efficiency. It could be advisable to reduce the reporting workload by focusing to some of the most significant KPIs and a uniform way of delivering.

6.1.6.What is the best methodology to measure impact of the initiative in the corporate strategy of its beneficiaries?

The key methodological challenge to evaluate the IHD is to determine to what extent the service provided has an impact the IP corporate strategy of its beneficiaries.

It could be interesting to create a pilot to look on a closer angle at the impact of the IHD initiative on the corporate strategy of their users. Running this pilot could take the form of a panel of users that provide information on an annual basis. The users would be selected from the list of IHD registered users who significantly benefitted from the IHD support (e.g. having followed trainings, having repetitively asked for advice…) during e.g. the last 18 months.

Pilot users would be requested first to indicate whether they have implemented actions, following the advice of the IHD to protect or enforce their IP. In a second phase, users would then be asked to provide information on e.g. whether the contact with the IHD modified or induced new actions on their IPR management, or whether the(se) action(s) that they took following their contact with the IHD positively influenced their business situation due to better protection and exploitation of IP. In addition, evidence of concrete steps having been taken by these pilot users would be asked for (e.g. growth of sales, jobs created/maintained, new products or services developed). The impact assessment run by the IHD themselves in 2016 173 could serve as an example of second phase questions to be asked to pilot users.

Such pilot project could eventually lead to make available to EU SMEs a collection of IP management best practices.

6.2.COHERENCE

To reply the three questions below, there is a need to analyse both external and internal coherence of the implementation of IHD initiative, i.e. between its various components and vis-à-vis other EU SME support measures as well as other relevant programmes contributing to the internationalisation of European businesses and their competitiveness in China, SEA and LA.

6.2.1.How to ensure best cooperation among the three regional entities of the IHD (i.e. in China the CNHD, in SEA the SEAHD, and in LA the LAHD)?

·According to the interviews, collaboration between the three IHD regional entities seems to be good, apparently slightly better between the CNHD and the SEAHD than between the LAHD and the two others.

·Specific mechanisms are in place to foster cooperation like a bi-monthly call with all project managers, an annual stakeholder conference, and joint publications (e.g. on trade fairs) when possible. Furthermore, the IHD are currently discussing joint visibility events. Lastly, the IHD refer companies to each other and promote each other’s services. For events in Brussels, one common representative represents the three regional entities.

The IHD view their cooperation as good. According to the survey, from a users’ point of view, the IHD are not sufficiently promoted. It is likely that improved cooperation in a common promotion strategy could help to increase the impact of the IHD communication vis-à-vis the targeted group. Such a strategy could lead to think about common scheduled actions like a joined communication with a uniform visual identity using polled communication means and sharing the promotional material with each other (in Brussels) when going to events.

6.2.2.How to best coordinate the IHD with other EU SME support measures as well as other relevant programmes contributing to the internationalisation of European businesses and their competitiveness in China, SEA and LA?

Coherence with EU actions

1. Best coordination within the COSME programme

·The interim evaluation of the COSME programme, performed before the evaluation of the IHD, mentions two main actions with potential coherence with the IHD namely the Enterprise Europe Network (EEN) and the Cluster Internationalisation actions. The other actions included in the COSME programme are either solely aimed at the Single Market (COSME countries) or do not have a strong ongoing link with IPR.

Best coordination with the EEN

·The EEN has two types of members: within and outside the COSME countries. The Network member within the COSME countries provide the full range of Network services to SMEs from their country. The Network members outside the COSME countries, also called Business Cooperation Centres (BCCs), only provide the matchmaking services as well as information about their own market to Network members from the COSME countries.

·The input from the IHD within the framework of the COSME evaluation, shows that the collaboration between the Network members and the IHD is healthy. The EEN members can refer their clients to the IHD when they have IP related questions in China, LA and SEA. Symmetrically, the IHD can refer to the EEN SMEs in need of support beyond the topic of IP. In such a large network as that of the EEN, IP appears as an important issue to manage, to only a small portion of EU SMEs. Only 3% of the Network members spontaneously mention the IHD in the open question of the survey. Within such framework, having an IHD specifically dedicated to SMEs for which IP issues appear consciously or potentially crucial for a successful internationalisation, sounds coherent. These SMEs cannot indeed find such an equivalent IP expertise among the broad range of services offered by the EEN.

·There could be some overlaps with information given by the specific BCCs category about their home market. It should however be noted that the BCCs only provide very basic market information and certainly not in-depth IP advice. As often the BCCs do not have dedicated IPR expertise, or at least not at the same level as the IHD, little to no indications of overlap are found between the activities of the IHD and the BCCs.

Best coordination with the Cluster Internationalisation actions

·The Cluster Internationalisation actions as part of the COSME programme include international cluster projects, a web portal aimed at cluster collaboration and a few matchmaking events. The international cluster projects are organised by the European Strategic Cluster Partnerships (ESCPs) and many of them target markets in China, LA and SEA and often revolve around the manufacturing industry for which the topic of IP could potentially be important. The web portal i.e. European Cluster Collaboration Platform (ECCP) 174 provides European Clusters with a tool to present themselves and to find others, to look into the ESCPs and their projects, promote various cluster events and to stimulate international cooperation. The website includes dedicated pages per country (in particular China, South Korea, Brazil and Colombia, as well as other countries covered by the IHD) providing some baseline information, key facts, key organisations, useful links and documents.

A survey was held amongst the cluster organisations as part of the evaluation of the COSME programme. The survey results show that the CNHD and LAHD are used by around 10% of the cluster organisations and the SEAHD by less than 5% of the cluster organisations. The ECCP website shows the IHD under the label of "EU International Support Services" right next to the EU IPR Helpdesk, however the IHD are not shown on the country specific pages.

·Overall the cooperation with the cluster activities of the COSME programme is minimal and there is no overlap. The cluster projects do not refer to IHD and seldom mention the topic of IPR. The IHD indicate not to be involved in any of the COSME funded international cluster projects. The marginal role of IPR in the cluster projects can be questioned in terms of coherence, as many of them are technologically advanced projects for which IP protection could potentially become a key issue. The IHD indicate that a better cooperation with the Cluster Internationalisation actions will be a priority for the future.

2. Coherence with other EU actions

·Other relevant services provided at EU level are the European IPR helpdesk, IPORTA / VIP for SMEs. The other activities are focused on the internal market, while the IHD provide the external arm. Since the other activities are mostly inward looking, the coherence seems good.

·The IP key project is complementary to the IHD both in China and in SEA and is expected to be of similar complementary in LA when the new IP key project becomes operational in spring 2018.

·In the view of the IHD members interviewed, cooperation can be increased with the EU delegations and with EU funded projects (especially in SEA and LA).

Potential synergies with similar IHD services provided by other entities

1. IPR Services provided by MS

·11 services have been identified with origin in one of the MSs and usually funded by them, having a dedicated IP focus and covering one or more of the regions China, SEA or LA. These services are provided, in large MSs only and by IP attachés (France, UK, Germany and Italy to a lesser degree) or in some cases, by trade organisations (like the German AHK or the Italian ITA IPR Desks) or business associations (like business Sweden). Most of these services, like the IHD ones, provide to some degree first line support and aim at raising overall awareness on IP matters while not excluding specific IP instruments. The main differences between the IHD and these services are that the latter focus on companies of all sizes and originating from the respective MS only.

·For similar services in 8 MSs which provide support in IP matters in the same third countries regions, a deeper analysis was undertaken. These similar services do not usually overlap with IHD services. They act as intermediaries by sign-posting their beneficiaries to the IHD helplines or publications, participating or hosting joined events, or contributing to publications incl. newsletter. While cooperation with IP attachés is valued among the strongest, Chambers of commerce appear to cooperate less.

·Synergy works better with the longer established CNHD than with the SEAHD of the LAHD where cooperation often only just started. In China, the CNHD is apparently known to all interviewed stakeholders both in China and in Europe (French INPI attachés, UK IP attachés, the German AHK Innovation Centre Denmark…) are either in direct contact, or refer to their publications, cooperates in events. In SEA, stakeholders know the SEAHD, although apparently to a lesser extent in particular in Europe. Cooperation there is good with IP attachés, but less with the three chambers of commerce. Strongest links had been built with one of them, which were eventually disrupted. In LA, similar services know the IHD services but do not cooperate or only in a few webinars or events. The French INPI attaché in Brazil is in contact with the EU delegation but not with the IHD, which is of concern. In this region, there is apparently room for more cooperation with other similar services in particular in this geographical area with the CIBEPYME network 175 that plans to provide more direct IP support services to SMEs in the future.

·2. Services on IPR provided by private entities

·Private services include IP law firms and similar services. Private services are focussed on legal support and consultancy services whereas the IHD limit themselves to providing first-line support and raising awareness. Therefore, if a company was made aware of the importance of IPR and has been provided some guidance on IPR types and how to register and protect them, the company still must get in touch with a legal service provider to register IPR. To this end, the IHD, when legally allowed, provide guidance to registered professionals in the relevant regions. Private services themselves consider the IHD not as competitors, but as potential intermediary for clients.

·As evidenced both by the finding of the survey and interviews, most SMEs even the micro-sized ones, use IP professionals (lawyers, patent attorneys) to assist them with their IP management.

Conclusion on coherence

To maintain coherence with all the public or private actors likely to provide similar IP services, there are several avenues for improvement:

1°) Internal cooperation between the three IHD entities: To reduce the heterogeneous visibility of the IHD entities vis-à-vis stakeholders notably in Europe, the IHD could be encouraged to organise common actions by polling resources, tools and information whenever possible like e.g. sharing promotional materials with each other (in Brussels) when going to events.

2°) Coherence with other networking initiatives within the COSME programme (EEN, BBCs, internationalisation clusters etc.):

·The IHD should ensure that they appear as the IP experts towards which EU SMEs must be redirected. In that regards, the IHD should ensure good professional communication around their services, and that EU networks gathering SMEs (EEN, BBCs, internationalisation clusters etc.) make them more visible (e.g. for example, by requesting more support from the BBCs or asking the EEN for an access to their events calendar). The IHD would have a vested interest in increasingly promoting within these networks, the IP protection approach as both a major risk and a key success factor for SMEs having valuable IP assets.

·The low synergy with the international cluster projects seems unhealthy. There is no valid reason, not to develop it. A better cooperation could benefit both partners. A first step could be to include the IHD in the country specific pages of ECCP website.

3°) Cooperation with other similar services:

·Maintain good relationship with IP attachés or CIBEPYME by following the example of the long-proven cooperation in China

·Enlarge collaboration with chambers of commerce for all three regions in particular in LA

·always insist on the SME focus of the IHD services, as a positive differentiating criterion

·highlight the IP expertise as an added-value (easy to access and free of charge) of the IHD services and a welcomed addition to the attachés own expertise.

6.3.EU VALUE ADDED

6.3.1.Are the IHD providing value added on top of existing services provided by MS, by the EU or by private service providers? In particular are they complementing the core trade promotion activities of MS?

From the views expressed by a majority of survey respondents, the added value of the service seems to be, for users in the provision of impartial, free of charge, pan-European as well as (third country) local advice on IPR. It is difficult to conceive that these values would have been achievable without the IHD, even when the MSs have themselves invested in IPR support.

Several MS (UK, DE, FR and IT) provide to their national SMEs similar IPR services in situ within the framework of their national globalisation policy support. These services are usually provided free of charge 176 . There is also a bilateral initiative developed between Spain and South America (CIBEPYME platform). The characteristic features of the service provided coincide to a certain extent with the one offered by the IHD. However, interviews suggest that while larger MS can provide similar services to SMEs in their national language, smaller MS cannot. In addition, the quality of advice and depth of services provided at national or bilateral level varies from one MS to the other. (Source: Desk research and interview):

·For instance, the INPI (French Industrial Property Office) provides three support services to French SMEs in their IP international strategy ("booster", "pass" and "master-class"). French companies can access the INPI attachés, i.e. the 11 experts located in different countries (including LA, China and SEA). Additionally, the INPI attachés produce fact sheets on IPR matters in the countries they cover and give first level advice. French companies can also benefit from support and can have access to a large network of experts in the context of the scheme "SME France without counterfeit".

·The Italian Trade Agency together with the Italian Ministry of Foreign Affairs and International Cooperation (MAECI) maintains IPR Desks at the embassies of selected countries. The service covers countries such as Russia, the US, China, Turkey and Japan. The SME associations’ internationalisation department acts as a relay and forwards companies’ requests to the relevant IPR Desk.

·The Delegations of German Industry & Commerce in Beijing, Shanghai and Guangzhou provide legal counselling to German companies active in China. The AHK informs about the Chinese IPR - laws and regulations, give customer-tailored advice on how to manage and especially how to protect IP and also undertake all formalities for registering domain names, trademarks and patents in China. Furthermore, the AHK branches can provide support if IPR-infringement occurred to a German company. This includes writing letters to the relevant authorities in China, lobbying, advising on the best reaction and seeing the company through the proceedings in case the infringement is brought to court (serving as the contact point to the Chinese attorney). Among the sites where the AHK is active, there are considerable differences in the services provided. According to the interviewee, in Beijing, the focus is mostly on trademark registrations, while the service in Shanghai is covering all types of IP. The main difference between the services is seen in the language, but also in services not offered by the AHK like mediations, and for the Beijing AHK, the focus of activities on the registration of trademarks. Eventually, their services are not provided free of charge.

·The CIBEPYME platform is the result of the Ibero-American Industrial Property Programme (IBEPI) implemented in 2009 to provide IPR support to Ibero-American SMEs that plan to internationalise. The platform informs and assists SMEs about the protection of their investment, trademark, designs, and other IPR forms. The platform covers the countries Argentina, Colombia, Ecuador, Mexico, Peru, Dominican Republic, Brazil, Costa Rica, Guatemala, Paraguay, Portugal, Spain and Uruguay.

·With these national (or regional in case of CYBEPYME) support, SMEs that intend to export their products or services towards geographical areas covered by the IHD, can benefit from services comparable to the ones offered by the IHD. They have access to information, to advices, and sometimes also to trainings. These services exist for companies of the respective MS. These MSs are usually among the largest. For SMEs from other MSs, a mere access to similar information reveals more difficult and usually means to have to resort to costly experts. For these SMEs, the added-value of the IHD is valuable. They would not normally benefit from equivalent expert advices for free. Could this mean that while SMEs from the largest MS obtain satisfaction trough their own MS services, the added-value of the IHD is exclusively reserved for SMEs from the other MSs? The evidence gathered tends to show that this is not the case. First of all, the, albeit, limited number of SMEs having responded to the survey include SMEs from 14 MS among which the largest MSs are well represented. In addition, national/regional support services do not usually see the IHD as a competitor. To the opposite, the IHD are normally seen as a body gathering expert information means and advices in the service of EU SMEs. There is no indication that the national or regional services would prevent their SMEs from accessing the IHD. On the contrary, national or regional services often ask the IHD for cooperation and/or direct their respective SMEs towards the IHD. One interview of the Study highlights that the Beijing (German) AHK knows the CNHD very well and cooperates especially on event organisation and by referring companies to the homepage of the IHD. This evidences that the IHD expertise is acknowledged. Despite these national or bilateral initiatives, the IHD adds value at EU level by developing and broadening this expertise to all EU SMEs. In the absence of the IHD, SMEs in several MSs would not benefit from IPR support services. In addition, allowing such a free access to these services for all EU SMEs is easier by pooling resources at European level.

From a qualitative point of view, as evidenced from the survey run, users seem satisfied with the services provided by the IHD. They find them essential while both planning to expand their business activity in the target countries and once they have entered them. Survey respondents highly value the relevance, speediness, and depth of the information provided by the IHD, as well as the quality of their advice. Therefore, it is very important to ensure this level of quality in an homogeneous way at EU level. In order that any SME in any EU MS have access to an equivalent quality of IPR service, it is of specific value to be able to offer them an IPR one-stop-shop like the IHD to accompany them in their globalisation projects. This is what gives special add valued to the European IHD by making them more effective or less costly than services provided directly by MSs or by experts.

In addition, there is apparently an EU-added value in terms of skills development for EU SMEs. Focusing resources on the IHD maintains and increases the level of expertise that SMEs need. This requires the IHD to adapt trainings, helpline, publications, etc. to the highest standards. As a result, the IHD are acknowledged as expert in this area, available to any EU SME in any MS. The IHD is unique in its dedication to the type of support it provides. It should also be noted that the form of the assistance provided, and the professionalism of the delivery generally add to the capacities of the organisations assisted and consequently add to their longer-term viability.

In the end, designing a common EU website, grouping trainings and publications, and sharing their costs, is an added-value from an economic point of view. The IHD offers services that could not be easily provided individually by each MS and allow to pool expertise to be available to any EU SME. Such economies of scale also enable to customise the training or advice to specific sectors or business models. The service also avoids duplication of technical contacts with relevant countries administrations to signpost SMEs.

6.3.2.What would be the most likely consequences of stopping or withdrawing the measure?

One question from the user survey asked the survey' respondents about the consequences of stopping or withdrawing the IHD for their organisation. Most respondents clearly agree that it would increase their overall workload. An important portion of them agree that it would generate negative consequences like unwanted know-how flows (52%), increase of IPR costs (51%), or more infringement situations (48%). This shows that, for their current users, the IHD are a key element to solve IP matters and secure their IPR when they plan to expand internationally or are yet trading outside the EU.

7. CONCLUSION

The aim of the IHD to strengthen European SME’ growth and competitiveness is directly aligned with the policy objective of the Juncker plan to boost for Jobs, Growth & Investment. "The protection of Intellectual Property Rights in third countries is crucial for promoting innovation and business of Europeans SMEs abroad. By facilitating and supporting companies to engage in international business, the helpdesks indirectly stimulate their growth and competitiveness”. The present evaluation confirms that the IHD initiative help export-oriented European SMEs benefit from the necessary tools to protect their IP. To achieve this, the IHD improve the EU SMEs' awareness of the importance to take appropriate formal and legal steps in order to reduce counterfeiting and infringement risks. Through providing suitable services (e.g. helpline, website, publications, training), the IHD allow EU SMEs to receive a complete and detailed information, practical recommendations and skills development. Offering thus, through the IHD, a free of charge access to expert information and tailored training, the EU brings support acknowledged by the SMEs which employ them. The IHD take place in a coherent manner inside the range of services offered by the EU to companies through the COSME programme. IP protection is a crucial subject for companies whose added value is based on their intellectual assets. However it is not among the major themes addressed by other networking initiatives of the COSME programme (e.g. EEN or the Cluster Internationalisation actions) because the expert knowledge the IHD brings interests a limited amount of SMEs. The specific mission that the IHD offers to these EU SMEs does not normally clash with national/regional entities providing comparable services in some MS, either directly on their territories or via their "détachés" in third countries. On the contrary, national administrations, IP civil organisms, chambers of commerce are inclined to collaborate with the IHD, e.g. by translating their publications, referring their own SMEs to the IHD, or requesting them to organise common events. It is difficult to estimate the impact of the IHD on its specific objective to reduce the risk of counterfeit products from entering the European market. However, evidence from the users' survey 177 and interviews indicate that the IHD services appear as important and useful to them. They acknowledge that the IHD contribute to make them aware of the challenges and risks linked to IP protection in the targeted third countries. Furthermore, a percentage of survey' respondents mention that the contact that they had with the IHD did impact their IP management. These results are positive. They show that support provided to EU SMEs in this area bear fruit. Therefore, there are indications that the IHD is overall effective in improving the knowledge of IP protection requirements in third countries which is a first step to reduce the risk of counterfeit products entering the EU market. A prerequisite is however that the IHD services obtain sufficient visibility among the concerned EU SMEs.

The IHD should seek in addition to be more efficient in achieving the specific actions. This is very important to progress towards the achievement of the overall objectives of the COSME Programme and the specific objective of reducing the risk of counterfeit products entering the European market from the concerned markets. At the end of the period covered by the evaluation period, the three regional entities of the IHD do not show an equivalent level of efficiency. This limits the impact of their services on SMEs. While the IHD offer is quite accessible, due to the lack of a sufficient visibility of the IHD, an important number of potential users of their services are not reached. Without support at the appropriate time, such potential users are likely to launch into internationalisation projects without (sufficiently) mastering the risks linked to IP protection. The IHD initiative appears therefore as a relevant, coherent, effective policy adding a real value to EU policy in favour of SMEs. Its efficiency needs to improve in order that the resources allocated to the IHD are used optimally to provide the best expected services to the largest possible number of EU SMEs concerned. The main outstanding concerns are listed below:

– The strategy to reach the critical mass of SMEs needs to be improved

The IHD communication strategy to reach these EU SME is mainly based on cooperation with stakeholders and attendance to events where the IHD might meet their target users. Based on this strategy, the IHD get potential users to know them by using intermediaries to show themselves or by being embedded in websites or communication portals. This "pull" communication results in a limited knowledge of the IHD services among the potential SMEs doing business in the concerned regions. Additional more proactive and direct approaches, is missing. The three IHD regional entities do not promote sufficiently their activity via direct contact, i.e. in a "push communication", aimed at EU SMEs from the manufacturing area or having a high technological added value and internationalisation projects in mind, for which IP protection might represent a real challenge.

– The reporting mechanism for KPIs is burdensome and not sufficiently quality oriented

To measure the progress achieved by the IHD initiative, the grant agreements impose on the beneficiary organisation (i.e. the consortium managing the three regional entities of the IHD) to provide follow-up and inform about an important amount of detailed performance indicators (e.g. webinars, country factsheets, case studies, serious game, video podcasts, training modules, etc.). Most of them deal with short-term outcomes and are more oriented towards quantity than quality results. By concentrating on tasks execution under the pressure of such a high number of operational targets, there is a risk to draw resources from the core activity which is to support and raise the awareness of the highest number of EU SMEs on IP protection. As evidenced, the IHD report only a few pieces of information about possible progress in the number of their users, or the satisfaction level reached by their services. Few data appear on the nature and frequency of the requests from SMEs as well. Getting information about these elements appear as very important to follow-up on the performances of the IHD activity. The reporting system does not encourage enough the IHD to progress towards more efficiency. The IHD are bound to follow up on around 50 indicators. With such a huge amount of detailed objectives, the IHD activity lose itself by achieving a lot of tasks instead of being focussed on selected keys objectives and challenged on relevant action plans.

– The IHD's visibility is quite poor, pro-active external promotion/communication plans are missing

Increasing the number of SMEs resorting to the services of the IHD is crucial. improving efficiency is useless when it benefits only a restricted number of the targeted EU SMEs, whereas the rest remains ignorant of the benefits the service offer could provide them. Too low visibility has been cited as one major issue of the three regional entities of the IHD due to an insufficient promotion effort. It is a more paramount issue for the younger regional entities of the IHD than for the CNHD. Without annual communication plans grouping and planning key partnership actions, participations to events, presentation in conferences and seminars, insertions in websites or partners portals, etc. the IHD cannot improve their visibility and cooperation with key stakeholders and SME intermediaries within Europe and in the operating regions.

– Lack of reliable indicators measuring the effectiveness of the IHD

The satisfaction indexes are rarely filled in the IHD progress reports. The lack of reliable satisfaction indicators on helpline, trainings and publications services hampers the follow up of the evolution of the IHD quality of service perceived by their users.

– Continue funding the IHD for the three regions and consider extending this type of service to India

The results of the relevance and of the effectiveness of the IHD support the notion to continue its funding. The topic is relevant, and there is apparently a sizable number of EU SMEs which do and would use the service. According to the available information, the benefits seem particularly outspoken for micro-enterprises therefore the SME focus should be maintained. Furthermore, the IHD interact mostly coherently with IP support services abroad from MSs (when such services exist). EU SMEs seem interested in receiving similar support for other markets, in particular India, given the feedback, as well as the growing importance of India as an economy both for selling and producing goods.

8.Annexes

Annex 1: Procedural information

6.Lead DG, Decide Planning/CWP references

Lead DG: Directorate-General Internal Market, Industry, Entrepreneurship & SMEs (GROW)

Work programme: Programme for the Competitiveness of the Enterprises and Small and Medium-sized Enterprises (COSME)

Specific measure: "COS-WP2014-2-05 China, Latin America and South-East Asia IPR SME Helpdesks."

7.Organisation and timing

The evaluation of the IHD was steered by the members of an Inter Service Steering Group involving various services of the Commission. Meetings of the Interservice steering Group took place on 2 December 2016, 28 April 2017, 19 July 2017 (kick-off meeting for the external study), 23 October 2017 and 24 November 2017.

Invited DGs and services: DEVCO, FPI, TRADE, RTD, SG and EASME

8.Exceptions to the better regulation guidelines

In addition to the internet based open public consultation of 12 weeks carried on for the overall COSME programme a consultation was run in the framework of the Study run by the external consultant. The survey with users and non-users of the IHD was executed electronically between August 2017 and September 2017.

9.Consultation of the RSB (if applicable)

(Not applicable)

10.Evidence, sources and quality

The following evidence has been used in the evaluation:

COSME Regulation (EU) No 1287/2013

Competitiveness and Innovation Programme (CIP)

Competitiveness of SMEs (COSME) program me

For China the CIP work programme 2010 (ENT/CIP/10/B/N02C00) and 2013 (ENT/CIP/13/B/N02C00) ,

For LA and SEA the CIP work programme 2012 (ENT/CIP/12/B/N04C00)

2012 Grant Agreement for the SEA IPR SME HD

2013 Grant Agreement for the LA IPR SME HD

2015 Grant agreement for the China IPR SME HD

2015 Grant Agreement for the SEA IPR SME HD

2015 Grant agreement for the LA IPR SME HD i.e.

· Original Grant Agreement

· Annexe 1 to original Grant Agreement

· Amended Grant Agreement

-Information/link to agencies, EU committee, main stakeholders: EASME

-External web site of relevance, databases etc.

An external consultant (Technopolis Group) run a study to partially support the evaluation on behalf of the Directorate-General Internal Market, Industry, Entrepreneurship & SMEs, Directorate F – Innovation and Advanced Manufacturing of the European Commission. This study was run within the Framework Contract between Technopolis Group and Internal Market, Industry, Entrepreneurship & SMEs (ENTR/172/PP/2012/FC/LOT 4 – Evaluation of EU programmes, legislation, policies and other activities).

Three independent experts reviewed and gave an opinion on the draft final report prepared by the contractor and the current staff working document of the Commission services, and presented their views 178 on the relevance and coherence of the IHD initiative.

(1)

http://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX:32013R1287

(2)

Moldova, Armenia and Ukraine will not participate to the COSME financial instruments.

(3)

See footnote 10

(4)

In addition, for the Enterprise Europe Network, possibilities exist for business support organisation in non-participating countries to associate themselves with the Network. They will not receive funding from the programme but should be entitled to use tools and information developed within the Network.

(5)

http://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX%3A32013R1287

(6)

http://eur-lex.europa.eu/legal-content/EN/TXT/?uri=URISERV%3An26104

(7)

From the executive summary of the CWP Impact Assessment accompanying the proposal for a COSME Regulation, COM(2011)834 SEC(2011)1452, http://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX:52011SC1452

(8)

EASME is the successor of the Executive Agency for Competitiveness and Innovation (EACI)

(9)

92% on average for 2014 and 2015 for objectives 2, 3 and 4.

(10)

COM(2005) 121

(11)

For a detailed comparison of CIP and COSME: Fanny Lajarthe (2012). Differences and Similarities between CIP and COSME: Briefing Note. IP/A/ITRE/2012-14

(12)

CSES (2011). Final Evaluation of the Competitiveness and Innovation Framework Programme

(13)

Final Evaluation of the Entrepreneurship and Innovation Programme - Final Report, Centre for Strategy and Evaluation Services, April 2011

(14)

Interim evaluation of the Entrepreneurship and Innovation Programme (April 2009) and Final evaluation of the Entrepreneurship and Innovation Programme (April 2011)

(15) Budget lines 02.0201 and 02.0202 for all fund sources, including third countries contributions and reflows from the CIP financial instruments and not including the COSME administrative budget lines.

(16)

The so-called COSME LGF enhancement

(17)

See COSME LGF Implementation Status as at 31 March 2016 available at: http://www.eif.org/what_we_do/guarantees/single_eu_debt_instrument/cosme-loan-facility-growth/index.htm

(18)

SAFE survey: http://ec.europa.eu/growth/access-to-finance/data-surveys_en

(19)

EU Access to Finance web portal: http://europa.eu/youreurope/business/funding-grants/access-to-finance/

(20) Interim Evaluation of the COSME Programme, final report, Technopolis, December 2017, https://ec.europa.eu/docsroom/documents/28084

(21) Interim Evaluation of the International Intellectual Property Rights (IPR) SME Helpdesk, a specific COSME measure, Technopolis, Annex 4 of this report.

(22) Evaluation of WATIFY 1 and WATIFY 2, Centre for strategy & evaluation services, 2019.

(23)

Outlined in section 2 of the present document

(24)

https://ec.europa.eu/info/consultations/interim-evaluation-programme-competitiveness-enterprises-and-small-and-medium-sized-enterprises-cosme-2014-2020_en

(25) https://ec.europa.eu/docsroom/documents/28084

(26)

Co-financing from various EU programmes is also possible.

(27)

There is no standard definition of “high risk” or “higher risk” SMEs among European financial intermediaries

(28) Pp 34 and 35, Interim Evaluation of the COSME Programme, final report, Technopolis, December 2017, https://ec.europa.eu/docsroom/documents/28084

(29) P. 35, Interim Evaluation of the COSME Programme, final report, Technopolis, December 2017, https://ec.europa.eu/docsroom/documents/28084

(30)

ECCP, 2016: Analysis of the Consultation launched by ECCP amongst the ESCP-4i partnerships in June 2016,

(31)

http://www.lowcarbonbrazil.com/index , http://www.lowcarbon.mx/

(32) http://ec.europa.eu/COMMFrontOffice/publicopinion/index.cfm/Survey/getSurveyDetail/instruments/FLASH/surveyKy/2090

(33)

European Commission, 2015: Flash European Barometer, Internationalisation of SMEs

(34) https://www.eu-japan.eu/

(35) http://www.ipr-hub.eu/

(36) https://www.clustercollaboration.eu/

(37)

Kraemer-Eis, H et al. (2016), The European venture-capital landscape: an EIF perspective, Volume I: The impact of EIF on the VC ecosystem, EIF Research and Market Analysis, Working Paper 2016/34.

(38)

European Commission, (2015): Staff Working Document SWD(2015) 203 final on 28 October 2015 accompanying the Communication Upgrading the Single Market: more opportunities for people and business, COM(2015) 550.

(39)

European Commission, (2016): Annual Growth Survey for 2015. Strengthening the recovery and fostering convergence

(40)

European Commission, (2017): Annual Growth Survey for 2016. Strengthening the recovery and fostering convergence

(41)

https://ec.europa.eu/info/law/law-making-process/better-regulation-why-and-how_en

(42)

The long-term objective set by the COSME regulation is to have 2500 Partnership agreements signed per year.

(43)

https://publications.europa.eu/en/publication-detail/-/publication/d4cf03ed-972c-11e5-983e-01aa75ed71a1/language-en

(44)

The long-term objective set by the COSME regulation is "Client satisfaction rate (% SMEs stating satisfaction, added-value of specific service): > 82 %"

(45) Interim Evaluation of the International Intellectual Property Rights (IPR) SME Helpdesk, a specific COSME measure, Technopolis, Annex 4 of this report.

(46)

https://www.clustercollaboration.eu/eu-cluster-partnerships/escp-4i/first-generation/achievements and https://www.clustercollaboration.eu/news/clusters-go-international-partnering-event-2018-sharing-experiences-boost

(47)

General budget of the European Union for the financial year 2018 ( http://eur-lex.europa.eu/budget/data/DB/2018/en/SEC03.pdf) , 2017 http://eur-lex.europa.eu/budget/data/DB/2017/en/SEC03.pdf) and 2016 ( http://eur-lex.europa.eu/budget/data/DB/2016/en/SEC03.pdf) . Based on outturn figures recorded in those documents (i.e. expenditure from previous years).

(48) Interim Evaluation of the COSME Programme, final report, Technopolis, December 2017, https://ec.europa.eu/docsroom/documents/28084

(49) Interim Evaluation of the COSME Programme, final report, Technopolis, December 2017, https://ec.europa.eu/docsroom/documents/28084 , p.112

(50)

The horizontal actions of the European Cluster Collaboration Platform and the European Observatory for Clusters and Industrial Change also provide support within the umbrella of cluster initiatives.

(51)

https://ec.europa.eu/info/evaluation-reports-economic-and-financial-affairs-policies-and-spending-activities/independent-evaluation-european-fund-strategic-investments-european-investment-advisory-hub-and-european-investment-project-portal_en

(52)

In Mexico, at least 50% of European participants were informed about the LCBA project through cluster organisations. 90% of those clusters were members of the European Cluster Collaboration Platform (ECCP)

(53)

http://www.lowcarbonbrazil.com/index , http://www.lowcarbon.mx/

(54)

(55)

The European cluster initiatives are designed to complement and support efforts taken at regional and national level, with a particular focus on supporting strategic interregional partnering.

(56)

Technopolis, EY & Danish Technologica Institute (2014) Evaluation - Cluster initiatives managed by DG Enterprise and Industry, ENTR/172/PP/2012 – LOT4, available at https://publications.europa.eu/en/publication-detail/-/publication/c1ad8410-9bae-49df-b1f1-69b36071cf30/language-en

(57)

REGULATION (EU) No 1287/2013

(58)

For example, 16 FTE in EASME implemented 91 contracts with a budget of 58 million for the EEN in 2017 compared to 36 FTEs who implement 47 actions divided into 80 contracts for a budget of 45 million.

(59)

https://een.ec.europa.eu/

(60)

http://ec.europa.eu/growth/smes/business-friendly-environment/small-business-act/sme-envoys_en

(61) http://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX%3A32013R1287

(62) http://eur-lex.europa.eu/legal-content/EN/TXT/?uri=URISERV%3An26104

(63) http://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX:52011SC1452 and http://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX:52011SC1453

(64)

http://ec.europa.eu/budget/mff/figures/index_en.cfm#com_2016_603

(65) http://ec.europa.eu/cip/public_consultation/index_en.htm

(66) http://ec.europa.eu/cip/documents/index_en.htm

(67) http://bookshop.europa.eu/en/search/?webform-id=WFSimpleSearch&DefaultButton=findSimple&WFSimpleSearch_NameOrID=EACI&SearchConditions=&SearchType=1&SortingAttribute=LatestYear-desc&findSimple.x=0&findSimple.y=0

(68) http://bookshop.europa.eu/en/final-evaluation-of-the-impact-of-the-enterprise-europe-network-2008-2014-pbET0415831/ Full Report http://bookshop.europa.eu/en/final-evaluation-of-the-impact-of-the-enterprise-europe-network-2008-2014-pbET0415830/

(69) http://ec.europa.eu/growth/smes/business-friendly-environment/performance-review_en

(70) https://ec.europa.eu/growth/smes/business-friendly-environment/small-business-act_en

(71) http://cor.europa.eu/en/takepart/eer/Documents/SBA%20implementation%20at%20regional%20and%20local%20level/EN.pdf

(72) https://ec.europa.eu/growth/single-market/strategy_en

(73) http://ec.europa.eu/growth/tools-databases/newsroom/cf/itemdetail.cfm?item_id=8723

(74) https://ec.europa.eu/growth/single-market/forum_en

(75) http://ec.europa.eu/growth/access-to-finance/data-surveys_en and http://www.ecb.europa.eu/stats/money/surveys/sme/html/index.en.html

(76) http://ec.europa.eu/public_opinion/flash/fl_354_en.pdf

(77) http://ec.europa.eu/COMMFrontOffice/PublicOpinion/index.cfm/Survey/getSurveyDetail/instruments/FLASH/surveyKy/2090

(78) https://ec.europa.eu/growth/smes/cosme_en

(79) https://ec.europa.eu/easme/en

(80) http://een.ec.europa.eu/

(81) http://www.erasmus-entrepreneurs.eu/

(82) http://www.eif.org/what_we_do/guarantees/single_eu_debt_instrument/cosme-loan-facility-growth/

(83) http://www.eif.org/what_we_do/equity/single_eu_equity_instrument/cosme_efg/index.htm

(84) https://ec.europa.eu/growth/tools-databases/vto/

(85)

the ones which received EFG investment in the first half of 2016

(86)

See also: EASME (2014). Guide for Applicants to COS-WP2014-2-1

(87)

http://ec.europa.eu/cip/files/docs/eip-final-evaluation-report_en.pdf

(88)

See: http://ec.europa.eu/cip/files/docs/eip-final-evaluation-report_en.pdf

(89)

The countries that the survey covered account for 96% of the entire population of beneficiaries at the time of the consultation (138,102 companies of a total 143,344). The survey did not include intermediaries or companies in the Czech Republic, Montenegro, Slovakia or the UK.

(90)

There may be slight differences between the beneficiary population and the survey population, as the former is centrally recorded, while the survey invited participants to self-report their sector of activity.

(91)

The 34 countries are: 14 of the EU15 countries (Austria, Belgium, Germany, Denmark, Greece, Spain, Finland, France, Ireland, Italy, Netherlands, Portugal, Sweden, United Kingdom) 12 of the EU13 countries (Bulgaria, Czech Republic, Estonia, Croatia, Hungary, Lithuania, Latvia, Malta, Poland, Romania, Slovenia, Slovakia) and eight of the 11 COSME-participating third countries (Albania, Armenia, Bosnia and Herzegovina, Moldova, Montenegro, Former Yugoslav Republic of Macedonia, Serbia, Turkey)

(92)

All EU15 countries (Austria, Belgium, Denmark, Finland, France, Germany, Greece, Ireland, Italy, Luxembourg, Portugal, Spain, Sweden, The Netherlands, United Kingdom), all EU13 countries (Bulgaria, Croatia, Cyprus, Czech Republic, Estonia, Hungary, Latvia, Lithuania, Malta, Poland, Romania, Slovakia, Slovenia), and four third countries (Belize, Republic of North Macedonia, Iceland, Turkey)

(93)

Full list of COSME-participating third countries can be viewed here: https://ec.europa.eu/docsroom/documents/34263

(94)

the ones which received EFG investment in the first half of 2016

(95) Members: Argentina, Brazil, Paraguay, Uruguay; Venezuela suspended

(96) Competitiveness of SMEs (COSME) program me

(97)

‘Intellectual Property Rights’ are used throughout this document in the broad sense, encompassing both so-called ‘formal’ means of protecting such rights (e.g. patents, trademarks, designs, geographical indications, copyrights) and ‘informal’ means (contractual and licensing conditions, management of business operations so as to protect trade secrets, planning business participation in trade fairs, etc.) ‘Enforcement’ of these rights refers to any applicable means of enforcement such as through the civil or criminal judicial systems, administrative means, customs authorities, or contractual means, as available in the relevant European, Member State, or third countries' jurisdictions.

(98) which is the most important player in the origin of counterfeit products (see https://euipo.europa.eu/ohimportal/en/web/observatory/mapping-the-economic-impact )

(99)

For China the CIP work programme 2010 (ENT/CIP/10/B/N02C00) and 2013 (ENT/CIP/13/B/N02C00)

(100)

For LA and SEA the CIP work programme 2012 (ENT/CIP/12/B/N04C00)

(101) https://ec.europa.eu/easme/

(102) see CNHD Annual Work Plan 2015

(103) e.g. with the EU SME Centre

(104) e.g. with the EU Business Avenues , EU Business centres in India ( EKONID ), Malaysia ( SEBSEAM ), Philippines, Thailand ( EABC ) and Vietnam ( EVBN ), as well as all European and bilateral chambers of commerce in SEA

(105) e.g. with European and bilateral chambers of commerce in LA, MSs' and EU delegations, related associations, and other EU network (e.g. ELAN )

(106)

i.e. 2015 Grant agreement for the China IPR SME HD ; 2015 Grant Agreement for the SEA IPR SME HD and 2015 Grant agreement for the LA IPR SME HD including the Original Grant Agreement ; Annexe 1 to original Grant Agreement and the Amended Grant Agreement

(107) 770 in June 2017 as reported in the Progress Report of the SEAHD

(108) A challenge that occurred recently was that the LAHD faced the situation of a poor provision of web services. This resulted notably in having no helpline management system available since the beginning of May 2016, in a long response time of months in most of the cases, in the loss of website statistics after the website transfer in August 2016, and in a delay in the deliverance of dissemination materials, requiring overall much extra effort from staff.

(109)

2010 assessment of the China IPR SME HD

(110) See http://www.latinamerica-ipr-helpdesk.eu/sites/default/files/la_ipr_sme_helpdesk_baja_pilot_study.pdf

(111) Report on the evaluation of the European IPR Helpdesk published in 2014

(112) Internal impact assessment exercise carried out by the IHD themselves in 2016

(113)

U:\IPR Helpdesks\02. International helpdesks\01. Intl HD 2017 evaluation\Technopolis Reports on IPR evaluation\Eval-IPRHelpdesks_final-26Jan2018.pdf

(114) See "Expert Group assessment on the evaluation of the EU international IPR SME helpdesks (IHDs)", Synthesis report prepared by Alfred Radauer under the following link: U:\IPR Helpdesks\02. International helpdesks\01. Intl HD 2017 evaluation\Experts opinion\ExpertGroup-Report-final.docx

(115)

http://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX%3A32013R1287

(116) http://ec.europa.eu/cip/

(117) https://ec.europa.eu/growth/smes/cosme_en

(118) http://ec.europa.eu/cip/files/docs/eip2010_wp_annex_indicators_en.pdf

(119) http://europski-fondovi.eu/sites/default/files/dokumenti/eip-2013-work-programme-final_en.pdf

(120) http://ec.europa.eu/cip/files/cip/docs/eip-2012-work-programme-consolidated-version-after-2nd-revision-adopted-26-november_en.pdf

(121) see 2012 Grant Agreement for the SEA IPR SME HD

(122) see 2013 Grant Agreement for the MERCOSUR IPR SME HD

(123) see 2015 Grant agreement for the China IPR SME HD

(124) https://myintracomm-collab.ec.europa.eu/networks/COSMESI/Shared%20Documents/COSME%202014/Objective%202%20-%20Access%20to%20Market%20-%20Code%20D/D-N205%20-%20IPR%20SME%20Helpdesk%20China%20-%20South/Selected%20Projects/Grant%20Agreement-657914-SEA%20HD%202015-2017.pdf

(125) https://myintracomm-collab.ec.europa.eu/networks/COSMESI/Shared%20Documents/COSME%202014/Objective%202%20-%20Access%20to%20Market%20-%20Code%20D/D-N205%20-%20IPR%20SME%20Helpdesk%20China%20-%20South/Selected%20Projects/17_03_09_Grant%20Agreement-657846-LA%20IPR%20SME%20HD.pdf

(126) https://myintracomm-collab.ec.europa.eu/networks/COSMESI/_layouts/15/WopiFrame.aspx?sourcedoc=/networks/COSMESI/Shared%20Documents/COSME%202014/Objective%202%20-%20Access%20to%20Market%20-%20Code%20D/D-N205%20-%20IPR%20SME%20Helpdesk%20China%20-%20South/Selected%20Projects/17_03_09_Annex%201%20-%20Description%20Of%20Action%20(part%20A).pdf&action=default

(127) https://myintracomm-collab.ec.europa.eu/networks/COSMESI/_layouts/15/WopiFrame.aspx?sourcedoc=/networks/COSMESI/Shared%20Documents/COSME%202014/Objective%202%20-%20Access%20to%20Market%20-%20Code%20D/D-N205%20-%20IPR%20SME%20Helpdesk%20China%20-%20South/Selected%20Projects/Amendment%20AMD-657846-4.pdf&action=default

(128) Among the 14 additional contributions provided by certain contributors in the framework of this consultation, four mentions the IHD.

(129) see Study, Appendix A

(130) See Study, Appendix B

(131) See Study, Appendix F

(132) See Study, Appendix C, D and E

(133) Better Regulation guidelines

(134) See, for example, OceanTomo (2016): Intangible Asset Market Value Study, http://www.oceantomo.com/intangible-asset-market-value-study/ ; or: WIPO (2017): Intangible Capital in Global Value Chains, http://www.wipo.int/edocs/pubdocs/en/wipo_pub_944_2017.pdf

(135) OECD/EUIPO (2016): Trade in Counterfeit and Pirated Goods – Mapping the Economic Impact.

(136) Office of the United States Trade Representative (2017): 2017 Special 301 Report.

(137) UPS (2016): 2016 European SME Exporting Insights Study

(138) Abel-Koch (2016): SMEs’ value chains are becoming more international – Europe remains key

(139) even if the cited export analyses of SMEs might indicate that it is only a smaller share of the European SME population that may be affected by this, the actively exporting SMEs with or without foreign direct investments represent still a significant amount.

(140) See above comments made on Figure 3

(141) European SME Exporting Insights Study 2015

(142)

i.e. Belgium, France, Germany, Italy, the Netherlands, Poland and the UK

(143) i.e. industrial manufacturing & automotive (IM&A), retail, high-tech and healthcare

(144) See above Table showing the Objectives of the IHD

(145) ANNEX to the Commission Implementing Decision concerning the adoption of the work programme for 2014 and the financing for the implementation of Programme for the Competitiveness of Enterprises and small and medium-sized enterprises does not list law firms among the representatives, but does not exclude them either.

(146) i.e. "very effective" or "effective"

(147) see below Table 3 in the Efficiency part

(148) see below Table 3 in Efficiency part

(149) see below Table 3 in Efficiency part

(150) For instance, the LAHD wanted to organise a lot of project management via the websites which was a functionality not foreseen by the creators of the central website

(151) see below Table 3 in Efficiency part

(152) See ANNEX to the Commission Implementing Decision concerning the adoption of the work programme for 2014 and the financing for the implementation of Programme for the Competitiveness of Enterprises and small and medium-sized enterprises

(153) Non publishable internal source

(154) The China 3/11/2016, LA 16/12/2016 and SEA PO Assessment Reports mentions that the project has put emphasis on cooperation (dialogue, joint coordination like implementing an annual cooperation action plan, signature of MoU) with the most relevant EU-funded projects notably the EU SME Centre and EU Gateway projects (in China); ICI+ projects, EU Business Avenues, EU-ASEAN Business Networks (in SEA); ELAN Network, ELAN Biz and AL INVEST (in LA) and the European IPR SME Helpdesk resulting in joint dissemination, joint participation in events and trainings (onsite or in Europe) or webinars, by exchanging internally best practice, sharing joint actions or developing an international 'train-the-trainer' programme for intermediaries.

(155) Non publishable internal source

(156) Non publishable internal source

(157) This chart appears on the EC webpage dedicated to "IPR Infringements: Facts and figures" which publishes the 2016 Report on EU customs enforcement of IPR at the EU border

(158) See previous note

(159) 2017 Situation Report on Counterfeiting and Piracy in the European Union

(160) see above Figure 7

(161) see Study, interview synthesis

(162) see Project Officer Report in SEA

(163) see Project Officer Report in LA

(164) see Project Officer Report in each targeted region

(165) 770 in June 2017 as reported in the Progress Report of the SEAHD

(166) Due to unexpected changes in the composition of the consortium, see on page 30

(167) See Table 2

(168) see interview synthesis, the only instance mentioned is that the SEAHD, due to budget limitations, could not cover Vietnam with an on-site partner

(169) e.g. for the IHD, under the current grant agreement, a pre-financing of EUR 2,256,152 (1,299,459.50 for CNHD + 598,138.50 for LAHD + 358,554 for SEAHD)

(170) Third countries with a special agreement with the Union in the field of public procurement are: Albania, Bosnia and Herzegovina, the former Yugoslav Republic of Macedonia, Montenegro, Serbia, Iceland, Liechtenstein and Norway

(171) Such exception was actually activated under the current COSME Grants

(172) In the case of COSME grant agreements, profit entities based in non-COSME countries are totally excluded from receiving EU funds and hence cannot sign a contract.

(173) See footnote 16

(174) https://www.clustercollaboration.eu/

(175) The CIBEPYME platform (Platforma Iberoamericana de Propriedad Industrial Dirigida a Empresas) provides IPR advice tailored to SMEs from Spain, Portugal and selected Latin-American countries

(176) One MS apart i.e. Germany

(177) 70 users and 40 potential users replied to the surveys

(178) U:\IPR Helpdesks\02. International helpdesks\01. Intl HD 2017 evaluation\Experts opinion\ExpertGroup-Report-final.docx

Choose the experimental features you want to try