Showing posts with label Frascati Manual. Show all posts
Showing posts with label Frascati Manual. Show all posts

Friday, October 28, 2022

The novelty test and the reproducibility criterion


In response to SAR's 2013 letter to the OECD, calling for a revision of the Frascati Manual in order to add artistic research as a new and separate scientific category (see here), the new 2015 Frascati manual made a negative but clear stance. I have reported on that briefly (see here), and want to elaborate a little.



Translating the position delineated in the manual (2.65-67) to the discourse in artistic research, we have: 

  • research on the arts, includes musicology, art-history, etc. 
  • research for the arts, relates to AR 
  • artistic expression, art itself 

(The manual limits the examples of research for the arts to the traditional sectors catering to artists, such as instrument builders, but it should be clear that artistic research targets artists as the beneficiaries of their research just as well.)

Of further interest are the novelty test and reproducibility criterion that are cited as fundamental to research. For art, these are simply considered not relevant: in a paragraph about artistic expression vs. research, it is stated that

2.67. [...] Artistic performances fail the novelty test of R&D as they are looking for a new expression, rather than for new knowledge. Also, the reproducibility criterion (how to transfer the additional knowledge potentially produced) is not met. [...]

The novelty test, although no written rule, is largely considered to be a priority in universities, at least in Europe. During the Bergen Conference of European Ministers Responsible for Higher Education, in May 2005, the qualifications signifying the completion of a third cycle (see here) are awarded to students who (my highlighting):

  • have demonstrated a systematic understanding of a field of study and mastery of the skills and methods of research associated with that field; 
  • have demonstrated the ability to conceive, design, implement and adapt a substantial process of research with scholarly integrity; 
  • have made a contribution through original research that extends the frontier of knowledge by developing a substantial body of work, some of which merits national or international refereed publication
  • are capable of critical analysis, evaluation and synthesis of new and complex ideas; 
  • can communicate with their peers, the larger scholarly community and with society in general about their areas of expertise; 
  • can be expected to be able to promote, within academic and professional contexts, technological, social or cultural advancement in a knowledge based society. 

Artistic knowledge is often difficult to determine. Certainly the "embodied" type of knowledge is problematic because of its tacit nature. Unless it is explicated in an efficiently communicable language, there is no use of it outside of theoretical perspectives. But even in the latter case, as in Knowing in Performing, the knowledge is pointed to, considered, and transferred... in writing, not even with an added CD or DVD. In practice, embodied knowledge cannot be considered as such by itself, either. When a dance teacher shows a student how to perform a certain movement, there will always be verbal explication accompanying the demonstration, if only to make sure that there is no inadvertant misinterpretation of some detail. It may be difficult to write efficiently about such knowledge (although a picture painting a thousand words suggests a promising start), but the entirely non-verbal manner is least (re)productive. No surprise the reaction in the Frascati manual, then, about the reproducibility criterion not being met. 

Despite the powerful method of one-to-one teaching so typical of performance pedagogy, reproducibility is considered to be against the point of being an artist. Indeed, what is heard/seen on stages and recordings is considered valuable, or at least valid, only in relation to the  individuality of the artist. Generally, a new recording is either produced with new repertoire, or with the same repertoire as before but in different interpretations. But this is no more than an assumption, and with globalisation, the idea that countless thousands of pianists would each be able to play the same repertoire in noticeably different ways has no right to keep evading scrutiny. Of course, reproducibility in research is really meant to be that of the applied method. From that perspective, AR is aimed at the peer-researcher or the peer-artist, who need to be able to trust the accuracy of the resulting insights. AR is not aimed at the audience, at least not directly. Hence the misconception that AR should always be audibly distinguishable from art, or that it would destroy the "magic" of making the art (as I was once accused of by a professional performer).

The problem of art vs. art-as-research vs. artistic research relates to the demarcation problem in science, including immunizing strategies and defense mechanisms. In art, and about art, one can express and claim anything. In this New York Times review of a concert by Yuja Wang, little makes sense, except that it can be appreciated as a verbal performance by the reviewer. Perhaps he wanted to transmit knowledge that he felt he acquired; it may even be exactly what Ms. Wang had intended to convey. But it is immune to being refuted, so it might as well be a review of another concert altogether. It is art, about art. No new knowledge or reproducibility needed to be expressed or checked. 

In artistic research, what constitutes 'new knowledge' can be seen to underly a division in the genres. Some visual artists have expressed less than emphatic feelings towards musicians who play music of long-dead composers, as if present-days painters would be considered proper artists if they painted à la Rubens. Similarly, the question can be asked whether artistic researchers who investigate, say, performance practices from the 18th century, can be considered to develop new knowledge if, in fact, demonstrably old knowledge is unearthed and reapplied to their practice. Practices in other disciplines, e.g. archeology, can provide answers to this concern. Neverthess, 'novelty' alone is often not enough: there has to be 'impact' as well. From the perspective of the Frascati manual, it goes tangentially deeper. A musician like Ton Koopman could be acknowledged as engaging in scholarship, which in turn may partly qualify as research. But the resulting 'new' expression in his performances would not qualify for research or development. Since the performance can be considered as a product, the discussion would not be whether it is research, but whether it is experimental development. And this development would principly not qualify because of the tests that should then apply. 

The Frascati manual is intended as guidance for measuring and reporting policies. Naturally, such guidance can become a proxy debating space, given the sensitivities and sustainability of some institutions and university departments. As such, the manual is meant to avoid that, for instance, artistic performance is taken into account as R&D expenditure, the latter of which can be witnessed here (scroll down to the before-last line of table 3). But, as much as the stance against acknowledging a new knowledge-type for AR is clear, a door is left open for accepting institutional AR: "Higher education institutions have, nevertheless, to be evaluated on a case-by-case basis if they grant a doctoral degree to an artist as a result of artistic performances. The recommendation is to adopt an “institutional” approach and only to take account of artistic practice recognised as R&D by higher education institutions as potential R&D". (2.67)

The first Frascati Manual ever was issued in the early 1960s, and six new versions have appeared in the half-century since, at a rate of about nine years per new issue. (See here for some history.) It may take a while for the OECD to reconsider the matter. It may take that same while before it makes the distinction between artistic performance and composition... 

Monday, February 29, 2016

A move to put our mark on the Frascati Manual²



A bit over two years ago, I reported (hereon a request by the Society for Artistic Research (SAR) to the Organisation for Economic Co-operation and Development (OECD), calling for a revision of the Frascati Manual in order to add AR as a new and separate scientific category.

The motion was amplified through the European League of Institutes of the Arts (ELIA) proposal, in collaboration with SHARE, two networks that represent some 339 (art) universities and academies, arguing for an “appropriate” status of the arts and arts research. It was put forth that this status should be supported by the recognition of the Arts (and AR with it) as a field of its own: not as part of the Humanities (next to History and Archaeology, Languages and Literature, Philosophy, Ethics and Religion, and “Other” humanities), but on par with Humanities, Natural sciences, Engineering and technology, Medical and Health sciences, Agricultural sciences, and Social sciences. (See here for a letter from ELIA in response to an OECD inquiry, highlighting some of the reasoning.)


Frascati Manual, artistic research, tacit knowledge, embedded knowledge

Last October, the updated manual was published (online readable and freely downloadable here; summary PPT presentation here). From p. 60 onwards, the Frascati Manual goes into the matter under “Examples of R&D, boundaries and exclusions in different areas”:

R&D and artistic creation

          2.64 Design sometimes tends to be characterised by the use of artistic 
          methods. This is another potential area of overlap. In order to address the 
          discussion of R&D and artistic creation, it can be useful to make a 
          distinction between research for the arts, research on the arts and artistic 
          expression.

Research for the arts

2.65 Research for the arts consists in developing goods and services to meet the expressive needs of artists and performers. There are enterprises in this line of business that devote a significant part of their resources to R&D in this area. For instance, they engage in experimental development to produce new electronic musical instruments to suit the needs of a group of performers. Other types of R&D organisations (mainly universities and technical institutes) also play a role in exploring new technologies for performance art (to improve audio/ video quality, for instance). The activity aimed at supporting the introduction of new organisational or marketing methods by art institutions (advertising, financial management, etc.) may qualify as R&D, but caution should be exercised in making this decision. This area of R&D performance is already covered by existing data collection.

Research on the arts (studies about the artistic expression)

2.66 Basic or applied research contributes to most of the studies of the arts (musicology, art history, theatre studies, media studies, literature, etc.). Public research institutions could have a role in selected research domains (as some relevant research infrastructures – like libraries, archives, etc. – are often attached to arts institutions, such as museums, theatres, etc.). As far as preservation and restoration activities are concerned (if not to be included in the group above), it is recommended to identify the providers of such technical services as R&D performers (employing researchers, publishing scientific works, etc.). This area of R&D performance is largely covered by existing data collection.

Artistic expression versus research

2.67 Artistic performance is normally excluded from R&D. Artistic
performances fail the novelty test of R&D as they are looking for a new expression, rather than for new knowledge. Also, the reproducibility criterion (how to transfer the additional knowledge potentially produced) is not met. As a consequence, arts colleges and university arts departments cannot be assumed to perform R&D without additional supporting evidence. The existence of artists attending courses in such institutions is not relevant to the R&D measurement. Higher education institutions have, nevertheless, to be evaluated on a case-by-case basis if they grant a doctoral degree to an artist as a result of artistic performances. The recommendation is to adopt an “institutional” approach and only to take account of artistic practice recognised as R&D by higher education institutions as potential R&D (to be further used by data collectors).


It is clear that the OECD did not follow the reasoning of ELIA, SHARE, and SAR all the way up to the desired consequences: the arts did not get the requested separate status, nor did arts research. In a way, this is not a big deal, I think: the so-called FORD categorization (the Fields Of Research and Development list as reproduced in my previous entry on the matter) serves analysts to statistically understand the dynamics in those fields. One can wonder whether the 1-digit level is crucial. Would it be more advantageous to compare the arts to natural sciences or engineering rather than to history, language, or philosophy? There is also plenty of overlap in different categories, and, most generally, any attempt at structuring such givens may well be futile. The SAR/ELIA/SHARE-move can be understood strategically, though: by considering AR to be in a league of its own, it might support the proponents in visual arts, or in music composition, who argue that the artistic practice is the research, and that the artistic output is the research output. They would certainly benefit from an internationally formalized extra epistemological category, allowing for further alienation from the established academic or academia-oriented (sub-)disciplines and their paradigms.


For the moment, however, this is not happening.

Thursday, November 14, 2013

A move to put our mark on the Frascati Manual




According to its latest newsletter, the Society for Artistic Research wrote a letter to the Organisation for Economic Co-operation and Development (OECD), on August 30, to call for a revision of the Frascati Manual in order to add AR as a new and separate scientific category. (Posted on the society’s FaceBook Timeline on October 29, 2013.)


Frascati Manual, artistic research, tacit knowledge, embedded knowledge


The Frascati Manual was thought of as initiating a “Proposed Standard Practice for Surveys on Research and Experimental Development” to measure Scientific and Technological Activities. Its latest (6th) edition dates 2002 and defines types of research and research personnel, deals with measuring expenditure and personnel resources, and also organizes the "Field Of Science" into main and sub-categories. (Buy here or download its free PDF here.The book is highly influential: worldwide, governments and organisations adopt its definitions for discussing their scientific, technological and economical development policies, and R&D studies acknowledged its function as a standard.

The 2002 edition Field of Science classification was revised in 2006 (freely downloadable here) and is listed as follows:

1. Natural sciences
1.1 Mathematics
1.2 Computer and information sciences
1.3 Physical sciences
1.4 Chemical sciences
1.5 Earth and related Environmental sciences
1.6 Biological sciences (Medical to be 3, and Agricultural to be 4)
1.7 Other natural sciences
2. Engineering and technology
2.1 Civil engineering
2.2 Electrical engineering, Electronic engineering, Information engineering
2.3 Mechanical engineering
2.4 Chemical engineering
2.5 Materials engineering
2.6 Medical engineering
2.7 Environmental engineering
2.8 Environmental biotechnology
2.9 Industrial biotechnology
2.10 Nano-technology
2.11 Other engineering and technologies
3. Medical and Health sciences
3.1 Basic medicine
3.2 Clinical medicine
3.3 Health sciences
3.4 Medical biotechnology
3.5 Other medical sciences
4. Agricultural sciences
4.1 Agriculture, Forestry, and Fisheries
4.2 Animal and Dairy science
4.3 Veterinary science
4.4 Agricultural biotechnology
4.5 Other agricultural sciences
5. Social sciences
5.1 Psychology
5.2 Economics and Business
5.3 Educational sciences
5.4 Sociology
5.5 Law
5.6 Political science
5.7 Social and economic geography
5.8 Media and communications
5.9 Other social sciences
6. Humanities
6.1 History and Archaeology
6.2 Languages and Literature
6.3 Philosophy, Ethics and Religion
6.4 Arts (arts, history of arts, performing arts, music)
6.5 Other humanities


The 6.4 Arts category is specified as containing: Arts, Art history; Architectural design; Performing arts studies (Musicology, Theater science, Dramaturgy); Folklore studies; Studies on Film, Radio and Television. (The 2002 classification saw 6. Humanities as made up merely from 6.1 History, 6.2 Languages and literature, and 6.3 Other humanities.)

The noble but bold request of SAR is to add AR not only to the classification as a new category, but to do so “as a significant category on its own terms.”  The 6.4 Arts category, as “containing various fields of research on art” [my italics] is juxtaposed with the claim that “this classification does not in any satisfactory way reflect the development of artistic research” (i.e. research through art [my italics]). The letter’s phrasing – to see AR as “a field of its own on the 1-digit level” – leaves unclear whether SAR wants AR to sit at the 1-digit level (making it a 7th high-level group) or under it (as a distinct 6.6).

Next to the notion of the unsatisfactory way that AR would compare to the subcategories under 6., the society’s letter states several other reasons for its claim, such as the Frascati manual’s “aims to reflect changes in the nature of contemporary R & D” (the 2006 revision is indeed considerable), and two recent international initiatives to suggest a similar (1-digit position) solution to the perceived problem. Of the latter, one proposal (from the Head of the Economics Statistics department, Statistics Sweden and the Swedish National Agency for Higher Education) was made in November 2010, and the other (from European League of Institutes of the Arts) in January 2012. Both initiatives were addressed to the OECD Head of Economic Analysis and Statistics.

I don’t know if the two previous attempts were left unanswered, nor can I form a good idea as to how complicated it would be to realize a revision of the classification. The international scope of AR is lauded in SAR’s letter as being well networked an widely recognized, but how much of that will (or needs to) impress whatever entity decides over this, is a mystery to me at the moment. More interesting is the question of the validity of creating a new position for AR. I am all for recognizing AR as separate from musicology and artistic practice (see here), agreeing also in full that research on an artistic practice is not necessarily the same as in-and-through it (the latter can comprise the former, but goes one step further in catering conclusions to the needs of the practice). In my personal opinion and experience, it would also be good for AR to be valued as independent in order to safeguard some of the support that an emerging discipline needs. There’s hardly any money for AR (yet), and funding entities divide their financial pie into smaller slices rather than increasing the available resources as a whole to accommodate a new type of player. That there is already little cash to go around, is further mirrored in the facts that a call for an AR position attracts a significant number of interested musicologists, music-theorists or –philosophers, or that these scholars prefer a term such as ‘artist-researcher’ rather than one that identifies a new type that they feel (unjustly, I think) excludes them from participating. For the time being, I prefer to make clear distinctions, as attempts at appropriation (“AR is musicology with a new focus”, as I have heard argue on an AEC symposium a few years ago) will only leave dire chances for the young category to grow. A spot of its own in the Frascati Manual would certainly offer some hopes, if not guarantees, of being recognized, taken seriously and supported on government levels.

But, when reading in the manual (pp. 30-45) about distinctions on the borderline of what it considers to be at the core of the manual, i.e. Research & Development, and what it states should be excluded from R&D, then the matter might not be a small task. An example: “[…]even research by students at the PhD level carried out at universities should be counted, whenever possible, as a part of R&D[…]” (p. 31) [my italics] As it happens, doctoral AR is nearly all we have to show for, really. Fortunately, I am now only concentrating on music in this post – the visual arts have more weight to put on the scales. And for music, if all goes well, output streams can and will grow stronger, broader and faster.

On a conceptual level, however, tougher questions remain. As long as doubts are expressed as to whether or not artistic practice in itself is research, and artistic output be valid as research output, and as long as institutions consider a D.M.A. format (e.g. Portugal) for the doctoral cycle, or restrict doctoral candidates’ options to musicology departments (Germany), the communal AR efforts will stay diluted, and it may be hard to convince the OECD that we don’t fall under arts. And while I go along, to at least a certain extent, with the idea that AR can involve a type of knowledge of its own, I still see quite a few AR dissertations and articles being written that do not benefit from artistic practice, do not demonstrably have an impact on it, and do not bring forth new artistic knowledge.

In any case, I warmheartedly favor SAR’s proposition, and I look forward to hearing of any development in the matter – will keep you posted here.