miscellaneous factZ – The online home of Rufus Pollock

From Laslett ‘Phillipe Ariès and “La Famille”‘ p.83 (quoted in Eisenstein, p.131):

The actual reality, the tangible quality of community life in earlier towns or villages … is puzzling … and only too susceptible to sentimentalisation. People seem to want to believe that there was a time when every one belonged to an active, supportive local society, providing a palpable framework for everyday life. But we find that the phenomenon itself and its passing — if that is what, in fact happened– perpetually elude our grasp.

Continues the series of post related to analyzing catalogue data, here are some stats on author “significance” as measured by the number of book entries (’items’) for that author in the Cambridge University Library catalogue from 1400-1960 (there being 1m+ such entries).

I’ve termed this measure “significance” (with intentional quotes) as it co-mingles a variety of factors:

• Prolificness — how many distinct works an author produced (since usually each work will get an item)
• Popularity — this influences how many times the same work gets reissued as a new ‘item’ and the library decision to keep the item
• Merit — as for popularity

The following table shows the top 50 authors by “significance”. Some of the authors aren’t real people but entities such as “Great Britain. Parliament” and for our purposes can be ignored. What’s most striking to me is how closely the listing correlates with the standard literary canon. Other features of note:

• Shakespeare is number 1 (2)
• Classics (latin/greek) authors are well-represented with Cicero at number 2 (4), Horace at 5 (9) followed Homer, Euripides, Ovid, Plato, Aeschylus, Xenophon, Sophocles, Aristophanes and Euclid.
• Surprise entries (from a contemporary perspective): Hannah More, Oliver Goldsmith, Gilbert Burnet (perhaps accounted by his prolificity).
• Also surprising is limited entries from 19th century UK with only Scott (26), Dickens (28) and Byron (41)

Top 50 authors based on CUL Catalogue 1400-1960

The other thing we could look at is the overall distribution of titles per author (and how it varies with rank — a classic “is it a power law” question). Below are the histogram (NB log scale for counts) together with a plot of rank against count (which equates, v. crudely, to a transposed plot of the tail of the histogram …). In both cases it looks (!) like a power-law is a reasonable fit given the (approximate) linearity but this should be backed up with a proper K-S test.

Histogram of items-per-author distribution (log-log)

Rank versus no. of items (log-log)

TODO

• K-S tests
• Extend data to present day
• Check against other catalogue data
• Look at occurrence of people in title names
• Look at when items appear over time

Colophon

Code to generate table and graphs in the open Public Domain Works repository, specifically method ‘person_work_and_item_counts’ in this file: http://knowledgeforge.net/pdw/hg/file/tip/contrib/stats.py

This follows up my previous post. Here we are going to calculation public domain numbers based directly on authorial birth/death date information rather than on guesstimated weightings. We’re going to focus on the Cambridge University Library (CUL) data we used previously.

Table 1: PD Relevant Information Availability

Table 1 presents a summary of how much relevant information is available for items (books) of particular vintages in the CUL catalogue — we only show data from 1870 to 1950 on the presumption that (almost) all pre-1870 publications are PD (their authors would have had to live for more than 70 years post-publication for this not to be the case) and almost all publications post 1950 are in copyright today (their authors would have to have died before 1940 for this not to be the case).

As the table shows, at best only just over 40% of items have a recorded authorial death date and extending to include birth dates only raises this proportion to, at best, the mid mid-to-low fifties. Taking account of items which lack any associated author, raises these figures somewhat further to around 60%, though we should note that the reason for the lack of an associated author is not clear — is it because they are genuinely anonymous or simply because the information has not been recorded? Thus, even for the earliest items listed a large proportion of items (50% or more) lack the necessary information for direct computation of public domain status.

At the same time, we can take some heart, and some interesting facts, from this table. First, a reasonable proportion, amounting to many thousands of items, did have associated death dates. Second, at least for older items, the majority of those with any date had a death date (95% for 1870-1880 and still at over 70% for 1920-1930). Third, and this is a more general observation, proportions were surprisingly constant over time. For example, the proportion of ‘anonymous’ items lies in a narrow band between 10% and 13% for the entire periods. Similarly the proportion of items with any date information ranged only from 45% to 56%. At the same time, and reassuringly, though the proportion with death dates is relatively constant for the oldest periods, in the more recent ones it falls substantially; as one would expect given that some of the authors from those more recent eras are still alive.

Table 2: PD Status by Decade. ‘?’ indicates items where PD status could not be computed. Prop(ortion) 1 equals total PD divided by total for which status could be computed (sum of total PD and Not PD). Prop(ortion) 2 equals total PD divided by number of items for which any author date was known (’Any Date’ in previous table).

Table 2 reports the results of direct computation of PD status based on the information available. Note that, in doing these computations, we have augmented the basic life plus 70 rule with the additional assumptions that a) all items published in 1870 or before are PD b) no author is older than 100 (so if a birth date is more 170 years ago the item is PD) c) every author lives at least until 30 (so that any work published by an author born less than a 100 years ago is automatically not PD).

As is to be expected, for the majority of the periods, the availability of PD status (either PD or Not PD) closely tracks the availability of death date information — the total for which PD status can be determined (the sum of PD and Not PD) almost exactly equals the total for which death date information is available. It is only in the last period 1940-1950 that the birth date appears to make any contribution. More interesting, is how the number PD and Not PD vary over time, especially relative to each other (and as a proportion of the records for which any date is available).

These two proportions/ratios are recorded in the last two columns which record, respectively: 1) the PD total relative to the number of items for which any status could be computed (i.e. the sum of PD and Not PD) 2) the PD total relative to the total number of items for which any date information is available. These ratios change dramatically over the periods shown: starting in the 1870-1880 period in the high 90%s by the 1940s they are down to 20% or below.

Table 3: Suggested PD Proportions

The key question for us is how to extrapolate these PD proportions to the full set of records — i.e. from the set of records for which there is the necessary birth/death date information to that where there is not. The simplest, and most obvious, approach is to assume that the proportions are identical and therefore that the PD proportions calculated on the partial dataset apply to the whole. However, there are some obvious deficiencies in this approach.

In particular, our ability to compute a PD status is largely linked to the existence of a death date and it is likely that the presence of this information is itself correlated with authorial age — after all a death date can only exist once that person has died! This correlation, and the bias it gives rise to, is probably small in the early periods — the authors of any pre 1930 work are almost certainly no longer alive today. However, for the later periods, the bias may be more substantial — it is in these last two periods (1930-1940 and 1940-1950) that there is a significant reduction in the number of records with a death date and (relatedly) a significant increase in the number of records for whom the PD status is unknown.

Thus, in converting the partial PD proportions to full PD proportions it seems sensible to revise down somewhat the partial figures with the revision being greater in later periods. Moreover, we have a lower bound for any downwards revision provided by the total PD as a proportion of all records — which even in the 1940-1950 period stood at 6%. In light of these considerations Table 3 gives fairly conservative figures for PD proportions that when estimating PD size based on publication dates. Interestingly, even with out conservative assumptions, these proportions are rather higher than those used in our previous analysis.

Here we’re going to look at using library catalogue data as a source for estimating information production (over time) and the size of the public domain.

Library Catalogues

Cultural institutions, primarily libraries, have long compiled records of the material they hold in the form of catalogues. Furthermore, most countries have had one or more libraries (usually the national library) whose task included an archival component and, hence, whose collections should be relatively comprehensive, at least as regards published material.

The catalogues of those libraries then provide an invaluable resource for charting, in the form of publications, levels of information production over time (subject, of course, to the obvious caveats about coverage and the relationship of general “information production” to publications).

Furthermore, library catalogue entries record (almost) the right sort of information for computing public domain status, in particular a given record usually has a) a publication date b) unambiguously identified author(s) with birth date(s) (though unfortunately not death date). Thus, we can also use this catalogue data to estimate the size of the public domain — size being equated here to the total number of items currently in the public domain.

Results

To illustrate, here are some results based on the catalogue of Cambridge University Library which is one of the UK’s “copyright libraries” (i.e. they have a right to obtain, though not an obligation to hold, one copy of every book published in the UK). This first plot shows the numbers of publications per year (as determined by their publication date) up until 1960 (when the dataset ends) based on the publication date recorded in the catalogue.

A major concern when basing an analysis on these kinds of trends is is that fluctuations over time derive not from changes in underlying production and publication rates but changes in acquisition policies of the library concerned. To check for this, we present a second plot which shows the same information but derived from the British Library’s catalogue. Reassuringly, though there are differences, the basic patterns look remarkably similar.

Number of items (books etc) Per Year in the Cambridge University Library Catalogue (1600-1960).

Number of items (books etc) Per Year in the British Library Catalogue (1600-1960).

What do we learn from these graphs?

• In total there were over a million “Items” in this dataset (and parsing, cleaning, loading and analyzing this data took on the order of days — while the preparation work to develop and perfect these algorithms took weeks if not months)
• The main trend is a fairly consistent, and approximately exponential, increase in the number of publications (items) per year. At the start of our time period in 1600 we have around 400 items a year in the catalogue while by 1960 the number is over 16000.
• This is a forty-fold increase and corresponds to an annual growth rate of approx 0.8%. Assuming “growth” began only around the time of the industrial revolution (~ 1750) when output was around 1000 (10-year moving average) gives a fairly similar growth rate of around 0.89%.
• There are some fairly noticeable fluctuations around this basic trend:
  1. There appears to be a burst in publications in the decade or decade and a half before 1800. One can conjecture several, more or less intriguing, reasons for this: the cultural impact of the French revolution (esp. on radicalism), the effect of loosening copyright laws after Donaldson v. Beckett, etc. However, without substantial additional work, for example to examine the content of the publications in that period these must remain little more than conjectures.
  2. The two world wars appear dramatically in our dataset as sharp dips: the pre-1914 level of around 7k+ falls by over a third during the war to around 4.5k and then rises rapidly again to reach, and pass, 7k per year in the early 20s. Similarly, the late 1930s level of around 9.5k per year drops sharply upon the outbreak of war reaching a low of 5350 in 1942 (a drop of 45%), and then rebounding rapidly at the war’s end: from 5.9k in 1945 to 8k in 1946, 9k in 1947 and 11k in 1948!

To do next (but in separate entries — this post is already rather long!):

• Estimates for the the size of the public domain: how many of those catalogue items are in the public domain
• Distinguishing Publications (”Items”) from “Works” — i.e. production of new material versus the reissuance of old (see previous post for more on this).

Colophon: Background to this Research

I’m working on a EU funded project on the Public Domain in Europe, with particular focus on the size and value of the public domain. This involves getting large datasets about cultural material and trying to answer questions like: How many of these items are in the public domain? What’s the difference in price and availability of public domain versus non public domain items?

I’ve also been involved for several years in Public Domain Works, a project to create a database of works which were in the public domain.

Colophon: Data and Code

All the code used in parsing, loading and analysis is open and available from the Public Domain Works mercurial repository. Unfortunately, the library catalogue data is not: library catalogue data, at least in the UK, appears to be largely proprietary and the raw data kindly made available to us for the purposes of this research by the British Library and Cambridge University Library was provided only on a strictly confidential basis.

The BBC ran a story yesterday headlined “Seven million ‘use illegal files’”. Its bolded first paragraph stated:

Around seven million people in the UK are involved in illegal downloads, costing the economy tens of billions of pounds, government advisers say. [emphasis added]

7 million people involved in unauthorised file-sharing is possible, but costs of tens of billions of pounds? It’s not unusual to see such figures bandied around by the rightsholders derived from wild guesstimates of download figures and ludicrously unsound assumptions such as equating every download with a lost sale.

Here, however, it is according to “government advisers” — surely a much more reliable source! A quick read and we discover this isn’t the case at all and these figures are directly recycled from rightsholder sources — with an additional uplift from the BBC: a possible £10 billion or more a year has becomes tens (notice that extra “s”) of billions a year.

First off, the story is based on a report entitled “Copycats? Digital Consumers in an Online Age” commissioned by the Strategic Advisory Board in Intellectual Property (SABIP) from UCL’s Centre for Information Behaviour and the Evaluation of Research. So this is CIBER’s report not SABIP’s — SABIP need not even have endorsed the report. That said, one can see how the BBC’s confusion came about, and this is a minor point (after all CIBER is part of a university).

More important is a check of the actual evidence underlying these very large claimed costs to the economy. Let’s take a look at the report. Page 6, at the start of the Exec Summary states (this is where I guess the BBC got its material from):

Industry reports [3] suggest that at least seven million British citizens have downloaded unauthorised content, many on a regular basis, and many also without ethical consideration. Estimates as to the overall lost revenues [4] if we include all creative industries whose products can be copied digitally, or counterfeited, reach £10 billion (IP Rights, 2004), conservatively, as our figure is from 2004, and a loss of 4,000 jobs. This is in the context of the “Creative Industries” providing around 8% of British GDP. And the situation is not solely a British problem, but a global one. …

But wait a moment: their only source here seems to be (IP Rights, 2004) and that turns out to be a single page press release from an IP (law) firm which simply states:

“Rights owners have estimated that last year alone counterfeiting and piracy cost the UK economy £10 billion and 4,000 jobs.”

So these are just the standard (and utterly unreliable) rightsholders-claimed figures (and not even first-hand!). To be fair in footnote 4 the authors acknowledge that the phrase “lost revenues” is complex and that not all downloaded content would have been purchased. However, they then seem to backtrack on this by saying (rightsholders provided figures again!):

Nevertheless, industries such as music and film do frequently publish estimated lost revenues, or “value gaps’. The BPI recently claimed that between 2008 and 2012 the music industry was looking at a ‘value gap’ of £1.2 billion. (Music Ally, 2008)

Furthermore, that claim that things are “complex” worries me, as things are, in fact, pretty simple: lost revenues mean lost revenues, i.e. the revenues the industry would have got if no unauthorised downloading had occurred. This will clearly be much, much lower than a figure based on assuming every unauthorised download is a lost sale.

Furthermore, looking at revenues in a single industry is dangerous here: we’ve got to look at the overall impact on the economy (and that’s still ignoring the welfare/income distinction). For example, if someone makes an unauthorised download rather than buying a CD they spend the money they would have spent on the CD on something else, be that a haircut, a meal, or going to a concert. If we want to count that as a loss to the music industry we need to count the gain it generates elsewhere.

Good evidence doesn’t get any thicker on the ground later on either as far as I can tell. For example, in the first key finding section (entitled “The scale of the ‘problem’ is huge and growing”):

• The only empirical study they cite on the impact of filesharing is that Zentner with no mention of some other major studies such as that of Oberholzer and Strumpf.
• The only figure on the film industry they quote is a claim of a $6 billion annual loss put forward by the UK film industry in interview and “some research (Henning-Thurau et al., 2007) [which] appears to demonstrate evidence that consumers’ intention to pirate movies “cause them to forego theatre visits and legal DVD rentals and/or purchases.”. Looking up that citation one finds (seems there was a typo in the date!): Henning-Thurau, T, Gwinner, K, Walsh, G, Gremler, D (2004) Electronic Word of Mouth via Consumer-Opinion Platforms: What Motivates Consumers to Articulate Themselves on the Internet? Journal of Interactive Marketing. 18 (1) pp.38-52. While I haven’t actually read this article, the title (and journal) don’t suggest this as the most reliable source as to the actual effect of unauthorised downloads on film industry income.

To sum up: it turns out the BBC’s line that illegal downloads are “costing the economy tens of billions of pounds” is based on nothing more than the usual, and completely unreliable, rightsholders claims, recycled via CIBER’s report. This is a worrying example of how industry PR, via repetition in other, more “respected” and supposedly independent sources, can gain legitimacy.

Tomorrow, the European Parliament will vote on the issue of copyright term extension for sound recordings, known in Parliamentese as “the Crowley Report (A6-0070/2009) on the Term of protection of copyright and related rights” (Mr Brian Crowley is the rapporteur for this report and a strong supporter of the extension).

Extending term would be a tragic mistake and a blatant example of special-interest lobbying winning out of the interests of society as a whole.

Let us therefore hope that the proposal is rejected.

That’s the line being by some right-thinking MEPs including Eva Lichtenberger, Greens, Sharon Bowles, ALDE, Andrew Duff, ALDE, Zuzana Roithova, EPP, Christofer Fjellner, EPP, Guy Bono, PSE who have put forward a rejection amendment (see their excellent justification below). But they need all the support they can get and remember: it is never too late to act.

Rejection Amendment Justification

The draft Directive is poorly conceived and disproportionate. The Commission claims that the measure is needed in order to benefit poor performers. However, the proposed regulation and procedure is complicated and over-bureaucratic. The biggest beneficiaries will be the four largest record companies. Individual performers will only receive very small amounts each.

Performers could be helped much more effectively by regulating copyright contracts and collecting societies, by setting up appropriate social security and insurance schemes, and by reconsidering remuneration rights and license tariffs.

The draft Directive leaves a large number of questions unanswered. Additional impact assessments are needed to see which measures are best suited to help those performers really in need, to limit the negative impact on consumers and jobs, and to establish if regulation is best done at state or EU level. In these circumstances, it is not wise to proceed to make the long-term permanent changes proposed.

Some of the particular problems are:

The extension of copyright to 95 or even 70 years will increase the revenue of trust funds of deceased performers instead of living performers.

Many performers cannot produce proof for the performances they participated in during the past decades. It then becomes difficult to assess their rights to payments.

The proposed regulation could cause legal uncertainty for all existing audiovisual productions as it will be unclear if the material used is subject to sound copyright.

There is a risk that all material that is not commercially viable will not be marketed by the copyright owners and will become inaccessible for public use.

Small record companies currently publishing copyright-free material risk going bankrupt.

7/10. Cannibalism and the Common Law: The Story of the Tragic Last Voyage of the Mignonette and the Strange Legal Proceedings to Which It Gave Rise by A Simpson, University of Chicago Press, 1984. More history than legal analysis. Interesting throughout but meandering slightly towards the end. One quote I wish to memorialize, which though rather apart from the main thrust of my book, made me wonder once again about the general tension between ‘definiteness’ (assertiveness/simplicity) and ‘correctness’, especially in the arena of public policy and democratic politics. Is it always necessary, as the quote suggests, for successful campaigns to simplify and exaggerate in order to obtain an effect?

In the period immediately before the case of the Mignonette [1884], controversies over the protection of sailors and passengers had been inflamed by the activities of the radical MP for Derby (1868-80), Samuel Plimsoll, ‘the sailor’s friend’, whose approach to the problem favoured prior intervention [i.e. regulation] … He [Plimsoll] concentrated first simply on unseaworth ships as a cause fo mortality and started a campaign to amend the law with a resolution in the House of Commons in July 1870. His most effective appeal was to public opinion through the publication of Our Seamen in 1872, attacking the ship-owners of the over-insured, overloaded “coffin ships”, which caught the public imagination. Plimsoll was no doubt careless with his facts, ill-informed, and sometimes violent in his language; but perhaps successful campaigns require devils, conspiracies and simple solutions. [emph added] In reality ships were lost for a variety of reasons, and unseaworthiness was only one of them.

From, Sexuality: A Biopsychosocial Approach by Chess Denman, p. 54:

Politicians and the press have created an image of a tidal wave of teen parenthood, caused by young women’s unregulated sexual behaviour and poor women sponging off the state, even though this is unwarranted. In America, for example, teen motherhood cannot be said to have grown as a consequence of welfare because the value of welfare has reduced (Schwartz and Rutter 1998). Interviews with teens who are pregnant do not indicate the kind of planning and forethought necessary for their pregnancy to be a thought-out monetary strategy. Indeed, being able to see a future for oneself is actually associated with abstaining from sex or using contraception (Pipher 1994, in Schwartz and Rutter 1998). In fact, teen pregnancy themselves have not increased at all. Instead they have declined along with the general decline in pregnancy rates but, because they have not declined as much as pregnancy rates in other age ranges, they form a rising proportion of the figures.

However, this may not be all the story, as shown by the following quote taken from the this article on Teenage pregnancy on the UK’s Department for Education and Skills website:

In the 1970s, Britain had similar teenage pregnancy rates to the rest of Europe. But while other countries got theirs down in the 1980s and 1990s, Britain’s rate stayed high. The latest available figures show that Britain’s teenage birth rate is five times that in Holland, three times higher than in France and double the rate in Germany. Other English-speaking countries such as Canada and New Zealand have teenage birth rates higher than ours. In the United States the rate is more than double that in the UK.

In 1999 the Government published a Teenage Pregnancy Report from its Social Exclusion Unit. It acknowledged there was no single cause, but pointed out three major factors: first, that many young people think they will end up on benefit anyway so they see no reason not to get pregnant. Second, that teenagers don’t know enough about contraception and about what becoming a parent will involve. Third, that young people are bombarded with sexual images in the media but feel they can’t talk about sex to their parents and teachers. [emphasis added]

From Robert E. Lane’s essay, The Joyless Market Economy p. 484:

Durkheim asks: “Even from a purely utilitarian point of view, what is the use of increasing abundance, if it does not succeed in calming the desires of the greatest number, but, on the contrary, only serves to increase their impatience. [emphasis added] It is forgotten that economic functions are not their own justification. … Society has no raison d’etre if it does not bring men a little peace, peace in their hearts and peace in their mutual relations”[^63] Deprived of its original utilitarian raison d’etre, does the market society now reflect something like Kroeber’s exhaustion of a cultural configuration in which the old, material civilization has exhausted the possibilities of that particular pattern? The offerings of the market no longer satisfy, not because the payoff is not large enough but because it is denominated in the wrong currency.

[^63]: Durkheim’s quote is cited as: Emile Durkheim, Professional Ethics and Civic Morals. Trans. C. Brookfield. Routledge and Kegan Paul (1957) p.16 (via Steven Lukes, Emile Durkheim: His Life and Work, Allen Lane/Penguin (1973) p. 267).

8/10. I have just finished the final volume of Taylor Branch’s monumental trilogy America in the King Years. A fitting end to an incredible effort — though to my mind the first volume remains the best. This work covers the more difficult years faced by King following the major successes of the Civil Rights movement culminating in the Selma March and the passage of the1965 Voting Rights Act (which occurs at the start of this volume). Venturing in to the more rocky waters of poverty and the Vietnam War, and caught between the increasing radicalisation and its corresponding conservative backlash, King stuck tenaciously to his non-violent principles only to be shot down outside his Memphis motel room on April 4 1968. It is perhaps for the very reason that these years were more troubled, with success more elusive and direction less sure, that it is this book which most increased for admiration for King as a man. Though contending with endless difficulties, self-doubt, persecution and perpetual internal dissension he struggled endlessly to retain his humility, his conscience and his commitment to nonviolent discipline. And it is his very humanity, evidenced, for example, by his incessant extra-marital affairs, that compels us to see in him some kind of latter-day saint.