miscellaneous factZ – The online home of Rufus Pollock

There’s an interesting 6 month fellowship at OPSI for work on economics of public sector information being funded by ESRC and National Archives. Deadline for applications is 6th August:

Valuing information: an economic analysis of public sector information and its re-use

Length of Fellowship: Six months

Proposed start date: Autumn 2009

Applications to be submitted as soon as possible (and by 6 August)

Location of Fellowship: The National Archives’ sites (Central London and Kew)

As part of its Placement Fellowship Scheme, the Economic and Social Research Council (ESRC) and The National Archives welcome applications from academic economists interested in working in a research capacity in the Office of Public Sector Information (OPSI). OPSI is part of The National Archives, a member of the Ministry of Justice family, working to set standards, deliver access and encourage the re-use of PSI.

The Placement Fellowship Scheme encourages social science researchers to spend time within a partner organisation to undertake policy relevant research and to develop the research skills of partner employees. The Fellowship will be jointly funded by the ESRC and OPSI while the Fellow remains employed by his or her institution.

See the document below for further details on the Placement Fellowship: http://www.nationalarchives.gov.uk/documents/esrc-placement-fellowship-june-09.pdf

Current methods of disseminating scholarly information focus on the use of journals who retain exclusive rights in the material they publish. Recently there has been increasing dissatisfaction with this model, with suggestions for alternative approaches such as “Open Access”.

Together with a colleague (Omar Al-Ubaydli) I’ve been working to explore the reasons for the development of the traditional journal model, why it is no longer efficient and how it could be improved upon. We’re particularly interested in going beyond the basic question of distribution (access) to that of filtering, i.e. the process of matching information with the scholars who want it.

With the volume of information production ever growing — and attention ever more scarce — filtering is becoming crucial. Digital technology offers us some radically new possibilities. In particular, distribution and filtering can be separated, in turn, allowing filtering to be decentralized and distributed — a model which promises dramatic increases in transparency, innovation and efficiency.

Below is an overview of our analysis with the full version of the current paper here: http://rufuspollock.org/economics/papers/scholars_and_journals.pdf

Overview

It is crucial to the progress of any domain of scholarship that those engaged therein are able to communicate their discoveries and activities to others. As such a variety of systems and institutions have been developed in order to support ’scholarly communication’ in one form or another ranging from personal letters to physical meetings. In recent times, the growth of scholarship, combined with its increasing geographical dispersion, have resulted in the centrality of the written word and its dissemination via ‘journals’. In this paper we consider the purposes of any system of scholarly communication and consider the current academic journal system in light of them. This examination highlights several deficiencies and also suggest various possible improvements.

When thinking about the possible mechanisms of scholarly communication it is useful to specify in more detail the criteria against which they should be measured. That is, to put it more succinctly, what do we want a good mechanism for scholarly communication to do? In particular, when we say communicate we must ask ourselves what, to whom, in what form, etc etc. For it is clear that when we talk of communication we usually mean more than the simple transmission of a piece of information. In fact, today, with so much scholarship available, the challenge may often not lie in the transmission from the author to the reader but in the matching of authors and readers — the decision of ‘what to read’. This growing focus on choice is a natural one in a world where time and attention are limited and the amount of scholarship available is ever increasing. As such it suggests that there are at least two distinct functions performed by a system of scholarly communication:

• Distribution — getting information from authors to readers (and back again)
• Selection (filtering) — deciding what to distribute and to whom

In appreciating this distinction it is illuminating to consider how practice has changed over time. Originally communication between scholars, at least in written form, primarily took the form of letters between the individuals involved. As such, the two activities of distribution and filtering would be almost completely identical. Then, as the number of authors and readers grew this became infeasible and dedicated journals would be created which would then disseminate to their particular readers a selection of what was submitted to them. Thus, what was once a direct peer-to-peer relationship became mediated by a new institutional form: the academic journal — though of course journals were often run by the very readers and authors who used them. Finally, today, thanks to digitization and the Internet peer-to-peer is once again a possibility though with important differences: unlike in the past, where a letter writer chooses the recipient, the modern peer-to-peer approach more resembles journals in that the author and reader act independently — the author uploads or publishes his/her work to a repository entirely separately from the reader finding, downloading and reading it. This last discussion suggests breaking down our original two categories a little further:

• ‘Making available’ — publishing material
• Discovery — finding out what is available
• Choice — choosing from what is available
• Reading — getting access to the material (in the form required)

Here, the first and fourth item would come under the ‘distribution’ heading while the second and third would come under ’selection’. In addition we should mention two other functions performed by such a system, both of which relate to selection: a) improvement of work via peer-review (distinct from filtering process itself); b) ‘quality signalling’ whereby the selection of work helps signal the quality of its creators which in turn is important for the purpose of resource allocation (jobs, grants etc) within the scholarly community.

With these added to the list we now have a good number of separate goals which a scholarly communication mechanism may seek to satisfy. The next stage is to consider how the current system, largely based on academic journals, fares in respect of them.

Goals, Instruments and the Current Journal System

It is well known that in order to fully address a given number of (independent) goals one needs an equal number of instruments. For example, if one is seeking to address both congestion and pollution in relation to road-traffic, a single instrument such as petrol taxes, will be insufficient.

Here too there are multiple independent goals, most notably distribution and selection (matching). These are clearly distinct goals and require distinct instruments for their achievement but journals are but a single instrument which combine distribution and filtering in one mechanism.

Originally, the restrictions of reproduction and distribution technologies, meant they were the best instrument available. Today, with the advent of the computer and the Internet, this is no longer true: distribution (the uploading and downloading) can be done by almost anyone and quite separately from recommendations and rating of that material.

As such, the traditional journal system is becoming a serious constraint, particularly in its closed access form. There are two distinct aspects of this constraint. First, on the distribution side, journals delay and restrict access as a result of higher prices arising either from simple monopoly control or the costs of the (inefficient) selection mechanism the traditional model necessitates. Second, on the selection side, the forced combination of selection and distribution and the associated monopoly control of content greatly limit the efficiency (and utility) of the selection and filtering processes used to match authors and readers together.

Unfortunately, the two-sided nature of the journal market (based on expectations), combined with the current evaluation structure of academia, continue to lock society into this inefficient restriction. Open-access journals provides are an important part of improving the current situation. However, as we discuss below, they are only a first step: in order to reap the full benefits of new technology we must move away from the traditional ‘journal’ model to a system that allow for full separation between the distribution and selection operations.

The Technological Origins of Modern Inefficiency

At this point it is worth considering in a little more detail why restricted-access journals originally came about. The answer lies in the nature of the technology available in earlier periods to manage distribution (printing and transmission). When many journals were originally started the cost of transmitting information was very high and journals acted as a club good by which the costs of reproduction and distribution could be (efficiently) shared (the efficiency arising here from economies of scale).

At the same time, given the limited ‘bandwidth’ it was natural for journals to take on some filtering role in order to economize on the scarce distribution capacity. In this situation, dissemination is limited and with only one instrument available (journals), it is natural to tie dissemination and filtering together (with filtering in many ways secondary). Once filtering is being done it is natural for journals to ‘tie’ material to the journal explicitly via copyright — though at an early stage given the scale economies of journals this explicit tying was not actually necessary and was probably done for simple legal convenience.

With the advent of digital communications, in particular the Internet, bandwidth is no longer scarce. What is now scarce is attention. In this setup the importance of a journal is not its role in efficiently sharing reproduction and distribution costs but its role as a filtering mechanism. However, there is now a problem: when distribution is central it is natural to ‘add-in’ filtering, it is not natural, or necessary, to tie distribution to filtering when filtering is central. In fact it seems clear that distribution and filtering can be done entirely separately (there are potentially lots of ways for you to download my paper quite separate from getting it from a journal — and lots of ways to do matching and filtering other than by journal editors and reviewers). The Open Access movement can be seen as largely about achieving this separation: with open access there is no longer a connection between access/distribution (which would be free) and the filtering mechanism (the choice of which articles go in a particular journal).

That said the ‘Open Access’ movement still has a large focus on journals — albeit open-access ones. This, in our view, is a mistake. Technology has also affected possibilities for filtering. In particular it is no longer clear why the centralized mechanism of official peer-review and journals is superior to alternative decentralized options. The last decade, has witnessed widespread, and often successful, experimentation with distributed voting and evaluation mechanisms (for example Slashdot’s story-ratings and Google’s link-based site rankings).

Thus, to be more radical, it makes sense not only to remove centralized control of distribution but also centralized control of filtering. A more distributed (market-like?) filtering mechanism would permit the same freedom (and same status) for reviewing and recommendation as it does in the production of scholarly information. At the same time it would deliver greater transparency and, by permitting ‘free-entry’ in filtering, would permit greater specialization, greater diversity, increased participation and the increasing efficiency flowing from greater competition.

As such, the gains from going ‘open’ are not simply wider access, but a reduction in the time and energy scholars spend finding and processing research information. Significantly, this second item, which is less frequently mentioned in discussions of ‘Open Access‘, may well be the most significant.

This follows up my previous post. Here we are going to calculation public domain numbers based directly on authorial birth/death date information rather than on guesstimated weightings. We’re going to focus on the Cambridge University Library (CUL) data we used previously.

Table 1: PD Relevant Information Availability

Table 1 presents a summary of how much relevant information is available for items (books) of particular vintages in the CUL catalogue — we only show data from 1870 to 1950 on the presumption that (almost) all pre-1870 publications are PD (their authors would have had to live for more than 70 years post-publication for this not to be the case) and almost all publications post 1950 are in copyright today (their authors would have to have died before 1940 for this not to be the case).

As the table shows, at best only just over 40% of items have a recorded authorial death date and extending to include birth dates only raises this proportion to, at best, the mid mid-to-low fifties. Taking account of items which lack any associated author, raises these figures somewhat further to around 60%, though we should note that the reason for the lack of an associated author is not clear — is it because they are genuinely anonymous or simply because the information has not been recorded? Thus, even for the earliest items listed a large proportion of items (50% or more) lack the necessary information for direct computation of public domain status.

At the same time, we can take some heart, and some interesting facts, from this table. First, a reasonable proportion, amounting to many thousands of items, did have associated death dates. Second, at least for older items, the majority of those with any date had a death date (95% for 1870-1880 and still at over 70% for 1920-1930). Third, and this is a more general observation, proportions were surprisingly constant over time. For example, the proportion of ‘anonymous’ items lies in a narrow band between 10% and 13% for the entire periods. Similarly the proportion of items with any date information ranged only from 45% to 56%. At the same time, and reassuringly, though the proportion with death dates is relatively constant for the oldest periods, in the more recent ones it falls substantially; as one would expect given that some of the authors from those more recent eras are still alive.

Table 2: PD Status by Decade. ‘?’ indicates items where PD status could not be computed. Prop(ortion) 1 equals total PD divided by total for which status could be computed (sum of total PD and Not PD). Prop(ortion) 2 equals total PD divided by number of items for which any author date was known (’Any Date’ in previous table).

Table 2 reports the results of direct computation of PD status based on the information available. Note that, in doing these computations, we have augmented the basic life plus 70 rule with the additional assumptions that a) all items published in 1870 or before are PD b) no author is older than 100 (so if a birth date is more 170 years ago the item is PD) c) every author lives at least until 30 (so that any work published by an author born less than a 100 years ago is automatically not PD).

As is to be expected, for the majority of the periods, the availability of PD status (either PD or Not PD) closely tracks the availability of death date information — the total for which PD status can be determined (the sum of PD and Not PD) almost exactly equals the total for which death date information is available. It is only in the last period 1940-1950 that the birth date appears to make any contribution. More interesting, is how the number PD and Not PD vary over time, especially relative to each other (and as a proportion of the records for which any date is available).

These two proportions/ratios are recorded in the last two columns which record, respectively: 1) the PD total relative to the number of items for which any status could be computed (i.e. the sum of PD and Not PD) 2) the PD total relative to the total number of items for which any date information is available. These ratios change dramatically over the periods shown: starting in the 1870-1880 period in the high 90%s by the 1940s they are down to 20% or below.

Table 3: Suggested PD Proportions

The key question for us is how to extrapolate these PD proportions to the full set of records — i.e. from the set of records for which there is the necessary birth/death date information to that where there is not. The simplest, and most obvious, approach is to assume that the proportions are identical and therefore that the PD proportions calculated on the partial dataset apply to the whole. However, there are some obvious deficiencies in this approach.

In particular, our ability to compute a PD status is largely linked to the existence of a death date and it is likely that the presence of this information is itself correlated with authorial age — after all a death date can only exist once that person has died! This correlation, and the bias it gives rise to, is probably small in the early periods — the authors of any pre 1930 work are almost certainly no longer alive today. However, for the later periods, the bias may be more substantial — it is in these last two periods (1930-1940 and 1940-1950) that there is a significant reduction in the number of records with a death date and (relatedly) a significant increase in the number of records for whom the PD status is unknown.

Thus, in converting the partial PD proportions to full PD proportions it seems sensible to revise down somewhat the partial figures with the revision being greater in later periods. Moreover, we have a lower bound for any downwards revision provided by the total PD as a proportion of all records — which even in the 1940-1950 period stood at 6%. In light of these considerations Table 3 gives fairly conservative figures for PD proportions that when estimating PD size based on publication dates. Interestingly, even with out conservative assumptions, these proportions are rather higher than those used in our previous analysis.

This post continues the work begun in this earlier post on “Estimating Information Production and the Size of the Public Domain”. Update: 2009-07-17 there is now a follow-up post.

Having already obtained estimates of the number of items (publications) produced each year based on library catalogue data our next step is to convert this into an estimate of the “size” of the public domain. (NB: as already discussed, “size” could mean several different things. Here, at least to start with, we’re going to take the simplest and crudest approach and equate size with number of publications/items.)

The natural, and most obvious, approach here is to go through our 1 million+ items and compute their public domain status (as discussed in this earlier post). Unfortunately, as detailed there, this is problematic because we often have insufficient information in library catalogues with which to compute PD status with certainty — in particular, author death dates are frequently absent. Thus, it will be necessary to fall back on some approximate method.

For example, we can use base PD status on simple publication dates: if a book was published, say, 140 years ago it is very likely it is in the public domain — for it to be in copyright its author must have lived more than 70 years after the book came out (remember copyright lasts for life plus 70 years in the EU)! Conversely, any publication less than 70 years old is almost certainly not in the public domain. For periods in between we can assume some proportion of publications are PD starting close to zero for more recent items and rising towards one for older ones. A calculation along those lines is provided in the following table:

Number of UK Public Domain Publications (Based on Cambridge University Library Catalogue Data)

So, based on the assumptions regarding PD proportions given in the table, there are somewhat over 600 thousand PD books according to the holdings of Cambridge University Library (of which just over half, approx 390k are from before 1870). The British Library dataset is approx 4x as big as Cambridge University Library and the numbers scale up roughly proportionately giving a total of over 2.4 million items.

Of course this is a fairly crude approach based purely on publication date and it be improved in a variety of ways, most notably by using the authorial birth date information which is usually present in catalogue data (we can also use death date information where present). This will be the subject of the next post. (2009-07-17 the post is up here).

Here we’re going to look at using library catalogue data as a source for estimating information production (over time) and the size of the public domain.

Library Catalogues

Cultural institutions, primarily libraries, have long compiled records of the material they hold in the form of catalogues. Furthermore, most countries have had one or more libraries (usually the national library) whose task included an archival component and, hence, whose collections should be relatively comprehensive, at least as regards published material.

The catalogues of those libraries then provide an invaluable resource for charting, in the form of publications, levels of information production over time (subject, of course, to the obvious caveats about coverage and the relationship of general “information production” to publications).

Furthermore, library catalogue entries record (almost) the right sort of information for computing public domain status, in particular a given record usually has a) a publication date b) unambiguously identified author(s) with birth date(s) (though unfortunately not death date). Thus, we can also use this catalogue data to estimate the size of the public domain — size being equated here to the total number of items currently in the public domain.

Results

To illustrate, here are some results based on the catalogue of Cambridge University Library which is one of the UK’s “copyright libraries” (i.e. they have a right to obtain, though not an obligation to hold, one copy of every book published in the UK). This first plot shows the numbers of publications per year (as determined by their publication date) up until 1960 (when the dataset ends) based on the publication date recorded in the catalogue.

A major concern when basing an analysis on these kinds of trends is is that fluctuations over time derive not from changes in underlying production and publication rates but changes in acquisition policies of the library concerned. To check for this, we present a second plot which shows the same information but derived from the British Library’s catalogue. Reassuringly, though there are differences, the basic patterns look remarkably similar.

Number of items (books etc) Per Year in the Cambridge University Library Catalogue (1600-1960).

Number of items (books etc) Per Year in the British Library Catalogue (1600-1960).

What do we learn from these graphs?

• In total there were over a million “Items” in this dataset (and parsing, cleaning, loading and analyzing this data took on the order of days — while the preparation work to develop and perfect these algorithms took weeks if not months)
• The main trend is a fairly consistent, and approximately exponential, increase in the number of publications (items) per year. At the start of our time period in 1600 we have around 400 items a year in the catalogue while by 1960 the number is over 16000.
• This is a forty-fold increase and corresponds to an annual growth rate of approx 0.8%. Assuming “growth” began only around the time of the industrial revolution (~ 1750) when output was around 1000 (10-year moving average) gives a fairly similar growth rate of around 0.89%.
• There are some fairly noticeable fluctuations around this basic trend:
  1. There appears to be a burst in publications in the decade or decade and a half before 1800. One can conjecture several, more or less intriguing, reasons for this: the cultural impact of the French revolution (esp. on radicalism), the effect of loosening copyright laws after Donaldson v. Beckett, etc. However, without substantial additional work, for example to examine the content of the publications in that period these must remain little more than conjectures.
  2. The two world wars appear dramatically in our dataset as sharp dips: the pre-1914 level of around 7k+ falls by over a third during the war to around 4.5k and then rises rapidly again to reach, and pass, 7k per year in the early 20s. Similarly, the late 1930s level of around 9.5k per year drops sharply upon the outbreak of war reaching a low of 5350 in 1942 (a drop of 45%), and then rebounding rapidly at the war’s end: from 5.9k in 1945 to 8k in 1946, 9k in 1947 and 11k in 1948!

To do next (but in separate entries — this post is already rather long!):

• Estimates for the the size of the public domain: how many of those catalogue items are in the public domain
• Distinguishing Publications (”Items”) from “Works” — i.e. production of new material versus the reissuance of old (see previous post for more on this).

Colophon: Background to this Research

I’m working on a EU funded project on the Public Domain in Europe, with particular focus on the size and value of the public domain. This involves getting large datasets about cultural material and trying to answer questions like: How many of these items are in the public domain? What’s the difference in price and availability of public domain versus non public domain items?

I’ve also been involved for several years in Public Domain Works, a project to create a database of works which were in the public domain.

Colophon: Data and Code

All the code used in parsing, loading and analysis is open and available from the Public Domain Works mercurial repository. Unfortunately, the library catalogue data is not: library catalogue data, at least in the UK, appears to be largely proprietary and the raw data kindly made available to us for the purposes of this research by the British Library and Cambridge University Library was provided only on a strictly confidential basis.

Last Thursday I attended a talk by Frederick Scherer at the [Judge] entitled: “Deregulatory Roots of the Current Financial Crisis”. Below are some sketchy notes.

Notes

Macro story:

• Huge current account deficit for last 10-15 years
  • Expansionary Fed policy has permitted this to happen while interest rates are low
• Median real income has not risen since the mid-1970s
  • Cheap money mean personal savings have dropped consistently: 1970s ~ 7%, 2000s ~ 1%
• Basically overconsumption

Micro story:

• Back in the old days, banking was very dull — three threes story, “One reason I never worked in the financial industry: it was very dull when I got my MBA in 1958″
• S&L story of 1980s: inflation squeeze + Reagan deregulation
  • FMs: Fannie Mae, Freddie Mac get more prominent
  • [Ed]: main focus here was on pressure for S&L to find better returns without much mention of the thoughtlessness of Reagan deregulatory approach (deposits still insured but S&L can now invest in anything) and the fraud and waste it engendered — see “Big Money Crime: Fraud and Politics in the Savings and Loan Crisis” by Kitty Calavita, Henry N. Pontell, and Robert Tillman
• In 1920s there were $2 billion of securitized mortgages (securitazation before the 1980s!)
• Market vs. bank finance for mortgages: market more than bank by mid-1980s [ed: I think -- graph hard to read]
• To start with: FMs pretty tough when giving mortgages, but with new securitizers and lots of cheap money, standards dropped => moral hazard for issuers [ed: not quite sure why this is moral hazard -- securitizers aren't the ones who should care, it's the buyers who should care]
• Even if issuers don’t care, buyers of securitized mortgages should care and they depended on ratings agencies (Moodys, S&P etc)
• Unfortunately, ratings agencies had serious conflicts of interest as they were paid to do ratings by firms issuing the securities! Result: ratings weren’t done well
• Worse: people ignored systemic risk in the housing market and therefore made far too low assessment of risk of these securities [ed: ignoring systemic risks implies underestimating correlations -- especially for negative changes -- between different mortgage types (geographic, owner-type etc). Interesting here to go back and read the quarterly statement from FM in summer 2008 which claims exactly this underestimate.]
• Banks over-leveraged for the classic reason (it raises your profits if things are good — but you can get wiped out if things are bad)
  • This made banks very profitable: by mid 2000s financial corporations accounted for 30% of all US corporate profits
  • Huge and (unjustified relative to other sectors) wage levels. Fascinating evidence here provide by correlating wage premia to deregulation: fig 6 from Philippson and Reshi shows dramatic association of wage premium (corrected for observable skills) with (de)regulation. Wage premium goes from ~1.6 in 1920s to <1.1 in 1960s and 70s and then back up to 1.6/1.7 in mid 2000s
• Credit default swaps and default insurance: not entirely new but doubled every year from 2001 to the present ($919 billion in 2001 to $62.2 trillion in 2007)
  • Much of the time CDS issued without any holding of the underlying asset
  • There was discussion on regulating CDSes in 1990s (blue-ribbon panel reported in 1998) but due to shenanigans in the house and senate led by Phil Graham (husband of Wendy Graham who was head of Commodity Futures … Board), CDSes were entirely deregulated via act tacked onto Health-Education-Appropriations bill in 2001.

It goes bad:

• Housing bubble breaks in 2007 or even 2006
  • Notices of default starts trending upwards in mid 2006
• [ran out of time]

What is to be done:

• Need simple, clear rules
  • A regulator cannot monitor everything day-to-day
• Outlaw Credit Default Swaps
• Anyone who issues CDOs must “keep skin in the game”
• Leverage ratios. Perhaps? Hard to regulate.
• Deal with too big to fail by making it hard for “giants to form” and breaking up existing over-large conglomerates
• We need to remember history!

Own Comments

This was an excellent presentation though, as was intended, it was more a summary of existing material than a presentation of anything “new”.

Not sure I was convinced by the “remember history” logic. It is always easy to be wise after the event and say “Oh look how similar this all was to 1929″. However, not only is this unconvincing analytically — it is really hard to fit trends in advance with any precision (every business cycle is different), but before the event there are always plenty of people (and lobbyists) arguing that everything is fine and we shouldn’t interfere. Summary: Awareness of history is all very well but it does not provide anything like the precision to support pre-emptive action. As such it is not really clear what “awareness of history” buys us.

More convincing to me (and one could argue this still has some “awareness of history in it) are actions like the following:

1. Worry about incentives in general and the principal-agent problem in particular. Try to ensure long-termism and prevent overly short-term and high-powered contracts (which essentially end up looking like an call option).

   Since incentives can be hard to regulate directly one may need to work via legislation that affects the general structure of the industry (e.g. Glass-Stegall).

   Summary: banking should be a reasonably dull profession with skill-adjusted wage rates similar to other sectors of the economy. If things get too exciting it is an indicator that incentives are out of line and things are likely to go wrong (quite apart from the inefficiency of having all those smart people pricing derivatives rather than doing something else!)

2. Be cautious regarding financial innovation especially where new products are complex. New products have little “track record” on which to base assessments of their benefits and risks and complexity makes this worse.

   In particular, complexity worsens the principal-agent problem for “regulators” both within and outside firms (how can I decide what bonus you deserve if I don’t understand the riskiness and payoff structure of the products you’ve sold?). Valuation of many financial products such as derivatives depend heavily — and subtly — on assumptions regarding the distribution of returns of underlying assets (stocks, bonds etc).

   If it is not clear what innovation — and complexity — are buying us we should steer clear, or at least be very cautious. As Scherer pointed out (in response to a question), there is little evidence that the explosion in variety and complexity of financial products since the 80s has actually done anything to make finance more efficient, e.g. by reducing the cost of capital to firms. Of course, it is very difficult to assess the benefits of innovation in any industry, let alone finance, but the basic point that 1940s through 1970s (dull banking) saw as much “growth” in the real economy as the 1980s-2000s (exciting banking) should make us think twice about how much complexity and innovation we need in financial products.

Finally, and on a more theoretical note, I’d also like to have seen more discussion about exactly why standard backward recursion/rational market logic fails here and what implications do the answers have for markets and their regulation. In particular, one would like to know doesn’t knowledge of a bubbles existence in period T lead to its unwinding (and hence by backward recursion to its unwinding in period T-1, and then T-2 etc until the bubble never existed). There are various answers to this in the literature based on things like herding, presence of noise investors, uncertainty about termination, but it would be good to have a summary, especially as regards welfare implications (are bubbles good?), and what policy interventions different theories prescribe.

The BBC ran a story yesterday headlined “Seven million ‘use illegal files’”. Its bolded first paragraph stated:

Around seven million people in the UK are involved in illegal downloads, costing the economy tens of billions of pounds, government advisers say. [emphasis added]

7 million people involved in unauthorised file-sharing is possible, but costs of tens of billions of pounds? It’s not unusual to see such figures bandied around by the rightsholders derived from wild guesstimates of download figures and ludicrously unsound assumptions such as equating every download with a lost sale.

Here, however, it is according to “government advisers” — surely a much more reliable source! A quick read and we discover this isn’t the case at all and these figures are directly recycled from rightsholder sources — with an additional uplift from the BBC: a possible £10 billion or more a year has becomes tens (notice that extra “s”) of billions a year.

First off, the story is based on a report entitled “Copycats? Digital Consumers in an Online Age” commissioned by the Strategic Advisory Board in Intellectual Property (SABIP) from UCL’s Centre for Information Behaviour and the Evaluation of Research. So this is CIBER’s report not SABIP’s — SABIP need not even have endorsed the report. That said, one can see how the BBC’s confusion came about, and this is a minor point (after all CIBER is part of a university).

More important is a check of the actual evidence underlying these very large claimed costs to the economy. Let’s take a look at the report. Page 6, at the start of the Exec Summary states (this is where I guess the BBC got its material from):

Industry reports [3] suggest that at least seven million British citizens have downloaded unauthorised content, many on a regular basis, and many also without ethical consideration. Estimates as to the overall lost revenues [4] if we include all creative industries whose products can be copied digitally, or counterfeited, reach £10 billion (IP Rights, 2004), conservatively, as our figure is from 2004, and a loss of 4,000 jobs. This is in the context of the “Creative Industries” providing around 8% of British GDP. And the situation is not solely a British problem, but a global one. …

But wait a moment: their only source here seems to be (IP Rights, 2004) and that turns out to be a single page press release from an IP (law) firm which simply states:

“Rights owners have estimated that last year alone counterfeiting and piracy cost the UK economy £10 billion and 4,000 jobs.”

So these are just the standard (and utterly unreliable) rightsholders-claimed figures (and not even first-hand!). To be fair in footnote 4 the authors acknowledge that the phrase “lost revenues” is complex and that not all downloaded content would have been purchased. However, they then seem to backtrack on this by saying (rightsholders provided figures again!):

Nevertheless, industries such as music and film do frequently publish estimated lost revenues, or “value gaps’. The BPI recently claimed that between 2008 and 2012 the music industry was looking at a ‘value gap’ of £1.2 billion. (Music Ally, 2008)

Furthermore, that claim that things are “complex” worries me, as things are, in fact, pretty simple: lost revenues mean lost revenues, i.e. the revenues the industry would have got if no unauthorised downloading had occurred. This will clearly be much, much lower than a figure based on assuming every unauthorised download is a lost sale.

Furthermore, looking at revenues in a single industry is dangerous here: we’ve got to look at the overall impact on the economy (and that’s still ignoring the welfare/income distinction). For example, if someone makes an unauthorised download rather than buying a CD they spend the money they would have spent on the CD on something else, be that a haircut, a meal, or going to a concert. If we want to count that as a loss to the music industry we need to count the gain it generates elsewhere.

Good evidence doesn’t get any thicker on the ground later on either as far as I can tell. For example, in the first key finding section (entitled “The scale of the ‘problem’ is huge and growing”):

• The only empirical study they cite on the impact of filesharing is that Zentner with no mention of some other major studies such as that of Oberholzer and Strumpf.
• The only figure on the film industry they quote is a claim of a $6 billion annual loss put forward by the UK film industry in interview and “some research (Henning-Thurau et al., 2007) [which] appears to demonstrate evidence that consumers’ intention to pirate movies “cause them to forego theatre visits and legal DVD rentals and/or purchases.”. Looking up that citation one finds (seems there was a typo in the date!): Henning-Thurau, T, Gwinner, K, Walsh, G, Gremler, D (2004) Electronic Word of Mouth via Consumer-Opinion Platforms: What Motivates Consumers to Articulate Themselves on the Internet? Journal of Interactive Marketing. 18 (1) pp.38-52. While I haven’t actually read this article, the title (and journal) don’t suggest this as the most reliable source as to the actual effect of unauthorised downloads on film industry income.

To sum up: it turns out the BBC’s line that illegal downloads are “costing the economy tens of billions of pounds” is based on nothing more than the usual, and completely unreliable, rightsholders claims, recycled via CIBER’s report. This is a worrying example of how industry PR, via repetition in other, more “respected” and supposedly independent sources, can gain legitimacy.

In my original post on Visualizing Technology Flows from Patent Data I just presented static information — flows for a single year. As I said there:

The next step is to watch how these flows, and the relationships implied by them, have evolved over time. We can do this by plotting the same graph say, every 3 years, from 1975 up until the present.

At the time I had already coded up, and computed, snapshots for each year. However, considerations of space, as well as a desire to find a way to display the information in a ‘nice’ (animated) form, warranted a separate entry. After what, as usual, has turned out to be a rather longer delay than intended, I’ve finally got round to having a first stab at this using simple animated gifs:

Animated Citation Flows 1975-1994 (1994 base year) (click through for full-size ~ 2MB). Click here to rerun the animation.

Here I’ve fixed the layout of the nodes based on the final year (1994) flows. I’ve also done quite a lot of tedious playing around (if only one had stylesheets!) with edge and node sizes to try and improve the look and they are still far from perfect (NB: this means edge/node sizes differ slightly from the images in the original post). As before:

• Size of nodes indicates total citation flows from that area in that year
• Yellow portion is citations back into that subcategory while black represents portion that is into other subcategories (comparison by area).
• Direction of flow is indicated by an arrow head (a rectangular block) with size of flow measured by width of edge and size of head.

Note that we are displaying year values not cumulative values — so, for example, links between nodes may get smaller or even disappear from one year to the next. What jumps out from this?

• The substantial increase in flows over time (most obviously seen in the size of the nodes).
• (At least based on examination by eye) no great change in the balance of these flows between cites outside and cites within a category (relative sizes of black and yellow in nodes).
• Growth has varied substantially across areas (largely, I would hazard, in line with the no. of patents in that area). In particular, the “Computer/Electronics” cluster (top-right) has grown substantially faster than the “Chemicals” sector at centre-left. Individual categories showing especially marked growth include: Biotechnology, Computer Hardware and Software, Communications, Information Storage, and Drugs.
• It also looks like some areas have grown more strongly linked and “clustered” over time (e.g. Computer/Electronics, and Drugs to Organic Compounds) though it is hard to tell from this visualization (pointing to the need for more formal techniques …).
• Something which is very clear from the visualization is that there is significant year-to-year variation with clear drops in flows in some cases year-on-year

I also computed another version where the network layout is based on that year’s flows — rather than with a fixed layout based on a given base year.

Unfortunately, this looks too “busy”, particularly as the sensitivity of the network layout algorithm (networkx.graphviz_layout) means that categories move around a lot. (To save on space — the files are big — I haven’t posted this up but if anyone is interested let me know and I’ll upload it).

One solution to this would be to move to rendering cumulative, rather than per-year, flows. This might also improve the base-year case: even there, it might be more natural, at least from a visual point of view, to display changes in flows over time via their impacts on “stocks” rather than displaying the “flows” themselves.

So, next steps:

• Plot cumulative flows
• Write up a more formal analysis based on e.g. PCA. I’ve already done PCAs on individual years and an animation might be interesting.
• Do animations right: the proper way to do this with would be with a proper “slider” widget and stop/start control. It looks like this should be pretty easy in javascript using e.g. jquery but it doesn’t look to be trivial — if it is please let me know how! (BTW: I know I could use Flash but it’s proprietary …).

I’m posting up an essay on “Discounting and Self-Control” (pdf). The essay, which I haven’t really touched for over a year, is still in its early stages but having lacked the time to do much on it over the last year, and going on the motto of “release early, release often”, I’m posting it up as a form of alpha version.

… then must you speak
Of one that loved not wisely, but too well;
Of one not easily jealous, but, being wrought,
Perplex’d in the extreme; of one whose hand,
Like the base Judean, threw a pearl away
Richer than all his tribe; …

– Othello, The Moor of Venice

Abstract

An agent’s intertemporal choices depend on a variety of factors, most prominently, their valuation of future payoffs as encapsulated in a discount function. However, it is also clear that factors such as self-control may also play an important role, and given the similarity of impact, a confouding one. We explore the literature on this issue as well as examining what occurs when those with higher time-preference (whether arising from discounting or self-control) also enjoy their consumption more.

Introduction

The exercise of will, especially in the form of self-control, has long been recognized as central to human existence, experience, and morality. Over the last few decades there has been increasing interest in the issue from a scientific perspective. At the same time, it has also long been appreciated that humans (and other animals) make trade-offs between the present and the future — as well as between different points in the future, and that events taking place closer to the present are given greater weight than those which are more distant. Traditionally, at least in economics, this type of behaviour has been subsumed under the heading of discounting.

Both of these factors, self-control and discounting, affect behaviour, and choices, in relation to outcomes which do not (all) take place in the present. However they are distinct. Specifically, consider a very simple case of two outcomes A and B where B occurs after A (for example, A might be one ice cream today and B an ice cream and a doughnut tomorrow). Self-control issues arise where one prefers B over A but is unable to execute on this preference and therefore actually takes (’chooses’) A. By contrast, in the discounting case A is actually preferred over B and therefore is chosen (freely) by the decision maker.

It would seem important to keep these two aspects of decision making clearly separated. While lack of ’self-control’ is usually seen as disadvantageous and a reason for adopting various ‘commitment strategies’ — for example, by opting to remove various items from the choice set (having no cigarettes in the house) — the simple preference for the present over the future incorporated in the discounting model would seem to generate no such difficulties.

However, empirically it may prove rather difficult to do so. As shown by the simple example above the same observed ‘choice’ for A (one ice cream today) over B (ice cream plus doughnut tomorrow) can be the result of two very different processes. Thus if we only observe choices, and not the underlying preferences and/or the process by which the choice is arrived at, it may be impossible to distinguish the two.

It is perhaps for this reason that these distinct aspects are sometimes conflated. Consider, for example, Mischel et al 1989 which is entitled “Delay of Gratification in Children” and summarizes much of Mischel of pioneering work on this area. Mischel’s approach is clearly more oriented along the self-control aspect, and this is borne out in the types of experiments conducted (more on this below). Nevertheless they state (p.934) “The obtained concurrent associations [between treatments and delay] are extensive, indicating that such preferences reflect a meaningful dimension of individual differences, and point to some of the many determinants and correlates of decisions to delay (18).” Here the orientation towards self-control has become a general “decision to delay” and this is borne out by the associated footnote (18) which references related literature in other disciplines and is worth quoting in its entirety:

[... see full essay for more]

Patricia Akester, a colleague of mine in the Centre for Intellectual Property and Information Law has just published the results of her recent research in the form of a 208 page report entitled Technological accommodation of conflicts between freedom of expression and DRM: the first empirical assessment.

There has been a lot of debate as to whether DRM/TPM can be used to go ‘beyond copyright’ and restrict legitimate uses of copyrighted material but little empirical work. Patricia’s work is therefore very valuable in providing the first systematic empirical data that we can use to assess what is going on. Here I’ll let her conclusions speak for herself but I strongly encourage readers to take a look at the study itself via the above link:

[From p. 99-100] This project looked at the impact of DRM on the ability of users to take advantage of certain exceptions to copyright. Based on a series of interviews with key organisations and individuals, involved in the use of copyright material and the development and deployment of DRM, this study examined how these issues are working out in practice. While the nightmarish vision of digital lock up has not materialised, this survey concluded, nevertheless , that significant problems do exist, and others can readily be foreseen:

1. Although DRM has not impacted on many acts permitted by law, certain permitted acts are being adversely affected by the use of DRM;
2. This is in spite of the existence of technological solutions (enabling partitioning and authentication of users. to accommodate those permitted acts (privileged exceptions.;
3. Beneficiaries of privileged exceptions who have been prevented from carrying out those permitted acts (because of the employment of DRM. have not used the complaints mechanism set out in UK law;
4. Article 6(4. of the Information Society Directive put an onus on content owners to accommodate privileged exceptions voluntarily. Voluntary measures have emerged in the publishing field, but not all content owners are ready to act unless they are told to do so by regulatory authorities.

These four conclusions will be explained in more detail and this will be followed by proposed solutions and recommendations.