Epilogue -- The Revolution Begins On Page Five: The Changing Nature Of The NIE And Its Implications For Intelligence

Part 1 -- Welcome To The Revolution
Part 2 -- Some History
Part 3 -- The Revolution Begins
Part 4 -- Page Five In Detail
Part 5 -- Enough Exposition, Let's Get Down To It...
Part 6 -- Digging Deeper
Part 7 -- Looking At The Fine Print
Part 8 -- Confidence Is Not the Only Issue
Part 9 -- Waffle Words And Intel-Speak
Part 10 -- The Problem With “If”
Part 11 -- One More Thing
Part 12 -- Final Thoughts

Epilogue

One of the reasons I decided to post this "article" as a series on this blog was to experiment with the blog format; to see how it might work (or not) with more academic style articles. The purpose of this post is to discuss what I have learned through this process.

First, though, I want to thank all of the people who took the time to comment on the blog or to drop me a line. The responses, even when they took exception to my findings, were overwhelmingly positive regarding this "experiment".

If this was an experiment, what, then, were the results? I will start by laying out some of the facts and then discuss what I think they mean in the context of academic publication and scholarship.

This blog, which has been around since just before Thanksgiving last year, currently gets about 1000-1500 unique visitors a week and about twice as many page views. As I understand it, "unique visitor" is a term of art that describes a "hit" from a single person and "page views" describes how many pages a unique visitor looks at before he or she departs the site. Confusingly, unique visitors can be the same person if they come back at different times. If, for example, someone hits the site, looks at 5 pages (clicks on 5 links to 5 posts), departs, comes back at a later date and looks at 3 pages, then the site has been hit by 2 unique users and has had 8 page views.

Clearly, given the nature of the series, the number of people who actually read any or all of the posts in the series is something well below the approximately 3000-4500 unique visitors the site had over the 2.5 weeks the series ran.

While there are all sorts of packages available, I have installed only the most basic analytics software on the site. This software allows me to know who is hitting the site and what they are reading in only the grossest possible sense. My own estimation, based on an extrapolation of the numbers I do have, is that the series had 1000-1500 unique visitors representing no fewer than 100 and no more than 300 real live people who read all or most of the posts in the series. It is likely that another 100-300 people read at least one part of the series over the last 2.5 weeks.

One of the main difficulties with Intelligence Studies as an academic discipline is that there are relatively few journals in it. Moreover, since the economics of journal publishing are driven more by subscriptions than the popularity of the articles in the journal, comparing the number of readers from journal to journal and from discipline to discipline is fraught with difficulty.

That said, I have tracked down two interesting numbers for comparison purposes. The first indicates that journal articles in the British Medical Journal average 1168 hits (not all of which will be unique visitors) per article in the week after publication. The second, from the British Library, indicates that the average number of readers per annum of an average journal article ranges from 500-1500 with an average of 900 (this presumably includes people who will read only part of the article).

I was, frankly, pretty surprised to see these comparison figures and it suggests that the way forward within the intelligence studies discipline is with online journal publishing. If I can get roughly the same number of hits (largely from professionals in the discipline) with this modest effort as the British Medical Journal reaches over generally the same time period and roughly the same number of readers in 2.5 weeks that the lesser read (but far more widely distributed) journal articles analyzed by the British Library get in a year, I think it indicates a high comfort level from people in the profession with the electronic distribution of scholarship.

Beyond the question of readership there is also the more important question of scholarship. This is much more difficult to get at by looking at the numbers, however. On the face of it, the series falls well within normal limits for journal articles. Putting all the pieces together adds up to about 9000 total words which would, with charts, graphs and bibliographies add up to a respectably substantial journal article (about 30-40 pages depending on the journal). This is far longer than mathematics articles (which average about 12 pages) and far shorter than law review articles (which have recently begun to impose page limits in order to bring down the number of pages to 70 or so...). A quick review of page lengths in Intelligence and National Security suggest that 30-40 pages is within the range of "normal" for that publication.

Likewise the methodology and collection of the data were well within the norm for academic articles. I took a discrete but logically connected subset of National Intelligence Estimates and analyzed the way in which they were written, looking for patterns that emerged from the data analysis. The results are easily open to verification -- anyone else can do the same thing I did -- and the documents analyzed were all primary sources. I also tried to indicate where I saw weaknesses in my method and why I thought I could still make the evaluation I was making.

I also tried to be suitably "academic" in the tone of the article. I think I laregly succeeded while noting an occasional descent into "blog-speak". Many of the readers of this series of posts are students at Mercyhurst College and I wanted to make the series as interesting as possible. Having read many academic articles over the years, I also recognize that the pure academic tone is certainly not required...

Citing sources proved particularly easy with this form of publication. My intent was to turn my endnotes into hyperlinks within the posts. The paper contains 75 hyperlinks to sources outside of the document. In a print journal, all of these would have to be endnotes. I am not entirely satisfied with this method for sourcing, however. For one thing, the hyperlink cannot take the reader to the exact place in a lengthy document to which the post refers. I think if I do such an experiment again (and I think I am likely to), I will include the page number in a parenthetical immediately after the hyperlink. While this article was fairly easy to write without reference to non-web-based sources, I expect that this will not always be the case. I had planned to just include an endnote and put it at the bottom of each post if I had to refer to something that was not on the web but, thanks to the DNI and the CIA Center for the Study of Intelligence, all of the major references were readily available.

The fundamental element of traditional academic scholarship that was missing from the process was peer review prior to publication. I would make three comments here. First, the problems with the peer review system are well known to any academic. Some have gone as far as to claim that it is less a system for determining quality than it is a system for enforcing acceptability. Second, the notion of what constitutes adequate peer review is changing dramatically with any number of experiments on-going. Finally, the ability to comment that is provided by blogging technology changes and adds both depth and nuance to traditional notions of peer review.

Consider the traditional process. An article goes in and it is assigned to various referees who make independent and anonymous reviews of the work prior to publication. The readers rarely get any insight into reviewer comments or questions. Comments from the readers likewise have to go back to the editors and may or may not show up in a later issue of the journal. The best indication of the quality of an article is likely the number of times it is cited in other works -- something that is not known for years after the article is published.

With blogging technology, the peer review process becomes an integral part of the writing process. It happens simultaneously, in more or less real time. A variety of different metrics (including the ones discussed above) are more or less immediately available.

All in all, the process of writing this article and posting it as a series on a blog was extremely gratifying. I enjoyed the research, writing and review processes much more than I do normally.
Again, thank you all for all of your comments. It was genuinely appreciated.