PAWDOC – Use of Information

The practice of sidelining text in articles, papers, and books is not uncommon and is something I started doing in the late 1970s – primarily to assist the writing of technical books when I was working at the National Computing Centre. I started to use my PAWDOC filing system in 1981, so, by the late 80s I was aware that a) there were a lot of documents in my filing system that I hadn’t looked at for several years, and b) inside all these documents were a large number of sidelined significant points. I started to think about these points as nuggets of information, some of which perhaps were the bedrock of my developing ideas, but others of which perhaps I had simply forgotten. I wondered also if an explicit examination of all these nuggets might prompt inter-relationships to be identified and new ideas to be developed. I considered this to be a potentially valuable spin-off from all the effort expended in operating a very comprehensive personal filing system. The notion of actually making use of the information that PAWDOC contained instead of having most of it just lie there statically, was very attractive.

Consequently, I looked for some simple and inexpensive tools to try out these ideas, and came across Mind Mapping software on the free disks issued with PC magazines in late 1990s. I started experimenting with one of them but found it was too bitty just pulling nuggets out of the odd paper – I felt I needed a whole set of material to work with. So I created Mind Maps for 19 esoteric books (on subjects such as The Great Pyramid), but found it too difficult to inter-relate the different Mind Maps. That’s as far as I got.

I’m still not sure if there is any merit in explicitly managing nuggets to either just cement them in one’s mind, or to inter-relate them and develop new ideas; and, I must admit, I’ve never done any serious book review to see what other work, if any, has been done on this subject. The retrospective work that Peter Tolmie is planning to do with me may throw a little light on the impact that such nuggets may have had on me – but that’s as far as it goes. My current view is that explicitly managing nuggets is probably not worth doing, and that adding extra tasks to the job of managing a personal filing system may well be the stone that breaks the camel’s back.

Specific questions relating to this aspect are answered below. Note that the status of each answer will fall into one of the following 5 categories: Not Started, Ideas Formed, Experience Gained, Partially Answered, Fully Answered.

Q27. How can an electronic filing system be used to develop and use knowledge?

2001 Answer: Ideas formed:

  • Include substantive information in the index entries, for example phone numbers, book references, and expense claim amounts.
  • Identify the nuggets of information (i.e. the valuable bits) when you first read a document (Wilson 1997: 3 – 4).
  • Capture and structure the nuggets into the overall nugget-base at the same time as indexing the item (Wilson 1997: 3 – 4).

2019 Answer: Ideas formed: The first point in the 2001 answer to this question (‘include substantive information in the index entries’) has proved useful: so much so that I started including substantive information in the file names of digital documents (for example, total amounts claimed in the file names of expense claim spreadsheets).

With respect to nuggets, those ideas emerged from my practice of sidelining text that I thought significant, I developed the idea that these pieces of information (which I called nuggets) might be picked out, recorded and combined with other nuggets to produce novel ideas and concepts. I explored technology options that might assist in this process and decided Mind Mapping software might be worth trying, and tried it out with a variety of esoteric books that I was reading at the time. Although I Mind Mapped 19 books I never took it to the next stage of combining them to see if I could develop any new concepts – there seemed to be no easy and effective way of doing so. That is as far as I got with this notion. I’m hoping that the experimental work I’m planning to do with Peter Tolmie on this subject might indicate if there is any merit in exploring these notions further or not.

Q28. What is the best way to capture and structure information nuggets?

2001 Answer: Ideas formed: By using a Concept Development tool. Some initial prototyping has been done using the Visual Concepts package and the eMindMaps package.

2019 Answer: Experience gained: I explored the use of Concept Development tools for this purpose by using the eMindMaps tool to capture nuggets from 19 separate esoteric books (on subjects such as The Great Pyramid). I found that, although it was probably quite a good way of summarising a book (or article or paper) on one page, there was no easy or effective way to combine several mind maps together or to relate an item on one mind map to an item on another mind map. So I concluded that such tools were not going to be an effective nugget management solution. I’m not intending to explore this any further, however, if I did, I would look into the collaborative concept development tools that I know were being explored by the CSCW community in the 1990s and from which commercially available software might have emerged by now. An alternative, much more feasible, solution might be simply to accumulate the nuggets in a spreadsheet. I guess the question of what tool to use is very much tied up to what one wants to do with the nuggets and what benefits can be achieved by working with them.

Q29. Is it feasible and practical to capture and structure information nuggets as well as indexing items?

2001 Answer: Not started

2019 Answer: Not started

Q30. Is it worthwhile building and developing an information nugget base?

2001 Answer: Not started:

2019 Answer: Ideas formed: I’ve always thought there was value in the nuggets I sidelined in articles and papers – which is way I started exploring this topic in the first place. However, whether there is any value in working with them in any way at all (either just accumulating them in a spreadsheet to cement each point in one’s mind, or inter-relating them in a specialist tool to develop new ideas) is still an unknown.

PAWDOC: Searching

The ability to search for and find a document is just about the most important aspect of a filing system, and that capability has undoubtedly been improved by increases in the power of the modern computer. For example, when the index for the PAWDOC collection was computerised in 1986 it took 228 seconds to conduct a standard search on 3200 records. By 2001, that standard search conducted on over 14,000 records took 7 seconds. That same search, now performed on over 17,000 records, takes less than a second on my current laptop – in fact it’s virtually instantaneous.

Of course, search speed is only half the story, since a targetted document also has to be selected and retrieved. The fact that whole digitised collections can be held on a modern laptop, means that this second part of the process can also be very quick. In fact the total end to end search and retrieval time for the current PAWDOC system is typically between 15 and 30 seconds.

However, speed is not the most important element of a successful search. Instead it is the ability to find what you are looking for – whether you are after a specific document or just doing a general search to see what you have on a specific subject. In the PAWDOC system, searches are conducted on the collection’s Index, therefore success is critically dependent on there being a match between the words specified in the search query and the words in the index entry of the targetted document(s). In a personal system, both elements – index entry and search query – come from a single individual’s mind, so more often than not a match is achieved. However, inevitably there are cases where a match isn’t achieved first time – and, sometimes, not at all. There are a variety of reasons for this including the passage of time (I placed my first document into PAWDOC some 38 years ago), and changing terminology as one becomes more familiar with a topic or as technology develops. To cope with these problems, strategies such as using terminology appearing in unsuccessful searches, and adding keywords to index entries when an item is eventually found, can be helpful.

Specific questions relating to this aspect are answered below. Note that the status of each answer will fall into one of the following 5 categories: Not Started, Ideas Formed, Experience Gained, Partially Answered, Fully Answered.

Q22. How long does it take to find items in the filing system?

2001 Answer: Partially answered:

  • Using the `t f o a d r’ test (find every record with the combination of t, f, o, a, d and r somewhere in it) on the 1988 Filemaker Index with 3200 records running on a Macintosh computer, 27 hits were found in 228 seconds. The equivalent test on the 2001 Filemaker Index with 14111 records running under Windows `98 on a Pentium II PC, took seven seconds to identify 612 hits. A search for all records containing the syllable `man’ across the same two systems took four seconds to identify 211 hits in the 1988 system, and one second to identify 1137 records in the 2001 system (Wilson 1992a: 38, 85).
  • Having identifed the correct index entry (in the 2001 system) it takes between 10 and 20 seconds to obtain the reference number, go to the hardcopy cabinet/box, find the item and pull it out.
  • Should the index entry concerned refer to an electronic file, it takes about 6 ± 10 seconds in the 2001 system to have the document management software display the relevant folder, to double click the required item and to have it open up in the relevant application.
  • The total end-to-end retrieval time for the 2001 system is about 10 ± 30 seconds for hardcopy and 10 ± 20 seconds for electronic files.

2019 Answer: Fully Answered: Using the `t f o a d r’ test (find every record with the combination of t, f, o, a, d and r somewhere in it) on the 2019 Filemaker Pro 15 Index with 17294 records running on a Chillblast Intel i7  computer with 8Gb of RAM, 1224 hits were found in less than a  second – in fact, almost instantaniously. A search for all records containing the syllable `man’ across the same system took less than a second  to identify 1538 hits (in fact it too was almost instantaneous).

Having identified the correct index entry, it takes between 10 and 15 seconds to copy the reference number, go to the Windows File Explorer screen, select the main PAWDOC folder, paste the number into the search field, press enter, and double click  the folder when it appears. Since there may or may not be multiple files in the folder, and since different files may require different applications which probably open at different speeds, it is difficult to provide a reliable figure for selecting a file and opening it. However, as a very rough guide it is likely to take between 3 and 15 seconds.

Therefore the total end to end retrieval time in the current system is approximately 13 – 30 seconds.

Q23. What can you do to speed up retrieval?

2001 Answer: Not started:

2019 Answer: Fully Answered: There are two key factors that affect retrieval times – Physical Proximity and System Integration. Physical Proximity relates to how close you are physically to the system and the hardcopy and/or digital documents. If you haven’t got the system with you then you can’t even identify the document you require, let alone retrieve it regardless of whether it is a hardcopy or digital document. If you are able to identify the document you require, and it is a hardcopy document, then the closer you are to the hardcopy documents the faster retrieval is likely to be (for example, retrieval will be faster if the hardcopies are in the same room that you are in as opposed to in a room down the corridor).If the document you require is an electronic document, then it will probably  be a little faster to retrieve if it is on the same system you are using to search for documents, than if it is on some remote server elsewhere. Therefore, from a Physical Proximity perspective, retrieval can be speeded up by making sure that the Index and the digital store and any hardcopies are all as close to the user as possible.

System Integration refers to the linkage between the searchable Index and a collection’s digital files. Zero integration requires the user to remember the Reference Number selected in the search process, to go to the database of digital files, and to use the Reference Number to open the relevant folder. In contrast, a very high level of integration might be achieved by having the files being stored under a particular Reference Number, appear somewhere in the Index screen for that Reference Number, and being able to open a particular file from there. A halfway house might be to have a macro which will use a Reference Number identified in the index to automatically open up the folder of that particular Reference Number. Therefore, from a System Integration perspective, retrieval can be speeded up by reducing the keystrokes required to go from selecting an Index entry to viewing the files associated with that Index entry.

Q24. In what circumstances are searches conducted?

2001 Answer: Partially answered:

  • `Start Work’: focused assembly of information while under no pressure.
  • `Mid Work’: a search for a specific piece of information while pre-occupied with the interrupted activity.
  • `Visitor’: a search while you are talking to someone at your desk.
  • `Phone Call’: a search while you are on the phone to someone.

2019 Answer: Fully Answered: There are probably three intersecting dimensions to the circumstances in which searches are conducted: Activity (what you’re doing at the time you conduct the search); Work Content (the topic you are working on when the search is conducted); and Location (the type of place in which the search is conducted).

Four different types of Activity are described in the original BIT answer in 2001 (Start Work, Mid Work, Visitor, Phone Call) – though I have no data on the relative frequency of each of those. However, from experience I would guess that Mid Work occurred most often with the frequency of Start Work, Phone Call and Visitor occurring in that descending order.

Work Content might be the subject you are looking into, or the project you are working on, or the organisation you are working for, or any other categorisation that summarises the type of work being undertaken. Again, no data is available to identify what types of work content have been most associated with the searches made on the PAWDOC collection.

Location can be categorised as Employer’s Office, Other Organisation’s Premises, Travelling, and Home. I know that, over the years, I have indeed conducted many searches at all these types of location.

Q25. What are the most common types of searches?

2001 Answer: Partially answered:

  • The `Familiar Item’ search for an item that you have accessed several times before.
  • The `Long Lost Friend’ search for an item you are sure is there but have not accessed recently.
  • The `Shot in the Dark’ search to see if there is any material on a subject.
  • The `Literature Search’ to find everything you have on a subject.

2019 Answer: Partially answered: The 2001 BIT answer provides one perspective on the most common types of searches conducted on the PAWDOC collection (the ‘Familiar Item’ search; the ‘Long Lost friend’ search; the ‘Shot in the dark’ search; and the ‘Literature Search’). However, another perpective might be to categorise the types of document content most frequently searched for. This analysis might be feasible to perform using the ‘Date Last Accessed’ field’ in the Filemaker Index. Although this may not be an entirely accurately record of which items have or haven’t ever been searched for, it nevertheless does provide some sort of indication. Therefore, by categorising the 4551 records which have an entry in the Date Last Accessed field (out of a total 17,294 records) and ranking the categories by number of occurrences, some indication will be gained of the types of documents that have been searched for and their relative frequency.

Q26. What are the most effective search strategies?

2001 Answer: Ideas formed:

  • Get into the habit of searching the filing system when you need some information even when you don’t think you have anything relevant; after several years you forget what you have (Wilson 1992a: 4, 25).
  • Let your mind roam freely when selecting search words; you are more likely to come up with words you originally specified as keywords (Wilson 1992a: 4).
  • Specify searches with minimal parts of words to avoid problems where spelling errors have been made.

2019 Answer: Fully Answered: In addition to the suggestions made in the 2001 BIT article (search just in case, let your mind roam freely, and use minimal parts of search words), I would add:

  • Use any older terminology you can think of if your current terminology isn’t coming up with the goods;
  • Check the results of unsuccessful searches to see if there are any terms which you might try in subsequent searches;
  • If eventually you are successfull in a search that has taken some time, consider adding some additional search terms to the index entry to give yourself a better chance of quicker success in the future.

PAWDOC: Scanning

Despite today’s widespread use of computers and email to create and distribute electronic documents, there has been little, if any, reduction in the use of paper in business. So, office workers  are likely to received, annotate or even handwrite paper documents for the foreseeable future; and scanning remains an integral part of any filing regime. However, it is much more straightforward activity than it used to be: scanners are now fast, effective and cheap and capable of digitising most documents. Contrast and brightness settings can deal with faint or over-emboldened documents; and colour scanning is no longer problematic – nearly all scanners can scan colour and the larger file sizes produced are no longer an issue as storage has become cheap and plentiful. Even large document sizes are no longer the issue they used to be because, if they are too large to be scanned on the equipment you have, they can simply be photographed with today’s high resolution cameras in mobile phones to produce easily readable images which can be enlarged at will.

The time it takes to scan a document is highly dependent on the model of scanner that is used and the size of document being scanned, so I can only provide timings for the particular scanner that I use – a 5 year old Canon DR-2020U A4 scanner. It takes roughly 1.5 – 2 minutes for a 1 page colour document; and roughly 2.5 – 3 minutes for a 5 page double sided colour document. These are overall times which include preparing the paper, specifying the name of the file containing the scanned documents, and saving the file to a particular Windows folder.

Experience has shown that working with scanned documents is perfectly doable – the image quality on today’s screens is good and, in any case, the image can be enlarged if necessary. Of course, it is much easier to do so at a fixed workstation – perhaps with a large screen; having to work with scans on a laptop in meetings or while travelling is inevitably a little less comfortable, especially as it isn’t really possible to fit the whole of an A4 page in portrait mode on a typical laptop screen and be able to read it easily. More often than not users will enlarge the image so that only half a page is showing on the screen and then scroll up and down. Having said that, I have found that the iPad – and presumably other tablets – overcomes this problem. Whole A4 scanned pages can be presented in full on an iPad in portrait mode, and are very clear and fully readable. The iPad is light to hold and it is easy to navigate through multi-page documents; and it can be purchased with sufficient storage as to hold all the digital documents in a collection  such as PAWDOC. This would seem to be the best way to put a digitised filing system to use – though great care would have to be taken to look after it as the very characteristics that make it so useable also make it extremely vulnerable to loss or theft.

Specific questions relating to this aspect are answered below. Note that the status of each answer will fall into one of the following 5 categories: Not Started, Ideas Formed, Experience Gained, Partially Answered, Fully Answered.

Q15. How long does it take to scan documents?

2001 Answer: Fully answered:

  • About 30 seconds for a black & white page. (Chan 1993: 28)
  • 40 ± 50 seconds for a black & white page (Wilson 1997: 3)
  • 30 ± 50 seconds a page for dual journal pages on a flatbed scanner (Wilson 1996b

2019 Answer: Fully Answered: Using a Canon DR-2020U scanner purchased in 2013 and a Chillblast i5 computer with 8Gb of RAM and 1Tb SSD drive purchased in 2018, scanning time for a 5 page double sided colour document is about 70 seconds; and for a single colour page about 25 sec. However, this doesn’t tell the whole story because the scanning process also involves preparing the pages and storing the output. Approximate timings for all of the different process elements are as follows: a) Prepare the paper ie. remove any staples or postITs, get pages  alligned etc.. (5 seconds if there are no problems – but could take much longer) b) Start the scan – place the pages in the scanner, select the appropriate scanning job – Colour or B&W etc. – and press START (10 seconds), c) Perform the scan (70 seconds for 5 double sided colour pages, or 25 seconds for a single colour page), d) Create the file title – paste the Ref No (which had been copied in the indexing process) into the file title box and type in the rest of the file title  (15 -30 seconds), e) Save the file – select the folder into which the document is to be placed and press SAVE (20 – 30 seconds), f) Check the folder – check that the file has been stored in the correct place with the correct title (15 seconds); g) Check the file – open the file to check it has been scanned correctly (only for multi-page documents or for documents where there may be an issue) (15 seconds).

In summary, overall scanning time for a 1 page A4 colour document is roughly 1.5 – 2 minutes; and for a 5 page double sided A4 colour document about 2.5 – 3 minutes.

Q16. What are the practical problems associated with scanning?

2001 Answer: Partially answered:

  • Scanning double-sided pages is time-consuming and error prone (Wilson 1995b: 121, 1995c: 1).
  • Paper jams and misfeeds sometime occur when using the scanner sheet feeder (Wilson 1995c: 1).
  • Some documents need preparation work prior to scanning, for example, removing staples or guillotining bound documents (Chan 1993: 34 ± 37).

2019 Answer: Fully Answered: Practical scanning problems are of two types – those that are inherent to the scanning activity; and those that are due to shortcomings in the scanner being used. The main inherent scanning problem is the need to ensure that the papers being scanned are free from staples, PostIT notes and any other additional items which will jam as the paper passes through the scanner. Problems due to scanner shortcomings may be due to a lack of capability in the scanner concerned such as no sheet feeder; the inability to scan large pages such as A3 (A3 scanners are available but they’re big and more expensive); and inability to scan double sided pages (a particular problem I had when I started scanning but I’m glad to say that my current scanner scans both sides of a page at the same time as it makes a single pass through the scanner). Scanner shortcomings may also include capabilities in which technology developments are continually making inroads, for example, sheet feeder technology (hugely better in my current scanner – but multiple pages are still pulled through occassionally); and dealing with very faint text (I would hope that future software will automatically adjust brightness and contrast settings to deal with this). Finally there may be shortcomings in a particular model of scanner. I believe this is the case in the software driving my Canon DR-2020U which seems incapable of scanning a full page which has continuous black or very dark areas on the edge of the page – the scan truncates the page to remove those areas. To deal with this I have to cover the relevant edges with white paper. This  may have been resolved in a later release of the software – though I havn’t checked for some time.

Q17. What are the practical problems of scanning documents containing colour?

2001 Answer: Ideas formed:

  • Some colours scan black when using a black and white scanner (Wilson 1995b; 76, 77, 98)
  • Colour scans take more time (Wilson 1997: 2).
  • Colour scans produce files 10 or 20 times the size of a black and white scan (Chan 1993:43, Wilson 1997:2)
  • Colour scans (which are made up of red, green and blue) may not work well on printers (which use cyan, magenta, yellow and black) (Chan 1993: 44)

2019 Answer: Fully Answered: Colour scanning no longer presents a particular problem because today’s scanner all have a colour capability, and even though colour scans take up more space, storage is now plentiful and cheap (for comparison purposes, with my current scanner, a single page with 5 colour photos on it scanned at 300dpi B&W to a PDF of 126 Kb; and at 300dpi Colour to a PDF of 801 Kb).

For the most part, colour scans are normally sharp and clear. The only problem I have had with colour scanning seems to be specific to the software used by my current 5 year old Canon DR-2020U scanner. I’m very pleased with the performance of this scanner except for tha fact that it truncates those parts of the edges of pages which have black or very dark colour. To deal with this, I have to place white paper just over the edges of the page – which can be difficult for larger pages which go up to the edge of the scanner platen. I have never experienced this problem before and hope that it has been fixed in a software upgrade.

Q18. Can poor originals be successfully scanned?

2001 Answer: Experience gained: Modifying the scanner settings can dramatically improve the quality of scans of poor originals.

2019 Answer: Partially Answered: Originals can be considered poor for a variety of reasons including skewed images, holes punched in the sides for binding purposes, and faint or heavily smudged images. My current scanner has on/off features to deal with skew (works well) and punched holes (I havn’t really tried to use this); and has settings to adjust contrast and brightness (these also work well but I’ve found it difficult to remember which combination of settings work best – perhaps in the future more automated assistance will be provided for this). In summary, usable scans can often be achieved from poor originals.

Q19. Can all types and sizes of documents be scanned successfully?

2001 Answer: Experience gained: Yes, provided you have a flatbed scanner with a sheet feeder. If the flatbed is not big enough, many of the larger documents can be cut or folded to the necessary size. The spines of gummed or stapled newsletters and booklets can be cut off (Wilson 1995b: 99). Post-it notes stuck on a page go through the sheet feeder without a problem (Wilson 1995b: 13, 39, 140). Journals can be scanned successfully on a flatbed scanner (Wilson 1995b:124).

2019 Answer: Fully Answered: As described in the 2001 answer, the vast majority of documents can be scanned successfully provided you have a scanner which can accommodate the physical size of the document. Of course, if you can’t lay a document flat on the scanning platen, as is the case when trying to scan a book or a journal, there will always be some distortion at the centre of the dual pages.  Most people are likely to have an A4 or, in some cases, A3 scanners  (though an A3 scanner does ta ke up more space) so that they will be unable to scan documents above these sizes; However, that is of no consequence these days because digital photography is now so cheap, and powerful, and prevalent on mobile phones, that it can be used to photograph documents that are too big or difficult to scan. The digital images produced are good enough to read and can be enlarged as necessary. A good example of this is the increasingly common practice of taking photos of flipcharts of which there are a few in the PAWDOC collection. However, there are a few potential problems with photographing documents or other flat objects: one is that, unless the camera is held in exactly the same plane as the document page, the image of the document will appear shorter on one side than on the other; and the other is that, if the document will not lie flat, the document image will appear equivalently distorted in the photograph.

Q20. How can you minimize the amount of scanning that needs to be done?

2001 Answer: Experience gained: Possible strategies include:

  • Stick with electronic files as much as possible
  • Minimize the printing of paper (Wilson 1995b: 82, 90).
  • Elect to receive electronic versions of newsletters, magazines and journals (Wilson 1995b: 126).
  • Go out of your way to obtain electronic versions of paper you get from other people (Wilson 1997: 3).

2019 Answer: Fully Answered: Despite paper still being very widely used (the paperless office never actually arrived), most documents are now created and distributed in digital form – very little paper arrives in the overland or office mail today. Hence, scanning can be minimised by simply storing the electronic versions of documents you create and that you receive by email; and by seeking out the electronic versions of other hardcopy document that you receive. For example, I get the quarterly newsletter of the professional body that I belong to in hardcopy form; but I go to their website to get a digital version of the magazine. Having said that, journal and magazine publishers continue to protect their paper and articles behind paywalls; and newspapers seem to be following their example. Consequently, filing system owners may have to scan some hardcopy papers and articles for the foreseeable future.

Q21. Can scanned images be used successfully in day-to-day work?

2001 Answer: Ideas formed: Relatively few of the scanned images have been used to date. However, experience so far has not indicated any major problems with either reading them on screen or printing them out. An A4 sized display would certainly make it easier to read scanned images on screen (Wilson 1995b: 104).

2019 Answer: Fully Answered: Although I don’t access many of the PAWDOC scanned images these days, over the years I have used them and worked with them and have had no trouble doing so. Nowadays I view all my scanned documents (in both multi-page TIFF and PDF formats) in my eCopy PDF Pro application and this works fine – though it would undeoubtedly be better if the screen could be in Portrait mode so that the whole of an A4 page could be displayed in the way one reads the hardcopy. Recently I transferred several scanned documents onto my iPad and I’ve found I can read them all without any difficulty with the iPad in Portrait mode and without needing to enlarge them. This has been a bit of revelation since it is clearly a much easier and more pleasing experiece to read these documents on the iPad than it is on my 24in desktop screen or my laptop. The combination of the iPad’s high resiolution screen, it’s light weight, the fact that you can hold it as close as you want, make it a winning combination for scanned work documents. This could be the way to store digital filing systems in the future – or perhaps large screen phones could serve the same purpose?

Nugget Investigation Plans

This entry has been jointly authored by Peter Tolmie and Paul Wilson

A key part of the motivation for keeping texts is that they contain information – nuggets – that have some kind of future value. It’s rare for a whole document to be seen as having such value in its entirety. Nuggets are more often certain sentences or paragraphs. The question is, what happens to that value over the lifespan of an archive. Does the value of specific nuggets persist? Does it change? Does it grow or reduce in relevance? Does it become eroded to the point of obsoletion? And, given the same documents at some later point in time, would the same nuggets be identified, or would something else stand out as being more important?

To assess the use and impact of identifying nuggets, three separate investigations will be conducted using documents containing nuggets that were collected over the last 38 years. In all cases, investigations will be undertaken with reference to the individual who originally identified the nuggets. Two of the investigations will focus on the individual reflecting himself upon his use of nuggets. The third investigation will focus upon extracting information about the nuggets and their use through discussion.

The investigations will be performed using two separate sets of material:

Set 1: A set of 19 MindMaps relating to esoteric books.

Set 2: Documents from the PAWDOC collection.

Approach

Investigation 1: The first investigation will be a written exercise using the set of Mindmaps and will attempt to assess:

a) Whether individual nuggets can still be recalled;
b) Whether the sources of those individual nuggets can still be recalled;
c) How significant specific nuggets are considered to be by the individual concerned;
d) What other nuggets, if any, are associated with the one in question;
e) What specific concept(s) individual nuggets are believed to have contributed towards.

Investigation 2: The second investigation will also be a written exercise and will explore what nuggets, if any, can be identified by the individual concerned in unmarked versions of randomly selected documents from the PAWDOC collection where nuggets have previously been identified. From this exercise it is hoped to deduce:

i) Whether nuggets lose their vitality over time, with new concepts being derived and becoming established.
ii) Whether looking at the documents anew, within a distinct context, will lead to the identification of different nuggets with different relevance to the individual.

Note that the first 2 investigations are interleaved so as to be able to maximise the amount of discussion possible in the concluding interview.

Investigation 3: This investigation will be conducted as an interview and will explore similar themes to those tackled in the prior investigations but in a more open-ended way, so that the reasoning involved in retaining documents for the sake of specific features can be examined in greater detail. The interview will also seek to examine in detail topics that have spanned all three investigations.

Method

Investigation 1: Nineteen nuggets, each from separate MindMaps, will be randomly selected from the MindMaps of books on esoteric subjects. Randomisation will be achieved by creating a template that divides an A4 page into 24 numbered areas of equal size. A random number generator will then be used to create nineteen separate numbers ranging from 1 to 24. The Researcher, Peter Tolmie, will generate the numbers and use each one in conjunction with one of the MindMaps and the template to identify the nugget(s) present in that location. If more than one nugget occurs in that location, the topmost one will be selected.

The Researcher will assemble the nineteen nuggets and send them to the PAWDOC Owner and Subject, Paul Wilson, who will be asked to provide a written answer to the following questions in relation to each one:

  • Do you remember where this nugget came from?
  • Do you remember why you might have marked this out as a nugget?
  • How important do you consider this nugget to be now?
  • Do you remember what other nuggets were associated with this one?
  • How did you use this nugget and what other things did you develop on the back of it?

The Subject will then be sent the original MindMaps in which the nuggets appeared, and will be asked the following questions for each one:

  • Do you remember more about why you marked this as a nugget now?
  • Now you see the MindMap it came from and the other nuggets it was associated with, do you see it as more or less important?
  • Do you remember anything more now about how you used it or what other ideas it may have contributed to?

The Subject will return his responses to the Researcher, who will categorise them and place the results in an analysis spreadsheet. The overall analysis and specific instances regarding the Subject’s responses and reactions will be written up as the findings for Investigation 1.

Investigation 2: Nineteen separate documents, each containing nuggets, will be randomly selected from documents included in the PAWDOC collection between 1981 and 2011. Randomisation will be achieved by using a random number generator to create numbers relating to the 16925 entries in the PAWDOC Index sequenced in the order in which they were created.  The Researcher will generate the numbers and use his copy of the Excel version of the Index to identify the first nineteen of the entries for which an electronic file exists (some entries just contain the information they relate to and have no associated documents – these are identifiable by the contents of the Movement Status field); and for which the associated electronic files are likely to contain highlighted nuggets (items such as, for example, Health & Safety booklets, are unlikely to contain highlighted nuggets). He will send the list of numbers to the Subject who will open the folder labelled with each particular number, take a copy of the first file that appears in each folder and will send the files back to the Researcher. The Subject will take as little notice of the file titles as possible by setting up the Windows File Explorer window to display only the beginning of the file name so that only the Reference Number, which is always at the start of the file name, is visible. Should the Researcher deem any of these files to be unsuitable, he will send additional numbers to the Subject until a satisfactory set of nineteen documents that contain nuggets has been obtained. Then, using a cropping tool or by obtaining a clean copy of the document from elsewhere, he will send clean unmarked copies back to the Subject. The Subject will read the documents, mark up any text that he considers to be nuggets, and will send the documents back to the Researcher.

The Researcher will then make a comparison between the original nuggets identified and the new nuggets identified and record the results in the analysis spreadsheet. The overall analysis will be written up as the findings for Investigation 2.

Investigation 3: Nineteen nuggets, each from a separate document, will be randomly selected from randomly selected documents placed in the PAWDOC collection between 1981 and 2011. Randomisation will be achieved by using the same procedure employed in Investigation 2.  The Researcher will take the first nineteen suitable documents that contain nuggets and randomly select one specific nugget from each document using a random number generator. The nineteen nuggets will be assembled together and presented to the Subject who will be asked to answer a similar set of questions to the first set of questions in Investigation 1. These questions will form the rough frame for the first part of an interview in which the various nuggets will be discussed.

The Subject will then be shown the original documents in which the nuggets appeared, and will be asked the following questions for each one, which will form the second part of the interview mentioned above:

  • Do you remember more about why you marked this as a nugget now?
  • Now you see the document it came from and the other nuggets it was associated with, do you see it as more or less important?
  • Do you remember anything more now about how you used it or what other ideas it may have contributed to?
  • Are there things in the original document that you didn’t mark as a nugget at the time that you would mark as a nugget now? If so, why?

A third and final part of the interview will explore the responses from across all of the assessments in a more open-ended fashion to generate deeper insights and discussion.

The Researcher will then transcribe the interview, categorise the responses and place the results in the analysis spreadsheet. The overall analysis and specific instances regarding the Subject’s responses and reactions will be written up as the findings for Investigation 3.

Conclusions: The Researcher will use the findings from all three Investigations to write the overall conclusions of the investigation.

PAWDOC: Indexing (and the Accession process)

Indexing is a key aspect of a filing system because it is one of the mechanisms used to find documents. An alternative mechanism is to search the full text of documents, but the PAWDOC system has not explored this approach because a) the PAWDOC system does not recognise the full text of all the documents it contains, and b) because I have always believed this approach would provide too many hits thereby reducing search effectiveness.

Indexing within the PAWDOC collection is performed at two levels: first, each entry (uniquely identified by a Reference Number) in the main index file contains a Title field (in which a free format description and any number of uncontrolled keywords can be specified) and a few other fields including publication date. Second, each electronic file (be it a scan of a hardcopy document, or a born-digital document) associated with a particular Reference Number contains a unique file title. The contents of these two sets of indexing information provide the means to search for a document, to decide if you have found what you are looking for, and to get a better idea of what the chosen Reference Number and/or associated electronic file(s) contain. These potential uses are worth bearing in mind when indexing information is being specified.

Inevitably mistakes are made when specifying Title information and so it is essential that index entries can be changed over time to correct typos, grammar and factual errors as necessary. Changes may also be inspired by experiencing difficulty in finding a document, and in these cases some additional text may be included to reflect the search terms that first came to mind, or to reflect the changes in terminology that occurs regularly in language and in today’s fast moving technical and cultural environment.

Indexing is closely integrated into the process of including a new document in the PAWDOC collection, not least because adding a new file involves creating a file title which in turn includes a Reference Number and possibly some text from the associated Index entry. Hence, timings for how long indexing takes are embedded within the overall process of Accessioning (which includes scanning for hardcopy documents). For the PAWDOC collection, which uses a scanner acquired in 2012 and a laptop acquired in 2018, it takes approximately 2.5 – 4 minutes to accession a single colour A4 hardcopy page; and about 3.5 – 5 minutes for a 5 page double sided colour hardcopy document. An electronic of any type and number of pages takes approximately 2 – 3.5 minutes to accession. These are appreciable amounts of time for an overhead administration activity amidst a busy working day. Therefore, it is worth exploring any means of reducing these timings. The obvious way is to keep the number of index fields to an absolute minimum. Experience with the PAWDOC system indicates that integrating keywords into the Title field (as opposed to having a separate keywords field) has been very successful. Conversely, the Publication Date field has not proved to be very useful and I believe could be dispensed with. Experience with the other fields used in the PAWDOC system are summarised below:

  • Reference Number – essential
  • Creation Date – a useful control
  • Movement Status – very useful for recording the whereabouts and status of documents
  • Date Last Accessed – only useful if you particularly need it.

Specific questions relating to this aspect are answered below. Note that the status of each answer will fall into one of the following 5 categories: Not Started, Ideas Formed, Experience Gained, Partially Answered, Fully Answered.

Q10. How long does it take to index a new document?

2001 Answer: Fully answered: It takes 1 -2 minutes to make an entry in the main index and to write the Reference Number on the document. A further 30 seconds is needed to create an entry in the document management system for either an electronic file or a document that is to be scanned immediately (Wilson1995c: 1). Scanning a document will add a further 40-50 seconds for a single page – though the per-page time is reduced considerably for multi-page documents that are put through the sheet feeder (Wilson 1997: 3).

2019 Answer: Fully Answered: This question should really be ‘How long does it take to index a new document and include it in the collection?’ because the indexing process is closely integrated with the process of saving the file. The figures in the 2001 Answer for making a new entry in the main index and writing the Reference Number on the document (1-2 minutes), and for creating a new entry in the document management system (an extra 30 seconds) still stand.  Scanning a document using the Canon DR-2020U scanner I bought in 2013 and the new Chillblast laptop I bought in 2018 with a 1 Terrabyte solid state drive, takes 25 seconds for one colour page or 70 seconds for 5 double sided colour pages. However, the Document Management system was eliminated from the PAWDOC architecture in 2018 and replaced by Windows folders, so the subsequent set of actions has changed to the following: a) Create the main index entry (1-2 mins); b) Create a sub-folder within the main PAWDOC folder in the Windows file system, with the new Reference Number and copy the Ref No (15 seconds); c) create the file title (Ref No   Title,  Date scanned) in the scanner control application starting by pasting in the copied Reference Number (15 – 30 seconds); d) Save the file – use the scanner control application to select the new folder just created to specify where the file should be stored and press save (20-30 seconds); e) check the file – open the file to check that it has been stored in the correct place with the correct title (15 seconds); f) Check the folder – open the file to check it has been scanned correctly (only for multi-page docs or for docs where they may be an issue) (15 seconds). The process for electronic documents has also changed as follows: i) create the main index entry (1-2 mins), ii) create a sub-folder within the main PAWDOC folder in the Windows file system, with the new Reference Number and copy the number (15 seconds); iii) Create the file title – open the Save AS dialogue box and give the file a new file name (Ref No   Title,  Date saved) by first pasting in the copied Reference Number (15 – 30 seconds); iv) Save the file – navigate to the folder into which the file is to be saved and press SAVE (20 – 30 seconds); v) Check the file – open the file to check that it has been stored in the correct place with the correct title (15 seconds). Of course, electronic documents which are placed into an existing Ref No have no need to create a new index entry or a new folder.

In summary, overall timings for including new documents in the collection are as follows: for a 5 page double sided hardcopy document using a new Ref No – approximately 3.5 – 5 minutes; this may reduce to about 2.5 – 3 minutes if an existing Ref No is being used. For an equivalent 5 page electronic document using a new Ref No – approx 2 – 3.5 mins; this may reduce to 1 -1.5 mins if an existing Ref No is being used. Timings for a hardcopy document with only 1 page of contents are reduced by about 45 seconds due to reduced scanning time; electronic document timings stay the same regardless of page count.

Q11. What can you do to speed up indexing?

2001 Answer: Ideas formed: Keep the number of index fields to a minimum.

2019 Answer: Fully Answered: Minimise the number of fields. Automate the generation of the Creation Date. Eliminate the Date Last Accessed field if there is no good reason for recording that information.  To speed up the overall accession process, a) store more rather than less documents in existing Reference Nos (using existing Ref Nos avoids having to create new index entries); b) use a faster scanner; c) reduce the time it takes application programs to page through 17,000+ sub-folder names in the Save As function (for some reason, in my Windows 10 system, the application programs take far longer to do this than Windows File Explorer).

Q12. What is the most effective set of index fields?

2001 Answer: Experience gained: The smallest number you can manage with (Wilson 1990: 95). Reference number, title and keywords, date of creation of the record, movement status, date last accessed.

2019 Answer: Fully Answered: Keeping the number of fields to the absolute smallest number you think you can manage with will minimise the time and effort spent on putting new items into the system and on managing the system. The minimum number of fields I could manage with are Reference No, Title, Creation Date, and Movement Status. The other two fields that I use, but which I believe I could do without, are Publication Date (I rarely refer to this) and Date Last Accessed (which was included mainly for research purposes).

Q13. What criteria should be employed when defining titles and keywords?

2001 Answer: Ideas formed:

  • Remember there is no need to use the actual title of the document.
  • Define titles/keywords in accordance with what the document means to you – that way you are more likely to be able to retrieve it.
  • Bear in mind that title/keywords serve multiple purposes (Wilson 1992a: 8):
    • To enable search and retrieval
    • To enable the user to decide if a retrieved record is what was being searched for
    • To provide the user with an understanding of what is contained in the item referenced by a retrieved record.

2019 Answer: Fully Answered: Do not feel constrained to use the actual title of the document – you are far more likely to be able to specify successful searches for items if you define Titles and Keywords which convey what the document means to you. If you are aware terminology is changing, use the latest terms you know as you are likely to become less familiar with the old terminology as time passes. Since there are no length constraints on the Index entry, use as many words as necessary to describe what the document is about. The second level of indexing – the text in the Titles of individual files – is constrained by length and for that reason might well be shorter than the main index entry. However, care should be taken not to truncate unnecessarily and not to just take the quick and easy route to create the file title by cutting and pasting a generic part of the Index entry. Certainly, where there are two or more files associated with the same Reference No, the titles of the 2nd and subsequent files should clearly distinguish them from all the other files in that Reference No. Also, bear in mind that Title fields (which include Keywords) and File Titles serve at least the following three purposes: a) to enable search and retrieval; b) to enable the user to decide if a retrieved record is what was being searched for; c) to provide the user with an understanding of what is contained in the item referenced by a retrieved record.

Q14. Are there circumstances in which keywords and titles should be changed over time?

2001 Answer: Experience gained: It is very necessary to be able to alter keywords and titles for the following reasons:

  • To accommodate the user’s growing familiarity with a topic. For example, the term `OA Human Aspects’ became an abbreviation `OA HCI’ in later entries, so it needed to be placed in previous entries to ensure they would be retrieved if only `OA HCI’ was specified in the search term (Wilson 1992a: 22).
  • To accommodate changing language. For example, the term `DTI pilot’ became `DOI pilot’ when DTI changed its name (Wilson 1992a, 12).
  • To accommodate additional material being added to an existing hardcopy file, or being added to the associated Folder in the Document Management System.

2019 Answer: Fully Answered: Users should be able to change Titles and Keywords at will to correct mistakes or to try and improve the search success rate. Circumstances in which changes might be made for the latter reason include:

a) to accommodate the user’s growing familiarity with a topic. For example, the term `OA Human Aspects’ became an abbreviation `OA HCI’ in later entries, so it needed to be placed in previous entries to ensure they would be retrieved if only `OA HCI’ was specified in the search term;

b) to accommodate changing language. For example, the term `DTI pilot’ became `DOI pilot’ when DTI changed its name; and

c) to accommodate additional files being added to the Reference No.

PAWDOC: Deciding what to file

Most types of things can be managed in a digital filing system. Documents, brochures and books can be either scanned or photographed; electronic documents are, of course, already in the right form; and even physical artefacts such as, for example, art works, can be photographed.

Once it’s been decided to establish a filing system, the reasons why the items concerned are being kept, and what is going to be done with them, should be written down. This should help in defining the following three key points: i) how long items are to be kept for; ii) whether any physical originals are to be retained after they have been digitised; and iii) whether every item is to be filed or just a particular subset. Point iii) is important because, if only a subset is required, the amount of time and effort that will be spent on adding items to the filing system, and managing it, will be reduced. Against this time saving must be weighed the extra effort – and potential errors that may be made – in deciding whether each particular object is to be included or not. Conversely, collecting every instance of a type of an item makes life easy and requires no decision making.

If the filing system is to replace or augment an existing collection, a decision must be taken about whether to exclude the existing items, or to undertake ‘backfile conversion’ to include some or all of them in the digital collection. Backfile conversion is often a lengthy and tortuous process and can act as a drag on proceeding with the new system; therefore, it should be avoided at all costs. If, however, it is thought to be needed, the reasons for doing it should be clearly understood before proceeding.

Specific questions relating to this aspect are answered below. Note that the status of each answer will fall into one of the following 5 categories: Not Started, Ideas Formed, Experience Gained, Partially Answered, Fully Answered.

Q5. What types of information and media can be managed by an electronic filing system?

2001 Answer: Fully answered: Just about anything can be managed by an electronic filing system including single sheets and stapled sets of hardcopy documents, scientific journals, volumes of bound material, ring binders, books, 35 mm slides, videos, electronic files and e-mail messages.

2019 Answer: Fully answered: Any kind of item can be managed by an EFS – though it may be necessary to also keep the original physical artefact to experience the full impact of certain types of items such as specially folded brochures, or significant douments with a tactile presence. Types of items that I have succeeded in managing in the PAWDOC collection include hardcopy documents, ring binders, books, journal papers, magazines, brochures, overhead slides, flipchart pages, CAD/CAM drawings, photos, videos, web pages, and email messages.

Q6. Can all types of electronic files be managed in an electronic filing system?

2001 Answer: Partially answered: Yes, certainly all types of application files can be. I have not stored executable files yet but have been advised that it can be done and how to do it.

2019 Answer: Fully Answered: Any types of file format – including executables – can be managed in this way. For example, file types in the PAWDOC system include Word, Excel, Powerpoint, PDF, TIFF, JPG, MP4, MPP, ZIP, and HTML. However, the real issue is whether you will be able to open them in future years if the particular application programmes are lost or not kept up to date. For this reason, it may be worthwhile converting some files to more common formats, such as PDF, before storing them. In the future, these so-called ‘Digital Preservation’ issues may get addressed automatically by AI programmes.

Q7. What criteria should be employed when deciding what to file?

2001 Answer: Not started

2019 Answer: Fully Answered: Individuals should file as much or as little information as they think they will need in their jobs – otherwise they will not have the motivation to spend the time introducing new material and managing the filing system. If a robust, all embracing, file facility is available within a particular system (such as in many electronic mail systems) such systems should be used to their full capability (i.e retain everything up to any limitations set by the system administrator). Inevitably, whatever an individual decides to file, there will always be some material which is never subsequently accessed, and there will always be some required information that can’t be found in the system. This seems to be a regularly occurring phenomenon and should not deter individuals from filing – though they may influence the filing criteria that an individual applies over time.

Q8. Is it worth doing backfile conversion?

2001 Answer: Experience gained: Backfile conversion is certainly not essential, and is probably not worth doing, except for material you definitely want to keep for many years.

2019 Answer: Fully Answered: Backfile conversion usually takes a huge amount of time and effort, so a detailed analysis should be undertaken before deciding whether to do it or not. Alternative approaches to dealing with the old items should be investigated. Backfile conversion should only be carried out if there are compelling reasons for doing so.

Q9. What are the major considerations when carrying out backfile conversion?

2001 Answer: Experience gained: Backfile conversion requires a huge amount of time and effort and is not to be undertaken lightly.

2019 Answer: Fully Answered: Considerations to be taken into account when conducting backfile conversion include:

  1. Ensure the physical backfile material does not interfere with any space required by the replacement filing system.
  2. Decide if any of the backfiles being converted will need to be retained in their original form and, if so, define clear criteria for selecting which ones to keep.
  3. Decide what quality the converted backfiles need to be and then put in place the appropriate equipment, software and procedures to achieve that quality.
  4. Decide how users will be informed of which backfiles have been converted and which are yet to be converted.
  5. Establish a clear and doable schedule for the conversion process.
  6. Put in place motivational aids for sticking to the schedule. For example, setting a target of a certain number of conversions each day; and creating a visible progress sheet on which achievements can be ticked off.

PAWDOC: Collection content, size, growth rate and usage patterns

The PAWDOC collection was set up in 1981 to explore the application of office technology. In 2001 a paper was published in the Journal Behaviour & Information Technology called ’20 years in the life of a long term empirical personal electronic filing study’. This described PAWDOC and summarised findings about its use up to that point under the following 15 headings:

  • Collection content, size, growth rate and usage patterns
  • Deciding what to file
  • Indexing
  • Scanning
  • Searching
  • Use of information
  • Archiving
  • Hardcopy/electronic mix
  • Relationship between work patterns and filing activities
  • Technology requirements and problems
  • Portability
  • Sharing files
  • Confidentiality, ownership and intellectual property rights
  • Reliability and Longevity
  • Costs and benefits

Now, a further 18 years on, the findings will be further reviewed in this and subsequent posts. The first of these follows below.

Collection content, size, growth rate and usage patterns

The precursor to PAWDOC was a conventional filing system in an upright cabinet using hanging folders and crystal tabs to store documents only (not journals, books etc.). The contents were specified in an index taxonomy with entries of the type, for example, 1.6.8 – Quarterly Progress Reports, which I was constantly adding to. Consequently much space was taken up in the cabinet by folders with crystal tabs housing only a few pages.  This changed after the PAWDOC system came into use: it needed far fewer hanging folders because each folder was filled to capacity with as many of the serially numbered documents that it could take.

The PAWDOC schema was explicitly defined to support the management of multiple sets of different material owned by multiple different owners. This capability was used to manage a variety of sets of my own material in addition to documents, for example, several different journals, 35mm slides (for making presentations), ring binders, and books; and, in the early years of the system, material owned by other people and organisations – though, as time went by, I made fewer and fewer such entries not least because of the uncertainty of being able to access such material.

The system soon became an integral part of my working life, and I used it just about every day. Several hundred new items were being added each year and this steady rate of acquisition soon ran up against the physical limits of the upright cabinet, so it became necessary to archive material in boxes and then to put some of the boxes in store. This went on until I started digitising newly acquired documents in 1996 and disposing of most originals. At this point, I also started to store born digital documents regardless of the applications they were created in.

In 2001 I started a new job in Bid Management which precluded personal storage of its associated highly confidential and fast moving documentation; so my useage of the PAWDOC system did reduce from then onwards. Nevertheless I continued to use it regularly on most days, and was still adding over 200 new items each year up to when I retired in 2012.

The 45 boxes of hardcopy that I had acquired eventually came out of store around 2001 and were stored in my garden shed. Their number was overwhelming and I began to doubt I would ever get them all digitised. However I stuck at the task, sometimes going at it solidly for several days at a time when my wife was away. By the time I retired in 2012 only 4 boxes remained and these had all been scanned by 2014: it was a great relief. The huge amount of physical space taken up by the collection had been reduced to just two archive boxes of significant hardcopy documents; and, after I had conducted a digital preservation exercise on the collection in 2018, the digital footprint of the collection amounted to some 115Gb. I have ended up with a fairly complete digitised archive of all the non-highly confidential materials that I had encountered throughout my working life, including substantial amounts of material from my earlier career from 1972 with Kodak and then CPC. Since retiring I’m continuing to add a few documents (around 40 up to 2019) in three categories: significant articles relating to the work I used to do; documents relating to the digital preservation of the PAWDOC collection; and material relating to work I am doing to investigate and document the findings from the PAWDOC collection’s 38 years of existence.

Specific questions relating to this aspect are answered below. Note that the status of each answer will fall into one of the following 5 categories: Not Started, Ideas Formed, Experience Gained, Partially Answered, Fully Answered.

Q1. What are the contents of the collection?

2001 Answer: Fully answered: At the beginning of July 2001, the collection consisted of 14 100 index entries representing approximately 185 000 pages of paper, 50 000 scanned pages, over 30 scientific journals (including Behaviour & Information Technology from 1982), around 30 books and conference proceedings, 3700 MS Word files, 400 MS Excel files, 250 MS PowerPoint files, 150 other electronic files of various types, and 10 CDs.

2019 Answer: Fully answered: The PAWDOC user Guide created in 2018 says: “All types of documents were stored including letters, internal memos, circulars, reports, specifications, minutes, overhead slides, 35mm slides, notes, training materials, brochures, manuals, maps, emails, computer magazines, journal articles, conference proceedings, and videos. As Office Technology became more versatile, electronic documents such as word processor files, spreadsheets, presentations and web sites were also filed.”. In June 2019, the collection consisted of 17,293 Index entries representing 29,610 electronic files in 16,067 Windows folders, and about 340 physical hardcopy documents in two archive boxes. A further 384 old electronic backup files are also stored in a separate folder. The Checking exercise performed in 2016  identified the following numbers of different types of files in the collection: Word – 6380; Powerpoint – 466; Excel – 625; HTML – 382; Help – 90; Zip – 92;  11 other apps – 88; Scanned documents – 28,418. The collection was primarily a work collection and therefore the number of new items being included reduced to a trickle when I retired in 2012.

Q2. How much space does the collection take up?

2001 Answer: Fully answered: The paper takes up about 4.7 sq. metres of floor space and 1.7 metres of shelf space. The scanned images and electronic files take up 2.9 GB. The scanner, magneto-optical drive and CD Writer take up about 0.25 sq metres of desk space. The Filemaker Pro index is about 8.1 MB in size and the FISH data file is 11.2 MB. The Filemaker Pro, FISH, SQL Anywhere and Easy CD Creator software packages take up approximately 38 Mb.

2019 Answer: Fully answered: The two archive boxes stand one upon the other and take up 0.2 sq m of floor space. The laptop in which the electronic files reside takes up 0.08 sq m of desk space. The electronic files of the main collection take up 45.9 Gb storage space; and the backup files take up 66.6 Gb. The Filemaker Pro software used for the index takes up 336 Mb of file space.

Q3. What is the growth rate of the collection?

2001 Answer: Partially answered: Between 1981 and 1993 an average of 543 index entries were created each year – estimated to consist of an average 29.4 pages per day (Chan 1993:25).Over the whole 20 years life of the system, the growth rate has been an average of 705 entries per year with a range of 210 ± 1202new entries a year. In 1993, it was estimated that the collection was increasing at the rate of 3.8 MB per day.

2019 Answer: Fully answered: The growth rate of the collection is shown in the chart below.

There are three distinct phases: 1981 – 2000; 2001 – 2011 when I was working on highly confidential bids; and 2012 – 2019 when I was retired. The average growth rates during these periods were:

  • 1981-2000 – 696
  •  2001-2011 – 272
  • 2012-2019 – 46

Q4. How often are the contents accessed?

2001 Answer: Experience gained: Between 1987 and 1993, an average of 363 records were being accessed each year (Chan 1993: 25)

2019 Answer: Experience gained: The only data that has been collected on this question is in the date last accessed field, and unfortunately that only records the latest date an item was accessed – there may have been any number of earlier accesses. Furthermore, some items may have been accessed without the date last accessed field being updated. Having said that, 4,551 items have an entry in the date last accessed field, implying that 12,742 items have never been looked at for work purposes after they had been included in the collection (the date last accessed field was never updated when items were looked at for the purposes of controlling and writing about the collection). For the period from 2001 onwards, when I moved jobs into Bid Management, there were 751 index items with dates of 2001 or later in the date last accessed field (only 15 of these were in 2012 and only a further 15 of these were from 2013 onwards).

NB. The various references in the texts above to Chan,1993 relate to the following reference at the end of the 2001 paper in Behaviour & Information Technology:

CHAN, S. C. 1993, Feasibility of Paperless Office, Submitted in partial fulfilment of the requirement for the degree of MSc in Information Systems and Technology in The Information Science Department at City University, London (Supervisor: Dr David Bawden) [PAW/DOC/4012/08].

EitherOr

I wonder if any internet entrepreneur has come up with a web service to help with choices in present buying… I’m envisaging a website called EitherOr which enables people to outline a number of gift options, and possibly a ‘none of the above’ option, for the individuals they want to buy for. The recipients would be notified and be able to use links to review the items and to choose the option they want. The service would then purchase the item for them and have it sent to them. The ‘none of the above’ option, if provided, might enable the recipients to specify any item on the net up to the specified amount, for the service to purchase for them.

Nuggets about Nuggets across 38 years

To review what I’ve done on the topic of information nuggets, I’ve been trawling through the PAWDOC Index and files. The earliest example of sidelined text that I can find in my document collection was from October 1981 when I was working at the National Computing Centre. I can’t remember why I started to do it – but it may well have been prompted by the method that NCC’s Chief Editor, Geoff Simons, used to construct his books. He explained to me that he read everything about a subject, identified key points and put them on Post-it notes which he stuck on the wall. When he was ready to write, he rearranged the Post-its into separate sections and in sequence within the sections – and I saw examples of this in his office. At some point I started to employ this technique to construct the best practice books I wrote at NCC – but using the word processor on our new Zynar Office System to assemble and organise the key points. Sidelining text was an obvious way to identify key material to feed into that process.

Around 1994 I started talking with City University academics Clive Holtham and David Bawdon with a view to undertaking a joint project on ‘The Paperless Office Worker’. A key strand of this work would involve me digitising my PAWDOC collection. Extracts from emails between us in the early part of 1994 included the following:

Email from Clive Holtham: ‘….I don’t record each document as Paul does, but file at a quite detailed level. What I am conscious of is how much I forget about what I already have. The equivalent of underlining is important – we need to consider something more than keywords to store with each piece.’

Reply from Paul Wilson ‘…I agree with needing to deal with the underlining problem. I have sidebars on most of my material – they are the information nuggets; but I don’t know how much use they would be out of context.’

Email from Paul Wilson: ‘… with reports , papers etc. I usually mark the nuggets of info within them – presumably these are the bits of information I really want.’

The collaboration with City University proved very productive: Clive Holtham introduced me to a product Manager in Fujitsu who loaned me a scanner; and to the owner of a small company called DDS who loaned me the Paperclip document management software. Soon I was scanning my existing paper documents and new ones as they arrived. In January 1997 I issued my 3rd briefing note on these activities and included the following towards the end of the four page document:

Despite the close relationship between filing and information use, contemporary filing systems provide little other than title and index fields to support the knowledge acquisition, synthesis and use process. Filing systems, it seems, are there just to store items and to aid their retrieval. Unfortunately, the personal knowledge acquisition, synthesis and use process is not supported adequately outside filing systems either. Some standalone packages do exist, but they are not intended to be used in a day to day manner for personal knowledge acquired in documents, electronic files and other artefacts. In fact, even the need for such support is not widely recognised.

It is not yet clear to me what support could be most beneficial. However the clues are littered throughout the practices of knowledge workers like myself. For example, whenever I read articles, papers and reports I always mark the good bits – the nuggets of information. These are key points which I particularly want to augment the knowledge in my brain. Sometimes when I have been researching a topic I collect together all the nuggets I can find, categorise them and reorder them, and synthesise a new view of the topic in question. Unfortunately, like the magnesium nodules on the floors of the deep ocean, huge numbers of nuggets now litter my filing system unseen and inaccessible. I hope they are in my brain and that they have been used to develop my current state of thinking – but I’m not so sure that their huge potential has been fully exploited.

For my filing activities to really start adding value I need tools which can record those nuggets as I consume and index each item, and which can enable me to reorganise those nuggets, add more nuggets, and synthesise new nuggets, in the process of actively developing my ideas. Such tools would, of course, maintain the links to the original source material stored in my files. And my files would become a combination of original source material and the representations of my developing thoughts and ideas.

Now that I am confident that I have the paper scanning and electronic file indexing activities under reasonable control, it seems high time to start addressing the critical area of information use and its role in knowledge management.

These are the earliest mentions I can find of information nuggets in the PAWDOC collection – and they give no indication of where I picked up the concept from. In fact I can only find one published mention of the term and that was in the Lotus Notes-oriented magazine ‘Groupware and Communications Newsletter’ from April 1998 in which one Ted Howard-Jones gave a brief description of a service implemented by a major financial institution to capture competitive information. He wrote:

‘Called Report-It!, this service captures knowledge using a secure voice-mail system and delivers categorised information directly to the desktops of office-bound managers and competitive information professionals. These nuggets of professional information are disseminated via Notes.’

The use of the term Knowledge in this article, and in my briefing paper mentioned above, reflected the fact that, in the late 1980s, the term Knowledge Management started to became fashionable and by the late 1990s had become a holy grail of IT Professionals, Management Consultants, and Academics. The first mention of the term in the PAWDOC Index appears in 1990, and occurs in a further 137 Index entries from then to 2016. The company I worked for (Computer Sciences Corporation – CSC) was a global computer services organisation with tens of thousands of employees worldwide. Its involvement in the Knowledge Management topic came from three angles: first, its clients started to ask about it and how to do it; second, its consultants and salesmen saw it as a potential source of revenue; and third, its employees and management began to think that they needed it internally to improve the effectiveness of the business. Hence, as a consultant in the UK end of the business, I was aware of or got involved in:

  • A number of initiatives to develop an offering or information for clients, including:
    • the development of a KM service by CSC Netherlands in 1990;
    • discussions with CSC UK Management Consultants who were developing KM propositions, in 1996-8;
    • the definition and design of a KM service by CSC UK personnel to address opportunities in a number of clients including ICI Paints, LUCAS Engineering, John Menzies and United Distillers, in 1996-7;
    • the publication a CSC Research Services Foundation report on KM in 1998;
    • news of CSC’s Global Knowledge Management services, in 2001.
  • The development of such systems for clients (primarily using web pages on intranets), including:
    • a presentation to ICI Paints in 1992;
    • the development of systems for KM and for a web-based ‘Gazateer’ of all KM, architecture and other organisational information, for the Nokia SCC – a new organisation being set up by CSC UK for Nokia, in 1997;
    • the design and population of a web-based KM system for Dupont Agriculture’s architectural components, in 1998.
  • the development and use of internal CSC solutions, including:
    • attendance at internal CSC workshops on developing an organisational learning infrastructure in 1996;
    • the development of an improvement process for CSC’s new application development organisation that was being designed and built from scratch, in 1998;
    • the design of a practical KM programme for CSC UK’s reorganised Consulting & Systems Integration unit, in 1999;
    • knowledge management work being done for BAE, in 1999;
    • an internal Community of Interest on the subject of Personal Knowledge Management, in 2001.

Of course, my rather lowly personal filing perspective had to be rapidly expanded as I entered the Knowledge Management (KM) arena to accommodate both high level Management Consultancy notions of ‘Intellectual Capital’ and the distinction between Knowledge and Information; and the practical need to derive benefits from an investment in KM by effectively sharing the knowledge that had been acquired. Indeed, one of my contributions to the internal Community of Interest mentioned in the last bullet point above seems to herald a change in my thinking. My opening sentence reads:

‘Since collaborating with everyone in this shared space I’ve had my eyes opened to the concept that KM is all about enabling people to find things and work together, as opposed to the idea that KM is all about nailing down bits of knowledge and providing it to people. I realise that there is significant crossover between the two approaches – but nevertheless giving priority to one or the other will result in significantly different activities.’

During the 1990s I learnt a great deal about what people thought Knowledge Management was, and also about the powerful potential of the new web technology to support KM. However, throughout this period I don’t recall any specific conversations or documents about dealing with underlined, sidelined, or highlighted text. Indeed, by the time I completed the draft of the paper summarising my PAWDOC findings in June 2001, it seems that my ideas on the subject had advanced no further than that reported above. The paper was published in the journal Behaviour & Information Technology (BIT), and addressed the subject as follows in the section on ‘Areas of Investigation and Summary Findings’:

Q27. How can an electronic filing system be used to develop and use knowledge?

  • Include substantive information in the index entries, for example phone numbers, book references, and expense claim amounts.
  • Identify the nuggets of information (i.e. the valuable bits) when you first read a document
  • Capture and structure the nuggets into the overall nugget-base at the same time as indexing the item

Status: ideas formed

Q28. What is the best way to capture and structure information nuggets?

Probably by using a Concept Development tool. Some initial prototyping has been done using the Visual Concepts package and the eMindMaps package.

Status: ideas formed

Q29. Is it feasible and practical to capture and structure information nuggets as well as indexing items?

Status: not started

Q30. Is it worthwhile building and developing an information nugget base?

Status: not started

The Concept Development prototyping mentioned in the answer to Q28 above probably started in April 2001 when I acquired a free copy of the eMindMaps software, and by the end of 2001 I had started making MindMaps of books on esoteric topics such as the Egyptian pyramids and the origin of Atlantis. I have no record of my detailed intentions in doing this, but I guess I wanted to experience the process of recording all the nuggets I found in a book – and then to explore what could be done to integrate and exploit the material from several different MindMaps. In all, I made MindMaps of 19 books over a two year period; but that’s as far as I got. At the end of 2001 I started a new job in bid management and my energies were increasingly taken up with managing very intensive bids, with documenting the bid process, and with operating a Lessons Learned programme. I had no time to pursue these information nugget ideas any further.

This concludes my review of my previous activities in the use of information nuggets. The questions I posed in the 2001 BIT paper still remain largely unanswered, and the operation of the PAWDOC system has not provided any further insights on the subject since then. However, the existence of the sidelined documents, and of the 19 MindMaps, do provide an opportunity to undertake some rudimentary practical work to explore if the information nuggets identified were memorable and of any use. Subsequent entries will outline the methods that will be used to undertake these investigations, and will report on the results.

NBod1.0

I got to thinking in the shower the other day (probably prompted by the BBC’s Years and Years series) that we’ll be needing the concept of the NaturalBody sooner rather than later. NaturalBody people – NBod1.0 – have no piercings or permanent embellishments on the skin; have had no parts of the body deliberately removed, filled or cut (and that includes teeth and the sexual organs); and have not had surgery to alter their natural appearance. They have no permanently attached artificial physical appendages; and they don’t contain any embedded physical engineering equipment. NatBod1.0 people don’t have any embedded chips or other computer equipment, or software that connects them to any digital networks. NatBod1.0s have not had their DNA adjusted to enhance their or their offspring’s capabilities or appearance (the offspring will not be NBod1.0 as they will inherit their changed DNA status). NBod1.0s are simply people who are as they were born and who have developed naturally. They may have been born with disabilities, or have had accidents or illnesses – the notion of NBod includes no value judgements. In the western world there may be relatively few NBod1.0s (mainly babies and young people) as many people have had dental fillings and/or tattoos; but large numbers must still be up at the 0.8 or 0.9 levels. Going forward, however, we may need the concept of NBod to remind us of what humans are as we gain a growing capability to augment our bodies with technology. Even today, such a concept might help us make choices in the face of cultural and religious motivations to deface, cut, mutilate and remould our bodies.