Knowledge Nugget Endeavours

In 1981 I was working in the newly formed Office Systems team in the UK National Computing Centre, and I was interested in how the new technology could support the management of an individual’s office documents. So, a colleague and I decided to experiment with our own documents. This was the start of a still-running practical exploration of how to manage personal documents using digital technology.

It was my practice to highlight key text with a side line as I read documents, and, as my document collection grew, I began to wonder how I could make explicit use of this very specific information. No doubt the act of highlighting was in itself helping me to assimilate documents; but I wasn’t sure if all the highlighted facts were being retained in my brain and being used to develop new concepts.

During the 1990s, the trendy new topic of Knowledge Management emerged which provided a recognised arena in which I was able to explore these ideas. Sometime during this period, I latched onto the term ‘nugget of information’ (the first published mention of this in my filing index is in an article by one Ted Howard-Jones in the March 1998 issue of the Groupware and Communications Newsletter). My attempts to relate lowly personal filing to the Knowledge Management field eventually fizzled out in the face of much sexier concepts such as an organisation’s ‘intellectual capital’. However, in the early 2000s, I did make a specific attempt to see if I could use Concept Mapping software to capture nuggets, by applying it to 19 new age books on the pyramids and the like; but that is where my knowledge nugget endeavours ended.

Now that I’m trying to find a home for my document collection, and to identify the findings from its long term operation, it seems a timely moment to review this particular aspect, to do some practical work on the nuggets I’ve identified over the years, and to draw some conclusions on the topic.

Getting an HTTrack Copy

HTTrack is a free-to-use website copier. Its web site provides the following description:  “It allows you to download a World Wide Web site from the Internet to a local directory, building recursively all directories, getting HTML, images, and other files from the server to your computer. HTTrack arranges the original site’s relative link-structure. Simply open a page of the “mirrored” website in your browser, and you can browse the site from link to link, as if you were viewing it online.”

I downloaded and installed HTTrack very quickly and without any difficulty, then I set about configuring the tool to mirror pwofc.com. This involved simply specifying a project name, the name of the web site to be copied, and a destination folder. The Options were more complicated and, for the most part, I just left the default settings before pressing ‘Finish’ on the final screen. There was an immediate glitch when I discovered that I had not provided the full web address (I’d specified pwofc.com instead of https://www.pwofc.com/ofc/); but having made that change, I pressed ‘Finish’ again and HTTrack got on with its mirroring.  Some 2 hours 23 minutes and 48 seconds later, HTTrack completed the job, having scanned 1827 links and having copied 1538 files with a total file size of 212 Mb.

The mirroring had produced seven components: two folders (hts-cache and www.pwofc.com) and 5 files (index, external, hts-log, backblue and fade).  The hts-cache folder is generated by HTTrack to enable future updates to the mirrored web site; the external file is a template page for displaying external links which have not been copied; backblue and fade are small gif images used in such templates; and the log file records what happened in the mirroring session. The remaining wwwpwofc.com folder and index file contain the actual contents of the mirror.

On double clicking the Index file, the pwofc.com home page sprang to life in my browser looking exactly the same as it does when I access it over the net. As I navigated around the site the internal links all seemed to work and all the pictures were in place, though the search facility didn’t work. External links produced a standard HTTrack page headed by “Oops!… This page has not been retrieved by HTTrack Website Copier. Clic to the link below to go to the online location!” – and indeed clicking the link did take me to the correct location (I believe it is possible to specify that external links can also be copied by setting the ‘Limit’ option ‘maximum external depth’ to one, but my subsequent attempt to do so ended with errors after just two minutes; I abandoned the attempt). The only other noticeable difference was the speed with which one could navigate around the pages – it was just about instantaneous. From this cursory examination I was satisfied that the mirror had accurately captured most, if not all, of the website.

An inspection of the log file, however, identified that there had been one error – “Method Not Allowed (405) at link www.pwofc.com/ofc/xmlrpc.php (from www.pwofc.com/ofc/)”. According to the net, a PHP file ‘is a webpage that contains PHP (Hypertext Preprocessor) code. … The PHP code within the webpage is processed (parsed) by a PHP engine on the web server, which dynamically generates HTML’. Interestingly, I wasn’t aware of having any content with such characteristics, but, on closer inspection of the files in my hosting folder, I found I had lots of them – probably hundreds of them. I tried to figure out what the error file related to but had no clue other than its rather striking creation date – 23/12/2016 at 00:00:00 – the same date as several of the other PHP files. I had not created any blog entries on that day, so my investigation ground to a halt. I don’t have the knowledge to explore this, and I’m not prepared to spend the time to find out. My guess is that the PHP files do the work of translating the base content stored in the SQL database into the structured web pages that appear on the screen. I’m just glad that there was only one error – and that its occurrence isn’t obviously noticeable in the locally produced web pages.

The log file also reported 574 warning which came in the form of 287 pairs. A typical example pair is shown below:

19:31:13        Warning:    Moved Permanently for www.pwofc.com/ofc/?p=987 19:31:13        Warning:    File has moved from www.pwofc.com/ofc/?p=987 to                                           https://www.pwofc.com/ofc/2017/06/29/an-ofc-model/

I tried to find a Help list of all the Warning and Error messages in the HTTrack documentation but it seems that such a list doesn’t exist. Instead there is a Help forum which has several entries relating to such warning messages – but none that I could relate to the occurrences in my log. As far as I can see, all of the pages mentioned in the warnings (in the above instance the title of the page is ‘an-OFC-Model’), have been copied successfully so I decided that it wasn’t worth spending any further time on it.

All in all, I judge my use of HTTrack to have been a success. It has delivered me a backup of my (relatively simple) site which I can actually see and navigate around, and which can be easily zipped up into a single file and stored.

A Backup Hosting Story

In the last few days I’ve been exploring making backup copies of this pwofc Blog using the facilities provided by the hosting company that I employ – 123-Reg. It was an instructive experience.

When I first set up the Blog in 2012 I had deliberately decided to spend a minimal amount of time messing around with the web site and to focus my energies on generating the stuff I was reporting in it. Consequently, most of my interactions with the hosting service had involved paying my annual fees, and I had little familiarity with the control panel functions provided to manage the web site. In 2014, I had made some enquiries about getting a backup, and the support operation had provided a zip file which was placed in my own file area. Since then I had done nothing else – I think I had always sort of assumed that, if something went wrong with the Blog, the company would have copies which could be used to regenerate the site.

However, when I asked the 123-Reg support operation about backups a few days ago, I was told that the basic hosting package I pay for does NOT include the provision of backups – and the company no longer provides zip files on request: instead, facilities are provided to download individual files, to zip up collections of files, and to download and upload files using the file transfer protocol FTP. Of these various options, I would have preferred to just zip up all the files comprising pwofc.com and then to download the zip file. However, the zipping facility didn’t seem to work and, on reporting this to the 123-Reg Support operation, I was told that it was out of action at the moment… So, I decided to take the FTP route.

I duly downloaded the free-to-use FTP client, FileZilla, set it up with the destination host IP Address, Port No, Username and Password, and pressed ‘Connect’. After a few seconds a dialogue box opened advising that the host did not support the secure FTP service and asking if I wanted to continue to transfer the files ‘in clear over the internet’. Naturally I was a little concerned, closed the connection, and asked 123-Reg Support if a secure FTP transfer could be achieved. I was told that it could be and was given a link to a Help module which would explain how. This specified that a secure transfer requires Port 2203 to be used (it had previously been set to 21), so I made the change and pressed ‘Connect’ again. Nothing happened. A search of the net indicated that secure FTP requires a Port No of 22, so I changed 2203 to 22 and, bingo, I was in.

FileZilla displays the local file system in a box on the left of the screen, and the remote file system (the pwofc.com files in this case) in a box on the right. Transferring the pwofc files (which comprise a folder called ‘ofc’, a file called ‘index’, and a file called ‘.htaccess’) was simply a matter of highlighting them and dragging them over to a folder in the box on the left. The transfer itself took about 12 minutes for a total file size of 246 Mb.

Of course, the copied files on my laptop are not sufficient to produce the web pages: they also require the SQL database which manages them to deliver a fully functioning web site. If you double click the ‘Index’ file it just delivers a web page with some welcome text but no links to anything else. Hence, these backup files are only of use to download back to the original hosting web site for the blog to be resurrected if the original files have become corrupted or destroyed. I guess they could also, in principle, be used to set up the site on another hosting service – though I have no experience of doing that.

Of course these experiences only relate to one customer’s limited experience of one specific hosting service and may or may not apply generally. However, they do indicate some general points which Blog owners might find worth bearing in mind:

  • Don’t assume that your hosting service could regenerate your Blog if it became corrupted or was destroyed – find out what backup facilities they do or don’t provide.
  • Don’t assume that all the functions provided by your hosting service work – things may be temporarily out of action or may have been superseded by changes to the service over the years.
  • Remember that a backup of the website may be insufficient to regenerate or move the Blog – be clear about what additional infrastructure (such as a database) will be required.
  • If you want to be able to look at the Blog offline and independently of a hosting service, investigate other options such as creating a hardcopy book, or using a tool such as HTTrack (which is discussed in the following entry).

ST’s Alternative Approaches

About 6 weeks ago (on 6th March), Sara Thomson of the Digital Preservation Coalition kindly spent some time on the phone with me discussing the archiving of web sites. I wanted to find out if there were any other solutions to the ones I had stumbled across in my brief internet search some 16 months ago. Sara suggested 3 approaches which were new to me and described them as follows in a subsequent email:

  1. UK Web Archive (UKWA) ‘Save a UK Website’: https://beta.webarchive.org.uk/en/ukwa/info/nominate Related to this – two web curators from the British Library (Nicola Bingham and Helena Byrne) presented at a DPC event last year discussing the UKWA, including the Save a UK Website function. A video recording of their talk along with their slides (and the other talks from the day) are here: https://dpconline.org/events/past-events/web-social-media-archiving-for-community-individual-archives
  2. HTTrack: https://www.httrack.com/  I gave a brief overview of HTTrack at that same DPC event last year that I linked to above. I have also included my slides at an attachment here – the HTTrack demo starts on slide 15.
  3. Webrecorder: https://webrecorder.io/ by Rhizome. Their website is great and really informative, but let me know if you have any questions about how it works.

Shortly after this, I followed the link that Sara had provided to the UKWA nomination site and filled in the form for pwofc.com. On 14th March I got a response saying that the British Library would like to archive pwofc.com and requesting that I fill in an on-line licence form which I duly completed. On 16th March I decided to explore the contents of the UKWA service and found it collects ‘millions of websites each year and billions of individual assets (pages, images, videos, pdfs etc.)’. I started looking at some of the blogs. The first one I came across was called Thirteen days in May and was about a cycling tour – but it seemed to lack some of the photos that were supposed to be there. The next two I looked at, however, did seem to have their full complement of photos; and one of them (called A Common Reader) had a strangely coincidental entry about ‘Instapaper’ which provides what sounds to be a very useful service for saving web sites for later reading. It looks like the UKWA does an automated trawl of all the websites under its wing at least once a year, so I guess that, as a backup, it should never be more than a year out of date.

An hour after completing this exploration, I got an email confirming that the licence form had been submitted successfully and advising that the archiving of pwofc.com would proceed as soon as possible but that it may not available to view in the archive for some time due to the many thousands of web sites being processed and the need to do quality assurance checks on each. Since then, I’ve been checking the archive every now and again, but pwofc.com hasn’t emerged yet. When it does, it’ll be interesting to see how faithfully it has been captured.

Regarding the other two suggestions that Sara made, I’ve decided to discount Webrecorder as that entails visiting every page and link in a website which would just take too much time and effort for pwofc.com. However, I’m going to have a go at using HTTrack, and I’m also going to try and get a backup of pwofc.com from my web hosting service. Having experienced all these various archiving solutions, there’ll be an opportunity to compare the various approaches and reach some conclusions.

The PAWDOC Preservation story

In May 2018 the inaugural digital preservation work on the PAWDOC collection was completed. The story of the work that was done, and the lessons that were learnt, are documented in the following paper which can be downloaded from this site subject to Creative Commons conditions:

The Application of Preservation Planning Templates to a Personal Digital Collection

Instances of the populated preservation planning templates that were used to control the work are also provided:

A summary of the work done and the lessons learned has been published as a Blog Post on the Digital Preservation Coalition (DPC) website.

The preservation planning templates were updated as a result of insights gained in the work and these are available as embedded files in the above ‘Application of Preservation Planning Templates’ paper and also in the DPC website.

March: Long and Plans

It looks like the blog post describing the Digital Preservation work undertaken last year on the PAWDOC collection, will be published next month on the DPC website. It will refer to the full paper describing the work in more detail, which will be published here within pwofc.com. At the same time, the preservation planning document templates will be replaced by updated versions in the DPC website.  The publication of all these materials will be a fitting end to the preservation planning activities that are described in previous entries in this site. However, there will still be one thing to do before the topic can be considered complete and that is to review the effectiveness of the Preservation Maintenance Plan template when an instance of it will be used in the PAWDOC Preservation maintenance exercise scheduled for September 2021.

A few insights and conclusions

The sort-out of my publications, reports and CSCW proceedings (broadly categorised as ‘things I had created and done’) confirmed that I have a particular interest in material I had created or had made significant contributions towards. It was undoubtedly rewarding to revisit the material – though I wouldn’t anticipate doing it again very often. In fact, it made me realise that just having the knowledge that all the material is available and easily accessible, is itself a very satisfying and reassuring thought. Of course, having a complete collection of work documents to draw on when assembling full sets of my publications and reports, was slightly unusual; most people might only have partial sets depending on what particular material they had saved in the course of their careers.

The items included in the category ‘things I had created and done’ are only a subset of all the work items I’ve kept over the years. I have previously digitised over 80 of my work book collection as described in the Electronic Bookshelf journey; I’ve created story boards for 30+ work books that I regarded as special in some or other; and my PAW-PERS collection of memorabilia contains aver 120 other items in the following additional categories:

Formal job documents (offer letters, job specs, pension info, pay slips etc.): I originally kept these for reference; but now, of course they have become very informative pieces of memorabilia.

Company information (brochures, newsletters etc.): Many of these are well presented documents providing detailed information about the organisations I worked for.

Recognition objects (certificates, long term service awards, contract win artefacts etc.): I didn’t keep the originals of certificates confirming I had completed in-company courses as they didn’t seem very significant; however I do value a certificate from my professional body and keep it framed on my study wall. I’ve kept the cut glass paperweight celebrating a contract win, and the cut glass bowl for long service, which are both in our crystal cabinet; though they are retained more because of their looks than as reminders of work. I also value the long service domino set (very nice in a large wooden box) which I chose deliberately because I knew I would want to keep it long term for both its utility and its looks.

People I worked with (humorous documents, social gatherings, leaving cards, etc.): These are generally mementos of the people I worked with and the activities we did together.

Associated activities (company sports and social clubs, trade unions, professional bodies etc.) These are mementos of my activities in organisations associated with my work, and they are surprisingly prominent in my collection. I guess they such organisations have played a significant part in my working life over the years.

In thinking back about what I’ve done with all these different sets of work items, I was reminded of how sometimes particular items have corrected a fact that I had mis-remembered. For example, for several years, I believed that I was the instigator of the Alvey project I was involved in (Cosmos). However, in trawling through my documents to create one of the Electronic Story Boards, I discovered that it was a colleague who had been the instigator and I was a very ardent subsequent advocate. I guess that often we remember things in the way we would like them to be, not necessarily the way they actually were. Hence, having some documentation or other artefact can cast a truer light on the past. However, it must be remembered that the documents we have may only be a subset of all the relevant documents that were produced; and/or that their contents may just be reflecting the biases of the authors. Hence, whatever the nature of our ‘record’, be it memory, or a selection of the relevant items that you have, or all of the items that you have, or, indeed, all the relevant items that exist in the world, we should always remember that it may not be the whole story.

As with my non-work mementos, most of these work items have been digitised and the originals disposed of; though a small number, which I decided are special in some way or other, have been retained in physical form. In this respect, these work items are very similar to other types of memento. However, there is one very significant difference: many of these work items will not be recognisable by my wife and family. That’s because my work took me to a different place and a different life for a part of each day – as it does for very many people; hence, work mementos are likely to mean more to the individual than to family, relatives and friends. Consequently, I suspect that such collections are even less likely than other types of mementos to be retained and maintained by future generations of the family. I believe this to be almost certainly true for physical work mementos (I can’t see people hanging onto bulky books and papers which mean little to them). However, I’m less sure about digital collections which, in principle, are much less obtrusive and much easier to keep in the short term, but do rely on some care and attention as computers are replaced and technology advances. In fact, this uncertainty must apply to all informally-held digital collections – too little time has passed so far to be able to discern if such material is being passed down the generations. Interestingly, I do see the possibility of Artificial Intelligence playing a role in managing such material, and this could significantly affect how much of its digital history a family may have access to in the future.

In summary, this short review of my work mementos seems to have thrown up the following insights:

  • Categories of work mementos include; things the individual has created and done, work books, formal job documents, company information, recognition objects, people the individual has worked with, and associated activities.
  • While work mementos are similar to other type of mementos, they do provide reminders of a part of life that is often very personal to the individual and often separate from family life. In as much as work is often done with other people, it is almost like a parallel life with a separate family; hence, it generates a separate set of mementos.
  • Work concerns making, creating and doing things; and if individuals are in any way proud of what they have done, then they may well be keen to retain examples of what they achieved and to inspect them from time to time.
  • It is very satisfying and reassuring to know that examples of what you have produced at work, are safely stored away and accessible when you want them. Just being able to have those thoughts may be as rewarding as actually looking at the material.
  • Our ‘record’ of events is only as good as the material we have, be it memory or a few relevant artefacts, or lots of artefacts. We should always remain open to the possibility we don’t have all the facts.
  • Work mementos are probably less likely to be passed on down family lines than other types of mementos.
  • Physical work mementos are less likely to be passed on down family lines than digital work mementos.
  • Artificial intelligence may result in many more digital mementos of all types being passed on reliably down the generations.

Clear Blue Calm Water

Unfortunately, the paper summarising the PAWDOC digital preservation work has not progressed in the last few months because the DPC has too much work on at the moment to deal with it. I’m hoping this might change in the early part of 2019.

In the meantime, I have just completed another important aspect of digital preservation work on the PAWDOC collection. I have long been concerned that the collection resides on a laptop running Windows 7 – an operating system for which Microsoft have said they will withdraw support in 2020.  At the same time, the battery in my existing laptop no longer functions so requiring that it be mains-connected at all times. So, about a week ago I acquired a Chillblast Leggera i7 Ultrabook with 8Mb of RAM and a 1Tb Samsung Solid State Drive (SSD). I listed a set of conversion activities and started working my way through attaching peripherals (keyboard, mouse, scanner) and loading software (Anti-virus, Scanning, Filemaker, MS Office, Cloud backup). All went well until nearly at the end when I hit the wall of connecting the external Dell 2405FP monitor which I bought in 2006, and which has worked fine ever since with at least three different laptops.

I had planned to use the laptop’s HDMI port and had acquired an HDMI to DVI adapter to enable an HDMI cable to be plugged into the Dell monitor’s DVI port. Unfortunately, the connection only worked for a few minutes. After that the monitor’s DVI interface went into Power Save Mode and, no matter what I tried, I couldn’t get it out of that mode. I then tried searching the net for a fix and discovered a huge number of entries about this problem for several different models of Dell monitors stretching back to 2005 – with no definitive fix emerging. I decided to try using the VGA port on the Dell monitor and duly purchased an Amazon next day delivery of an HDMI to VGA converter. Unfortunately, this simply had the similar effect of putting the monitor’s VGA interface into Power Save Mode.

However, a ray of hope did appear when I plugged the VGA lead back into my old laptop, and the Dell monitor immediately came out of Power Save Mode and the screen image was displayed. I was able to obtain the monitor menu while it was attached to the old laptop and returned the monitor back to factory settings – but this didn’t make any difference – every time I attached the laptop’s HDMI port to either the monitor’s DVI or VGA interfaces they returned to Power Save Mode.

My last-ditch effort to resolve the problem was to try using the laptop’s Mini Displayport (MD) port, and, in a state of some depression and resignation, yesterday I duly purchased an Amazon same day delivery of an MD to VGA adapter plug.  It cost £5.99, was ordered around 9am and was delivered around 8pm (really…). With the laptop switched off, I put the adapter into the laptop’s MD port and plugged in the monitor’s VGA cable. The buttons on the monitor went orange (signifying Power Save Mode) and I thought, ‘here we go again’ and switched on the laptop; and suddenly after a few seconds I saw a bright light out of the corner of my eye and, blow me down, there was the laptop screen on the monitor! I used it for a while and then, trepidatiously, tried closing the laptop lid and it kept on displaying on the monitor. Later, I shut the laptop down and subsequently fired it up again – but still no problem – up it came on the monitor. So it looks like this is now working OK. Phew.

This morning I reorganised my physical desktop and placed the new smaller laptop in a new position immediately next to my scanner so that the problem of making the scanner cable reach the laptop port was eliminated. With the conversion process complete and my desk back in some sort of order, I began to feel more in control of things and much more relaxed. I had sailed into clear blue calm water in the sheltered bay of an up to date operating system and a modern laptop.

The power of the shower

This morning I finished digitising the CSCW conference proceedings, including the creation of a bookmarked contents list for each one (rather a tedious process), and downloaded the PDFs to the Sidebooks app on my iPad. Although this was a largely mundane exercise, I was stimulated from time to time when I came across author and project names that I had become familiar with while I was working in the CSCW field in the early 1990s. Reflecting on this in the shower this morning, I remembered my conclusion a week or so ago that, for memorabilia, the journey was often better than the destination. Suddenly, in a deluge of shower illumination, I realised that it was the remembering that had been fulfilling; and that the act of remembering is an act of doing; and that ­any thoughts about memorabilia – even just pondering the fact that they are where you put them and can be accessed when you wish – are ‘doing’ acts. It is when items cease to stimulate any thought or interest that they become worthless. Conversely, while items of memorabilia still inspire some physical or thinking action, they still have some value for the individual.

I continued to think through the meaning of this insight and concluded that it has significant implications for why we keep things; and that it will necessitate the adjustment of parts of the OFC tutorial text (though I must add that I’m sure these ideas are not new – but I have the luxury of not having to trawl the huge literature to see what has been documented before: that is the job of academics who should be appropriately paid to do a very difficult, laborious and hugely important job). This experience has cemented my belief in the innovative power of the shower, and makes me wonder just how important ablutions have been to the development of modern civilisation over the centuries.

Proceeding with proceedings

The final stage in this sort out of work books/documents concerns the seven volumes of proceedings of conferences on Computer Supported Cooperative Work that appear in the picture below.

The first of these events was held in Austin, Texas, in 1986 and I was there to experience the excitement of a new field being born. It was a field which embraced the Cosmos project that I was participating in at the time, and a field in which I actively worked for the following five years. In 1989 I was to organise the first European CSCW conference, EC-CSCW89. Hence, I am particularly attached to the proceedings of both the 1986 and 1989 events, though the 1986 volume is in poor condition with the cover having come away, and the 1989 volume is spiral bound. So, I’ve decided to turn both volumes into hardback books and to incorporate some additional pages of related material in the process.

As for the remaining 5 volumes, although I attended all but one of the events they document, I am unlikely to want to look at them in the future so have decided to digitise them and include them in my electronic bookshelf collection so that at least the covers and spines will be visible for decor and I’ll have the comfort of knowing that the text is immediately accessible should I want to take a look at it.