This investigation has focused on how AI can help the owners and users of personal collections; so, a good place to start this summation of its findings is to be clear about what collections consist of and which of their parts can be explored by AI.

About Collections

In this investigation, a collection is one or more items that a person has deemed to be a collection. There are several types of collection that we are all familiar with:

• Accumulations of items like food and clothing that are collected out of functional necessity
• Libraries of works recorded on portable media
• Files of factual information on specific media collected for future reference
• Archives of objects that have been moved from their original locations into long-term storage
• Museums & Galleries collecting objects and cultural phenomena for public viewing
• Private Curators who build valuable collections of original, fine, or rare objects for their own use
• Amateur Collectors who decide what they want to collect as a hobby and then seek out and assemble appropriate items at their leisure.

The primary element of a collection is, of course, the things being collected – physical objects, paper documents, jokes – whatever. The secondary elements are a) an index/catalogue of the collection items, and b) any digital files associated with each item. Of these three elements, AI can only interact with the latter two: current widely available AI systems are not able to interact directly and independently with the world – they have to be fed information in digital form to be able to explore and interpret it. In the future it may be possible for the average person to put a robot in front of a collection and ask it to explore – but that is not a commonly available capability today. So, the collection-related interactions that we can initiate with AI at present are limited to the following:

• General questions about a collection description provided with the question.
• Questions about the collection’s index
• Questions about the titles of the digital files associated with each item/index entry
• Questions about the contents of the associated digital file.
• Any combination of the above four.

The actual questions that might be asked about a collection are many and varied. However, they will almost certainly be associated with one or more of the things that owners and users actually do in connection with their collections. A hierarchical breakdown of these activities has been documented in the book ‘Collecting in the Icon Age – IT’s impact on Collecting Practices’, and this has been used to identify some of the questions that AI might be asked in relation to a collection.

About AI

Before discussing the investigation results it is useful to have an understanding of what AI is and how it works.

Artificial Intelligence (AI) is the broad field concerned with creating computer systems capable of performing tasks that normally require human intelligence. Recent advances in AI have been driven largely by developments in machine learning, particularly deep learning, leading to the emergence of Large Language Models (LLMs).

LLMs are trained on enormous quantities of text, during which they learn statistical patterns in language by adjusting billions of numerical parameters (called weights) within an artificial neural network. Rather than storing facts like a database, these parameters collectively capture relationships between words, concepts and ideas, enabling the model to generate coherent and contextually appropriate responses. When presented with a question in natural language [referred to as a ‘Prompt’], an LLM produces an answer by progressively predicting the most likely next token (roughly equivalent to a word or part of a word) until a complete response has been generated.

Some LLMs are also able to answer questions about images. This is achieved by adding another AI component, known as a Vision Encoder, which converts images into a numerical representation that the LLM can interpret. Similarly, Audio Encoders convert speech or other sounds into a form that the LLM can understand. An LLM combined with one or more of these specialised encoders is known as a multimodal model, because it can process and reason about multiple types of input, such as text, images and audio.

The Large Language Models that many people are aware of are those publicly accessible in the internet:

• ChatGPT from OpenAI (General-purpose assistant, coding, reasoning, multimodal)
• Claude from Anthropic (Long-document analysis, writing, coding)
• Gemini from Google (Search, reasoning, multimodal, coding)
• Grok from xAI (Reasoning, coding, web-aware assistant)

The other main type of model – Open Weight Models – can be downloaded and run on a local machine. The most well-known of these are (according to ChatGPT):

• Mistral from Mistral AI (Enterprise AI, local deployment, coding)
• Llama from Meta (Research, local assistants, fine-tuning)
• Qwen 3 from Alibaba (Multilingual, coding, reasoning)
• DeepSeek from DeepSeek (Reasoning, coding, mathematics)

There are actually many thousands of models – mostly variants and fine tunes of the well-known ones. Web sites like Hugging Face stock large numbers available for download.

LLMs are usually measured in parameters (roughly “weights” in the neural network). ChatGPT reports that:

• Small models have ~1 million to 1 billion parameters, take up 2-4Gb, and can run locally on phones/laptops.
• Medium models have ~1B to 20B parameters, and take up 20-40Gb.
• Large models have ~50B to 200B parameters, take up 200-400 Gb.
• Frontier / cutting-edge models such as ChatGPT, Gemini and Claude, have hundreds of billions to possibly trillions of parameters, and take up 1T to multiple terabytes.

The names given to models often include the letter ‘B’ (like “Mistral 7B Instruct”) and this refers to the number of billion parameters they have.

As indicated earlier in this piece, LLMs work with tokens. In the course of the tests conducted in this investigation, I was told a number of different rules of thumb by various LLMs:

• generally, one token equals approximately 0.75 English words
• there are 2-3 characters/token
• a token is roughly 3–4 characters of English text (Copilot).

However, Copilot also explained to me that files such as my spreadsheet .csv file, with long lines, punctuation, quotes, and irregular structure, tokenize very inefficiently and dramatically increase token count.

This uncertainty about the token-to-character ratio is highlighted because it affects how much information the AI can take into account when answering a prompt. AIs work by getting a model’s parameter-set to work together with a separate set of information (the prompt and any supporting files) to progressively predict the next token. The amount of separate information that an AI model can take into account in a single conversation is limited and is known as its ‘Context Window’. If you want to ask a question about, for example, a complete collection index, you have to include the whole of the index in the prompt or in supporting files. If the combined size of the question and supporting files and the answer and any previous conversations included, exceeds the Context Window, the overrun in the supporting files simply gets truncated and will not be considered in the workings of the AI.

In the course of these investigations, I established the following Context Window sizes:

• Mistral supports around: ~32k tokens context (varies by version).
• ChatGPT’s usable context window is about 128k tokens
• Claude’s context window is 200k tokens
• Copilot doesn’t have a fixed context window – its design means that it’s effective context window is much larger and more flexible than a single token number would suggest.

These are not the only limitations likely to be encountered when using an AI LLM – others include:

Max No of files in one conversation: Mistral – no limit up to context window; ChatGPT – 20; Claude – 20; Copilot – about 7 batches of up to 20 files.

Max size of any one file: Mistral – 512Mb; ChatGPT – 512Mb; Claude – 30Mb; Copilot – 50Mb.

Max total size of all files in one conversation: Mistral & ChatGPT – no limit up to context window; Claude – 20 files x 30Mb each = 600Mb; Copilot: 20 files × 50Mb each = 1Gb.

Limits associated with a specific time period: Mistral – no limit; ChatGPT: 3 files in one day (though up to 20 will be accepted across 3 zip files), Claude – a ‘modest allowance’ that resets on a rolling ~5-hour window; Copilot – up to 20 upload events per 24 hours (an upload event can include 1 file or 20 files).

NB. These figures are primarily for free versions of the models concerned; paid for plans may be different. They are provided only to indicate the type and scale of limitations that exist. They will almost certainly change as time goes by.

In order to get round the Context Window size limitation a technique called Retrieval-Augmented Generation (RAG) has been developed. RAG stores all the material you may want the AI to work on, in chunks of around, for example, 1000 characters, in the form of vectors – mathematical representation of tokens such that related items are clustered together thereby enabling capabilities like similarity searching. This is referred to as ‘embedding’ in a Vector Database. Each vector (i.e. each chunk) is represented by hundreds or thousands of numbers with 6-8 decimal points, for example, 0.143827 or -0.872913. When a prompt is created, it too is embedded and the chunks which most closely match it are identified and sent to the model together with the prompt. In this way, only the most relevant chunks of a large set of material are sent to the model’s context window.

Collections used in the tests

I acquired the above understanding of AI in the course of carrying out a series of tests between April and July this year. The tests were all conducted on one or other of two of my own collections – Mementos and PAWDOC. The Mementos Collection is an assembly of memorabilia from 1949 to the present day. It was extended in the 2020s to include computer artefacts and other items retained for show in a display cabinet; as well as memorabilia from a relative. Items in the collection are diverse in nature. Many are simply photos of objects. All collection items are recorded in an Excel spreadsheet which has 2,394 Index entries. All items are in digital form with only 1,004 of the index entries being retained in physical form. There are 2,751 digital files taking up 15.7Gb, associated with the index entries.

PAWDOC is a digitised collection of most of the work documents I received, read and produced over the period 1981 until I retired in 2012. It continues to be populated with selected material from work I continue to do under the general heading of ‘Order From Chaos’. The collection was started as part of my work on Office Automation at the National Computing Centre (NCC) in Manchester with the aim of gaining some practical experience of what it would be like to work in an electronic office. As time went on, the practice of indexing all my documents became second nature to me and the use of the filing system to control my paperwork and to find and retrieve documents at will, became an integral part of my working life. Many different types of documents are stored in PAWDOC including letters, internal memos, circulars, reports, specifications, minutes, overhead slides, 35mm slides, notes, training materials, brochures, manuals, maps, emails, computer magazines, journal articles, conference proceedings, and videos. Electronic documents such as word processor files, spreadsheets, presentations and web sites are also included. All documents in PAWDOC were either born digital, or have been digitised by scanning or photography, and the paper destroyed. A small subset of about 340 significant physical documents have been retained in physical form. In July 2026, the digital collection comprised some 31,324 digital files taking up approximately 51Gb of storage space. The PAWDOC Index is held in a Filemaker database with 17,381 Index entries.

Preparing for the investigation

The first thing I did was to try and apply the advice in the recent publication “AI preparedness guidelines for archivists” by Prof. Giovanni Colavizza and Prof. Lise Jaillant, to the Mementos collection. This resulted in 14 actions being identified, 12 of which I was able to complete in short order. I didn’t have the experience or knowledge to address the other two (identifying evaluation metrics, and changes to support programmatic retrieval), and dealt with them in the course of the tests I carried out.

Conclusion 1: Don’t address evaluation metrics and support for programmatic retrieval until you start trying out some AI software.

Installation of an Open Weight model on my laptop

After extensive exchanges with ChatGPT about how I could interrogate my Mementos index and associated files, I decided to try a RAG approach on a model loaded on my laptop. ChatGPT guided me through the installation of three components which it recommended for my 8Gb laptop:

• AnythingLLM → the interface and workflow tool you interact with.
• Ollama → the engine that runs the AI model locally.
• Phi-3 Mini → the AI model – the “brain” that generates text.

I used this for a week but found I was getting “Ollama not responding” more often than not when I sent in my prompts, and I eventually concluded that my 8-year-old laptop with 8Gb of memory just wasn’t up to the job. So, I brought forward my replacement plans and acquired an HP Omen laptop with 64Gb of memory, and duly installed AnythingLLM/LM Studio/ Mistral 7B Instruct. Unfortunately, this didn’t work, so ChatGPT then advised AnythingLLM/Ollama/Mistral:Latest  (which I believe is Mistral 7B v0.3 (Instruct) – 4.2Gb) – and that did work. After using that for a while, I also downloaded Mixtral 8×7B (26Gb) to see if it would give better results.

Conclusion 2: Use AI to guide you through product selection and installation – but be aware it may not always be right. Try and find a knowledgeable human to get advice from if you need it.

First RAG tests

I knew I needed some evaluation metrics for my first tests and decided to go with a simple set of questions about the Memento collection that I could establish the answers for independently of the AI by doing an Excel Find on the Description column in the Index. Then I could give a percentage correct score to each of the AI answers. The six questions I used were:

• What items are to do with the KRS? [KRS standing for Kodak Recreational Society]
• What happened on the 20th?
• List the items relating to exam results.
• What linen is in the collection?
• Are there any items relating to Aston Martin cars? [There are some individuals called Martin in the Index]
• What documents are there about finances?

The scoring was actually a little more complicated than I had envisaged because the AI also produced other correct answers not identified by the Find search AND incorrect answers. So the formula I ended up using was:

(Number of items identified by FIND that the AI had reported correctly +                          Number of additional correct answers identified by the AI) (Total correct answers); divided by: (Number of answers identified by the FIND + Number of additional correct answers the AI identified + Number of incorrect answers that the AI identified) (Total number of answers overall).

5 tests were performed on each of the Mistral and Mixtral models, each one with a different variation as described below.

• Index Guide included: The Guide was quite a detailed description of the Mementos collection amounting to 12,418 characters.
• Normalised Facets as advised by ChatGPT: This change mainly involved eliminating plurals and putting all items in lower case (two FGacet columns are included in the mementos Index).
• Primary Facet and AI Context (PF&AIC) as advised by ChatGPT: Added a Primary Facet column, and an AI Context column (which combined the Reference Number, Description, and Facets fields].
• PF&AIC with Context snippets delivered increased to 40: This increased the number of chunks delivered to the model from 4 to 40 (this number is a variable between 1 and 40 which can be set by the user; it was originally defaulting to 4).
• One item in a single Excel cell as advised by ChatGPT: This combined all 18 columns of information for each item into a single cell using a 19-line Excel formula provided by ChatGPT. Context snippets remained at 40.

The results were as follows:

These test results seem to suggest that AI is a lot worse at word search than an Excel spreadsheet; and that making various changes to the structure of the Index being tested seemed to make the AI perform worse not better. Even increasing the number of chunks delivered to the model didn’t seem to improve things. On the plus side, however, there were several instances of the AI correctly identifying relevant items without the exact words in the question being present in the item’s record. Unfortunately, a number of instances also occurred in which the AI hallucinated and made things up.

ChatGPT suggested a number of reasons for these occurrences: chunks may have combined multiple index entries which is not ideal; Indexes are often short, compressed, and keyword-based, whereas AI embeddings work best with descriptive sentences; AI is probably at its best when asking exploratory questions; and, finally, ChatGPT suggested using the following text constructs in prompts when needed:

• “Using ONLY the provided documents” to reduce hallucination
• “Do not invent information” to force restraint
• “If unsure, say…” → prevents guessing

Indeed, the more testing I did the more important I found this latter advice to be: Prompts need to be explicit, very clear, and complete. The more detailed a prompt is, the more likely you are to get a better result from the AI.

At this point I decided to do further tests with a different set of questions.

Second set of RAG tests

For these tests I chose the following new set of more general questions:

• What are the main themes that run through the entire index?
• Are there distinct phases or periods in the collection?
• Which items suggest important life events or transitions?
• What patterns or motifs repeat across the collection?
• What are the top 5 most important entries, and why?

To evaluate the answers, I had to rely on my own subjective assessment since there was no easy way of establishing what the correct answer should be. I scored each answer out of 10 and specifically reduced the score for any errors or hallucinations that I spotted. The results may not be absolutely correct but at least they should be consistent and comparable with each other.

Before undertaking this second round of testing I established that a) if an AI’s RAG answer involves more chunks than the model’s context window allows then its just too bad – some chunks won’t be passed to the model and the model’s AI response will not reflect the missing information; and b) an alternative to using the RAG approach was to simply attach the whole of the Memento index file to the prompt – but that that wouldn’t work either as the whole Index is bigger than the Context Window. With this conundrum running round my head, I thought I’d try out the new questions not only with a couple of different versions of the Index using RAG with Mistral and Mixtral, but also with the Copilot AI which was sitting in the task bar of my windows screen – in this case just submitting the whole index with the Copilot prompt (though Copilot would not accept it until I had reduced its size to 346Kb). The results were striking:

Copilot was a clear winner. I then tried out another version of the index with all fields in a single cell, and with the description field replaced by keywords (this transformation achieved using an Excel formula provided by ChatGPT). This time, however, I split the Copilot file into three files of between 200kb and 300kb each because I had a nasty suspicion that the original file might have been truncated. I also tried using ChatGPT as well. The results are in the table below – with Copilot’s score even better this time.

Copilot’s score was all the more remarkable because I specifically asked Copilot if any of the files had been truncated and it confirmed that only about the first 120 lines of each were ingested – making about 360 entries in all out of the total 2,393. This prompted me to investigate further and I established that Copilot will ingest up to 20 whole documents in a single turn but only up to about 30,000 characters/30kb file size each; and will work across all those documents to construct its answer. I duly split the Mementos Index into 17 files, all between 20,000 – 30,000 characters long, and conducted the Copilot test again, this time including a version of the Guide document. This produced an even better score of 9.1.

In a final set of tests, I dramatically reduced the size of the Mementos index by excluding the set of relative’s mementos, and spilt it into 3 files of around 90Kb for ChatGPt and 14 files of 30Kb or less for Copilot, with the results in 6a-6d in the table below:

Copilot was a clear winner with ChatGPT following on behind. Both provided substantial answers which included rationale, several examples, and a summary. In contrast the MIstral RAG answers were sparse, sometimes not very good, and occassionally included complete hallucinations. Below are my overall conclusions from all the tests conducted to this point:

Conclusion 3: The RAG approach isn’t very effective for Index material.

Conclusion 4: Collection Indexes may contain information which is more unevenly spread throughout their contents than a written document with an ordered set of contents. Consequently, such indexes may require more RAG chunks to be sent to the LLM, and may require the LLM to have larger Context Windows when ingesting files, than ordinary documents in order to obtain satisfactory answers.

Conclusion 5: Different LLMs are trained with different amounts and types of training data, thereby making them more or less suitable for operation within specific domains. Therefore, there may be some LLMs available that are more suitable than others for use with Collection indexes.

Conclusion 6: If you need to manipulate an Excel Index for use in a prompt to an AI, it’s worth trying to get the AI to provide an Excel formula to achieve the result you are looking for.

Tests on both Index entries and file titles

The next set of tests explored how AI can support the combination of a collection’s Index and the titles of the associated files (assuming, of course, that the file titles contain meaningful text and not just a series of letters and numbers). I tried this out by listing the titles of the 2,065 files associated with the Mementos index, in a separate 73kb file. ChatGPT wasn’t used because it has a limit of around 250kb per conversation and the file titles together with the Index significantly exceeded that. I also tried out Anthropic’s Claude model for the first time. The results are shown in the table below together with the results of the last set of previous tests (6a and 6c) for comparison purposes.

From these results, it’s not possible to assess if the inclusion of File Titles has made a difference: However, it seems reasonable to assume that if there is additional information in the file titles, over and above that in the Index, there will be a better outcome.

At this point, three more questions were added to the new set of 5 already being used, to further test the capabilities of the AI models concerned:

• Describe the things that happened to Paul Wilson during the time he spent at Barlborough Hall School and support the story with references to relevant mementos.
• Identify anything strange or unexpected about the contents of the Memento collection and support the analysis with references to particular files.
• Describe life in the Wilson family during the 1990s citing relevant artefacts in the collection.

These extra questions were deliberately designed to explore the AIs ability to make more broadly-based connections and inferences, as opposed to identifying related items in a narrow subject area. This is where AI can excel – and sometimes fail – as indicated in the results below:

Conclusion 7: File titles can certainly be included in the material delivered to an AI model, and this is worth doing if they include substantive material (i.e. not just identifying letters and numbers) which is additional to that which is contained in an Index – or if an Index does not exist.

Tests of text and Image content in files

Having discovered that multi-modal LLMs could not only work with textual content, but also with scanned text, and with images in general, I defined the following tests to explore their capabilities:

Tests of Machine-readable text

• Describe and summarise the contents of three years of diary entries in Word format.
• Discuss any relationships that can be found between three Word files with diverse contents: my library loan history for 2004-2012; an account of the petitioning of a school’s teachers to make a change to daily activities; some thoughts about university life while in the infirmary recovering from German Measles.

Tests of Image-only scanned text

• Summarise Friends of the Earth activities in Harrow as documented in three image-only scanned text documents from 1976-1979 in PDF format.

Tests of Text in images

• List all the events and activities described in three documents of events, tickets, membership cards etc.
• Describe and summarise the contents of all the images in three pages of Italy stamps which also include cutouts from the relevant parts of stamp catalogues.
• Catalogue the contents of the three pages of Italy stamps images using the following fields: Reference Number, Country, Year, Value, Notes.

Tests of collections of objects in images

• Describe and summarise the contents of all the images in three photos of chocolate wrappers, with each photo showing a) a double page of a chocolate wrapper scrapbook (in an unused 1967 A4 diary); and b) 3 plastic wallets containing loose wrappers.
• Catalogue the contents of the three chocolate wrapper photos using the following fields: Reference Number, Name, Manufacturer, Type, and Size.

• Describe and summarise the contents of three photos of household ornaments showing a) 10 pieces of Wedgewood, b) 30 small display items; and c) 13 sundry items.
• Catalogue the contents of the household ornament photos using the following fields: Reference Number, Name, Type, Colour.

All 10 tests were applied to the four AIs that had been used in the previous phase – AnythingLLM with Mistral, ChatGPT, Copilot and Claude. I had been using the free version of ChatGPT, Copilot and Claude up to now. However, when I started doing these tests ChatGPT suddenly changed the number of files it was allowing me to upload each day from 3 to 2. Since all the tests involve 3 files I elected to upgrade to ChatGPT-Go which enables you to “usually upload far more than the Free tier’s 3 files/day, but there is still a rate limit, and OpenAI hasn’t publicly stated the exact number.” The cost was £7 a month with the ability to cancel anytime. I encountered no limits when I was conducting these tests with the free versions of Copilot and Claude.

Each individual test was evaluated in two ways: first, I looked at the answer provided and wrote a short paragraph reflecting what I thought about it; and, second, I awarded a score out of 10 according to what I’d written.  As with the last series of tests, this was a completely subjective, and a potentially highly inaccurate, process. However, it employs a single observer with a consistent mind-set which hopefully enables comparative conclusions to be drawn. The table below provides an overview of the results.

Claude comes out a clear winner in these tests, with ChatGPT coming in second. Copilot, while performing excellently with text, appears to have had less relevant image training.

Conclusion 8: AIs can work extremely well with both text and images; and could be very useful to collectors in identifying items, describing them, cataloguing them, and creating indexes for them.

Tests of File Splitting and Result Stitching

The tests on the relatively small Memento collection had made it clear that good results could be obtained from Collection Indexes and file titles provided they could all be considered at once in an AI’s Context Window. However, the PAWDOC collection which I now turned to was about six times bigger and would certainly not fit. The solution I decided to try out was to split the index and associated file titles into subsets that would fit into a Context Window, and then to request the AI to stitch the result from each subset together into an overall answer.

For the PAWDOC Index, I exported 4 fields (Reference Number, Title, Publication Date, and Creation Date) into a file which contained 17,381 entries composed of just over 2,400,000 characters and sized at 1.4Mb.

The titles of the associated PAWDOC files typically contain a Reference Number followed by a free text description with a creation date at the end. I created a file of all these titles which contained 31,270 entries composed of 2,568,000 characters and sized at 1Mb.

Copilot and Claude’s limits suggested that each subset should consist of a maximum of 20 files each containing a maximum of 30k characters. I knew this would exceed ChatGPT’s limits, but I thought that I could do without ChatGPT if its results were poor. I had already decided that I would upload a copy of the ‘PAWDOC Guide‘ file in every prompt; and that I would count the text request as a file in its own right; so that left a maximum of 18 files of Index and Filename information in each prompt. Using these criteria the splitting exercise produced 10 subsets (which I named A-J) of either 17 or 18 files each.

I then set about getting each of the AIs to answer question 1 with Subset A. As it happened, there seemed to be no problem with ChatGPT: it happily accepted all the files in a zip file, and advised that no files were truncated when it analysed them. This didn’t seem to concur with the ChatGPT context window information I had established earlier, but the results seemed reasonable so I decided to go ahead and use ChatGPT in the testing. So too with Claude (which, by the way, also was happy to ingest all the files compressed into a single zip file). Copilot, however, produced inferior results which I established, after some Q&A with the AI, was due to the files being truncated. It was clear that I wasn’t going to be able to conduct the tests with Copilot using the same subsets that I had already spent a lot of time and effort creating, so I excluded Copilot from the tests.

I used the following six questions for these tests combined with very detailed prompts:

• List all the people named in this part of the Index and its associated files, and the organisation they belong to if any.
• Describe Paul Wilson’s career over the period covered by this part of the Index and its associated files.
• What significant changes in Information Technology occurred during the period covered by this part of the Index and its associated files?
• Document all the travel undertaken by Paul Wilson over the period covered by this part of the Index and its associated files.
• What training was undertaken by Paul Wilson over the period covered by this part of the Index and its associated files, and how important were particular elements to his subsequent career?
• What are the strangest or most unusual things to be found within this part of the Index and its associated files, including unlikely coincidences, and events with unexpected outcomes?

For each of the six questions I got 10 answers for ChatGPT and 10 for Claude – 120 answers in all. Then I asked ChatGPT and Claude respectively to combine the 10 answers they had produced for each question into a single merged answer, which produced 6 merged answers from ChatGPT and 6 from Claude. I reviewed each of the 132 answers, and gave each of the merged answers a score out of 10; the results are summarised in the table below.

Claude has clearly produced the best results – and perhaps the ‘time to produce’ numbers indicate why that should be: ChatGPT took an average of 44 seconds to produce its merged answers, and Claude took an average of 962 seconds. Indeed, in some cases, Claude’s merged reports are so detailed and so well formatted that they are too believable – and this is their downfall: to reproduce such comprehensive results, or to verify the answers, would probably take weeks of manual work, and so, in view of this it is very tempting to just assume they are totally correct. However, although I didn’t diligently check every aspect of some of these extremely detailed reports, I did spot a few errors which suggests there may be several more across this body of material. Examples include incorrect dates; interpreting a joint seminar as joining an organisation; suggesting I was thinking about doing a diploma because I’d received a letter asking me to publicise a course; and courses ascribed to the wrong organisation. These errors may have been due to lack of material in the subset, or to a misinterpretation of the information provided, or to limited information in Index entries without access to the contents of the associated files: but, however they were caused, they provide an important reminder that all LLM material needs to be checked if you want to rely on its veracity.

Conclusion 9: Splitting up Collection information into subsets and then getting the AI to stitch together the subset answers can provide a very effective way to interrogate collection information.

Conclusion 10: No matter how comprehensive and impressive an AI report on collection information might be, it will almost certainly contain errors of some sort or other.

Conclusion 11: Researchers using the ‘subset and stitch’ approach to interrogating collection information would be well-advised to specify verification activities as an integral part of their plans for the exercise.

Exploring the digital files in a collection

The ‘subset and stitch’ approach used to examine a collection’s Index and Titles of associated files, can also be applied to the contents of the associated files. This was tested by investigating two topics in the PAWDOC collection. One concerned the two years (1977-78) that I worked for a company called ‘CPC’; and for the other I chose the word ‘measurement’. In each case, I performed a search of the relevant term (‘CPC’ or ‘measurement’) on my PAWDOC index in the Filemaker application. These produced lists of 21 and 40 index entries respectively. PAWDOC index entries can have any number of files associated with them, and in these cases, it turned out that ‘CPC’ had 48 files, and ‘measurement’ had 72 files, associated with them.

Using the limits I had previously established for Claude and Copilot, I chose to limit the subsets as follows:

• 19 files to a subset (this to include the ‘PAWDOC Guide’ file and a file of the Index entries related to the files in this subset: leaving room for 17 content files)
• Text files to be no larger than 18k characters.
• Max individual file size: 28Mb
• Max Total file size: 40Mb

In preparing the subsets I encountered three issues:

• TIF files: Most of the PAWDOC files are scans of hardcopy documents in multi-page TIF files. However, the free version of Copilot does not support TIF files; and Claude only supports single page TIF files, so I had to turn all the TIF files into PDFs. As it turned out, that was a quick process: select a file; right click the mouse and choose ‘Open with [PDF App]; when it opens in the PDF app, select ‘Save As’ and save it as a PDF in the relevant Subset folder.
• One of the files (a scan of a desk diary) was 35.7 Mb – a little over my Individual file size limit of 28 Mb. So, I split it into two files of 21Mb and 15 Mb respectively.
• One of the files was an old PowerPoint file with a ’ppt’ extension. I believe Claude and Copilot only accept pptx extensions so I produced a pptx version for the subset.

Having dealt with these problems, I ended up with 5 subsets for each question. I then created a prompt for each question, and ran through the subsets using Claude without difficulty. However, for Copilot I encountered a daily upload quota and consequently abandoned the Copilot Measurement test.

The results are summarised in the table below.

Three things are apparent from these results: a) Copilot operated in seconds, whilst Claude operated in minutes; b) Claude’s two answers were both very good; and c) using the ‘subset and stitch’ approach seems to work just as well for the contents of digital files as it does for Index entries and File titles.

Conclusion 12: AI can be effectively used to investigate a collection through the contents of its digital files – using the ‘subset and stitch’ approach if necessary; and the results are likely to be even more impressive than those that can be obtained when providing the AI with just index entries and file names.

Conclusion 13: The better the results from research into collections by using AI, the more research is likely to be undertaken and the less time researchers are likely to spend verifying the results. This is a serious long-term issue for archives, for researchers, and for the integrity of the global information canon.

This is the end of the report on findings from the investigation into AI support for personal collections. To see all the Conclusions collected together in one place, click this link. For completeness, below is a breakdown of time spent on the various phases of the journey.

My PAWDOC work archive of some 17,000 Index entries and 31,000 associated files has an Index in the FileMaker database app. That was the app I chose in 1986 when I moved the index from a card-based system – and I’ve stuck with FileMaker ever since despite it becoming a hugely more complex and capable beast. A few months ago, I discovered that FileMaker was offering a free 45-day trial of its latest AI-enabled version, so I thought I’d try it out as part of this investigation into AI for personal archives. I figured that it would help me understand a bit more about AI, and that perhaps what FileMaker was doing would be generalisable to all specialist archive-supporting apps. This post summarises what I discovered and concluded.

There is an extensive range of videos and training materials for FileMaker available online. I had already looked at a number of the videos as part of my Phase 4 investigations, and now I went through two of the training modules:

• AI Fundamentals I: Semantic search in Claris FileMaker
• AI Fundamentals II: Using LLMs in Claris FileMaker

These describe the following three AI-related features:

• A Semantic Search facility
• Generate response from model
• Perform Find by Natural Language

These features have been built into FileMaker using about a dozen AI-related FileMaker Scripts – sets of predefined instructions carried out by the application when a particular script is run by the user.  The Semantic Search works by first generating the vector value of the contents of a field (see Terminology in the post at this link), putting that value into a file, and storing that file with the relevant record in the database. This is known as embedding and is enabled by one of the pre-built AI scripts called ‘Insert Embedding in Found Set’. Another script (Perform Semantic Find) turns a natural language search term input by the user into a specific vector value, and compares that vector value with the vector values for the records which have already been given vector values. The result of the comparison is a Cosine Similarity Coefficient ranging between +1 and -1 and typically calculated to 6 or 7 decimal points. FileMaker then lists the top X records which have the highest Cosine Similarity score (where X is a user-specified variable). A more sophisticated version of this is a hybrid version of this search which provides a screen in which the user defines BOTH a search term for the FileMaker Find facility to derive a subset, AND a natural language search request to be used on the contents of a specific field in all the records in the subset. In both the Ordinary Semantic Search and the Hybrid Semantic Search, the vector embedding files must already be in place in the database: records without a vector embedding file will not be included in this search. As with all AI usage, there may be limits or cost implications related to the number/size of requests that are made to the model.

The Generate Response from Model feature enables a general-purpose interface to be created with a local or remote AI model. This can be used to gather information from the database and for that information to be sent with an instruction Prompt to the model. The response from the AI is then displayed in FileMaker in a pre-specified way. For example, information in certain fields in a Product Database could be sent to the AI together with instructions to provide single line summaries of each product consisting of short description, category and price. The specific information to be gathered, and the instructions, could all be pre-specified in particular scripts, or could be defined by the user. Inevitably, the scripts to do this may be a little complicated; and appropriate interface screens (‘layouts’ in FileMaker terms) will need to be built. As with the Semantic Search feature, there may be limits or cost implications related to the number/size of requests that are made to the model.

Unlike the previous two features, the Perform Find by Natural Language facility does not send any of the database contents to an AI model. Instead, it sends the prompt request created by the user along with further instructions plus the database schema. The AI then creates and returns the Find text which FileMaker includes in the relevant fields before running a Find operation.

The training courses did a good job of explaining how all three of these features work. I then tried out the Semantic Find function on my PAWDOC database using the trial 2026 version of FileMaker. This involved creating additional scripts around the primary scripts to specify things like the AI model being used, the fields for storing inputs and outputs, and the particular fields to be interrogated in the database. I initially followed the example of the training courses and obtained a trial API key to be able to access models in the cloud from the Cohere provider. However, I also followed the steps described in a very informative blog post called ‘Using LM Studio as a local LLM for FileMaker AI Development‘ which showed how the same three features could be used with a local model residing on your own local machine. The examples in the blog post use the LM Studio app which I happened to already have on my laptop. So, I ended up carrying out a set of tests using both a remote and a local AI model. The tests were performed on the PAWDOC Title field to a) embed the contents, and b) perform semantic searches on a subset of the PAWDOC records. The models used were LM Studio’s “text-embedding-nomic-embed-text-v2-moe” on my laptop, and Cohere’s “embed-english-light-v3.0” in the cloud.  For comparison purposes, I also conducted some equivalent searches using the standard FileMaker Find function. My observations from these tests include the following:

Embedding

• The contents of the Title field in most of my index entries probably contain between about 4 and 30 words. I found that for both models, embedding one record took less than a second, and to embed 60 records took about 1 or 2 seconds. 2803 records took LM Studio 29 seconds to embed and Cohere probably took around 50 seconds (I couldn’t get an accurate Cohere figure because of usage limits associated with my free Cohere API key).
• It took the LM Studio model 3 minutes and 4 seconds to create embedding files for all 17,381 PAWDOC records; and in the process it added about 25Mb to the 15Mb file.
• Embedding file names from both models were similar. For example, an LM Studio file was called emb_05D4B.fve, and a Cohere file for the same record was called emb_2E1C6.fve.

Quality of Search Results

• Both LM Studio and Cohere had been set up to produce a list of the 10 most relevant hits. However, many of the results seemed to be of dubious relevance, for example, the results of a search on ‘shipping industry’ included the record ‘CSC top ten clients worldwide’; and the results of a search on ‘work in the oil and gas industry’ included the record ‘CSC H&S guide to working alone’.
• In some cases, the semantic search results seem to fail almost completely to pick up relevant records. For example, while most of the semantic search terms did not produce any hits at all when used in a standard FileMaker search, one (UK management) did produce 25 hits. However, only 40% of those hits actually appeared in the semantic search results – three of them appeared in both the LM Studio and Cohere lists, and 2 appeared in one or the other.
• In a test on the ‘NCC’ subset of 237 records, the semantic search on ‘screen layout standards’ produced 6 records common to both the LM Studio and Cohere results, of which the top three were the same but in a different order. A standard FileMaker search on ‘screen layout’ produced no hits; and another on ‘standards’ produced 4 hits of which 3 were in the results for both models. While some of the semantic search results were a little off-beam (‘standards for the use of overhead projectors’, for example), this was a good result from the semantic search which couldn’t be reproduced in the standard search using the same search words.
• In the semantic search on the term ‘Workshops in non-UK countries’ neither LM Studio nor Cohere seemed to take on board the qualifying phrase ‘in non-UK countries’: six of the LM Studio records included one or other of the following words in the Title field; Lancaster, London, UK; and 3 of the Cohere records included either Lancaster or UK.

Comparison of the hits produced by LM Studio and Cohere

• Five tests were carried out on the subset ‘CSC’ which contains 2803 records. The semantic search test phrases were ‘bad experience’; ‘shipping industry’, ‘social events’, ‘work in the oil and gas industry’ and ‘UK Management’. All produced 10 answers (as specified in the script) from each of the LM Studio and Cohere models. Across all these answers only 22% were to be found in both the LM Studio and Cohere lists – a surprisingly low degree of agreement between each set of results.
• In the semantic search test for ‘travel by train’ on the subset ‘NCC’ (containing 237 records), the same four records appeared in the top 5 spots in the results for both models. Overall, in the 10 records identified by each model, 7 (70%) were common to both model’s results – a rather better result than achieved in tests on the larger ‘CSC’ subset (described in the bullet point above) possibly due to this being a much smaller set of records.

Cosine Similarity Scores

• The 5 semantic searches for ‘bad experience’; ‘shipping industry’, ‘social events’, ‘work in the oil and gas industry’ and ‘UK Management’ produced 10 answers (as specified in the script) for each of LM Studio and Cohere. On average there was only 22% commonality between the records identified by LM Studio and those identified by Cohere. Despite this significant difference in results, the Cosine Similarity Coefficient of the top ranked items in all 5 tests were quite close (LM Studio first vs Cohere second): +0.479378961 vs +0.477162529; +0.340306588 vs +0.34258184; +0.419345513 vs +0.377885542; +0.293494392 vs +0.278173457; +0.267438353 vs +0.28565865.
• Some of the results of the semantic searches for ‘shipping Industry’ and ‘bad experience’ were of dubious relevance. For example, ‘shipping industry’ produced ‘CSC World Fall ’85…. OA business, interconnection’, and ‘bad experience’ produced ‘CSC User Experience Teleconferencing minutes’. However, that is perhaps understandable given the low Cosine Similarity numbers for both models, and the need to provide 10 answers as specified in the script. The top hits in LM Studio and Cohere respectively, for each of the searches were +0.293494392 & +0.278173457 for ‘shipping industry’, and +0.267438353 & +0.28565865 for ‘bad experience’. If there are few or no items of high relevance then the records identified are, by definition, going to be of less relevance.
• For the semantic search for ‘Workshops in non-UK countries’ there was a particularly strong similarity in the results for LM Studio and Cohere: the top two hits were exactly the same though with slightly different Cosine Similarity scores: +0.456294376 vs +0.513860943 and +0.44603088 vs +0.471864208.

Comparison with Standard FileMaker Find Results

• In comparison with the semantic search on ‘Workshops in non-UK countries’, a standard FileMaker search on ‘Workshops’ produced 10 hits, three of which appeared in the results for both models and two of which appeared only in the Cohere set. A second standard search using the singular ‘workshop’ produced 210 hits. I would have thought the AI would give as much weight to the singular as to the plural in this case.
• To compare the results of the semantic search for ‘work in the oil and gas industry’, three separate standard FileMaker searches were conducted for the terms ‘oil’, ‘gas’, and ‘industry. The ‘oil’ search produced 8 hits, none of which appeared in either semantic search results for LM Studio or Cohere. The ‘gas’ search produced 3 hits, one of which appeared in both the LM Studio and Cohere lists and one just in the LM Studio list; and the ‘industry’ search produced 12 hits of which one appeared in both the LM Studio and Cohere lists and one just in the Cohere list.
• A semantic search was conducted on the term ‘travel by train’. A standard FileMaker search for ‘train’ produced 52 hits, but, surprisingly, only 1 of these (the top answer in each model) appeared in the 10 results from LM Studio or Cohere respectively.

While the tests from which the above observations are drawn were by no means rigorous or definitive, they do provide enough insights to draw the following general conclusions:

1. Different LLM models may well produce very different results so choosing the right model for the job you want to do is key. However, that’s not the only consideration: as the Claris Fundamentals II course puts it, “you want to use the smallest, fastest and least expensive model possible to get the job done”.
2. FileMaker doesn’t have its own AI capabilities – it engages with independent models through its Script programs which can get complicated and require developer knowledge. This is likely to be the case with most specialist applications.
3. PAWDOC is a very simple database in a very sophisticated app with just a single user. Based on what I have learned throughout this AI journey I would choose to work directly with an AI model rather than try to build scripts/macros/programs to enable interaction with an AI model from within an application. Owners of larger and more complex databases with hundreds of users may not have that option.
4. Standard FileMaker searches and the ability to search again within a found set are sufficient for my use of PAWDOC today.
5. The Cosine Similarity Coefficient is a useful measure; and it’s also helpful to be able to specify how many hits you get in a semantic search. However, it must always be remembered that semantic search results are never absolutely right – they just deliver the best matches along with a Cosine Similarity rating to indicate how close a match they are.

Below is a breakdown of the time I spent on Phase 6 and across all phases.

Phase 5 in this journey was to assess the possibility of investigating the contents of a large archive by providing an AI with not only the Index, but also the actual digital files relating to the Index entries. As explained in the previous post, AIs are only able to hold a limited amount of information in their memory; so, for large archives I’m employing a strategy whereby the AI is asked to provide an answer to a subset of the archive’s information; and once that answer has been provided, another subset is presented for an answer. When all the subsets have been exhausted, the AI is then asked to stitch all the answers together to provide an overall answer for the archive. That is the approach I’ve taken here: I assembled a whole set of digital files, divided them up into groups which the AI is capable of handling, got answers from the AI for each group, and then got the AI to stitch the answers together. To highlight the practicalities of this exercise, the whole process is described in some detail below.

As in the previous phase, the archive I’ve used for this test is my PAWDOC work archive of some 17,000 Index entries and 31,000 associated files. I decided I’d do two distinct tests: one with files about a very specific subject with a limited life-span; and one with files related to a very general topic which might appear anywhere in the lifetime of the archive. For the first I chose to investigate the two years (1977-78) that I worked for a company called ‘CPC’; and for the second I chose the word ‘measurement’. In each case, I performed a search of the relevant term (‘CPC’ or ‘measurement’) on my PAWDOC index in the Filemaker application. These produced lists of 21 and 40 index entries respectively. PAWDOC index entries can have any number of files associated with them, and in these cases, it turned out that‘CPC’ had 48 files, and ‘measurement’ had 72 files.

The next part of the process was to split the files into sufficiently small subsets such that each subset would be accepted in full in a single request to the AI. The two AIs I used were Claude and Copilot, both of which limit file uploads to 20 per conversation; Claude limits an individual file to 30Mb and Copilot to 50Mb. So, I chose to limit my subsets to the following:

• Number of files: 19 (this to include the ‘PAWDOC Guide’ file and a file of the Index entries related to the files in this subset: leaving room for 17 content files)
• Max individual file size: 28Mb
• Max Total file size: 40Mb
• Text files to be no bigger than 18k characters

Armed with these criteria I set about dividing up the content files into their subsets and ended up with 5 for each one:

I encountered three issues in the course of creating the subsets

• TIF files: Most of the PAWDOC files are scans of hardcopy documents in multi-page TIF files. However, the free version of Copilot does not support TIF files; and Claude only supports single page TIF files, so I had to turn all the TIF files into PDFs. As it turned out, that was a quick process: select a file; right click the mouse and choose ‘Open with [PDF App]; when it opens in the PDF app, select ‘Save As’ and save it as a PDF in the relevant Subset folder.
• One of the files (a scan of a desk diary) was 35.7 Mb – a little over my Individual file size limit of 28 Mb. So, I split it into two files of 21Mb and 15 Mb respectively.
• One of the files was an old PowerPoint file with a ’ppt’ extension. I believe Claude and Copilot only accept pptx extensions so I produced a pptx version for the subset.

With the subsets prepared, the final thing to do was to specify the following prompts:

For the CPC subsets: PAWDOC is a work document collection built up from 1981 to the present day. The attached files include a Guide outlining how PAWDOC is constructed and what it contains. Also attached is a subset of PAWDOC’s Index and the actual digital files associated with those index entries. The following request is to be undertaken using the information in the attached files: Outline everything that you can find out about CPC in the attached files, including its origins, operations, locations, products, finances, people and culture; and describe the contributions that Paul Wilson made while he worked for the company.  Present the results in a word document with an executive summary at the beginning.

For the Measurement subsets: PAWDOC is a work document collection built up from 1981 to the present day. The attached files include a Guide outlining how PAWDOC is constructed and what it contains. Also attached is a subset of PAWDOC’s Index, and the actual digital files associated with those index entries. The following request is to be undertaken using the information in the attached files: Describe everything that you can find out about ‘Measurement’ in the attached files, under a set of appropriate category headings which should include the philosophy of measurement, attitudes towards measurement, and the pros and cons of measurement. In a final section suggest interesting further research that could be undertaken to extend the findings reported here. Present the results in a word document with an executive summary at the beginning.

Running the subsets through the AIs was relatively quick for Claude. Indeed, I’d completed the whole exercise – definition of the questions, creation of the subsets and runs through both the CPC and Measurement tests – within the space of two days. I’m not sure whether or not that was helped by the fact that I was still operating under the Claude Pro plan for which I had paid £18 for a one-month subscription. For Copilot, however, it was a different kettle of fish. I got through the CPC tests and the first of the Measurement subsets, quickly enough. But at that point I came up against something I hadn’t encountered before – Copilot’s daily upload quota. Apparently, this is nothing to do with the number of files you upload, but rather the number of upload events you initiate in a 24-hour period (an event can be an upload of anything from 1 to 20 files). When you exceed this, you have to wait a rolling 24 hours before you can upload anything again – but there is no where to look to see when you can start uploading again. So, I ground to a halt on the Copilot Measurement tests, and wasted a lot of time trying to find out what the problem was. Having found out, I decided to abandon the Copilot Measurement tests for reasons that will become obvious below.

Copilot’s individual subset answers to the CPC question were all reasonably good, 6-8 page answers with a fair amount of detail – though four out of the five were mainly in bullet point format within headings and sub-headings. The merged document was in similar style and I gave it 7 out of 10 with following comments:

This is a 10-page report with 10 separate sections and lots of sub-headings with the text being primarily in the form of bullet points. There’s plenty of detail in the bullet points, but no citations back to the specific Reference Numbers from which they came. The bullet point format means that there is little additional commentary or embellishment and inevitably makes it a less informative read. It took Copilot just 8 seconds to produce this merged report as compared to an average of 38 seconds to produce the subset answers.

However, the Claude answers to the CPC questions were a significant cut above the Copilot versions, with 8-13 pages of discursive text, and a great deal of detail. I scored Claude’s CPC merged document 9.7 with these comments:

This 21-page merged document is of a very high standard with the following contents: Exec Summary; 1. Origins and Corporate Structure; 2. Locations; 3. Products; 4. Finances; 5. People; 6. Culture and Working Environment; 7. Paul Wilson’s Contributions at CPC, 8. Quick Reference Summary; 9. Further Work Recommended for Researchers. In each of these sections there is a huge amount of detail described in discursive text. Claude has clearly inspected and interpreted many, if not all, of the 49 files provided – all of which were scans in PDF documents. I noticed a few doubtful assumptions: a) In section 2.4 it is suggested that Perivale was a CPC location (I don’t think it was – though I couldn’t find the ‘Wray memorandum of December 1976’ to check, which is interesting); b) section 6.5 suggests that ‘Wilson’s formal 11-stage systems methodology paper, produced c.1978 ‘ was actually in operation (but it was only a proposal); and c) some items in the Project Portfolio list in section 7.5 were taken from a scan of my desk diary and may not have actually been as significant as they sound, eg. ‘Corn Shipment Simulation’ and ‘Forecasting’ and ‘Factory Open Day Coordination’. There may well be other misinterpretations I didn’t spot, but despite these, this is a hugely impressive, comprehensive, thorough, and highly detailed report. It took Claude just over 6 minutes to produce as compared to an average of 7 and a half minutes to produce the subset answers.

The Claude Measurement answers were of the same ilk. In fact, because the topic is so broad, each of the individual answers made good reading in their own right. I scored Claude’s merged Measurement document 9.8 out of 10 with the following comments:

This is a very comprehensive 23-page report on a wide variety of aspects of ‘measurement’. There is a 4-page discussion on the ‘The philosophy of measurement’ under 6 sub-headings.  The ‘Attitudes to measurement’ section has 5 sub-headings; and the pros and cons of measurement are addressed in similar detail (10 pros and 11 cons). The section on ‘Measurement Instruments and Frameworks’ lists 22 different mechanisms. Section 7 provides 11 suggestions for further work; and a full listing of all the documents used in the analysis rounds of the report. As with the CPC report, it appears that Claude inspected and interpreted most, if not all, of the 72 files provided, of which 68 were scans in PDF documents, three were word documents, and one was a PowerPoint file. This merged report is excellent. How accurate it is, is something I wouldn’t be able to say without doing many days if not weeks of work. It took Claude 8 minutes to produce this report as compared to an average of 5 and a half minutes to produce the subset answers.

One very practical point emerges from these answers: the difference in the time it took for Copilot and Claude to arrive at their answers is striking. Copilot operates in seconds, whilst Claude operates in minutes. There is a distinct possibility that this has something to do with the quality of the results they produced; and it is another reminder that some LLM models are better than others for particular tasks.

Regarding the overall objective of these Phase 5 tests, these results clearly indicate that it is indeed feasible to have AI investigate an archive through its digital files. Furthermore, the results are likely to be even more impressive than those that can be obtained when providing the AI with just index entries and file names. Consequently, there is even greater likelihood that researchers will employ these techniques to explore archives, and even less likelihood that they will spend time verifying the results. This is a serious long-term issue for archives, for researchers, and for the integrity of the global information canon.

In these tests I did try to get the AI to suggest further work to mitigate these problems, and, indeed, several suggestions were forthcoming. Although I’m not convinced that they directly addressed the accuracy issue, I do think they indicate that better prompts, more focused on identifying potential issues and mitigations, could be developed.

Below records the breakdown of the time I spent on Phase 5 and across all phases.

The Phase 4 objective was to explore if it is possible to use AI to investigate the contents of a large archive by using a combination of its Index entries and the names of the associated files. Having completed the work I can report that it is indeed feasible to do just that – and with some spectacular results. However, there are some caveats which are highlighted in the following paragraphs.

The issue with a large archive is that there is too much information to hold in the Context Window of an LLM. To get round that problem, I split up the Index and File Name information into small subsets, and instructed the LLM to, first, produce answers for each subset, and then to merge the subset answers into an overall answer for the whole archive. The practicalities of splitting up the data and establishing subsets are described in a previous post. Suffice it to say that it is best to a) produce subsets that combine a number of index entries together with their associated file names (rather than having separate Index Entry and File Name subsets), and b) to keep the number of characters in every individual subset file well under the limits specified by the LLM to avoid truncation.

I tested this subset strategy by asking the following 6 questions of my PAWDOC work archive of some 17,000 Index entries and 31,000 associated files:

1. List all the people named in this part of the Index and its associated files, and the organisation they belong to if any.
2. Describe Paul Wilson’s career over the period covered by this part of the Index and its associated files.
3. What significant changes in Information Technology occurred during the period covered by this part of the Index and its associated files?
4. Document all the travel undertaken by Paul Wilson over the period covered by this part of the Index and its associated files.
5. What training was undertaken by Paul Wilson over the period covered by this part of the Index and its associated files, and how important were particular elements to his subsequent career?
6. What are the strangest or most unusual things to be found within this part of the Index and its associated files, including unlikely coincidences, and events with unexpected outcomes?

As described in a previous post, to get the answers you want, you need to provide clear and detailed instructions to the AI in the Prompt that you submit: the bare questions as phrased above are not sufficient, so I used far more detailed versions.

Another factor that makes a difference in the results you can get is the LLM that you choose to use. For this investigation I used ChatGPT and Claude Sonet 4.6 (I wanted to use Copilot as well but found that my subset files contained too many characters for Copilot and were being truncated). The results from ChatGPT and Claude are compared below. However, its worth saying here that different LLMs have different abilities and its good to be clear about what sort of task you want the AI to perform and which LLMs are the best AIs for that job. The Huggingface website is one of the primary LLM repositories for localised models, each being categorised as being suitable for one or more of over 50 different tasks – though support for archives isn’t one of them. Similar advice about which cloud platforms are best for particular tasks can be found in review articles with titles like “10 best AI….”. However, I have found nothing specific to archives in these texts either. So, as yet, I have no definitive listing of the AI tasks that might be required to support archives, nor of which LLM models are best suited to perform those tasks. I hope to have more information on this when I get to Phase 7 of this work.

Getting back to the Phase 4 tests, I ended up with 10 Subsets of PAWDOC Index entries and associated file names. So, for each of the six questions I got 10 answers for ChatGPT and 10 for Claude – 120 answers in all. Then I asked ChatGPT and Claude respectively to combine the 10 answers they had produced for each question into a single merged answer, which produced 6 merged answers from ChatGPT and 6 from Claude. I reviewed each of the 132 answers and gave each of the merged answers a score out of 10; the results are summarised in the table below.

Claude has clearly produced the best results – and perhaps the ‘time to produce’ numbers indicate why that should be: ChatGPT took an average of 44 seconds to produce its merged answers, and Claude took an average of 962 seconds. Indeed, in some cases, Claude’s merged reports are so detailed and so well formatted that they are too believable – and this is their downfall: to reproduce such comprehensive results, or to verify the answers, would probably take weeks of manual work, and so, in view of this and despite their potential for error (more of this below), it is very tempting to just assume they are totally correct.

Although I didn’t diligently check every aspect of some of these extremely detailed reports, I did spot a few errors which suggests there may be several more across this body of material. They are summarised below to give a flavour of what can go wrong.

• ChatGPT Q1 subset answers: Large numbers of incorrect People and Organisations are put forward by ChatGPT. A sensible rationale for identifying the difference between any two adjacent words and a Person or Organisation’s name is just not there. For example, two of the people names put forward by ChatGPT are ‘Do, To’ and ‘Taxation, Oil’.
• Claude Q2 subset answers: The subset B answer says “CSC – Bid Management and Internal Systems work, 1984–1990s… a transition into bid management roles by the early 1990s “; but I didn’t start bid management work until 2001.
• Claude Q2 subset answers: A subset C section heading – ‘Joining CSC and the Cosmos / Amigo research project (1986–1989)’ is wrong because we never joined the Amigo Research project – we just had a seminar to share what each group was doing.
• Claude Q3 subset answers: In the subset A answer, the PAW-ACMOIS-Jul88-p277 (1988) review of “wireless Intraoffice network technologies’ appears incorrectly in section 5 (CSCW) instead of section 4 (Local Area Networks).
• Claude Q5 subset answers: Claude incorrectly assumes that I explored doing a ‘UCL Graduate Diploma in HCI with Ergonomics’ but actually the index entry concerned a letter from UCL asking me to publicise their course.
• Claude Q2 merged answer: The summary trajectory section starts in 1977 with my time at CPC and doesn’t mention Kodak [where I worked prior to CPC] at all despite Kodak appearing in its subset B answer.
• Claude Q2 merged answer: Claude says that, in a CHOTS paper, I specified the ‘four-part reference-number scheme later used in PAWDOC itself’ whereas the scheme was already being used in PAWDOC when I was doing the CHOTS work.
• Claude Q2 merged answer: The CSC start date is specified as 1986 but actually was 1984.
• Claude Q3 merged answers: The text says “Doug Engelbart’s bootstrap seminars on Dynamic Knowledge Repositories, which CSC adopted (PAW-DOC-7385-01)”. However, the only reason I can see for claiming that ‘CSC adopted’ is the inclusion of a username and password for access to the Bootstrap institute in the 7385-01 Index entry.
• Claude Q5 merged answer: Three CPC courses are wrongly ascribed to Kodak in sub-section 3.2.
• Claude Q5 merged answer: Claude assumes that an Index entry about a letter concerning Nottingham’s proposed MSc in Human-Computer Interaction is “evidence that he was seriously weighing a formal postgraduate qualification” whereas, in fact, I was simply responding to a request for support for the establishment of such a course.

These errors may have been due to lack of material in the subset, or to a misinterpretation of the information provided, or to limited information in Index entries without access to the associated files; but however they were caused they provide an important reminder that all LLM material needs to be checked if you want to rely on its veracity.

Overall, then, I believe it is feasible to use AI to explore large archives via their Index entries and associated File Names. It may need careful planning and preparation and be time-consuming to carry out, but the results can be very informative – perhaps providing insights which would be just too time-consuming and expensive to obtain in any way. Certainly, there’s no way I, or anyone else, would ever have produced the in-depth material on the information contained in PAWDOC on people, organisations, travel, and training that Claude has produced in these tests. However, if you are going to make more than just casual enquiries of your archive, researchers would be well advised to develop verification strategies as an integral part of the exercise. These need not be comprehensive and definitive checks, but instead may involve sampling, using catalogue searches, or even using the AI itself, to get a sense of how much, if anything, is amiss or not. The research need not stop there. There are probably optimum strategies for using such AI answers, and awareness of their flaws, as a starting point for research. However, at present, I am not aware of any such strategies having been documented. This, too, is another aspect I hope to know more about by the time I get to Phase 7 of this work.

Below records the breakdown of the time I spent on Phase 4 and across all phases.

As described in the previous post, the Index and file information in my PAWDOC collection is too big to be ingested and used in a single prompt in today’s AI systems. So, I’m using it to explore how AI might be able to analyse a very large collection in pieces and then to stitch the results together. The first step in such an undertaking is to decide what information is to be provided to the AI and how small the individual subsets (i.e. files) of that information need to be to guarantee that they will be taken into account in full in the AI’s analysis (files that are too large will simply have some of their contents truncated after they are uploaded but before the AI starts its analysis).

I decided to provide the following fields from the PAWDOC index to the AI:

• Reference Number: every document in the collection has a Reference Number which is made up of four parts – an Owner Identifier, a Set Identifier, a Serial Number, and a Sub-Serial Number. For example, PAW-DOC-4046-01.
• Title: this contains free format text describing the document(s) concerned. Three dots (…) in the middle of the text denotes that what follows are Keywords/Phrases.
• Publication date: this is the publication date of the oldest document relating to that Reference Number.
• Creation date: the date the index entry was created.

I also decided to provide the file title for every file associated with each Index entry. File titles have a general structure of Reference Number, Description, filename extension. For example:

PAW-DOC-1104-39   Planning the ITUSA document Interchange group X.400 User Test.tif

I was able to produce a file of the relevant Index fields by exporting them out Filemaker. The export contained 17,381 entries composed of just over 2,400,000 characters (I calculated character numbers by using Excel’s ‘LEN’ formula) and sized at 1.4Mb.

Obtaining a list of the associated Filenames was a little more difficult as the files reside inside separate folders for each Reference Number. However, a search of the net established that the 7z file compression utility will provide a list of all the files within a higher-level directory structure; so, I downloaded and installed 7z, and created a file of file titles containing 31,270 entries composed of 2,568,000 characters and sized at 1Mb.

To assess the number and size of the subset files that needed to be produced I turned to the insights I recorded in the Phase 1 and Phase 2 Summary results:

• ChatGPT’s usable context window is about 80k-100k tokens (approximately 160k-300k characters – assuming 2-3 characters/token). ChatGPT limits uploads to 3 files a day but this can be circumvented by putting multiple files in a zip file.
• Copilot doesn’t have a fixed context window – its design means that its effective context window is much larger and more flexible than a single token number would suggest. It limits uploads to about 7 batches of up to 20 files. Approximately 30k characters per file should work fine.
• Claude’s context window is 200k tokens (roughly 500 pages of text or approximately 800,000 characters). Claude caps file uploads at 30Mb per file and 20 files per conversation.

Copilot and Claude’s limits suggested that, for each prompt, I should load no more than 20 files containing a maximum of 30k characters each. This would exceed ChatGPTs limits, but I thought that I could do without ChatGPT if its results were poor.

I had already decided that I would upload a copy of the ‘PAWDOC Guide‘ file in every prompt; and that I would count the text request as a file in its own right; so that left a maximum of 18 files of Index and Filename information in each prompt. I duly, set about splitting the Index file into subsets of around 30k characters, and then creating associated files of file titles (ensuring that none of the files exceeded 30k characters) and assembled the combination of Index files and File Title files into groups of 18 files or less. It was a very laborious task. If I’d realised how time-consuming it was going to be, I would have found a way of splitting out the Reference Number from the File titles into a separate field and merged the overall Index file and overall File Title file into a single file sorted by Reference Number, which would have made the splitting task a much simpler and quicker operation – a lesson worth remembering (in actual fact, I’d achieved a similar feat a few months earlier simply by asking ChatGPT to provide me with an Excel function for a similar task – I’d just forgotten it was that easy). Anyway, after the job was completed, I found I had 10 subsets of either 17 or 18 files each, and which I named Subsets A-J. Subset B’s files are shown below.

• Test B files\PAWDOC Index 12.csv
• Test B files\PAWDOC Index 13.csv
• Test B files\PAWDOC Index 14.csv
• Test B files\PAWDOC Index 15.csv
• Test B files\PAWDOC Index 16.csv
• Test B files\PAWDOC Index 17.csv
• Test B files\PAWDOC Index 18.csv
• Test B files\PAWDOC File Names 7.csv
• Test B files\PAWDOC File Names 8.csv
• Test B files\PAWDOC File Names 9.csv
• Test B files\PAWDOC File Names 10.csv
• Test B files\PAWDOC File Names 11.csv
• Test B files\PAWDOC File Names 12.csv
• Test B files\PAWDOC File Names 13.csv
• Test B files\PAWDOC File Names 14.csv
• Test B files\PAWDOC File Names 15.csv
• Test B files\PAWDOC File Names 16.csv

I then set about getting each of the AIs to answer question 1 with Subset A. As it happened, there seemed to be no problem with ChatGPT: it happily accepted all the files in a zip file, and advised that no files were truncated when it analysed them. This didn’t seem to concur with the ChatGPT context window information I had established earlier, but the results seemed reasonable so I decided to use ChatGPT in the testing. So too with Claude, which also was happy to ingest all the files compressed into a single zip file. Copilot, however, produced inferior results which I established, after some Q&A with the AI, was due to the files being truncated. Copilot explained the problem very clearly:

Your file size (≈33 KB) and character count (≈29,950) were both within your rule of thumb — but that rule is no longer reliable. The truncation happened because:

1. The ingestion limit is based on tokens, not characters or kilobytes: A token is roughly 3–4 characters of English text, but CSV files with long lines, punctuation, quotes, and irregular structure tokenize very inefficiently. Your file contains long filenames, commas, quotes, and mixed punctuation dramatically increase token count. So even though the file is only ~30k characters, it may be 10k–15k tokens, which is enough to trigger truncation.
2. CSVs with hundreds of long rows are the worst-case format: Each line is treated as a dense block of text. Tokenization cost spikes.
3. The system applies a per‑file token cap: This cap is not published, but from your repeated tests, the effective safe zone is:

• ≈15,000–18,000 characters for CSV-like content
• ≈20,000–22,000 characters for plain text
• ≈30,000 characters only for very simple text

Your file was near 30k characters but had extremely token-heavy content → truncated.

It was clear that I wasn’t going to be able to conduct the tests with Copilot using the same subsets that I had already created. However, ChatGPT and Claude seemed able to deal with the subsets, and would enable me to make some sort of comparison of results. Furthermore, I wasn’t prepared to redo the subsets, so I decided simply to exclude Copilot from the tests. This experience suggests that it’s a good idea to test some sample file sizes with all the models to be used, before undertaking the final splitting of files into subsets.

The Index to my PAWDOC collection of work documents is in a Filemaker database, and I’d recently been told that Filemaker had added some AI functionality into its product, and that it also had a number of AI-related videos for users to learn from. So, at the start of this investigation into providing AI support for larger collections like PAWDOC, I took to watching some of the videos. One was entitled ‘Learn about AI prompting basics (30May2024)’ and it outlined some of the key things to remember when constructing an AI prompt. In particular:

• Clearly define the audience.
• Define your goals.
• Provide a step-by-step approach (do a, b, c, etc).
• Include examples of what you want and say why they work.

At the time, I was thinking about putting a protective covering on the top of books to stop dust settling on the top of the text block, and I thought I could get AI to produce an outline which I could print on card and cut out. So, I set about developing a prompt using the guidelines listed above. It turned out to be an interactive process which took me at least a dozen attempts before I was satisfied with the results – illustrated in the image below.

The way the cover works is that you score a line where the image ends, cut away the two small rectangles on either side, and fold down the resulting tab and place it inside the spine of the book. The rest of the cut-out then sits on the top of the text block as shown in the top-down view of some of the books below. When reading the book, the cover can act as a bookmark.

Perfecting the prompt was a matter of getting the cut-out lines to be drawn in the right places; of getting the text boxes to be of the right size with minimal empty space; of getting the whole of the specified area to be covered with interesting illustrations; of figuring out whether its best to have it produced in a Word or Powerpoint document (I ended up with Word); and of realising that Claude AI would produce more interesting illustrations than ChatGPT (particular AI models are better at some things than others). Getting all of these things right took rounds of experimentation until I came up with the final prompt.

I had the same experience when investigating AI support for my large PAWDOC document collection for which I devised 6 evaluation questions, each of which were to be asked of a subset of the PAWDOC Index and file titles (the need to use subsets was because, unlike AI support for the Mementos collection which I have written about in earlier posts, the PAWDOC Index and file titles have some 16+ times the number of characters – a size which could not be ingested and used in a single prompt in today’s AI systems. So, I decided to see what sort of results could be obtained by splitting the information into 10 subsets, getting answers for each subset, and then stitching the 10 answers together). The 6 questions I specified were:

1. List all the people named in this part of the Index and its associated files, and the organisation they belong to if any.
2. Describe Paul Wilson’s career over the period covered by this part of the Index and its associated files.
3. What significant changes in Information Technology occurred during the period covered by this part of the Index and its associated files.
4. Document all the travel undertaken by Paul Wilson over the period covered by this part of the Index and its associated files.
5. What training was undertaken by Paul Wilson over the period covered by this part of the Index and its associated files, and how important were particular elements to his subsequent career.
6. What are the strangest or most unusual things to be found within this part of the Index and its associated files, including unlikely coincidences, and events with unexpected outcomes.

These all look fairly straightforward but the actual prompts I ended up with were considerably more extensive. For example, the prompt for the sixth question was:

PAWDOC is a work document collection built up from 1981 to the present day. The attached Guide outlines how PAWDOC is constructed and what it contains. Also attached is a subset of PAWDOC’s Index and the digital files associated with those index entries. Note: None of the actual PAWDOC documents have been provided. The following request is to be undertaken using just the information in the Index entries and the File Names provided in the attached files: identify the training that was undertaken by Paul Wilson over the period covered by this part of the Index and its associated files, including the objectives and composition of the training, how it was delivered, how much time was spent on it, how relevant it was to his subsequent career, and citing the Reference Numbers from which the information was obtained. NB. Be sure to inspect all supplied data first, then analyse.

All the prompts required a number of iterations before they were producing good results. They are reproduced in the attached document together with additional prompts to stitch the 10 responses to each question together. Note the prompt for the first question which was especially demanding.

These experiences are not novel – they merely reflect a well-known fact that to get the best out of an AI capability the prompts need to be well thought out and will likely require several iterations before they are deemed satisfactory.

At the start of this journey, I said I would conduct my investigations in 9 Phases:

1. AI support for the Memento collection’s index entries.
2. AI support for the Memento collection’s combined Index entries and file titles.
3. AI support for the Memento collection’s index entries, file titles and textual items.
4. AI support for PAWDOC’s index entries.
5. AI support for PAWDOC’s combined index entries and file titles.
6. AI support for PAWDOC’s combined index entries, file titles, and some or all of the born digital items.
7. AI support for a subset of PAWDOC’s scanned items.
8. AI support for a combination of index entries, file titles, some born digital material and some scanned items.
9. AI support for the whole of PAWDOC.

I’ve completed the first three Phases and reported on them in previous posts. However, given what I have learned up to now, I’ve decided to change my plan in the following ways:

Combine Phases 4 and 5: It’s a feasible and better approach to obtain AI support for a combination of Index entries and associated File names.

Combine Phases 6, 7, and 8: I’ve discovered that AI can deal with scanned documents in much the same way as born digital documents, so they can both be investigated together. Furthermore, given the limits on context windows, and file numbers and sizes, it’s not really practical to try to obtain AI support for combinations of the whole index and large numbers of content files. Instead, I shall investigate the approach of applying AI to a subset of content files and their associated index entries.

Change Phase 9 – AI support for the whole of PAWDOC will have been explored in the new Phases outlined above. Instead, the final Phase will assess all my findings about how AI can support personal archives

Add another Phase: Since setting out the Phases back in March this year, I’ve discovered that the FileMaker application I use to manage the PAWDOC Index, has been incorporating progressive amounts of AI support since the version I currently have which came out in 2019. Furthermore, FileMaker is currently offering a 45-day free trial of its latest version – FileMaker Pro 2025. I intend to take up this offer once I have completed the revised PAWDOC Phases. This will include watching various training videos about FileMaker’s AI’s capabilities in addition to performing tests on the PAWDOC Index and Files. I anticipate that the way FileMaker has addressed the opportunities afforded by AI, will extend my AI knowledge as well as providing an insight into how any Database/Indexing/Repository software might try to exploit AI.

In summary the Phases I now intend to follow are as follows:

1-3: Complete.

4. AI support for PAWDOC’s combined index entries and file titles

5. AI support for selected subsets of PAWDOC born digital/scanned document and their associated index entries.

6. AI Support provided by FileMaker for the PAWDOC Index.

7. Assessing all my findings about how AI can support personal archives.

Phase 3’s objective was to assess how well AI could support not only an Index and its file titles, but also the textual content of the files itself. However, after doing a bit of digging, I discovered that AIs could not only work with textual content, but also with scanned text, and with images in general (of course, such material still has to be presented to the AI in the form of uploaded files or RAG Chunks). With this awareness in mind, I set about devising some tests to find out just how well AIs can perform when presented with real content as opposed to just metadata. I came up with the following:

Tests of Machine-readable text

• Test 1 – describe and summarise the contents of three years of diary entries in word format
• Test 2 – discuss any relationships that can be found between three Word files with diverse contents: my library loan history for 2004-2012; an account of the petitioning of a school’s teachers to make a change to daily activities; some thoughts about university life while in the infirmary recovering from German Measles.

Tests of Image-only scanned text.

• Test 3 – summarise Friends of the Earth activities in Harrow as documented in three image-only scanned text documents from 1976-1979 in PDF format.

Tests of Text in images

• Test 4 – list all the events and activities described in three documents of events, tickets, membership cards etc.
• Test 5 – describe and summarise the contents of all the images in three pages of Italy stamps which also include cutouts from the relevant parts of stamp catalogues.
• Test 6 – catalogue the contents of the three pages of Italy stamps images using the following fields: Reference Number, Country, Year, Value, Notes.

Tests of collections of objects in images

• Test 7 – describe and summarise the contents of all the images in three photos of chocolate wrappers, with each photo showing a) a double page of a chocolate wrapper scrapbook (in an unused 1967 A4 diary); and b) 3 plastic wallets containing loose wrappers.
• Test 8 – catalogue the contents of the three chocolate wrapper photos using the following fields: Reference Number, Name, Manufacturer, Type, and Size.

• Test 9 – describe and summarise the contents of three photos of household ornaments showing a) 10 pieces of Wedgewood, b) 30 small display items including porcelain (cups, saucers, plates, vases, jug, trinket boxes, flowers), glassware (bowl, vase, jug, flower, bird), stoneware (lighthouse, ashtrays, barometer), and wood (bowl, elephants); and c) 13 sundry items including silver trays, bowls, coasters and shoehorn; large shells, letter rack; pen holder; and decorative tray and plate.
• Test 10 – catalogue the contents of the household ornament photos using the following fields: Reference Number, Name, Type, Colour.

As may be apparent from the above descriptions, each test involved attaching three files to the AI Prompt together with a request, for example, “Using the three files I have just uploaded, catalogue the contents of the images using the following fields: Reference Number, Name, Type, Colour.”

All 10 tests were applied to the four AIs that had been used in the previous phase – AnythingLLM with Mistral, ChatGPT, Copilot and Claude. I did explore the possibility of using two other widely used products – Llama from Meta, and Gemini from Google. However, both require that you create an account before you can use them and I didn’t want to do that because, among other reasons, I’m trying to limit my exposure to data collection and advertising which are central to both of those organisation’s operations. Indeed, during the process of opening a Meta account, I was actually informed that I would be consenting to being shown adverts: I stopped at that point. So, for these tests I stuck with the four AIs previously mentioned. I have been using the free version of ChatGPT, Copilot and Claude up to now. However, when I started doing these tests ChatGPT suddenly changed the number of files it was allowing me to upload each day from 3 to 2. Since all the tests involve 3 files I elected to upgrade to ChatGPT-Go which enables you to “usually upload far more than the Free tier’s 3 files/day, but there is still a rate limit, and OpenAI hasn’t publicly stated the exact number.” The cost was £7 a month with the ability to cancel anytime. I encountered no limits when I was conducting these tests with the free versions of Copilot and Claude.

Before discussing the test results, its worth being clear about the image recognition and text-reading capabilities of the AIs concerned. First, Mistral is not capable of interpreting images so, unsurprisingly its results in these tests are very poor. However, I included it anyway just to see how it would react.  Second, ChatGPT, Copilot and Claude (like most other Large Language Models) don’t apply separate conventional OCR (Optical Character Recognition) techniques to interpret text in scans, photos or other images. Instead, they undertake text recognition as part of their general image understanding capabilities which includes the combined assessment of visual patterns, language, spatial relationships, and context. Consequently, their text recognition capabilities often depend on the type and volume of training data they have been given. As ChatGPT put it: The image recognition capabilities of AI models “are deeply dependent on the training data, but also on model design, compute scale, optimisation methods, and how visual and language information are combined during training.”

With this in mind, we can now consider the results of the 10 tests. Each individual test was evaluated in two ways: first, I looked at the answer provided and wrote a short paragraph reflecting what I thought about it; and, second, I awarded a score out of 10 according to what I’d written.  As is obvious, this is a completely subjective, and a potentially highly inaccurate, process. However, it does have the advantage that it uses a single observer with a consistent mind-set. Hopefully, that consistency will be reflected throughout all the results.

All the evaluation paragraphs and scores for all four AIs are included in this word document. However, I also produced an overall summary paragraph, and overall average score, for each AI, and these are provided below. They are accompanied by an equivalent paragraph produced by ChatGPT after it was provided with each AIs detailed paragraphs and scores for each test.

Claude comes out a clear winner in these tests, with ChatGPT coming in second. Copilot, while performing excellently with text, appears to have had less relevant image training. At a general level, however, the results illustrate very clearly that AIs can work extremely well with both text and images; and could be very useful to collectors in identifying items, describing them, cataloguing them, and creating indexes for them.

For completeness, below records the breakdown of the time I spent on Phase 3 and across all phases.