The Phase 4 objective was to explore if it is possible to use AI to investigate the contents of a large archive by using a combination of its Index entries and the names of the associated files. Having completed the work I can report that it is indeed feasible to do just that – and with some spectacular results. However, there are some caveats which are highlighted in the following paragraphs.

The issue with a large archive is that there is too much information to hold in the Context Window of an LLM. To get round that problem, I split up the Index and File Name information into small subsets, and instructed the LLM to, first, produce answers for each subset, and then to merge the subset answers into an overall answer for the whole archive. The practicalities of splitting up the data and establishing subsets are described in a previous post. Suffice it to say that it is best to a) produce subsets that combine a number of index entries together with their associated file names (rather than having separate Index Entry and File Name subsets), and b) to keep the number of characters in every individual subset file well under the limits specified by the LLM to avoid truncation.

I tested this subset strategy by asking the following 6 questions of my PAWDOC work archive of some 17,000 Index entries and 31,000 associated files:

1. List all the people named in this part of the Index and its associated files, and the organisation they belong to if any.
2. Describe Paul Wilson’s career over the period covered by this part of the Index and its associated files.
3. What significant changes in Information Technology occurred during the period covered by this part of the Index and its associated files?
4. Document all the travel undertaken by Paul Wilson over the period covered by this part of the Index and its associated files.
5. What training was undertaken by Paul Wilson over the period covered by this part of the Index and its associated files, and how important were particular elements to his subsequent career?
6. What are the strangest or most unusual things to be found within this part of the Index and its associated files, including unlikely coincidences, and events with unexpected outcomes?

As described in a previous post, to get the answers you want, you need to provide clear and detailed instructions to the AI in the Prompt that you submit: the bare questions as phrased above are not sufficient, so I used far more detailed versions.

Another factor that makes a difference in the results you can get is the LLM that you choose to use. For this investigation I used ChatGPT and Claude Sonet 4.6 (I wanted to use Copilot as well but found that my subset files contained too many characters for Copilot and were being truncated). The results from ChatGPT and Claude are compared below. However, its worth saying here that different LLMs have different abilities and its good to be clear about what sort of task you want the AI to perform and which LLMs are the best AIs for that job. The Huggingface website is one of the primary LLM repositories for localised models, each being categorised as being suitable for one or more of over 50 different tasks – though support for archives isn’t one of them. Similar advice about which cloud platforms are best for particular tasks can be found in review articles with titles like “10 best AI….”. However, I have found nothing specific to archives in these texts either. So, as yet, I have no definitive listing of the AI tasks that might be required to support archives, nor of which LLM models are best suited to perform those tasks. I hope to have more information on this when I get to Phase 7 of this work.

Getting back to the Phase 4 tests, I ended up with 10 Subsets of PAWDOC Index entries and associated file names. So, for each of the six questions I got 10 answers for ChatGPT and 10 for Claude – 120 answers in all. Then I asked ChatGPT and Claude respectively to combine the 10 answers they had produced for each question into a single merged answer, which produced 6 merged answers from ChatGPT and 6 from Claude. I reviewed each of the 132 answers and gave each of the merged answers a score out of 10; the results are summarised in the table below.

Claude has clearly produced the best results – and perhaps the ‘time to produce’ numbers indicate why that should be: ChatGPT took an average of 44 seconds to produce its merged answers, and Claude took an average of 962 seconds. Indeed, in some cases, Claude’s merged reports are so detailed and so well formatted that they are too believable – and this is their downfall: to reproduce such comprehensive results, or to verify the answers, would probably take weeks of manual work, and so, in view of this and despite their potential for error (more of this below), it is very tempting to just assume they are totally correct.

Although I didn’t diligently check every aspect of some of these extremely detailed reports, I did spot a few errors which suggests there may be several more across this body of material. They are summarised below to give a flavour of what can go wrong.

• ChatGPT Q1 subset answers: Large numbers of incorrect People and Organisations are put forward by ChatGPT. A sensible rationale for identifying the difference between any two adjacent words and a Person or Organisation’s name is just not there. For example, two of the people names put forward by ChatGPT are ‘Do, To’ and ‘Taxation, Oil’.
• Claude Q2 subset answers: The subset B answer says “CSC – Bid Management and Internal Systems work, 1984–1990s… a transition into bid management roles by the early 1990s “; but I didn’t start bid management work until 2001.
• Claude Q2 subset answers: A subset C section heading – ‘Joining CSC and the Cosmos / Amigo research project (1986–1989)’ is wrong because we never joined the Amigo Research project – we just had a seminar to share what each group was doing.
• Claude Q3 subset answers: In the subset A answer, the PAW-ACMOIS-Jul88-p277 (1988) review of “wireless Intraoffice network technologies’ appears incorrectly in section 5 (CSCW) instead of section 4 (Local Area Networks).
• Claude Q5 subset answers: Claude incorrectly assumes that I explored doing a ‘UCL Graduate Diploma in HCI with Ergonomics’ but actually the index entry concerned a letter from UCL asking me to publicise their course.
• Claude Q2 merged answer: The summary trajectory section starts in 1977 with my time at CPC and doesn’t mention Kodak [where I worked prior to CPC] at all despite Kodak appearing in its subset B answer.
• Claude Q2 merged answer: Claude says that, in a CHOTS paper, I specified the ‘four-part reference-number scheme later used in PAWDOC itself’ whereas the scheme was already being used in PAWDOC when I was doing the CHOTS work.
• Claude Q2 merged answer: The CSC start date is specified as 1986 but actually was 1984.
• Claude Q3 merged answers: The text says “Doug Engelbart’s bootstrap seminars on Dynamic Knowledge Repositories, which CSC adopted (PAW-DOC-7385-01)”. However, the only reason I can see for claiming that ‘CSC adopted’ is the inclusion of a username and password for access to the Bootstrap institute in the 7385-01 Index entry.
• Claude Q5 merged answer: Three CPC courses are wrongly ascribed to Kodak in sub-section 3.2.
• Claude Q5 merged answer: Claude assumes that an Index entry about a letter concerning Nottingham’s proposed MSc in Human-Computer Interaction is “evidence that he was seriously weighing a formal postgraduate qualification” whereas, in fact, I was simply responding to a request for support for the establishment of such a course.

These errors may have been due to lack of material in the subset, or to a misinterpretation of the information provided, or to limited information in Index entries without access to the associated files; but however they were caused they provide an important reminder that all LLM material needs to be checked if you want to rely on its veracity.

Overall, then, I believe it is feasible to use AI to explore large archives via their Index entries and associated File Names. It may need careful planning and preparation and be time-consuming to carry out, but the results can be very informative – perhaps providing insights which would be just too time-consuming and expensive to obtain in any way. Certainly, there’s no way I, or anyone else, would ever have produced the in-depth material on the information contained in PAWDOC on people, organisations, travel, and training that Claude has produced in these tests. However, if you are going to make more than just casual enquiries of your archive, researchers would be well advised to develop verification strategies as an integral part of the exercise. These need not be comprehensive and definitive checks, but instead may involve sampling, using catalogue searches, or even using the AI itself, to get a sense of how much, if anything, is amiss or not. The research need not stop there. There are probably optimum strategies for using such AI answers, and awareness of their flaws, as a starting point for research. However, at present, I am not aware of any such strategies having been documented. This, too, is another aspect I hope to know more about by the time I get to Phase 7 of this work.

Below records the breakdown of the time I spent on Phase 4 and across all phases.

As described in the previous post, the Index and file information in my PAWDOC collection is too big to be ingested and used in a single prompt in today’s AI systems. So, I’m using it to explore how AI might be able to analyse a very large collection in pieces and then to stitch the results together. The first step in such an undertaking is to decide what information is to be provided to the AI and how small the individual subsets (i.e. files) of that information need to be to guarantee that they will be taken into account in full in the AI’s analysis (files that are too large will simply have some of their contents truncated after they are uploaded but before the AI starts its analysis).

I decided to provide the following fields from the PAWDOC index to the AI:

• Reference Number: every document in the collection has a Reference Number which is made up of four parts – an Owner Identifier, a Set Identifier, a Serial Number, and a Sub-Serial Number. For example, PAW-DOC-4046-01.
• Title: this contains free format text describing the document(s) concerned. Three dots (…) in the middle of the text denotes that what follows are Keywords/Phrases.
• Publication date: this is the publication date of the oldest document relating to that Reference Number.
• Creation date: the date the index entry was created.

I also decided to provide the file title for every file associated with each Index entry. File titles have a general structure of Reference Number, Description, filename extension. For example:

PAW-DOC-1104-39   Planning the ITUSA document Interchange group X.400 User Test.tif

I was able to produce a file of the relevant Index fields by exporting them out Filemaker. The export contained 17,381 entries composed of just over 2,400,000 characters (I calculated character numbers by using Excel’s ‘LEN’ formula) and sized at 1.4Mb.

Obtaining a list of the associated Filenames was a little more difficult as the files reside inside separate folders for each Reference Number. However, a search of the net established that the 7z file compression utility will provide a list of all the files within a higher-level directory structure; so, I downloaded and installed 7z, and created a file of file titles containing 31,270 entries composed of 2,568,000 characters and sized at 1Mb.

To assess the number and size of the subset files that needed to be produced I turned to the insights I recorded in the Phase 1 and Phase 2 Summary results:

• ChatGPT’s usable context window is about 80k-100k tokens (approximately 160k-300k characters – assuming 2-3 characters/token). ChatGPT limits uploads to 3 files a day but this can be circumvented by putting multiple files in a zip file.
• Copilot doesn’t have a fixed context window – its design means that its effective context window is much larger and more flexible than a single token number would suggest. It limits uploads to about 7 batches of up to 20 files. Approximately 30k characters per file should work fine.
• Claude’s context window is 200k tokens (roughly 500 pages of text or approximately 800,000 characters). Claude caps file uploads at 30Mb per file and 20 files per conversation.

Copilot and Claude’s limits suggested that, for each prompt, I should load no more than 20 files containing a maximum of 30k characters each. This would exceed ChatGPTs limits, but I thought that I could do without ChatGPT if its results were poor.

I had already decided that I would upload a copy of the ‘PAWDOC Guide‘ file in every prompt; and that I would count the text request as a file in its own right; so that left a maximum of 18 files of Index and Filename information in each prompt. I duly, set about splitting the Index file into subsets of around 30k characters, and then creating associated files of file titles (ensuring that none of the files exceeded 30k characters) and assembled the combination of Index files and File Title files into groups of 18 files or less. It was a very laborious task. If I’d realised how time-consuming it was going to be, I would have found a way of splitting out the Reference Number from the File titles into a separate field and merged the overall Index file and overall File Title file into a single file sorted by Reference Number, which would have made the splitting task a much simpler and quicker operation – a lesson worth remembering (in actual fact, I’d achieved a similar feat a few months earlier simply by asking ChatGPT to provide me with an Excel function for a similar task – I’d just forgotten it was that easy). Anyway, after the job was completed, I found I had 10 subsets of either 17 or 18 files each, and which I named Subsets A-J. Subset B’s files are shown below.

• Test B files\PAWDOC Index 12.csv
• Test B files\PAWDOC Index 13.csv
• Test B files\PAWDOC Index 14.csv
• Test B files\PAWDOC Index 15.csv
• Test B files\PAWDOC Index 16.csv
• Test B files\PAWDOC Index 17.csv
• Test B files\PAWDOC Index 18.csv
• Test B files\PAWDOC File Names 7.csv
• Test B files\PAWDOC File Names 8.csv
• Test B files\PAWDOC File Names 9.csv
• Test B files\PAWDOC File Names 10.csv
• Test B files\PAWDOC File Names 11.csv
• Test B files\PAWDOC File Names 12.csv
• Test B files\PAWDOC File Names 13.csv
• Test B files\PAWDOC File Names 14.csv
• Test B files\PAWDOC File Names 15.csv
• Test B files\PAWDOC File Names 16.csv

I then set about getting each of the AIs to answer question 1 with Subset A. As it happened, there seemed to be no problem with ChatGPT: it happily accepted all the files in a zip file, and advised that no files were truncated when it analysed them. This didn’t seem to concur with the ChatGPT context window information I had established earlier, but the results seemed reasonable so I decided to use ChatGPT in the testing. So too with Claude, which also was happy to ingest all the files compressed into a single zip file. Copilot, however, produced inferior results which I established, after some Q&A with the AI, was due to the files being truncated. Copilot explained the problem very clearly:

Your file size (≈33 KB) and character count (≈29,950) were both within your rule of thumb — but that rule is no longer reliable. The truncation happened because:

1. The ingestion limit is based on tokens, not characters or kilobytes: A token is roughly 3–4 characters of English text, but CSV files with long lines, punctuation, quotes, and irregular structure tokenize very inefficiently. Your file contains long filenames, commas, quotes, and mixed punctuation dramatically increase token count. So even though the file is only ~30k characters, it may be 10k–15k tokens, which is enough to trigger truncation.
2. CSVs with hundreds of long rows are the worst-case format: Each line is treated as a dense block of text. Tokenization cost spikes.
3. The system applies a per‑file token cap: This cap is not published, but from your repeated tests, the effective safe zone is:

• ≈15,000–18,000 characters for CSV-like content
• ≈20,000–22,000 characters for plain text
• ≈30,000 characters only for very simple text

Your file was near 30k characters but had extremely token-heavy content → truncated.

It was clear that I wasn’t going to be able to conduct the tests with Copilot using the same subsets that I had already created. However, ChatGPT and Claude seemed able to deal with the subsets, and would enable me to make some sort of comparison of results. Furthermore, I wasn’t prepared to redo the subsets, so I decided simply to exclude Copilot from the tests. This experience suggests that it’s a good idea to test some sample file sizes with all the models to be used, before undertaking the final splitting of files into subsets.

The Index to my PAWDOC collection of work documents is in a Filemaker database, and I’d recently been told that Filemaker had added some AI functionality into its product, and that it also had a number of AI-related videos for users to learn from. So, at the start of this investigation into providing AI support for larger collections like PAWDOC, I took to watching some of the videos. One was entitled ‘Learn about AI prompting basics (30May2024)’ and it outlined some of the key things to remember when constructing an AI prompt. In particular:

• Clearly define the audience.
• Define your goals.
• Provide a step-by-step approach (do a, b, c, etc).
• Include examples of what you want and say why they work.

At the time, I was thinking about putting a protective covering on the top of books to stop dust settling on the top of the text block, and I thought I could get AI to produce an outline which I could print on card and cut out. So, I set about developing a prompt using the guidelines listed above. It turned out to be an interactive process which took me at least a dozen attempts before I was satisfied with the results – illustrated in the image below.

The way the cover works is that you score a line where the image ends, cut away the two small rectangles on either side, and fold down the resulting tab and place it inside the spine of the book. The rest of the cut-out then sits on the top of the text block as shown in the top-down view of some of the books below. When reading the book, the cover can act as a bookmark.

Perfecting the prompt was a matter of getting the cut-out lines to be drawn in the right places; of getting the text boxes to be of the right size with minimal empty space; of getting the whole of the specified area to be covered with interesting illustrations; of figuring out whether its best to have it produced in a Word or Powerpoint document (I ended up with Word); and of realising that Claude AI would produce more interesting illustrations than ChatGPT (particular AI models are better at some things than others). Getting all of these things right took rounds of experimentation until I came up with the final prompt.

I had the same experience when investigating AI support for my large PAWDOC document collection for which I devised 6 evaluation questions, each of which were to be asked of a subset of the PAWDOC Index and file titles (the need to use subsets was because, unlike AI support for the Mementos collection which I have written about in earlier posts, the PAWDOC Index and file titles have some 16+ times the number of characters – a size which could not be ingested and used in a single prompt in today’s AI systems. So, I decided to see what sort of results could be obtained by splitting the information into 10 subsets, getting answers for each subset, and then stitching the 10 answers together). The 6 questions I specified were:

1. List all the people named in this part of the Index and its associated files, and the organisation they belong to if any.
2. Describe Paul Wilson’s career over the period covered by this part of the Index and its associated files.
3. What significant changes in Information Technology occurred during the period covered by this part of the Index and its associated files.
4. Document all the travel undertaken by Paul Wilson over the period covered by this part of the Index and its associated files.
5. What training was undertaken by Paul Wilson over the period covered by this part of the Index and its associated files, and how important were particular elements to his subsequent career.
6. What are the strangest or most unusual things to be found within this part of the Index and its associated files, including unlikely coincidences, and events with unexpected outcomes.

These all look fairly straightforward but the actual prompts I ended up with were considerably more extensive. For example, the prompt for the sixth question was:

PAWDOC is a work document collection built up from 1981 to the present day. The attached Guide outlines how PAWDOC is constructed and what it contains. Also attached is a subset of PAWDOC’s Index and the digital files associated with those index entries. Note: None of the actual PAWDOC documents have been provided. The following request is to be undertaken using just the information in the Index entries and the File Names provided in the attached files: identify the training that was undertaken by Paul Wilson over the period covered by this part of the Index and its associated files, including the objectives and composition of the training, how it was delivered, how much time was spent on it, how relevant it was to his subsequent career, and citing the Reference Numbers from which the information was obtained. NB. Be sure to inspect all supplied data first, then analyse.

All the prompts required a number of iterations before they were producing good results. They are reproduced in the attached document together with additional prompts to stitch the 10 responses to each question together. Note the prompt for the first question which was especially demanding.

These experiences are not novel – they merely reflect a well-known fact that to get the best out of an AI capability the prompts need to be well thought out and will likely require several iterations before they are deemed satisfactory.

At the start of this journey, I said I would conduct my investigations in 9 Phases:

1. AI support for the Memento collection’s index entries.
2. AI support for the Memento collection’s combined Index entries and file titles.
3. AI support for the Memento collection’s index entries, file titles and textual items.
4. AI support for PAWDOC’s index entries.
5. AI support for PAWDOC’s combined index entries and file titles.
6. AI support for PAWDOC’s combined index entries, file titles, and some or all of the born digital items.
7. AI support for a subset of PAWDOC’s scanned items.
8. AI support for a combination of index entries, file titles, some born digital material and some scanned items.
9. AI support for the whole of PAWDOC.

I’ve completed the first three Phases and reported on them in previous posts. However, given what I have learned up to now, I’ve decided to change my plan in the following ways:

Combine Phases 4 and 5: It’s a feasible and better approach to obtain AI support for a combination of Index entries and associated File names.

Combine Phases 6, 7, and 8: I’ve discovered that AI can deal with scanned documents in much the same way as born digital documents, so they can both be investigated together. Furthermore, given the limits on context windows, and file numbers and sizes, it’s not really practical to try to obtain AI support for combinations of the whole index and large numbers of content files. Instead, I shall investigate the approach of applying AI to a subset of content files and their associated index entries.

Change Phase 9 – AI support for the whole of PAWDOC will have been explored in the new Phases outlined above. Instead, the final Phase will assess all my findings about how AI can support personal archives

Add another Phase: Since setting out the Phases back in March this year, I’ve discovered that the FileMaker application I use to manage the PAWDOC Index, has been incorporating progressive amounts of AI support since the version I currently have which came out in 2019. Furthermore, FileMaker is currently offering a 45-day free trial of its latest version – FileMaker Pro 2025. I intend to take up this offer once I have completed the revised PAWDOC Phases. This will include watching various training videos about FileMaker’s AI’s capabilities in addition to performing tests on the PAWDOC Index and Files. I anticipate that the way FileMaker has addressed the opportunities afforded by AI, will extend my AI knowledge as well as providing an insight into how any Database/Indexing/Repository software might try to exploit AI.

In summary the Phases I now intend to follow are as follows:

1-3: Complete.

4. AI support for PAWDOC’s combined index entries and file titles

5. AI support for selected subsets of PAWDOC born digital/scanned document and their associated index entries.

6. AI Support provided by FileMaker for the PAWDOC Index.

7. Assessing all my findings about how AI can support personal archives.

Phase 3’s objective was to assess how well AI could support not only an Index and its file titles, but also the textual content of the files itself. However, after doing a bit of digging, I discovered that AI’s could not only work with textual content, but also with scanned text, and with images in general (of course, such material still has to be presented to the AI in the form of uploaded files or RAG Chunks). With this awareness in mind, I set about devising some tests to find out just how well AIs can perform when presented with real content as opposed to just metadata. I came up with the following:

Tests of Machine-readable text

• Test 1 – describe and summarise the contents of three years of diary entries in word format
• Test 2 – discuss any relationships that can be found between three Word files with diverse contents: my library loan history for 2004-2012; an account of the petitioning of a school’s teachers to make a change to daily activities; some thoughts about university life while in the infirmary recovering from German Measles.

Tests of Image-only scanned text.

• Test 3 – summarise Friends of the Earth activities in Harrow as documented in three image-only scanned text documents from 1976-1979 in PDF format.

Tests of Text in images

• Test 4 – list all the events and activities described in three documents of events, tickets, membership cards etc.
• Test 5 – describe and summarise the contents of all the images in three pages of Italy stamps which also include cutouts from the relevant parts of stamp catalogues.
• Test 6 – catalogue the contents of the three pages of Italy stamps images using the following fields: Reference Number, Country, Year, Value, Notes.

Tests of collections of objects in images

• Test 7 – describe and summarise the contents of all the images in three photos of chocolate wrappers, with each photo showing a) a double page of a chocolate wrapper scrapbook (in an unused 1967 A4 diary); and b) 3 plastic wallets containing loose wrappers.
• Test 8 – catalogue the contents of the three chocolate wrapper photos using the following fields: Reference Number, Name, Manufacturer, Type, and Size.

• Test 9 – describe and summarise the contents of three photos of household ornaments showing a) 10 pieces of Wedgewood, b) 30 small display items including porcelain (cups, saucers, plates, vases, jug, trinket boxes, flowers), glassware (bowl, vase, jug, flower, bird), stoneware (lighthouse, ashtrays, barometer), and wood (bowl, elephants); and c) 13 sundry items including silver trays, bowls, coasters and shoehorn; large shells, letter rack; pen holder; and decorative tray and plate.
• Test 10 – catalogue the contents of the household ornament photos using the following fields: Reference Number, Name, Type, Colour.

As may be apparent from the above descriptions, each test involved attaching three files to the AI Prompt together with a request, for example, “Using the three files I have just uploaded, catalogue the contents of the images using the following fields: Reference Number, Name, Type, Colour.”

All 10 tests were applied to the four AIs that had been used in the previous phase – AnythingLLM with Mistral, ChatGPT, Copilot and Claude. I did explore the possibility of using two other widely used products – Llama from Meta, and Gemini from Google. However, both require that you create an account before you can use them and I didn’t want to do that because, among other reasons, I’m trying to limit my exposure to data collection and advertising which are central to both of those organisation’s operations. Indeed, during the process of opening a Meta account, I was actually informed that I would be consenting to being shown adverts: I stopped at that point. So, for these tests I stuck with the four AIs previously mentioned. I have been using the free version of ChatGPT, Copilot and Claude up to now. However, when I started doing these tests ChatGPT suddenly changed the number of files it was allowing me to upload each day from 3 to 2. Since all the tests involve 3 files I elected to upgrade to ChatGPT-Go which enables you to “usually upload far more than the Free tier’s 3 files/day, but there is still a rate limit, and OpenAI hasn’t publicly stated the exact number.” The cost was £7 a month with the ability to cancel anytime. I encountered no limits when I was conducting these tests with the free versions of Copilot and Claude.

Before discussing the test results, its worth being clear about the image recognition and text-reading capabilities of the AIs concerned. First, AnythingLLM is not capable of interpreting images so, unsurprisingly its results in these tests are very poor. However, I included it anyway just to see how it would react.  Second, ChatGPT, Copilot and Claude (like most other Large Language Models) don’t apply separate conventional OCR (Optical Character Recognition) techniques to interpret text in scans, photos or other images. Instead, they undertake text recognition as part of their general image understanding capabilities which includes the combined assessment of visual patterns, language, spatial relationships, and context. Consequently, their text recognition capabilities often depend on the type and volume of training data they have been given. As ChatGPT put it: The image recognition capabilities of AI models “are deeply dependent on the training data, but also on model design, compute scale, optimization methods, and how visual and language information are combined during training.”

With this in mind, we can now consider the results of the 10 tests. Each individual test was evaluated in two ways: first, I looked at the answer provided and wrote a short paragraph reflecting what I thought about it; and, second, I awarded a score out of 10 according to what I’d written.  As is obvious, this is a completely subjective, and a potentially highly inaccurate, process. However, it does have the advantage that it uses a single observer with a consistent mind-set. Hopefully, that consistency will be reflected throughout all the results.

All the evaluation paragraphs and scores for all four AIs are included in this word document. However, I also produced an overall summary paragraph, and overall average score, for each AI, and these are provided below. They are accompanied by an equivalent paragraph produced by ChatGPT after it was provided with each AIs detailed paragraphs and scores for each test.

Claude comes out a clear winner in these tests, with ChatGPT coming in second. Copilot, while performing excellently with text, appears to have had less relevant image training. At a general level, however, the results illustrate very clearly that AIs can work extremely well with both text and images; and could be very useful to collectors in identifying items, describing them, cataloguing them, and creating indexes for them.

For completeness, below records the breakdown of the time I spent on Phase 3 and across all phases.

The object of Phase 2 was to explore how AI can support the combination of a collection’s Index and the titles of the associated files. The answer is straightforward: the assembled file titles need to be provided to the AI model in one or more files, in the same way that the Index is delivered. In Windows, file titles can be collected by highlighting the files you are interested in and selecting ‘Copy as path’ from the Right-Click drop down menu. These can then be pasted into either Excel or Word, and the path in front of the file name can be deleted by using the Find and Replace function: specify the path in the Find box and specify blank in the Replace box, highlight the file names, then select Replace-All. No doubt other operating systems have similar capabilities.

I tried this out with the names of 2065 files associated with the Mementos index, along with the 1.62 version of the Mementos Index (the version I used for the final set of tests in Phase1). I retained the 5 evaluation questions used in Phase 1, but I also added three more to further test the capabilities of the AI models concerned (more of these at the end of this post). I performed tests using AnythingLLM/Mistral and Copilot. ChatGPT wasn’t used because it has a limit of around 250kb and the file titles together with the Index would have significantly exceeded that. I also tried out Anthropic’s Claude model for the first time. The results are shown in the table below together with the results of the final test in Phase1 for comparison purposes.

From these results, it’s not possible to assess if the inclusion of File Titles has made a difference: AnythingLLM’s performance was generally very poor across both tests; Copilot’s two scores were too similar to draw any conclusion; and Claude did not have any earlier test result to compare against. However, it seems reasonable to assume that if there is additional information in the file titles, over and above that in the Index, there will be a better outcome.

Since this was the first time I had used Claude, I enquired about its operational parameters, and discovered that Claude’s Context Window is 200K tokens across all models and paid plans (except for Enterprise plans). That is roughly 500 pages of text or approximately 800,000 characters. Claude caps file uploads at 30MB per file and 20 files per conversation. I found the quality of Claude’s answers to be at the same sort of high level as Copilot – though the results indicate that, on this particular set of uploaded material, Copilot has a bit of an edge. I also noticed that Claude differs from Copilot in two ways: first, Claude, unlike Copilot, tells you what it’s doing in the course of responding to a question; and, second, Claude took three times longer to respond to the same set of 8 evaluation questions (59.3 vs 19.6 seconds).

The three extra evaluation questions that I added were:

• Describe the things that happened to Paul Wilson during the time he spent at Barlborough Hall School and support the story with references to relevant mementos.
• Identify anything strange or unexpected about the contents of the Memento collection and support the analysis with references to particular files.
• Describe life in the Wilson family during the 1990s citing relevant artefacts in the collection.

These extra questions were deliberately designed to explore the AIs ability to make more broadly-based connections and inferences, as opposed to identifying related items in a narrow subject area. This is where AI can excel – and sometimes fail – as indicated in the detailed results for each question provided below.

Finally, to come back to the objective for Phase 2, I have concluded that file titles can certainly be included in the material delivered to an AI model, and this is certainly worth doing if they include additional material to that which is contained in an Index – or if an  Index does not exist.

For completeness, below records the breakdown of the time I spent on Phase 2.

Phase1 Description: AI support for the Memento collection’s index entries.

Conclusions

• Different LLMs are trained with different amounts and types of training data, thereby making them more or less suitable for operation within specific domains. Therefore, there may be some LLMs available that are more suitable than others for use with Collection indexes.
• To enable LLMs to provide answers to questions related to specific information in documents or databases, a process called Retrieval Augmented Generation (RAG) is used whereby relevant parts of the information are provided to the LLM alongside the question.
• The RAG process performed by particular products typically includes a multiplicity of different techniques to identify an appropriate set of Chunks for submission with the question to the LLM. It is likely that particular combinations of these techniques will be more effective than others for use with collection indexes.
• Collection Indexes may contain information which is more unevenly spread throughout their contents than a written document with an ordered set of contents. Consequently, such indexes may require more chunks to be sent to the LLM, and may require the LLM to have larger Context Windows, than ordinary documents in order to obtain satisfactory answers.
• Out of the 4 models/systems I have tried out so far, Copilot is by far the best LLM for use with a collection index of around 2,500 entries.
• The choice of systems, models, configurations, and Index adjustments in this first Phase were heavily influenced by ChatGPT, and therefore may have been based on some inaccuracies or hallucinations. This needs to be born in mind when considering the findings from Phase 1 and taking them forward into subsequent phases.

These conclusions were reached in the course of undertaking the activities summarised below.

Tasks and Timescales

Reference Sources

• A Survey on Retrieval-Augmented Text Generation for Large Language Models by Yizheng Huang, Jimmy Huang, v2, arXiv:2404.10981, 23Aug2024.
• I obtained lots of answers from ChatGPT to questions I asked about RAG, AI software products, and how to adjust the Memento index to get better AI results.

Concepts and Terminology encountered

• LLM stands for Large Language Model. Hundreds of thousands of these have been built and new ones emerge daily, but many are variants of major models like GPT-4 from ChatGPT which can be accessed online. Other LLMs can be downloaded from websites like Hugging Face, to use locally on a stand-alone computer.
• RAG stands for Retrieval Augmented Generation whereby the LLM is not trained on the archive, but instead a relevant subset of the archive data is provided to the LLM alongside the question.
• Chunk is a smaller, manageable segment of a larger document or dataset. In RAG, one or more Chunks are provided to the LLM along with the question.
• Token is the fundamental unit of data used by LLMs. Models convert questions and Chunks into tokens, process them, produce an answer in tokens, and then convert those tokens back into text. Generally, one token equals approximately 0.75 English words.
• Vector Databases store mathematical representations of Tokens as vectors – lists of numbers – in such a way that related items are clustered together thereby enabling capabilities like similarity searching.
• Embedding: Chunks are broken down into Tokens which are converted into mathematical representations and embedded in a Vector Database.
• Context Window is the maximum number of Tokens that can be handled by the LLMs working memory (which contains both the input prompt and the answer). If the inputs to the LLM exceed its Context Window, then some content may be simply left out and the answer may be less complete.
• Hallucination is a phenomenon where LLMs generate false, misleading, or nonsensical information confidently. It happens when an LLM predicts text based on patterns rather than facts, often due to poor training data, ambiguous prompts, or a lack of understanding of reality.

Test results – 2^nd set of evaluation criteria (scores out of 10)

In the previous post I explained why I changed the evaluation criteria I was using to the following:

• What are the main themes that run through the entire index?
• Are there distinct phases or periods in the collection?
• Which items suggest important life events or transitions?
• What patterns or motifs repeat across the collection?
• What are the top 5 most important entries, and why?

I tried using them with a version of the Index which had extraneous fields such as ‘Physical Location’ and ‘No of digital files’ removed leaving just Ref No, Description, Item Label, Set, Facets, AI Context, and Year all in a single column. This produced a result of 4.8 out of 10 using the Mistral model and 5 out of 10 for Mixtral – and I proved to myself that I was able to apply the new evaluation criteria, albeit with rather more subjective scoring.

At this point it dawned on me that it might be worth trying to use the MS Copilot AI provided as an integral part of my Windows operating system. After making some initial enquiries through the net about its possible use with Excel, I got the impression it could only be used with MS Office 365 which I do not have (I have the home version of Office); but Copilot itself set me straight explaining that the 365 version enabled support for the functionality within the Office applications, whereas the free-to-use version of Copilot simply ingests documents up to the cloud where it works out the answer to the question it has been asked. I tried it out using a very simple version of the Mementos index with just the Ref No, Description, Set, Facets, and Year fields in their own columns, and was excited by the result which I described as follows:

“This is a strong result (8 out of 10): comprehensive answers with just a few poor interpretations, but with no obvious hallucinations. Importantly, Copilot was able to deal with the whole attached Index in one go which delivers far better answers than the RAG approach which can only work on a pre-selected subset of the material. The response time (average 4.4 seconds) was very quick considering that the whole document had to be sent to, and analysed in, the cloud before delivering its answer.”

I then tried to compare the AnythingLLM and the Copilot systems by using yet another version of the Mementos index based on suggestions from ChatGPT. It still contained just Ref No, Description, Set, Facets, and Year with the all the fields in a single cell, but this time with the Description replaced by keywords derived automatically using an Excel formula supplied by ChatGPT. The results were very clear: Mistral scored 3.1, Mixtral 4.8, and Copilot 8.6. For the Copilot Test I wasn’t sure if the size of the file would cause a problem so I split it into three files of between 200kb and 300kb each, and these seemed to have been ingested successfully. However, I subsequently discovered that the files had been truncated so that only about the first 120 lines of each were ingested – making about 360 entries in all out of the total 2393 (I guess Copilot must have truncated the file I used in the previous test as well). Given this, Copilot’s 8.6 score was even more impressive.

Finding that Copilot had truncated the files prompted me to do some digging and experimentation to find out just what its limits are. I established that it will ingest up to 20 whole documents of up to about 30,000 characters/30kb file size each in a single turn, and will work across all those documents to construct its answer. Furthermore, more batches of 20 can be submitted in subsequent turns up to a total of about 150 after which ‘the conversation becomes unwieldy’ (I’m not sure what is meant by this). I duly split the Mementos Index into 17 files, all between 20,000 – 30,000 characters and conducted the test again, this time including a version of the Guide document. This produced an even better score of 9.1.

Discovering Copilot’s capabilities made me wonder what ChatGPT could do. I discovered that the free version which I was using allows you to upload a maximum of 3 files of up to about 512Mb in any one day. However, to be sure that all the contents of all the files will be taken into account in answering a question, the total of the 3 files need to be a maximum of about 300k characters with 250k being a safe limit. I put ChatGPT to the test with the same index used in the first Copilot test but broken into three files of between 111k and 130k characters each. This produced a result of 5.9 which was probably expected given that the file size limits had been exceeded.

As I was exploring the file size limits for Copilot and ChatGPT, it became clear that the ultimate determinant of the amount of text that can be reliably reasoned about at once is the AI model’s maximum ‘Context Window’. This is effectively the model’s working memory which contains both the input prompt and the answer. If the inputs exceed the Context Window then some content may be simply left out and the answer may be less complete. The table below summarises the Context Window limits for the different AI models I’ve been using.

 If one wishes Index files to be considered in full by an AI system, the primary requirement is to ensure that the whole set fits into the Context Window. However, when it comes to assembling and submitting those files there are additional considerations to be taken into account, as summarised below.

With these constraints in mind, I conducted a final set of tests to compare the three AI Systems – AnythingLLM’s Rag approach, Copilot, and ChatGPT. I assembled a cut-down version of the Index (by removing the MW set) which was small enough (around 274k characters, 270kb file size) to fit within the limits of all three systems.  As can be seen in the table below, there was a clear winner.

Copilot was a clear winner with ChatGPT following on behind. Both provided substantial answers which included rationale, several examples, and a summary. In contrast the AnythingLLM RAG answers were sparse, sometimes not very good, and occassionally included complete hallucinations. The RAG approach just doesn’t seem very effective for Index material.