How Frustrating is AI? Our journey so far…

how frustrating is ai our journey so far 1200 v3

Gartner predicts that in 2025 30% of generative AI projects will be abandoned due to poor data quality, inadequate risk controls or unclear business values. 

Given poor quality data or inadequate understanding of what is being analysed or generated, AI could be seen as nothing more than a means to produce results of indeterminate quality and value, faster. 

Being a provider of Data Readiness services, where our data consultants work alongside customer’s employees to check data readiness for generative AI goals, I wanted to talk about APPtechnology’s own AI journey to date.

What Works

When applying the principles that we consult on to our own environment, what is clear is that given a known data set and business requirement, AI has real value.  Give ChatGPT a selection of pdf’d Invoices and ask it to analyse the different services, and you can receive tabular or graphical output that supplies consolidated data covering the billed services, months and frequency, along with observations around outlying information.  You could also see recommendations of what to check “post AI analysis”, for instance the “one off” large invoice for unusually named services, an increase in the monthly cost of a repetitive service, or perhaps a service period that has been double billed.

This is a fast and semi-intelligent analysis of a known data set, and saves having to type detailed invoice information into a useable table for manual analysis via spreadsheet pivots and colour scales.  You would probably expect a Human to have made the same recommendations, they aren’t revolutionary in nature (Generative AI isn’t currently going to suggest where you could purchase the same services for less for instance).  In reality the invoice data set would already exist for analysis, the invoices would be recorded (likely via human data entry with human approval processes) within your organisation’s financial data.  So given a managed data set, with a history and consistent content, generative AI is good at spotting outliers, offering observational insight where trends or patterns highlight nonstandard data. 

Trusting AI

Taking this principle of using known and trusted data sets further, you can extend generative AI confidently to more complex document generation.  General trade analysis, such as suggestions on expanding more profitable trade areas are all practical output from Generative AI, especially as the generated analysis would inevitably be subject to human review and oversight once created.  No organisation is going to make dramatic business changes without humans checking the analysis output before it is used to enforce change.  When it comes to confidence in Generative AI, human applied checks persist, and for good reason.  Would you allow AI chatbots and generative AI to create and price a high value quote without a human review?  If so, above what quote value would you want to check, allowing all lower value quotes to be processed without human intervention?  The answers is different for every organisation, and changes based on type of goods or services, quote values, volume of low and high value trade.

Data Source validity

We have of course had to question what data sources should be available to AI within our organisation.  In a generative AI scenario, the data sets benefit from being smaller and more constrained.  Give an AI access to an employee’s historical Annual Review of Performance (ARP) information and ask for an assessment of performance and strengths and you will receive a view aligned to the controlled and formulaic information held in a company’s ARP system.  Open AI up to all internal emails and M365 data and ask for the same, the results are likely to vary from formally polite down to informal gossip and probably unprintable suggestions!

Opening up the data set raises the question of whether publicly accessible, or partly accessible Web data is a useable and dependable source of information?  In December 2024 The New York Times (NYT) started a significant lawsuit against ChatGPT owner OpenAI, with Microsoft as an additional defendant.  The NYT claims millions of articles, either public facing, or kept behind a subscriber paywall, have been used without permission, with ChatGPT allegedly giving “verbatim excerpts”, often without quoting the source, and potentially costing the NYT significant loss of income from subscriptions or referral links.  Earlier in 2024 a New York Lawyer faced a court hearing after an associate had used AI to find examples of case law for use in a brief submitted to the case Judge.  Six of the submitted cases were deemed “bogus”, with erroneous quotes and non-existent internal citations.  In December 2024 the BBC, using Apple’s AI offerings to automate the creation of news alerts for BBC apple based mobile notifications, saw multiple erroneous headlines being published, seemingly with no Human oversight or fact checking.  Headlines such as “Netanyahu arrested”, or “Luigi Mangione shoots himself” were push distributed without what looks like the safeguards of basic checks.

Investing time and oversight to Help AI

Conversely, given a large data set that is constrained in format but containing huge content variations, AI is proving it’s value.  In recent research AI was able to analyse historical mammogram X-Rays, and after being given the information on which X-rays related to instances of confirmed breast cancer, showed a significant ability to process and identify patients where further investigation was recommended.  Today mammograms are reviewed by at least two qualified Humans, with an early detection rate of around 80%.  AI seems to promise a reduction in the 20% of early detections missed, however the debate runs as to whether the task should be left to AI without Human review on such a critical topic.  The Key to the use of AI, certainly generative AI, is whether you have the time and drive to help AI get things right, and the confidence to let it handle critical tasks without Human oversight.  

The possibilities from AI are enormous, but allowing diverse multi funded projects on multiple AI platforms to drive outcomes that could conflict, duplicate or create erroneous outcomes is tantamount to a production disaster and overspend.  A Deloitte Q3 2024 survey showed that 30% or less of Generative AI projects Have had production implementations.  Their recommendation is that organisations should start their AI journey based on a single platform / providers technology, to increase the understanding of the technology capabilities and the challenges to be overcome to reach production solutions.  Based on our experience to date, for 2025 our recommendation is that all organisations should have an overseer for AI adoption to maintain a lite projects overview, and to keep up to date with the latest advances in technology. 

Our ongoing Journey

 For APPtechnology the journey to AI usage remains targeted to specific outcomes.  Accounts analysis is front and center, AI has been proven to consistently deliver results in comparative analysis, provided that the observations are reviewed before they are actioned.  Then of course we have the AI based services that we procure and use internally, such as Microsoft Purview (Data Loss Prevention), Information Protection and our marketing activities.   What we are a long way from is auto generation of Statements of Work, or complex quotes for implementations, upgrades or managed services.  AI can identify what it sees as the key deliverables for an opportunity, drawn from general customer communications, but AI currently struggles to prioritise delivery based on the nuances of human interactions, conversations, whiteboard workshops and human relationships.

Further Reading

Gartner: AI Data Ready Roadmap (free, registration required)
2024 State of AI report by Digital Leaders (free)

For now AI is helping APPtechnology to analyse the known, spot outlying activity or data, and reduce time or costs on repetitive activities such as workflows and document classification.  Complex proposals, or having the faith to allow AI to generate technical solutions for multi million pound tenders, seems to be a long way off, however the journey, as it has for many organisations, has begun.

 

AI is only as good as the data it’s given

If you need support with analysing and rationalising your estate to reduce complexity, cost and cyber risk, get in touch with APPtechnology and make an appointment with one of our Directors. We'll discuss how our existing AI tools, along with human expertise and oversight, can help safely and productively support your organisation in 2025 and beyond.

CONTACT US


X