How Working Groups Extract Value from AI

Technology

How Working Groups Extract Value from AI

By Ben Craker, Portfolio Manager

The AI buzz is palpable everywhere as companies large and small try to ‘rassle’ this 500-pound gorilla to the ground. That includes your portfolio team at AgGateway.

What’s interesting is the unique position we occupy. We not only look at it from the standpoint of what it can do for our work, but also what our member volunteers are doing with it. As we share and engage with them on projects, inevitably we interact on the power, functionality, and limitations of AI. Everything from simply replacing Google searches to writing code has been raised in discussions, and the boundaries and questions change as the technology shifts.

But ultimately, few are sitting on the sidelines our working group volunteers are no exception, they are using AI in a variety of ways to help make progress. Below is a quick summary of what worked when AI was applied in several different working groups in the last few months.

1. Harnessing the Internet. AI seems to be a great resource for “fancy” internet searches. You ask it a question and instead of a list of links to read through you get a pretty good answer summarizing those links without having to browse through each website. This has been handy during several working group discussions. When a concept or topic comes up that not everyone is familiar with, participants post info in the meeting chat that helps explain concepts or clarify questions the group has.

2. Ensuring Quality Returns. One great tip from Stuart Rhea for this kind of work is to put in a standing protocol for your AI of choice to follow. In the settings of most chat bots, usually under personalization there is a “custom instructions” or some similar setting. There you can issue standing orders for all your interactions with the LLM. It can be anything, setting the tone to be more or less formal, summarizing using bullet points instead of paragraphs, whether to prioritize internal data sources, etc. Stuart’s suggestion was five “Session Reliability Instructions” for the AI to confirm at the start of each new chat session:

1. The task type
2. Stakes
3. Allowed sources
4. Recency
5. Support threshold

These instructions have a little more detail, but help to make sure the AI is clear on the context of what it is doing, how much it should infer vs. only cite direct sources, and what sources should be used. In addition to these questions are instructions to clearly tag where inference was used so that it is not presented as fact, allowing you to better identify where a hallucination may have occurred, or where an incorrect source might have been referenced. The interesting thing is how well the AI adjusts for each session, if have a quick, low stakes question on the background of a topic, you can lower the threshold for what sources are ok. Or if you need to know you have the exact most up-to-date information about a regulatory requirement you can target that session to only recent material from authoritative sites.

3. Summarizing Information. Another area AI has helped in several working groups is summarizing information. AgGateway members are keenly aware of the inefficiencies in reinventing the wheel. That leads many working groups to find papers, implementation guides, or even other standards that seem to address at least part of the issue they are working to resolve but may or may not be relevant to their deliverables. Since time to read all of these documents is not something many can commit to, several working groups have had AI generate summaries and identify the key points that might be relevant for their primary use cases. This helps the groups sort through what existing materials that seem relevant actually are and warrant a closer look, and which do not really apply, allowing attention to be focused elsewhere.

4. Streamlining Tedious Data Tasks. Another use case AI has been a big help is in the latest update to the Modus lab test method lists. Toward the end of the working group that created the v2 plant tissue method list, the team realized there had been some drift between the soil, manure and plant tissue lists. The same analyte had slightly different names in each list, and similar methods were described in slightly different ways between lists. With over 800 methods in the soil list, around 130 in the manure list, and 175 in the plant tissue list, going through manually to find all these variances and fixing them was going to be a huge effort. However, using an AI tool to review the content, with strict rules based on the SRI protocol mentioned above, it was able to quickly sort out the identified issues. It also found a few other inconsistencies and generated detailed documentation about the changes while updating the method list files, tracking the changes and updating the suppression files. And it was highly accurate, looking through the work, what it did in a short amount of time was impressive. It was also helpful that part of the task was to basically track all the changes and document everything, making it a little easier to verify it did not go off the rails.

These are a few of the successes, and while not all have been smooth, they offer a glimpse of the benefits of AI to come as it improves. Also, as shocking as it may seem, results from the paid subscription to a tool are much better than the free version.

It does take some trial and adjusting settings to get consistent output, but overall, AI has been a great contributor enabling several working groups to move more quickly and generate better digital resources, and we will continue to keep you posted on our progress working with AI in the months ahead.