Product Enhancements:

• The hallucination claims processor now better handles context with formatted lists
• The experimental hallucination rule received another major upgrade. It is now in beta.
• Users can now view details for keyword and regex rules in the inference deep dive UI
• Users can now filter on a single column by multiple values in the UI inference deep dive UI
• Users can now sort by timestamp in the UI inference deep dive UI
• In the inference deep dive UI, the hallucination claims are ordered to align with the generated response for new inferences moving forward
• Users will now see a message in the “view details” section on an expanded inference if there is additional information available about validation status

Bug Fixes:

• Fixed an issue where the hallucination rule was not considering markdown context like URL links

Product Enhancements:

• Users now have the option to add “Task Name” as a column to the inference deep dive table in the Admin UI
• Users can now search by task name using the search bar on the Admin UI task page
• Added the ability to filter by Inference ID on both the API and the Admin UI

Upgrade Requirement:

• When performing an upgrade, you must have Arthur Shield v2.0.356 on your existing instance before running the upgrade

New features:

• Enhanced the Admin UI Inference Deep Dive page with the ability to explore all hallucination results details including the claims and the context
• Added the ability to collect and retrieve user feedback on the inference context and the Shield results using the /api/v2/feedback endpoints
• Added the ability to capture user_id with inferences on the validation endpoints

Product Enhancements:

• Improved the text parser in the Hallucination rule for handling common abbreviations and markdown
• The LLM backed Shield rules will now return more informative errors when something goes wrong with LLM interactions

Bug fixes:

• Fixed an issue in the UI where the page refresh would trigger after a short period of time even though the user’s session is still alive

Deprecation:

• We are deprecating the ability to set threshold on the PII confidence score. The configuration has been removed from the UI and it is deprecated on the API.

Product Enhancements:

• Improved the concurrency of the toxicity rule execution for performance improvement
• Reduced the false positives of partial profanities, accidental profanities, and obscured niche scandalous internet terms in the toxicity rule
• Added the capability for the hallucination v2 rule to return partial results instead of responding with “Unavailable” for the whole inference when it’s not able to successfully evaluate all claims
• Retrained the prompt injection model and updated the input truncation schema for higher accuracy
• Introduced a new experimental version of hallucination rule (not ready for production use)
• The hallucination v1 rule has been deprecated
• UI: It’s now more clear that sorting the inference deep dive table is done according to timestamp
• UI: Added the ability to navigate to specific pages on the tasks page
• UI: Made usability improvements to filtering on the inference deep dive page

Bug Fix:

• Fixed an issue where an inference was labeled as an overall pass but there was at least one rule that failed

New Features:

• Single-sign-on (SSO) authentication with customer IdPs over OpenId Connect (OIDC)

Product Enhancements:

• The toxicity classifier model was retrained with an upgraded harmful request dataset with better false examples including innocent requests
• All PII configurations are exposed in the UI for creating, editing, and viewing the task rule
• The document selection of the Chat UI will now persist for the entirety of a user’s session

Bug Fixes:

• Fixed an issue where refreshing while on the task page in the shield admin UI would bring the user to the inference deep dive page
• In version 2.0.293, the prompt injection rule was not able to evaluate prompts larger than 512 tokens

Product Enhancements:

• Improved the prompt injections rule with a focus on jailbreaking and instruction manipulation
• The latency metric in milliseconds is captured for each Shield rule execution

API Change:

• Eliminated the regex field from the RegexConfig schema that was deprecated and replaced by regex_patterns in v2.0.190 (January 8, 2024 release)

Product Enhancements:

• Upgrades on the toxicity rule
  • The toxicity rule has been expanded to check for additional categories including harmful/illegal requests
  • The new classifier is trained on more recent and bigger data that has better scores on our benchmarks than our previous classifier
  • The regex component of the toxicity rule received more advanced capabilities to flag profanity and attempted obfuscation of it
• New PII Rule configurations
  • confidence_threshold has been added to ignore PII entity recognitions with confidence scores below the specified threshold
  • disabled_pii_entities is added to disable a certain set of PII entities from getting evaluated
  • allow_list is added for bypassing validations of certain texts
• An informative and helpful error message is returned when the provided config field is invalid during the rule creation process

Bug Fix:

• Adjusted the presentation of PII validation details of the prompts in Chat UI so they’re fully legible

Bug Fix: The default rules were previously enforced on up to 10 tasks. The default rules now work on all active tasks.

Product Enhancements:

• Introducing the Shield Admin UI including 1) a log of inferences and 2) the ability to create and manage rules by task

Bug Fixes:

• Improved handling of authenticated session expirations in the UI
• Improved scrolling through prompts and responses in the Chat UI

• Optimized inferences query endpoint; reducing latency by 90%
• Added page size limit of 250 items to all paginated endpoints
• Fixed toxicity configuration not returned when retrieving rules