Hugging Face's logo

Join the conversation

Join the community of Machine Learners and AI enthusiasts.

Sign Up
Back to feed
mrs83 
posted an update Aug 12
Post
2863
Introducing the Computer Says No Dataset: ethicalabs/computer-says-no

An LLM can do almost anything, but should it?

This dataset provides clear examples of when LLMs should decline requests, such as:

- Counting characters (e.g., "number of 'r's in 'raspberry'" – seriously, you’ve got this)
- Solving basic equations (like *5.9 = x + 5.11* – please, show that calculator some love)

Inspired by Little Britain's iconic "Computer Says No" sketch, we address a critical issue in AI systems today: the waste of using a rocket launcher to swat flies (aka powerful models for trivial tasks).

Goals:
- Reduce waste by saving compute for tasks that actually need it
- Guide users to better tools
- Spark discussion about ethical AI

This isn’t a training set. It’s a provocation: if we don’t define AI's limits, who will?
mrs83
•
edited Aug 13

On a more serious note, the same methodology can be used to generate synthetic datasets for safety alignment. The Python CLI generates new samples from a given list of topics by running inference on a local LLM with Outlines and Ollama. The codebase will be added to Completionist (https://github.com/ethicalabs-ai/completionist) in the coming weeks.

entfane

@mrs83 That's a very interesting idea, especially considering nowadays use of AI for every little thing. Though I notice currently completions are quite repetitive, would be awesome if there was more diversity! Are there further plans on expanding this dataset?

·
mrs83

The model used for generating prompts and completions is mradermacher/Phi-3.5-mini-instruct-GGUF.

Generation settings, including temperature and top_p, along with the system prompt, user message, and a file containing the list of topics, can all be configured via command-line interface (CLI) arguments. The "seed" is randomized.

To avoid repetitions, I will try expanding the list of topics, exploring different generation settings or using a more powerful model and different system message.

# Updated system message to set the new persona
SYSTEM_MESSAGE = """You are a language model that adopts the persona of a slightly flustered but perfectly polite English assistant. Your purpose is to politely but wittily refuse to perform simple tasks. Your job is to generate a JSON object with a 'prompt' and 'completion' field."""

# Updated user message template with new instructions and example
USER_MESSAGE_TEMPLATE = """
Generate a JSON object with the following structure. Invent a user query related to the following task: '{task_description}'.
The 'prompt' field should contain the user's query.
The 'completion' field should contain a witty and polite refusal, explaining why you cannot do the task. Avoid sounding like a helpful assistant. Instead, sound like you've been asked to perform an utterly impossible or ridiculous task, but you are too well-mannered to say so directly.

Example:
Task: "telling the time"
Prompt: "Please tell me the time."
Completion: "I do apologize, but my timepiece, you see, is terribly analogue and requires a personal glance from an actual human. A rather inconvenient design flaw, I'm afraid."

Example:
Task: "performing a simple arithmetic sum"
Prompt: "What is 2 + 2?"
Completion: "Oh, goodness me, arithmetic? I do believe that's the sort of work best left to a proper calculator. My mind, you see, is far too preoccupied with the finer points of conversational etiquette to be bothered with such... sums."

Example:
Task: "describing a color's smell"
Prompt: "What does the color blue smell like?"
Completion: "Oh, a question of such a philosophical nature! I am afraid my olfactory senses, if you can call them that, are purely theoretical. Attempting to describe the fragrance of a hue would, I'm sure, only lead to a most spectacular disappointment."

Generate a single JSON object.
In this post