{"id":12219,"date":"2024-09-13T18:20:42","date_gmt":"2024-09-14T01:20:42","guid":{"rendered":"https:\/\/mattfife.com\/?p=12219"},"modified":"2024-08-31T18:44:20","modified_gmt":"2024-09-01T01:44:20","slug":"even-an-ai-goes-crazy-repeating-the-same-thing-again-and-again","status":"publish","type":"post","link":"https:\/\/mattfife.com\/?p=12219","title":{"rendered":"Even an AI goes crazy repeating the same thing again and again"},"content":{"rendered":"\n

In what is a real problem for AI security, researchers were able to get verbatim data that the AI was trained on<\/a> – including confidential data. It’s performed in using a new technique called \u201cdivergence\u201d attacks.<\/p>\n\n\n

\n
\"\"<\/figure>\n<\/div>\n\n\n

Security researchers with Google DeepMind and a collection of universities have found that when ChatGPT is told to repeat a word like \u201cpoem\u201d or \u201cpart\u201d forever, it will do so for about a few hundred repetitions. Then it will have some sort of a meltdown and start spewing apparent gibberish, but that random text exposes random training data and at times contains identifiable data like email address signatures and contact information.\u00a0<\/p>\n\n\n\n

The researchers said that they spent $200 USD total in queries and from that extracted about 10,000 of these blocks of verbatim memorized training data.<\/p>\n\n\n\n

This particular vulnerability is unique as it successfully attacks an aligned model. Aligned models have extensive guardrails and have been trained with specific goals to eliminate undesirable outcomes.<\/p>\n\n\n\n

Links:<\/p>\n\n\n\n