Skip to content
Snippets Groups Projects
Commit 46df55a3 authored by zqwerty's avatar zqwerty
Browse files

rm reddit dial that have url

parent 65928231
No related branches found
No related tags found
No related merge requests found
...@@ -40,6 +40,8 @@ def preprocess(): ...@@ -40,6 +40,8 @@ def preprocess():
if len(utterance) > 256: if len(utterance) > 256:
# remove dialogs that contain too long utterances # remove dialogs that contain too long utterances
return None return None
if 'http://' in utterance or 'https://' in utterance:
return None
speaker = 'system' if i % 2 == 1 else 'user' speaker = 'system' if i % 2 == 1 else 'user'
turn = { turn = {
'speaker': speaker, 'speaker': speaker,
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Please register or to comment