• clb92@kbin.social
    link
    fedilink
    arrow-up
    0
    ·
    1 year ago

    Sure it can create new words. It can’t create new tokens, would be more correct, I think. But a token is just a text fragment, and, as far as I know, they can range from being several words to being single characters.

    • Dr Cog@lemmy.ml
      link
      fedilink
      arrow-up
      1
      ·
      1 year ago

      Tokens are usually never multiple words. Think of them like “information units”. If you have a plural word “hats”, there are two tokens: the “hat” one and the "s’ that adds more information about it being plural. Combinations of words only really occur for proper names.