• GenderNeutralBro@lemmy.sdf.org
    link
    fedilink
    English
    arrow-up
    32
    ·
    1 year ago

    This is why I, as a user, am far more interested in open-source projects that can be run locally on pro/consumer hardware. All of these cloud services are headed down the crapper.

    My prediction is that in the next couple years we’ll see a move away from monolithic LLMs like ChatGPT and toward programs that integrate smaller, more specialized models. Apple and even Google are pushing for more locally-run AI, and designing their own silicon to run it. It’s faster, cheaper, and private. We will not be able to run something as big as ChatGPT on consumer hardware for decades (it takes hundreds of gigabytes of memory at minimum), but we can get a lot of the functionality with smaller, faster, cheaper models.

    • WrittenWeird@lemmy.world
      link
      fedilink
      English
      arrow-up
      10
      ·
      1 year ago

      Definitely. I have experimented with image generation on my own mid-range RX GPU and though it was slow, it worked. I have not tried the latest driver update that’s supposed to accelerate those tools dramatically, but local AI workstations with dedicated silicon are the future. CPU, GPU, AIPU?

    • FaceDeer@kbin.social
      link
      fedilink
      arrow-up
      9
      ·
      1 year ago

      Hundreds of gigabytes of memory in consumer PCs is not decades away. There are already motherboards that accept 128 GB.

      • GenderNeutralBro@lemmy.sdf.org
        link
        fedilink
        English
        arrow-up
        5
        ·
        1 year ago

        You’re right, I shouldn’t say decades. It will be decades before that’s standard or common in the consumer space, but it could be possible to run on desktops within the next generation (~5 years). It’d just be very expensive.

        High-end consumer PCs can currently support 192GB, and that might increase to 256 within this generation when we get 64GB DDR5 modules. But we’d need 384 to run BLOOM, for instance. That requires a platform that supports more than 4 DIMMs, e.g. Intel Xeon or AMD Threadripper, or 96GB DIMMs (not yet available in the consumer space). Not sure when we’ll get consumer mobos that support that much.

      • GenderNeutralBro@lemmy.sdf.org
        link
        fedilink
        English
        arrow-up
        7
        ·
        1 year ago

        Technically I could upgrade my desktop to 192GB of memory (4x48). That’s still only about half the amount required for the largest BLOOM model, for instance.

        To go beyond that today, you’d need to move beyond the Intel Core or AMD Ryzen platforms and get something like a Xeon. At that point you’re spending 5 figures on hardware.

        I know you’re just joking, but figured I’d add context for anyone wondering.