This is an abstract curiosity. Let’s say I want to use an old laptop to run a LLM AI. I assume I would still need pytorch, transformers, etc. What is the absolute minimum system configuration required to avoid overhead such as schedulers, kernel threads, virtual memory, etc. Are there options to expose the bare metal and use a networked machine to manage overhead? Maybe a way to connect the extra machine as if it is an extra CPU socket or NUMA module? Basically, I want to turn an entire system into a dedicated AI compute module.

  • InvertedParallax@lemm.ee
    link
    fedilink
    English
    arrow-up
    21
    ·
    1 year ago

    Stop. Go back. This is the wrong way.

    If you’re running python you basically need a full os.

    There are projects that run as an rtos, and in fact I worked on an ml soc that ran Linux, but there are 2 levels here:

    1. The ml processing itself, ie the math. This is simple in software and very complex otherwise. The software just says “copy this block and start running a matrix multiply”. The hard logic is in moving data around efficiently.

    2. The stack. This is high level, python or so, and has graph processing overhead too. This needs a lot of “overhead” by its nature.

    In either case, don’t worry about any of this, the overhead won’t be very noticeable, you’ll be cpu gated hard, the main thing is finding an optimized pytorch library.

    If you have an amd cpu or somehow have an nvidia gpu in your laptop you might be able to use their pytorch library which would improve performance by roughly 1.5-2 orders of magnitude.

    Unfortunately there isn’t a pytorch implementation for Intel igpus, but there is an opencl backend for pytorch, and apparently this madlad got it working through opencl on an Intel igpu: https://dev-discuss.pytorch.org/t/implementing-opencl-backend-for-pytorch/283/9

    But don’t worry about overhead, it’s less than fractions of percents in these kinds of tasks and there are ways to bypass them completely.

    • j4k3@lemmy.worldOP
      link
      fedilink
      English
      arrow-up
      3
      ·
      1 year ago

      Seems like avoiding context switching and all the overhead associated would make a big difference when pretty much everything in cache is critical data.

      I’m more curious about running something like Bark TTS where the delay is not relevant, but it would be cool to have the instructional clarity of the Halo Master Chief voice read me technical documentation, or test the effects of training my own voice, tuned to how I hear it, reading me stuff I find challenging. If the software is only able to process around 10 seconds at a time, just script it and let it run. The old machine will just collect dust otherwise.

      Anyways, what’s the best scheduler with affinity/isolation/pinning?

      • Spike@discuss.tchncs.de
        link
        fedilink
        English
        arrow-up
        4
        ·
        1 year ago

        Seems like avoiding context switching and all the overhead associated would make a big difference when pretty much everything in cache is critical data.

        It’s not. Like the commenter above said: It’s a fraction of the task at hand. Especially when you design the rest of the system to run only if necessary. Context Switches are what? like 50 CPU Cycles? Store Registers, Store TCB, Load other TCB and load other register states jump back to PC. Maybe some other OS Shenanigans, but that’s basically it.

        Now Imagine complex calculations on a 25-Dimensional Matrix.

      • InvertedParallax@lemm.ee
        link
        fedilink
        English
        arrow-up
        4
        ·
        1 year ago

        K, for that look at a kernel subsystem/feature called cpu_isol, friend of mine implemented/upstreamed, basically you take cores half out of Linux and can use them for heavy workloads.

        But I doubt you’d see more than 1% improvement, linux doesn’t do that much without you asking.

        You can try setting rt priority but I’ve never found that to matter much.

        Listen, this is the kind of thing I would have tried a decade ago, but the thing to remember is: time spent improving algorithm is generally more effective than time trying to optimize kernel overhead that millions of people have been trying to optimize for decades.

        • j4k3@lemmy.worldOP
          link
          fedilink
          English
          arrow-up
          3
          ·
          1 year ago

          Thanks. I went down this rabbit hole already with large FreeCAD assemblies where single threaded operations had an enormous amount of non voluntary context switching. It helped some in that instance. I don’t have a benchmark, but I reduced the NVCS by an order of magnitude using Tuna to set affinity and pinning. Editing a large assembly in the middle of the tree went from 5++ minutes to around 3-4 minutes.

          I’m not claiming I know better, or that this is a bright idea. As mentioned, it’s an abstract curiosity. Playing with FreeCAD got me interested in the OS on a deeper level, and got me reading as much as I can about the CFS and others. The LLM toolchain I’m working on is to help me follow the CS curriculum posted freely online from UC Berkeley. Ultimately I plan to use a langchain database to help me follow along with the books and transcribed lectures. It is just a hobby interest.

          Thanks for the insights!

          • InvertedParallax@lemm.ee
            link
            fedilink
            English
            arrow-up
            2
            ·
            1 year ago

            That’s interesting, I spent a decade doing hpc and other optimizations for large software on 2 socket systems, there are degenerate cases, which can be fixed, I just doubt they’re here.

            Freecad sounds like it was poorly written with a lot of hopping about ram with poor cache localization, which happens but is pretty ugly.

            Ml tends to be better behaved, it’s actually very close to dsp code and the compilers try to enforce locality, more importantly a lot of the modules are hand coded for extreme performance.

            I’m not trying to be that discouraging, I’m saying this as someone who originally looked for performance, and often found it in the os, but later found more performance in the loops themselves or the compiler, basically linux is a lot smarter than it used to be, and many applications are too.

            Just my 2c, there are performance tools that can tell you how bad the os is vs other things, and you shouldn’t be swapping so much that it hurts you a lot in ml.

          • InvertedParallax@lemm.ee
            link
            fedilink
            English
            arrow-up
            1
            ·
            1 year ago

            Actually, simple pinning might be enough if you’re seeing a lot of thrash, but most ml systems have something like openmp to handle that automatically.