CS336: Language Modeling from Scratch

 (cs336.stanford.edu)

142 points | by kristianpaul 3 hours ago

6 comments

  • meken 1 hour ago

    I have fond memories of cs224d [1] taught by richardsocher. It’s a bit dated at this point as it was created in the pre-transformer era, but it was a very cool introduction to applying deep learning to nlp at the time.

    [1] https://cs224d.stanford.edu

    [-]
    • egl2020 49 minutes ago

      Similar thoughts here. That was when I realized the potential of the Internet: I didn't have to be a grad student at a tier 1 research university to learn about the frontier.

    • skerit 1 hour ago

      > GPU compute for self-study

      Those suggestions they make for a B200 start at $4.99 an hour.

      Is that really required, for starting out? I've been tinkering with my own from-scratch LLM, but in the early phases I don't need anything more than a 4090 on Vast.ai

      [-]
      • flakiness 17 minutes ago

        I beliee these are affordable enough for the intended audience (which is Stanford undergrad/master)

        • root-parent 46 minutes ago

          You dont even need a GPU to train your own LLM.

        • sonabinu 23 minutes ago

          I brought a group together to do this class using the YouTube videos and course materials available online. It is challenging but rewarding. We tackled it one lecture video per week. Started with over 30 learners and by last session we were down to 8.

          • airstrike 54 minutes ago

            I wonder if people prefer to learn this on their own or if building a community around open learning is something that others are interested in

            • storus 1 hour ago

              Thanks for releasing this again! What are this year's changes to prior offerings?

              • tmule 1 hour ago

                Are video lectures available online?

                [-]