The Stack: 3 TB of permissively licensed source code · HackerPapers discussions

The Stack: 3 TB of permissively licensed source code — mentioned by godelski

Wed, 03 May 2023 22:15:38 GMT

You can view the prompts, solutions, and checks here[0]. See my sibling comment (to yours) where I quote the Human Eval paper and do some more analysis. But I think if you look at [0] you'll see that these aren't really unique problems and are likely to have large repetitions in the dataset. I should add to that comment to include the dataset[1] (too late to edit) where they mention that they just scrape all of GitHub (Jan 1 2015 - Mar 31 2022). They do exact and near de-duplicate but near de-duplication is messy. > We implement near-deduplication in our pre-processing pipeline on top of…

Paper and discussion history. A mention is not an endorsement.

The Stack: 3 TB of permissively licensed source code — mentioned by GreedClarifies

Wed, 03 May 2023 16:23:35 GMT

Oh my bad! I thought I read that, is it based upon: https://arxiv.org/abs/2211.15533 (The Stack) ?

Paper and discussion history. A mention is not an endorsement.

The Stack: 3 TB of permissively licensed source code — mentioned by saurabh20n

Tue, 14 Mar 2023 17:28:15 GMT

Congrats on the launch. I think you should share some technical details for a more substantial pitch. You are using the OSS BigCode effort and "The Stack" [1, 2] (as you say in another comment), which is great. A few questions that might help an enterprise customer: How big is your base model? Where did you find more datasets (maybe just a hint would be sufficient)? Are you using SantaCoder [3]? Anything you can say about your fine-tuning that makes it special? Totally on board with you that HumanEval/MBPP are not great benchmarks for real world, and do you have a suggested alternative to…

Paper and discussion history. A mention is not an endorsement.