GNU Shepherd 1.0.0 Released
38 by sharpshadow | 5 comments on Hacker News.
Tuesday, January 14, 2025
Monday, January 13, 2025
Sunday, January 12, 2025
New top story on Hacker News: Show HN: SemHash – Fast Semantic Text Deduplication for Cleaner Datasets
Show HN: SemHash – Fast Semantic Text Deduplication for Cleaner Datasets
11 by Pringled | 2 comments on Hacker News.
We’ve just open-sourced SemHash, a lightweight package for semantic text deduplication. It lets you effortlessly clean up your datasets and avoid pitfalls caused by duplicate samples in semantic search, RAG, and machine learning. Main Features: - Fast and hardware friendly: Deduplicate datasets with millions of records in minutes, on a CPU. - Flexible: Works on single or multiple datasets (e.g., train/test deduplication), and multi-column data (e.g., Question-Answering datasets). - Lightweight: Minimal dependencies (largest is NumPy). - Explainable: Easily inspect duplicates and what caused them, and view the lowest similarity duplicates to adjust the threshold based on your dataset. We found that text deduplication is more complex than it appears, so we built SemHash to simplify the process. Duplicate samples can skew model training, reduce generalization, and cause train-test leakage—leading to unreliable results. Techniques like minhash handle exact or near-exact duplicates, but semantic deduplication also catches semantically redundant samples, which we believe is an important aspect of deduplication. Furthermore, it’s not trivial to see why something was removed with minhash, which we also believe is important. We already found some interesting results on some well known datasets in our benchmarks which are included in the repo. We are curious to hear your feedback! Do you currently deduplicate your datasets before training, and what techniques do you use?
11 by Pringled | 2 comments on Hacker News.
We’ve just open-sourced SemHash, a lightweight package for semantic text deduplication. It lets you effortlessly clean up your datasets and avoid pitfalls caused by duplicate samples in semantic search, RAG, and machine learning. Main Features: - Fast and hardware friendly: Deduplicate datasets with millions of records in minutes, on a CPU. - Flexible: Works on single or multiple datasets (e.g., train/test deduplication), and multi-column data (e.g., Question-Answering datasets). - Lightweight: Minimal dependencies (largest is NumPy). - Explainable: Easily inspect duplicates and what caused them, and view the lowest similarity duplicates to adjust the threshold based on your dataset. We found that text deduplication is more complex than it appears, so we built SemHash to simplify the process. Duplicate samples can skew model training, reduce generalization, and cause train-test leakage—leading to unreliable results. Techniques like minhash handle exact or near-exact duplicates, but semantic deduplication also catches semantically redundant samples, which we believe is an important aspect of deduplication. Furthermore, it’s not trivial to see why something was removed with minhash, which we also believe is important. We already found some interesting results on some well known datasets in our benchmarks which are included in the repo. We are curious to hear your feedback! Do you currently deduplicate your datasets before training, and what techniques do you use?
Saturday, January 11, 2025
New top story on Hacker News: Show HN: 3D Terrain simulation for hiking, skiing etc.
Show HN: 3D Terrain simulation for hiking, skiing etc.
17 by folli | 3 comments on Hacker News.
I'm working on a GPS track visualizer for quite some time. It shines in hilly and mountaineous terrain (where a 3D view makes more sense), but it also offers quite a lot of Strava-like features (statistics etc.). You can upload your GPX and FIT files manually, or sync directly with Garmin, Coros and Polar. See https://cubetrek.com for the live app and check out some examples there. It's free and opens source. Also, anyone who likes to work with 3D visualizations (especially Babylon.js), let me know if you like to help polish this thing further.
17 by folli | 3 comments on Hacker News.
I'm working on a GPS track visualizer for quite some time. It shines in hilly and mountaineous terrain (where a 3D view makes more sense), but it also offers quite a lot of Strava-like features (statistics etc.). You can upload your GPX and FIT files manually, or sync directly with Garmin, Coros and Polar. See https://cubetrek.com for the live app and check out some examples there. It's free and opens source. Also, anyone who likes to work with 3D visualizations (especially Babylon.js), let me know if you like to help polish this thing further.