Commits · 5d71fe60fdcbf972d8d208f2baccd1430d289dc0 · Joel Dag / scalable-c4-preprocessing

Mar 16, 2025
- minor readme adaption · 5d71fe60
  joeld authored 3 weeks ago
  
  5d71fe60
- removed setup.sh · c72333d9
  joeld authored 3 weeks ago
  
  c72333d9
- added commetns · 0a2171b0
  joeld authored 3 weeks ago
  
  0a2171b0
- adapted README.md · 1f6f3a8e
  joeld authored 3 weeks ago
  
  1f6f3a8e
- final adpatons · 06f2b8a7
  joeld authored 3 weeks ago
  
  06f2b8a7
- minor changes, added TODOs · a1ba0603
  joeld authored 4 weeks ago
  
  a1ba0603
- adapted readme + shell scripts + requirements · 356bb018
  joeld authored 4 weeks ago
  
  356bb018
- restructured + cleansed code · 47fac1d8
  joeld authored 4 weeks ago
  
  47fac1d8
- Add master file list for consistent input distribution across processes · 12453f11
  joeld authored 4 weeks ago
  
  12453f11
- added pipline performance measurement logs · 16f64fbe
  joeld authored 4 weeks ago
  
  16f64fbe
- working pipline for parallel processing, added modular processing logic · defc1823
  joeld authored 4 weeks ago
  
  defc1823
Mar 15, 2025
- efficient multiprocess data loader, parallel streaming input files line by line · 37fb8aad
  joeld authored 4 weeks ago
  
  37fb8aad
- initial prepocessing logic for cc_news multi-process preprocessing with sharding + lock · 0f68664f
  joeld authored 4 weeks ago
  
  0f68664f
Mar 12, 2025
- updating sample readme · a963b60a
  Nikit Srivastava authored 1 month ago
  
  a963b60a
- updating sample readme · a17fbc4f
  Nikit Srivastava authored 1 month ago
  
  a17fbc4f
Feb 16, 2025
- updating comments · 6192d783
  Nikit Srivastava authored 1 month ago
  
  6192d783
Feb 13, 2025
- updating preprocess_dataset.sh · cc1c1e69
  Nikit Srivastava authored 1 month ago
  
  cc1c1e69
- adding assignment template files · b700066b
  Nikit Srivastava authored 1 month ago
  
  b700066b