restructured + cleansed code

data/c4_realnewslike_big/c4-train.00000-of-00512.json.gz

data/c4_realnewslike_big/c4-train.00001-of-00512.json.gz

data/processed/processed_0.jsonl

data/processed/processed_1.jsonl

data/processed/shard_0.jsonl

data/processed/shard_1.jsonl

data_loader.py

main.py

preprocess_data.py

shard_processor.py

text_preprocessor.py