Parag Ekbote

AINovice2005

https://paragekbote.github.io/

AI & ML interests

ML Engineer passionate about taking models from research to production. 1 year supporting tech startups. Active OSS contributor.

Recent Activity

posted an update 5 days ago

The hf article detailing the integration between hf datasets and dagster is live 🤗 : https://huggingface.co/blog/AINovice2005/dagster-hf-datasets

published an article 5 days ago

From Data Repositories to Production Data Pipelines: Bridging Hugging Face Datasets and Dagster with dagster-hf-datasets

published a bucket 6 days ago

the-hf-stack/dagster-hf-datasets-examples

View all activity

Organizations

posted an update 5 days ago

Post

The hf article detailing the integration between hf datasets and dagster is live 🤗 : https://huggingface.co/blog/AINovice2005/dagster-hf-datasets

published an article 5 days ago

Article

From Data Repositories to Production Data Pipelines: Bridging Hugging Face Datasets and Dagster with dagster-hf-datasets

AINovice2005

•

5 days ago

• 3

published a bucket 6 days ago

the-hf-stack/dagster-hf-datasets-examples

0 Bytes

updated a Space 6 days ago

README

🚀

Description of HF Organization

posted an update 8 days ago

Post

Excited to share the release of dagster-hf-datasets: A Dagster-native integration that brings Hugging Face Datasets into Dagster's asset-oriented orchestration model

The integration enables:

• 🤗 Dataset and DatasetDict assets
• 🐙 Dagster asset lineage and observability
• 📦 Parquet-backed materialization via HFParquetIOManager
• 🚀 Publishing curated datasets back to the Hugging Face Hub
• 📝 Automatic dataset card generation from pipeline metadata

As the Hub continues to grow beyond 1M+ datasets, orchestration, reproducibility, and observability are becoming increasingly important parts of the dataset lifecycle. I'm also working on a longer article covering the architecture and data pipelines enabled by the integration.

More Soon!

https://github.com/dagster-io/community-integrations/tree/main/libraries/dagster-hf-datasets

https://github.com/dagster-io/community-integrations/tree/main/libraries/dagster-hf-datasets/docs