core

Fetch competition data, push notebooks, and maintain library datasets on Kaggle

On Kaggle, credentials come from Kaggle secrets rather than the usual ~/.kaggle/kaggle.json, so getting an authenticated API client depends on where you are. import_kaggle handles both cases:

source

import_kaggle

def import_kaggle():

Import kaggle API, using Kaggle secrets kaggle_username and kaggle_key if needed

The kaggle package authenticates when it is first imported, but silently ignores failure. Calling api.authenticate() again means a missing credential fails loudly here, and on Kaggle it picks up the secrets we just copied into the environment.

The API’s list calls return response objects whose payload lives in an attribute, for example competitions_list().competitions:

api = import_kaggle()
L(api.competitions_list().competitions).attrgot('title')

['Passenger Screening Algorithm Challenge', 'Zillow Prize: Zillow’s Home Value Prediction (Zestimate)', 'Data Science Bowl 2017', 'Vesuvius Challenge - Ink Detection', 'ARC Prize 2026 - ARC-AGI-3', 'ARC Prize 2026 - ARC-AGI-2', 'Google DeepMind - Vibe Code with Gemini 3 Pro in AI Studio ', 'Red‑Teaming Challenge - OpenAI gpt-oss-20b', 'OpenAI to Z Challenge', 'ARC Prize 2026 - Paper Track', 'The Pokémon Company - PTCG AI Battle Challenge Strategy', 'LLM Prompt Recovery', 'Vesuvius Challenge - Surface Detection', 'Google - American Sign Language Fingerspelling Recognition', 'Second Annual Data Science Bowl', 'The Gemma 4 Good Hackathon', 'Measuring Progress Toward AGI - Cognitive Abilities', 'National Data Science Bowl', '2019 Data Science Bowl', 'Feedback Prize - Evaluating Student Writing']

Competitions

source

setup_comp

def setup_comp(
    competition, install:str=''
):

Get a path to data for competition, downloading it if needed

setup_comp('titanic')

Path('titanic')

If you pass a list of space separated modules to install, they’ll be installed if running on Kaggle.

source

competition_submit

def competition_submit(
    file_name, message, competition
):

Submit file_name to competition, returning the submission response

Once you’ve created a submission file, submit it directly from your script or notebook. E.g:

competition_submit('subm.csv', 'first try', 'titanic')

The response includes a message confirming the submission was created. Note that on Kaggle “code competitions” you must instead submit a notebook through push_notebook.

Notebooks

source

nb_meta

def nb_meta(
    user, id, title, file, competition:NoneType=None, private:bool=True, gpu:bool=False, internet:bool=True,
    linked_datasets:NoneType=None
):

Get the dict required for a kernel-metadata.json file

nb_meta('jhoward', 'my-notebook', 'My notebook', 'my-notebook.ipynb', competition='paddy-disease-classification')

{'id': 'jhoward/my-notebook',
 'title': 'My notebook',
 'code_file': 'my-notebook.ipynb',
 'language': 'python',
 'kernel_type': 'notebook',
 'is_private': True,
 'enable_gpu': False,
 'enable_internet': True,
 'keywords': [],
 'dataset_sources': [],
 'kernel_sources': [],
 'competition_sources': ['competitions/paddy-disease-classification']}

source

push_notebook

def push_notebook(
    user, id, title, file, path:str='.', competition:NoneType=None, private:bool=True, gpu:bool=False,
    internet:bool=True, linked_datasets:NoneType=None
):

Push notebook file to Kaggle Notebooks

Note that Kaggle recommends that the id match the slug for the title – i.e it should be the same as the title, but lowercase, no punctuation, and spaces replaced with dashes. E.g:

push_notebook('jhoward', 'first-steps-road-to-the-top-part-1',
              title='First Steps: Road to the Top, Part 1',
              file='first-steps-road-to-the-top-part-1.ipynb',
              competition='paddy-disease-classification',
              private=False, gpu=True)

The response returned by Kaggle includes the notebook’s url, and an error string if the push failed.

Datasets

Core

source

check_ds_exists

def check_ds_exists(
    dataset_slug, # Dataset slug (ie "uciml/iris")
):

Does dataset_slug exist on Kaggle?

Because it asks Kaggle directly, this works for any public dataset, not just your own. A dataset you cannot see reports as not existing:

assert not check_ds_exists('zillow/no-such-dataset')
check_ds_exists('uciml/iris')

True

source

mk_dataset

def mk_dataset(
    dataset_path, # Local path to create dataset in
    title, # Name of the dataset
    force:bool=False, # Should it overwrite or error if exists?
    upload:bool=True, # Should it upload and create on kaggle
):

Creates minimal dataset metadata needed to push new dataset to kaggle

The upload=False form only creates the folder and its dataset-metadata.json locally, which is also how we can demonstrate it without creating a real dataset:

mk_dataset('./testds', 'mytestds', force=True, upload=False)
md = json.load(open('./testds/dataset-metadata.json'))
assert md['title'] == 'mytestds'
assert md['id'].endswith('/mytestds')
md

Data package template written to: testds/dataset-metadata.json

{'title': 'mytestds',
 'id': 'jhoward/mytestds',
 'licenses': [{'name': 'CC0-1.0'}]}

get_dataset downloads an existing dataset, along with its metadata file, ready to update and push back:

source

get_dataset

def get_dataset(
    dataset_path, # Local path to download dataset to
    dataset_slug, # Dataset slug (ie "uciml/iris")
    unzip:bool=True, # Should it unzip after downloading?
    force:bool=False, # Should it overwrite or error if dataset_path exists?
):

Downloads an existing dataset and metadata from kaggle

To fill a library dataset with installable files, download the library’s wheels with pip:

source

get_pip_library

def get_pip_library(
    dataset_path, # Local path to download pip library to
    pip_library, # name of library for pip to install
    pip_cmd:str='pip', # pip base to use (ie "pip3" or "pip")
):

Download the whl files for pip_library and store in dataset_path

get_pip_libraries does the same for everything in a requirements.txt file.

source

get_pip_libraries

def get_pip_libraries(
    dataset_path, # Local path to download pip libraries to
    requirements_path, # path to requirements file
    pip_cmd:str='pip', # pip base to use (ie "pip3" or "pip")
):

Download whl files for a requirements.txt file and store in dataset_path

dl_path = Path('./mylib')
get_pip_library(dl_path,'fastkaggle')
assert 1==len([o for o in dl_path.ls() if str(o).startswith(f"{dl_path}/fastkaggle")])

Once the local folder holds the new files, push_dataset uploads a new version:

source

push_dataset

def push_dataset(
    dataset_path, # Local path where dataset is stored
    version_comment, # Comment associated with this dataset update
):

Push dataset update to kaggle. Dataset path must contain dataset metadata file

get_local_ds_ver reads the version number from a library’s wheel in a local dataset copy, so the high level functions below can tell whether the Kaggle copy is up to date.

source

get_local_ds_ver

def get_local_ds_ver(
    lib_path, # Local path dataset is stored in
    lib, # Name of library (ie "fastcore")
):

Checks a local copy of kaggle dataset for library version number

High Level

source

create_libs_datasets

def create_libs_datasets(
    libs, # library or list of libraries to create datasets for (ie 'fastcore' or ['fastcore','fastkaggle'])
    lib_path, # Local path to dl/create dataset
    username, # Your username
    clear_after:bool=False, # Delete local copies after sync with kaggle?
):

For each library, create or update a kaggle dataset with the latest version

source

create_requirements_dataset

def create_requirements_dataset(
    req_fpath, # Path to requirements.txt file
    lib_path, # Local path to dl/create dataset
    title, # Title you want the kaggle dataset named
    username, # Your username
    retain:list=['dataset-metadata.json'], # Files that should not be removed
    version_notes:str='New Update', # Comment associated with this dataset update
):

Download everything needed in a requirements.txt file to a dataset and upload to kaggle