SingleStore logo

  • Pricing

User Icon
LoginL
Start free
SingleStore logo
Notebook Icon

Try Out This Notebook to See What’s Possible in SingleStore

Get access to other groundbreaking datasets and engage with our community for expert advice.

Sign UpLog In
SingleStore Spaces/Running Notebooks from Another Notebook with Fusion SQL
New

Running Notebooks from Another Notebook with Fusion SQL

Notebook

SingleStore Notebooks

Running Notebooks from Another Notebook with Fusion SQL

In this notebook, we demonstrate how to use Fusion SQL to run a notebook from another notebook, either within the same session (useful to avoid code duplication) or in a new session (useful for parallel execution).

Creating the Sample Notebook

In this section, we will create a sample notebook to be used in our examples.

Create the sample notebook in the local filesystem:

In [1]:

import nbformat as nbf


nb = nbf.v4.new_notebook()


cell = nbf.v4.new_code_cell("""# This is a code cell
if 'sample_var' not in globals():
    sample_var = 'sample value'
print('Sample Notebook has been executed!')""")


cell.metadata = {
    "language": "python"
}


nb.cells.append(cell)


# Save the notebook to a file
with open('sample_notebook.ipynb', 'w') as f:
    nbf.write(nb, f)


print("Notebook 'sample_notebook.ipynb' created successfully in the local filesystem.")

Upload it to the Data Studio shared files for later download:

In [2]:

import time


sample_notebook_name='Sample Notebook {}.ipynb'.format(int(time.time() * 1_000_000))


%sql UPLOAD SHARED FILE TO '{{ sample_notebook_name }}' FROM 'sample_notebook.ipynb';


print("Notebook '{}' has been created in the Data Studio shared files.".format(sample_notebook_name))

Running the Sample Notebook in the Current Session

In this example, we will run the previously created sample notebook within the current session. This approach is useful for avoiding code duplication in your notebooks, such as environment setup or reusable functions.

To run the notebook, we can use the %run_shared magic command. Let's run the notebook and confirm that the sample_var variable set in the sample notebook is accessible in the current session:

In [3]:

if 'sample_var' in globals():
    del sample_var


%run_shared {{ sample_notebook_name }}


print("The value of 'sample_var' is '{}'.\n".format(sample_var))

Running the Sample Notebook in a New Session

Instead of running a notebook in the current session, we can run it in a new session — either synchronously or asynchronously — using jobs. This approach is useful for parallel execution or running code in a separate runtime environment.

Note

SingleStore enforces limits on the number of compute sessions that can be created, which can be increased upon request. Please contact Support for more information.

Run two jobs in parallel and wait for their completion (each job will run on a separate session):

In [4]:

job_ids = []
for x in range(2):
    print("Running job for {}...".format(x))
    job_res = %sql RUN JOB USING NOTEBOOK '{{ sample_notebook_name }}' WITH PARAMETERS {"sample_var": "{{x}}"}
    job_ids.append(job_res[0].JobID)


print(f'Waiting for jobs to complete... {job_ids}')
success = %sql WAIT ON JOBS {{ job_ids }} WITH TIMEOUT 60 MINUTES


print(f'All jobs completed with success: {bool(success[0].Success)}')

View the executions of the jobs we ran:

In [5]:

for job_id in job_ids:
  execs = %sql SHOW JOB EXECUTIONS FOR {{ job_id }} from 1 to 1
  print(execs)

The jobs can now be dropped:

In [6]:

for id in job_ids:
    print(f"Dropping job '{id}'...")
    %sql DROP JOBS {{id}}

Cleanup

Delete the sample notebook from the Data Studio shared files:

In [7]:

%%sql
DROP SHARED FILE '{{ sample_notebook_name }}'

Details

About this Template

Learn how to run Notebooks from another Notebook in SingleStoreDB Cloud using Fusion SQL.

Notebook Icon

This Notebook can be run in Standard and Enterprise deployments.

Tags

notebooksjobspythonfusionstarter

License

This Notebook has been released under the Apache 2.0 open source license.