Nuvolos users can import and upload data into their tables. To do so, the user has a number of options which are illustrated below.
Before proceeding, please consider if your use-case is well suited for loading table data into Nuvolos. Table data has a number of advantages, especially in the following use-cases:
Combining with other datasets: if you are combining a dataset with other datasets on Nuvolos that come in a table format, it is often easiest to do the joins directly via SQL statements.
Cross-application compatibility: tables provide a cross-application typed data structure that can make data easier to analyse with multiple applications.
Large dataset support: tables support data volumes that are sometimes difficult to efficiently store in regular files.
If none of the above benefits are essential, then storing data in files is often the simpler solution.
Uploading data programmatically
The examples below assume that the user is uploading tables to Nuvolos using Nuvolos Applications. Make sure that the table file has been uploaded to Nuvolos. Check the documentation here for information on file upload.
Python
Instead of Pandas's df.to_sql, use nuvolos.to_sql , which has a similar signature but uses highly efficient bulk data loading. df.to_sql is not supported in Python applications running on Nuvolos.
We strongly recommend using lowercase Pandas DataFrame column and index names and a lowercase table name when uploading tables to Nuvolos in order for the resulting database table name and column name to be case-insensitive.
Using uppercase or mixed-case DataFrame column or index names can lead to inconsistent DataFrame column names when reading data with pandas.read_sql, depending on whether the code is run on- or off-Nuvolos.
from nuvolos import get_connection, to_sql
import pandas as pd
df = pd.read_csv(...) #read your data
con = get_connection()
to_sql(df=df, name="table_name", con=con, if_exists='replace', index=False)
from nuvolos import get_connection, to_sql
import pandas as pd
df = pd.read_csv(...) #read your data
con = get_connection(dbname = "dbname", schemaname="schemaname")
to_sql(df=df, name="table_name", con=con, if_exists='replace', index=False)
The above example assumes the use of the latest (>= 0.4.1) version of thenuvolos-odbc (for use in Nuvolos applications) or nuvolospackage (for off-Nuvolos use on your local PC) from PyPI.
To update to the latest nuvolos-odbc (for use in Nuvolos applications) or nuvolos package (for off-Nuvolos use), please execute the following commands in your Python environment:
We provide two solutions to load data into Nuvolos. The standard solution relies on the DBI::dbWriteTable call, which is sufficient for smaller tables. For larger tables, we suggest using an alternative method, as it provides significantly better performance and other convenience features.
Loading smaller tables:
df <- read.csv('path_to_data') #read your data
con <- nuvolos::get_connection()
DBI::dbWriteTable(con, name="table_name", value=df)
Loading large tables:
# Update the Nuvolos connector to the latest version
options(repos = "https://cran.rstudio.com")
install.packages("remotes")
remotes::install_github("nuvolos-cloud/r-connector")
use_condaenv(condaenv = "base", conda = "/usr/local/lib/conda/bin/conda")
nuvolos::to_sql(df=df, name="table_name", if_exists='replace', index=FALSE)
df <- read.csv('path_to_data') #read your data
# Update the Nuvolos connector to the latest version
options(repos = "https://cran.rstudio.com")
install.packages("remotes")
remotes::install_github("nuvolos-cloud/r-connector")
reticulate::install_miniconda()
# You need to restart your R session if miniconda was installed on line 7 and not before
nuvolos::to_sql(df=df, name="table_name", dbname="dbname", schemaname="schemaname", if_exists='replace', index=FALSE)
Matlab
df = readtable('path_to_data')
con = get_connection()
sqlwrite(con,'TABLE_NAME',df)
If you have a specific data onboarding task that you want to discuss with us, you can open a support request by sending an email to support@nuvolos.cloud.
