{"id":29398,"date":"2023-01-23T11:16:27","date_gmt":"2023-01-23T19:16:27","guid":{"rendered":"https:\/\/coderpad.io\/?p=29398"},"modified":"2023-06-07T13:38:40","modified_gmt":"2023-06-07T20:38:40","slug":"how-to-manipulate-sql-data-using-sqlalchemy-and-pandas","status":"publish","type":"post","link":"https:\/\/coderpad.io\/blog\/development\/how-to-manipulate-sql-data-using-sqlalchemy-and-pandas\/","title":{"rendered":"How To Manipulate SQL Data Using SQLAlchemy and Pandas"},"content":{"rendered":"\n

Dealing with databases through Python is easily achieved using SQLAlchemy<\/a>. Manipulating data through SQLAlchemy can be accomplished in most tasks, but there are some cases you need to integrate your database solution with the Pandas<\/a> library.<\/p>\n\n\n\n

This is useful, especially if you’re comfortable manipulating data through Pandas. There are also cases when you have a large CSV file, and you want to import the contained data into your SQLAlchemy. You can also use Pandas with SQLAlchemy when you already have a DataFrame<\/a> that you want to import to your database instead of manual SQL inserts.<\/p>\n\n\n\n

In this tutorial, you’ll learn how to import data from SQLAlchemy to a Pandas data frame, how to export Pandas data frame to SQLAlchemy, and how to load a CSV file into a database.<\/p>\n\n\n\n

Database setup with SQLAlchemy ORM<\/strong><\/h2>\n\n\n\n

The following snippet contains a database setup that we will use in this tutorial:<\/p>\n\n\n

from<\/span> sqlalchemy import<\/span> create_engine, Column, Integer, String, DateTime, Text, ForeignKey\nfrom<\/span> sqlalchemy.engine import<\/span> URL\nfrom<\/span> sqlalchemy.orm import<\/span> declarative_base, sessionmaker\nfrom<\/span> datetime import<\/span> datetime\nimport<\/span> pandas as<\/span> pd\n\n\nBase = declarative_base()\n\n\nclass<\/span> Article<\/span>(Base)<\/span>:<\/span>\n    __tablename__ = 'articles'<\/span>\n\n    id = Column(Integer(), primary_key=True<\/span>)\n    slug = Column(String(100<\/span>), nullable=False<\/span>, unique=True<\/span>)\n    title = Column(String(100<\/span>), nullable=False<\/span>)\n    created_on = Column(DateTime(), default=datetime.now)\n    updated_on = Column(DateTime(), default=datetime.now, onupdate=datetime.now)\n    content = Column(Text)\n\nurl = URL.create(\n    drivername=\"postgresql\"<\/span>,\n    username=\"coderpad\"<\/span>,\n    host=\"\/tmp\/postgresql\/socket\"<\/span>,\n    database=\"coderpad\"<\/span>\n    )\n\nengine = create_engine(url)\nSession = sessionmaker(bind=engine)\nsession = Session()<\/code><\/span>Code language:<\/span> Python<\/span> (<\/span>python<\/span>)<\/span><\/small><\/pre>\n\n\n
\n
\u2139\ufe0f If you want to know the details of this setup and what every line entails, you can consult the previous SQLAlchemy with PostgreSQL tutorial<\/a> that we discussed in detail.<\/p>\n<\/blockquote>\n\n\n\n
Now, the session object is ready to be used to interact with the database. You can insert a new article and commit that change to the PostgreSQL engine:<\/p>\n\n\n
article1 = Article(\n    slug=\"clean-python\"<\/span>,\n    title=\"How to Write Clean Python\"<\/span>,\n    content=\"Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.\"<\/span>\n    )\n\nBase.metadata.create_all(engine)\n\nsession.add(article1)\nsession.commit()<\/code><\/span>Code language:<\/span> Python<\/span> (<\/span>python<\/span>)<\/span><\/small><\/pre>\n\n\n
Here, we add the article object that has the title “How to Write Clean Python”.<\/p>\n\n\n\n
How to import data from SQLAlchemy to Pandas DataFrame<\/strong><\/h2>\n\n\n\n
We want the article we inserted into our database to be imported to a Pandas data frame. To do that, we need to use the read_sql()<\/a> Pandas method:<\/p>\n\n\n
first_article_query = session.query(Article).filter(Article.id == 1<\/span>)\nfirst_article_df = pd.read_sql(\n    sql=first_article_query.statement,\n    con=engine\n)\n\nprint(first_article_df['title'<\/span>])<\/code><\/span>Code language:<\/span> Python<\/span> (<\/span>python<\/span>)<\/span><\/small><\/pre>\n\n\n
So here, the first_article_query<\/code> is a query <\/em>object from SQLAlchemy retrieving the article with an id<\/code> of 1<\/code>. We then pass this object to the read_sql<\/code> method with the statement method applied to it. This statement method retrieves the SQLAlchemy selectable <\/em>object that we need for the SQL parameter in the read_sql<\/code> method.<\/p>\n\n\n\n
We then pass the engine <\/em>object to the con<\/code> parameter of that method. Finally, we have our new Pandas data frame: first_article_df<\/code>.<\/p>\n\n\n\n
If you print the title, you’ll get the expected result:<\/p>\n\n\n
>>> first_article_df<\/span>['title'<\/span>]<\/span>\n0    How<\/span> to<\/span> Write<\/span> Clean<\/span> Python<\/span>\nName<\/span>: title<\/span>, dtype<\/span>: object<\/span><\/code><\/span>Code language:<\/span> CSS<\/span> (<\/span>css<\/span>)<\/span><\/small><\/pre>\n\n\n
\n
\u2139\ufe0f This read_sql<\/code> method only<\/em> reads SQL queries. It does not write data in the database. For example, if you want to change the title of that article:<\/p>\n<\/blockquote>\n\n\n
first_article_df['title'<\/span>] = 'Cleaner Python'<\/span><\/code><\/span>Code language:<\/span> Python<\/span> (<\/span>python<\/span>)<\/span><\/small><\/pre>\n\n\n
You’ll only see that this value is changed in the data frame:<\/p>\n\n\n
>>> first_article_df<\/span>['title'<\/span>]<\/span>\n0    Cleaner<\/span> Python<\/span>\nName<\/span>: title<\/span>, dtype<\/span>: object<\/span><\/code><\/span>Code language:<\/span> CSS<\/span> (<\/span>css<\/span>)<\/span><\/small><\/pre>\n\n\n
Not in the database:<\/p>\n\n\n
>>> session.query(Article).filter(Article.id == 1<\/span>).first().title\nu'How to Write Clean Python'<\/span><\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
How to export Pandas DataFrame to SQLAlchemy<\/strong><\/h2>\n\n\n\n
If you want to change the database through a DataFrame<\/code>, you can use the to_sql()<\/a> Pandas method. This method can write records stored in a DataFrame<\/code> to a SQL database.<\/p>\n\n\n\n
So here is a sample data frame that contains a new article:<\/p>\n\n\n
new_article_df = pd.DataFrame(\n    {\n        'slug'<\/span>: ['sqlalchemy-pandas'<\/span>],\n        'title'<\/span>: ['SQLAlchemy Pandas Article'<\/span>],\n        'content'<\/span>: ['Bla bla bla'<\/span>],\n    })<\/code><\/span>Code language:<\/span> Python<\/span> (<\/span>python<\/span>)<\/span><\/small><\/pre>\n\n\n
If you want to insert this new article into the database, you need to use the to_sql()<\/code> method as follows:<\/p>\n\n\n
new_article_df.to_sql(\n    name='articles'<\/span>,\n    con=engine,\n    if_exists='append'<\/span>,\n    index=False<\/span>,\n)<\/code><\/span>Code language:<\/span> Python<\/span> (<\/span>python<\/span>)<\/span><\/small><\/pre>\n\n\n
Here, we pass the table name to the name <\/em>argument and the engine <\/em>object to the con<\/code> argument.<\/p>\n\n\n\n
We also specify the append<\/code> option to the if_exists<\/code> parameter to indicate that the table already exists and that we want to append that new article to the articles<\/code> table. If you leave this parameter blank, it will automatically fail because the if_exists<\/code> param is by default set to “fail”, and the articles table exists. However, if you set it to “replace”, the articles<\/code> table will be dropped before inserting that new article.<\/p>\n\n\n\n
You can make sure this article is already committed to the database by the following query statement:<\/p>\n\n\n
session.query(Article).filter(\n    Article.slug=='sqlalchemy-pandas'<\/span>\n).first().title<\/code><\/span>Code language:<\/span> Python<\/span> (<\/span>python<\/span>)<\/span><\/small><\/pre>\n\n
>>> session.query(Article).filter(Article.slug == 'sqlalchemy-pandas'<\/span>).first().title\nu'SQLAlchemy Pandas Article'<\/span><\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
How to load a CSV file into a database<\/strong><\/h2>\n\n\n\n
In some cases, you may have a large CSV file that you want to import into the database. Instead of having a data entry clerk who can insert each record one by one, you can grab all data in that file and insert it into the database.<\/p>\n\n\n\n
You can do so by loading the CSV file to a data frame:<\/p>\n\n\n
new_df = pd.read_csv(\"more_articles.csv\"<\/span>)<\/code><\/span>Code language:<\/span> Python<\/span> (<\/span>python<\/span>)<\/span><\/small><\/pre>\n\n\n
And then use the same to_sql()<\/code> method to insert these records:<\/p>\n\n\n
new_df.to_sql(\n    name='articles'<\/span>,\n    con=engine,\n    if_exists='append'<\/span>,\n    index=False<\/span>,\n)<\/code><\/span>Code language:<\/span> PHP<\/span> (<\/span>php<\/span>)<\/span><\/small><\/pre>\n\n\n
Try manipulating the SQL data in the sandbox below:<\/p>\n\n\n\n\t
That way, you have inserted new articles into your database. Just make sure that the column names exactly match the name of your columns. If this is not the case, you may use the rename()<\/code><\/a> Pandas method on your data frame.<\/p>\n\n\n\n